Philipp Schmid on agent skills: first eval, then post (Category AI4SDLC)

Watched a short talk by Philipp Schmid of Google DeepMindDon't Ship Skills Without EvalsThe basic thesis is simple: skill changes the behavior of an agent, so releasing it without checking is like changing code without testing.

Schmid suggests checking not whether the agent has read SKILL.md, but the result of the work. To do this, each skill needs a small set of tasks: where it should connect, where it should not and what result is considered correct. First, conventional script checks and regular expressions are enough; LLM-as-a-judge is only needed where the result cannot be determined.

Here’s what the author’s algorithm looks like.

1. Choose a frequently used skill.
2. Write five test queries: positive, negative and one real problem case.
3. For each, set a verifiable result, not a mandatory sequence of actions of the agent.
4. Run tasks several times with and without skill in an isolated environment.
5. Delete no-op instructions. If skill does not improve the result, remove it and leave it for regression.

This set should be expanded to 10–20 Cases and run with each skill change and model update.

SKILL.md is not just in case documentation, but part of the agent system. Its value is determined not by the volume of instructions, but by the measurable change in the result.

#AI #AI4SDLC #Agents #Evals #Engineering