How to test agent skills: examples and a checklist

The short answer

Test selection and output separately. Use realistic prompts, fixed input files, fresh sessions and observable assertions. Compare the skill with a no-skill baseline or its previous version, inspect actual artifacts, and repeat cases where the result is inconsistent.

Edit a reviewed copy in Skiller

Use Skiller's Skills Manager to inspect the skill you want to improve, its instructions and the agents that consume it. Read the current SKILL.md before editing. Save the current directory or source revision so you can compare the candidate with the previous behavior.

Skiller supports inline editing of SKILL.md. Change the step responsible for the failure: the trigger description for selection problems, the procedure for missing checks, or the output instructions for incomplete reports. Keep the edit small enough that the evaluation can explain whether it helped.

Run the candidate in the coding agent with the fixture inputs below. Skiller manages the instructions and copies; the coding agent executes the task. Synchronize the reviewed change to the intended agents after checking its result, then verify those clients separately.

Define success before tuning instructions

A valid SKILL.md can still produce poor results. Format validation checks structure, while a behavioral evaluation checks what the agent does with the instructions. Begin with one real workflow and write down the artifact you expect. For a release-note review, that might be supported findings tied to a supplied diff, without editing the source or publishing anything.

Choose checks a reviewer can observe. “The review is useful” is too vague. “The review catches the breaking API change in this fixture and cites the changed file” can be verified. Do not require an exact sentence if several phrasings would express the correct result.

Create a small test set

Use realistic inputs rather than prompts that repeat the entire skill. Start with a normal case, a nearby task that should not trigger the skill, a missing-input case and a failure case. For the release-note example, make the failure case an unavailable Git range and check that the agent reports the limitation.

Keep fixture data harmless and independent of production. Save the prompts and input files so later changes can be checked against the same cases. A small set with meaningful assertions is more useful than many variations that all test the same happy path.

Compare with a baseline

The official evaluation guide recommends comparing outputs with and without a skill, or comparing an updated skill with a snapshot of the previous version. Use the same task and inputs for both. Start each run in a fresh session so previous answers and authoring discussion do not supply missing instructions.

Record the client and model configuration as well as the skill version. If you change the model, prompt and skill together, you cannot tell which change caused the difference. Where your client exposes timing or token use, record it alongside quality rather than assuming a longer workflow is better.

evals/
├── prompts.md
├── fixtures/
│   ├── release-notes.md
│   └── change.diff
└── results/
    ├── baseline/
    └── candidate/

This layout is an example for your records, not a required Agent Skills directory convention. Store generated results outside the distributable skill when they contain private project material.

Grade the evidence

Inspect the saved output, not just the agent's summary. If the workflow creates a file, open it. If it claims a command passed, check the command result. Use deterministic checks for mechanical requirements such as valid JSON, required keys or unchanged input files. Use human review for clarity, relevance and unsupported advice.

Record pass or fail with a reason. A finding that merely repeats a comment in the fixture is weaker than one connected to the changed behavior. Review your assertions too: a check that passes identically with and without the skill may not measure any value added by the skill.

A worked example: evaluate a release-review skill

Create a fixture diff that removes GET /v1/export and release notes that only say “Export performance improved.” Ask the agent: “Review these release notes against the supplied diff. Report missing breaking changes; do not edit files.” Run it once without the skill and again with the candidate skill in a fresh session.

These are expected outcomes for an example fixture, not measured results for a particular model. Save both outputs and grade each condition. If both versions pass, try a harder representative case before claiming that the skill improved the workflow. Use the same cases after editing SKILL.md in Skiller.

Fix the step that owns the failure

If selection fails, work on the description. If the agent loads the skill but misses a required comparison, clarify that procedural step or provide an example. If a command is unavailable, document the dependency and fallback. Repeating “always be careful” throughout the file does not identify what evidence is missing.

Rerun the full small set after a meaningful change. Repeat inconsistent cases before calling an improvement reliable. Keep the previous result so regressions remain visible. Once the skill behaves acceptably, share the reviewed version with the clients and environments you actually checked.

FAQ

How should I test an agent skill?

Use a representative task, fixed inputs and a clear success criterion. Include a missing-input or failed-command case, and record which client and skill copy you tested.

How do I compare changes to SKILL.md?

Change one part of the instructions, rerun the same test inputs and compare results. Skiller lets you inspect and edit the content before synchronizing the tested version.

Does valid frontmatter prove a skill works?

No. Valid frontmatter helps discovery. You must also verify selection, procedure execution and the resulting artifact.

Your skills. One place.

Inspect, edit and sync your skill library across coding agents.

↓ Download Skiller

Sources

• Agent Skills specification
• Agent Skills: evaluating output quality
• Skiller product documentation