NVIDIA's Open Tool Checks If Your Agent Skill Really Helps

Share

NVIDIA open-sourced SkillEvaluator (Apache-2.0), which checks AI agent Skills in three layers: a basic check that scores without an API key, a duplicate check for repeated or conflicting instructions, and a real-task comparison of results with and without the Skill. Learn what each layer needs, how to read the scores, and get a prompt that has your AI agent check your Skill for you.

Machine translation. Read the original

SourceGitHub · NVIDIA/SkillEvaluatorReported Nov 15, 2026Not yet tested

NVIDIA open-sourced SkillEvaluator on GitHub (Apache-2.0). It checks how well an AI agent Skill is written, and whether the Skill actually helps once it is installed. The most basic round needs no API key and no GPU, and gives your Skill a score in a few minutes. The part worth paying attention to is the third layer: run the same set of tasks with and without the Skill, then compare how much the results differ.

What It Checks

A Skill is a folder. Inside are instructions and supporting files written for an AI agent. SkillEvaluator splits its checks into three layers, and each layer can run on its own:

Layer Question it answers What you need
Layer 1: Basic checks Is the format right, is it safe, and can mainstream agents such as Claude Code and Codex load it properly? No API key. It downloads its security scanning tools automatically.
Layer 2: Duplicate check Do the instructions repeat themselves, and do they contradict other Skills you have installed? A model connection for semantic comparison.
Layer 3: Real-task comparison With the Skill installed, does the agent actually do the same task better? Model credits, a supported agent, and a Docker sandbox.

Layer 1 gives a score out of 100 and a grade. The default passing mark is 70. It also lists exactly where points were deducted. The official sample Skill scored 86.2 (B).

A High Score Does Not Mean It Is Useful

Layer 1 only checks whether the writing follows the rules. Think of it as a checkup report. It catches hard problems, such as a Skill that fails to load, vague instructions, or a risk of leaking keys. Whether a Skill is worth using depends on whether the agent does the same job noticeably better with it installed. Only Layer 3 can answer that.

So the practical approach is:

  • Run Layer 1 after writing a Skill. It is free and fast, and it clears up format and security issues first.
  • Before you share a Skill publicly or give it to your team, run Layer 3. Use a few real tasks to compare results with and without the Skill. A Skill that makes no difference is better deleted, so it stops taking up context for nothing.
  • Once you have several Skills, run Layer 2. Check whether some of them cover the same ground or contradict each other.

How to Get Started

The easiest way is to let your AI agent install and run it, while you only read the results:

I want to check how good my own Agent Skill is. Please use NVIDIA's open-source SkillEvaluator (GitHub: NVIDIA/SkillEvaluator) to help me:

1. First read its README and confirm the current install method. Before installing, tell me what will be installed.
2. Run only Layer 1 (the one that needs no API key) on this Skill: [path to your Skill folder]
3. Tell me in plain language: the total score and grade, the three biggest deductions, and how to fix each one.
4. Only fix the obvious format problems directly. List any changes to the Skill's content or approach for me to review, and wait for my confirmation.
5. Do not run Layer 2 or Layer 3 yet. They use model credits or Docker. First tell me roughly what I need to prepare and whether they will cost money.

After the changes, run Layer 1 again and show me the before and after scores.

Run only Layer 1 first, because it costs nothing and needs no setup. That makes it the best way to find out what the problems are. Layer 3 actually runs the agent, which takes time and may cost money, so only do it once you have confirmed it is worth the effort.

Notes

  • Layer 3 may cost money. It calls models and runs the agent, and managed sandboxes also charge. Check the expected usage before you run it.
  • It downloads scanning tools automatically. If Layer 1 is missing a scanning tool, it downloads one by itself. If your work computer restricts downloads, ask your administrator first.
  • Sample scores are only a reference. 86.2 is the official sample Skill's score. Your Skill's score depends on how it is written, and the scoring rules may change between versions.

If you want to judge whether someone else's test results deserve trust, read this experience. The idea is the same: run the comparison again with your own tasks.

Done with this one? Mark it to track your path.Sign in
Save it for laterSign in
Questions about this piece?Ask AI about this

Comments (0)

No comments yet. Start the conversation.

Sign in to join the discussion.Sign in