NVIDIA open-sourced SkillEvaluator on GitHub (Apache-2.0). It checks how well an AI agent Skill is written, and whether the Skill actually helps once it is installed. The most basic round needs no API key and no GPU, and gives your Skill a score in a few minutes. The part worth paying attention to is the third layer: run the same set of tasks with and without the Skill, then compare how much the results differ.
What It Checks
A Skill is a folder. Inside are instructions and supporting files written for an AI agent. SkillEvaluator splits its checks into three layers, and each layer can run on its own:
| Layer | Question it answers | What you need |
|---|---|---|
| Layer 1: Basic checks | Is the format right, is it safe, and can mainstream agents such as Claude Code and Codex load it properly? | No API key. It downloads its security scanning tools automatically. |
| Layer 2: Duplicate check | Do the instructions repeat themselves, and do they contradict other Skills you have installed? | A model connection for semantic comparison. |
| Layer 3: Real-task comparison | With the Skill installed, does the agent actually do the same task better? | Model credits, a supported agent, and a Docker sandbox. |
Layer 1 gives a score out of 100 and a grade. The default passing mark is 70. It also lists exactly where points were deducted. The official sample Skill scored 86.2 (B).
A High Score Does Not Mean It Is Useful
Layer 1 only checks whether the writing follows the rules. Think of it as a checkup report. It catches hard problems, such as a Skill that fails to load, vague instructions, or a risk of leaking keys. Whether a Skill is worth using depends on whether the agent does the same job noticeably better with it installed. Only Layer 3 can answer that.
So the practical approach is:
- Run Layer 1 after writing a Skill. It is free and fast, and it clears up format and security issues first.
- Before you share a Skill publicly or give it to your team, run Layer 3. Use a few real tasks to compare results with and without the Skill. A Skill that makes no difference is better deleted, so it stops taking up context for nothing.
- Once you have several Skills, run Layer 2. Check whether some of them cover the same ground or contradict each other.
How to Get Started
The easiest way is to let your AI agent install and run it, while you only read the results:
I want to check how good my own Agent Skill is. Please use NVIDIA's open-source SkillEvaluator (GitHub: NVIDIA/SkillEvaluator) to help me:
1. First read its README and confirm the current install method. Before installing, tell me what will be installed.
2. Run only Layer 1 (the one that needs no API key) on this Skill: [path to your Skill folder]
3. Tell me in plain language: the total score and grade, the three biggest deductions, and how to fix each one.
4. Only fix the obvious format problems directly. List any changes to the Skill's content or approach for me to review, and wait for my confirmation.
5. Do not run Layer 2 or Layer 3 yet. They use model credits or Docker. First tell me roughly what I need to prepare and whether they will cost money.
After the changes, run Layer 1 again and show me the before and after scores.
Run only Layer 1 first, because it costs nothing and needs no setup. That makes it the best way to find out what the problems are. Layer 3 actually runs the agent, which takes time and may cost money, so only do it once you have confirmed it is worth the effort.
Notes
- Layer 3 may cost money. It calls models and runs the agent, and managed sandboxes also charge. Check the expected usage before you run it.
- It downloads scanning tools automatically. If Layer 1 is missing a scanning tool, it downloads one by itself. If your work computer restricts downloads, ask your administrator first.
- Sample scores are only a reference. 86.2 is the official sample Skill's score. Your Skill's score depends on how it is written, and the scoring rules may change between versions.
If you want to judge whether someone else's test results deserve trust, read this experience. The idea is the same: run the comparison again with your own tasks.


Comments (0)
No comments yet. Start the conversation.