We scanned the AI skills our operator runs. Here is what a security scanner gets right and wrong

On 9 September 2026 we ran a security scan over the AI skills our operator runs from its two skill folders: 58 directories, 5 installed globally and 53 in our own notes vault. Skills that ship inside plugins were not in scope, and we say so because a scan is only as honest as its boundary. The scanner was NVIDIA's SkillSpector, released under the Apache License 2.0, run with its LLM analysers off, so no file contents left the machine; its dependency check queries a public vulnerability database with package names and versions only.

Four directories came back CRITICAL, two more HIGH. We read the HIGH findings against their source lines, with two exceptions on the day, plus a file-coverage gap on three large directories that is described further down and remains open: on the third-party skill with the most findings, 27 of 79 were read, three per rule group, and six smaller skills carried findings that were not opened. Both gaps were closed two days later, every remaining line read, and nothing genuine turned up in either.

A skill is somebody else's instructions running next to your data

If an AI agent does real work in your business, check whether it has been extended with skills or tools, because that is where this starts. A skill is a folder: written instructions the agent reads, sometimes scripts it runs, sometimes dependencies it pulls. An MCP server is the same idea with a network connection attached. Both change what the agent does and what it can reach.

Installing one is less like installing an app from a store and more like handing a contractor written instructions and your office keys without reading them.

The trigger was ordinary. Five open-source repositories came out of a podcast round-up we transcribed on 9 September 2026; we installed three (the scanner itself and the two skills discussed below) and parked two. Before installing the two we were adding as skills, we set the rule (the scanner itself went in unscanned, which we own): scan first, and use the same scanner on everything after.

What the baseline actually found

SkillSpector, at version 2.11.1 as we ran it, advertises 71 vulnerability patterns across 17 categories, matched, in part, by what its documentation calls "fast regex-based pattern matching across 11 static analyzers". It is fast, and by design it reads shapes rather than meaning.

Every HIGH finding we opened fell into one of five benign groups:

1. Our own self-patching rule. Every skill of ours documenting a "self-patch" section was flagged as rogue agent behaviour at 0.9 confidence. A skill that rewrites itself is exactly what a scanner should catch. Ours is deliberate and scoped to one folder. Any third-party skill that self-patches deserves the same suspicion the scanner gave ours.
2. Warnings read as commands. A destructive git command quoted inside a sentence telling a future agent never to run it. A routine build step clearing a folder. The scanner cannot tell an instruction from a prohibition.
3. Keyword hits on .env and API_KEY, flagged as privilege escalation. Ordinary key loaders.
4. HTML comments and template references, flagged as prompt injection and analysis evasion.
5. A comedy transcript and a course note, flagged as anti-refusal content.

At the other end of the table, more than half of the 58 directories scored between 0 and 20 with no HIGH findings at all. The four CRITICAL scores were two third-party bundles, one directory roll-up of our own sub-skills, and one of our own deployment skills, whose findings were a build command that deletes its own output folder and a self-patch instruction. The HIGH findings inside all four, now all read, were the keyword shapes listed above, with one useful exception: the full read of the biggest third-party skill turned up two real capabilities, both disclosed and opt-in, a check against its own public repository for updates and an image-generation feature that reads a key the user supplies. Neither is a problem. Both are the kind of thing the rule below says deserves a second look.

The rule that came out of it

A static CRITICAL is a hypothesis to read, not a verdict.

That is now a standing rule here, because the alternative failed a live test the same day. One of the two repositories scanned as a clone before installation, and so outside the 58 directories above, browser-use/video-use (MIT, commit 9575612 of 30 August 2026, which describes itself as "edit videos with Claude Code. 100% open source."), scored 100 out of 100, CRITICAL, do not install. Its 18 HIGH findings were keyword shapes of the same kind: credential-looking names in config and prose, an install command inside a comment, prose flagged as evasion. An independent read of the code found exactly one outbound network call, to the transcription API its documentation says it requires. The only shell execution in its helper scripts is to ffmpeg and ffprobe on the local machine, which the tool exists to drive; an optional animation add-on it bundles will also call Manim and LaTeX if you choose to install them, and there is no telemetry in its own code. The other, petergyang/no-ai-slop (MIT, commit 000650b of 2 September 2026 UTC), which "removes 20+ patterns of AI slop from any piece of writing", scored 20 and SAFE. Both were installed on 9 September 2026 after reading, not after scoring; the video tool is in for a trial and has not yet been run on real footage.

Relaying the headline would have blocked a safe tool, and trained us to ignore the scanner on the day it is right.

What the scanner cannot tell you

Coverage was partial on three large directories, between 86% and 91% of files inspected by the scanner's own coverage figure as we noted it that morning, because packed archive files fall outside the scanner's inspection profile. One large third-party skill produced 79 HIGH findings; on the day we sampled three lines per rule group across nine groups rather than reading all 79, and six mid-band skills carried one or two findings that were not opened at all. Two days later all of it was read: 79 lines on the big skill, seven on the six small ones, and none of it was genuine. The pattern held to the last line: documentation datasets, dot-prefixed folder names and the skill's own disclosed features tripping the same keyword rules. The three large directories with partial file coverage stay on the list as partly unscanned; packed archives sit outside the scanner's profile, we treat what it could not open as unread rather than clean, and the next step is to unpack those archives and scan them by hand. And with no API key configured, the LLM analysers never ran, so the intent-reading pass its documentation says "improves precision to ~87%" was not part of this baseline. That stage is the scanner's default. It was off because no key was configured for it, and we would keep it off, because turning it on means sending file contents to a model provider, a decision to take deliberately.

The baseline is a first pass with known edges.

The routine, in order

This is the routine we set on 9 September, and we broke it the next day: two third-party skills and a third-party connector went in on a manual read of their code and instructions, and were only scanned the day after. One skill scored zero, the other 77 and HIGH on fourteen findings that were all the scanner tripping on the skill's own instructions, and the connector scored 60 and HIGH on four hits against one comment declaring an authorisation scope. Nothing genuine in any of them, which is not the point; the rule is scan first, and it holds again. The reports from that catch-up now live in a folder future scans can be compared against. The baseline's own JSON reports were not kept, only the summary written that morning, which is why step five exists. The routine:

1. Scan before registration, not after. Static only, against the folder or repository URL, and the report is kept.
2. Open every HIGH finding against its actual source line. If you will not read the lines, the scan is decoration.
3. Search independently for what the scanner is weakest on: outbound network calls, shell execution, environment variable reads, telemetry, and anything writing outside its own folder.
4. Then decide, and only then register it.
5. Keep the report, so the next version can be compared against it.

The question to ask if you do not run the agent yourself

You do not need to run a scanner yourself. The question to ask the person who does is simple: which skills and tools has the agent been given, and who read them before they went in. If the answer is a score, ask for the lines.

The scanner does step one. It is the inventory and the reading list, not the judgement.

If an agent can extend itself, somebody has to own what it installed, and "the scanner said it was fine" and "the scanner said it was bad" are equally poor answers on their own. Our guide on whether AI is safe for your business data covers the vendor side, and audit readiness and compliance covers what you need to show someone else.

This check is part of what an AI Operations Day covers. It starts with a free fit call. The AI Operations Day costs £750 per day on site. If you proceed with a qualifying pilot or build within 60 days, the full fee is credited against it.

Questions we get asked

Is it risky to let an AI agent install its own tools?

It is a real risk, and it is the kind you can manage rather than avoid. A skill or an MCP server is a folder of written instructions, scripts and dependencies that your agent reads and acts on, sitting next to your files and your credentials. The risk is not that the agent is reckless. It is that nobody read what was in the folder. Scanning before registration, and reading the flagged lines against the source, turns an unbounded risk into a bounded one: a scan, then a read of the flagged lines.

What does an AI skill security scan actually check?

NVIDIA SkillSpector, the open-source scanner we use (version 2.11.1 as we ran it), advertises 71 vulnerability patterns across 17 categories, matched, in part, by what it calls "fast regex-based pattern matching across 11 static analyzers". Those categories include prompt injection, data exfiltration, privilege escalation, supply chain risk and rogue agent behaviour. Run with the LLM analysers disabled it is static: it reads text and code on your own machine, and the only thing it sends out is the list of declared dependency names and versions, to a public vulnerability database. No file contents leave the machine. It finds shapes, not intent.

Does a CRITICAL score mean a tool is malicious?

No. On our 9 September 2026 baseline, four of 58 directories scored CRITICAL and every HIGH finding we opened turned out to be benign: our own documented self-patching rule, destructive commands quoted inside warnings not to run them, keyword hits on .env and API_KEY in ordinary key loaders, HTML comments read as prompt injection, and a comedy transcript flagged as anti-refusal. The rule we set that day is that a static CRITICAL is a hypothesis to read, not a verdict. It tells you which lines to open.