You spent months building an AI app. Before you ship it, have you thought about what happens when someone deliberately feeds it malicious prompts? promptfoo is an open-source red team testing tool built exactly for this -- it finds security holes in your AI application before attackers do.
What It Tests
promptfoo ships with over 50 vulnerability detection plugins. Here is a quick overview of what it covers:
| Category | What It Does |
|---|---|
| Jailbreak attacks | Generates adversarial prompts that try to bypass model safety guardrails |
| Prompt injection | Simulates attackers embedding malicious instructions inside user input |
| Data leakage | Checks whether the model leaks training data or system prompts |
| RAG poisoning | Runs targeted attacks against retrieval-augmented generation pipelines |
| Compliance | Tests against OWASP and NIST standards, plus EU AI Act requirements |
| Custom policies | Lets you define detection rules based on your own business logic |
It works on chatbots, RAG systems, and tool-calling agents alike.
How to Run It
Three commands and you are up:
npx promptfoo@latest redteam setup # opens a config UI where you describe your app
npx promptfoo@latest redteam run # generates hundreds of adversarial inputs and fires them
npx promptfoo@latest redteam report # shows the vulnerability report
During setup you specify what your app does and how to reach it (HTTP endpoint, Python script, or direct model connection), then pick which attack plugins to enable. The report grades each finding by severity and includes the exact input that triggered it, along with remediation suggestions.
Model coverage is broad -- OpenAI and Anthropic Claude go without saying, and Google Gemini and DeepSeek are on the list too. Locally hosted models via Ollama also work.
Why It Matters
promptfoo has 27k+ stars on GitHub and is fully open source under the MIT license. It was acquired by OpenAI this year and remains open source. A key selling point: all test data runs locally and your prompts never leave your machine.
For teams with CI/CD pipelines, it plugs in as an automated security scan on every commit, catching new vulnerabilities before they reach production.
If you have an AI app running, hand this prompt to your AI agent:
Run a red team security test on my AI app using promptfoo. Start with
npx promptfoo@latest redteam setupto configure the target, thenredteam runto generate adversarial tests, and finallyredteam reportto review results. Focus on jailbreak attacks and prompt injection vulnerabilities.
Requires Node.js 22.22 or above. One npx command gets you going -- no global install needed.

Comments (0)
No comments yet. Start the conversation.