This Open-Source Tool Acquired by OpenAI Breaks Into Your AI Apps

Share

promptfoo is an open-source AI red team testing tool with 50+ vulnerability plugins that automatically tests AI apps for jailbreaks, prompt injection, and data leakage. 27k+ GitHub stars, MIT license.

Machine translation. Read the original

SourceGitHub · promptfoo/promptfooReported Nov 15, 2026Not yet tested

You spent months building an AI app. Before you ship it, have you thought about what happens when someone deliberately feeds it malicious prompts? promptfoo is an open-source red team testing tool built exactly for this -- it finds security holes in your AI application before attackers do.

What It Tests

promptfoo ships with over 50 vulnerability detection plugins. Here is a quick overview of what it covers:

Category What It Does
Jailbreak attacks Generates adversarial prompts that try to bypass model safety guardrails
Prompt injection Simulates attackers embedding malicious instructions inside user input
Data leakage Checks whether the model leaks training data or system prompts
RAG poisoning Runs targeted attacks against retrieval-augmented generation pipelines
Compliance Tests against OWASP and NIST standards, plus EU AI Act requirements
Custom policies Lets you define detection rules based on your own business logic

It works on chatbots, RAG systems, and tool-calling agents alike.

How to Run It

Three commands and you are up:

npx promptfoo@latest redteam setup   # opens a config UI where you describe your app
npx promptfoo@latest redteam run     # generates hundreds of adversarial inputs and fires them
npx promptfoo@latest redteam report  # shows the vulnerability report

During setup you specify what your app does and how to reach it (HTTP endpoint, Python script, or direct model connection), then pick which attack plugins to enable. The report grades each finding by severity and includes the exact input that triggered it, along with remediation suggestions.

Model coverage is broad -- OpenAI and Anthropic Claude go without saying, and Google Gemini and DeepSeek are on the list too. Locally hosted models via Ollama also work.

Why It Matters

promptfoo has 27k+ stars on GitHub and is fully open source under the MIT license. It was acquired by OpenAI this year and remains open source. A key selling point: all test data runs locally and your prompts never leave your machine.

For teams with CI/CD pipelines, it plugs in as an automated security scan on every commit, catching new vulnerabilities before they reach production.

If you have an AI app running, hand this prompt to your AI agent:

Run a red team security test on my AI app using promptfoo. Start with npx promptfoo@latest redteam setup to configure the target, then redteam run to generate adversarial tests, and finally redteam report to review results. Focus on jailbreak attacks and prompt injection vulnerabilities.

Requires Node.js 22.22 or above. One npx command gets you going -- no global install needed.

GitHub · Docs

Done with this one? Mark it to track your path.Sign in
Save it for laterSign in
Questions about this piece?Ask AI about this

Comments (0)

No comments yet. Start the conversation.

Sign in to join the discussion.Sign in