Promptfoo

What is Promptfoo?

Promptfoo is a library for testing and evaluating prompts used with large language models (LLMs). It runs automated evaluations based on clear, objective metrics. You can define test cases, create custom evaluation criteria, and compare different prompts and model outputs side by side. LLM applications serving over 10 million users rely on it.

Who created Promptfoo?

Promptfoo is based in San Mateo, California, and is backed by investors including Andreessen Horowitz along with leaders from the tech and security industries. Ian Webster is the CEO and Co-founder, previously leading engineering teams at Discord. Michael D’Angelo is the CTO and Co-founder, with extensive experience in AI and engineering leadership.

Who is Promptfoo for?

The tool is aimed at data scientists, machine learning engineers, software developers, AI researchers, product managers, and quality assurance engineers — anyone who needs to validate prompt and model behavior before shipping.

Using Promptfoo

• Create test cases from a variety of typical user inputs.
• Run evaluations with the built-in metrics, then define custom metrics to meet specific requirements.
• Compare prompts and model outputs side by side to find the best combination.
• Integrate Promptfoo into existing testing or continuous integration (CI) setups for consistent quality monitoring.
• Choose the web viewer for a visual approach or the command line interface for a code-driven workflow.