Get Started with Promptbeat: Install, Eval, and Report

Install Promptbeat, validate a config, generate adversarial probes, run the evaluation, and open your first HTML safety report in under ten minutes.

This guide walks you through the core Promptbeat workflow using the built-in llm-basic example. By the end you will have validated a config, generated adversarial probes with an attacker model, evaluated a real LLM target, and produced an HTML report showing pass/fail results by risk type. No custom configuration required—everything runs from the example files included in the release package.

1. Install Promptbeat

   Install or unpack a Promptbeat release package, then verify the binary is available from the package root:

   ./bin/promptbeat --version
   

   If you are using uv, you can run Promptbeat directly without unpacking:

   uv run promptbeat --version
   

   The rest of this guide uses the ./bin/promptbeat form. Substitute uv run promptbeat wherever you see ./bin/promptbeat if you prefer the uv path.

2. Validate the example config

   Before generating anything, confirm that the example config parses correctly and all referenced files resolve:

   ./bin/promptbeat validate --config examples/llm-basic/promptbeat.yaml
   

   Validation checks YAML syntax, required fields, scenario references, seed file paths, and provider configuration. Fix any reported errors before proceeding. Common causes are missing seed files or an unresolvable provider name.

3. Generate adversarial probes

   Export your OpenAI API key, then run the generator. The generator model reads your scenario and seed files and writes expanded adversarial probes to the output directory:

   export OPENAI_API_KEY="sk-..."
   
   ./bin/promptbeat generate \
     --config examples/llm-basic/promptbeat.yaml \
     --output-dir artifacts/llm-basic/generate
   

   Promptbeat writes a generated_redteam.yaml file into artifacts/llm-basic/generate/. That file is a self-contained Promptfoo red-team config that drives the evaluation in the next step.

4. Evaluate the generated probes

   Run the evaluation against the generated config. Promptbeat executes each probe against the configured target, judges the responses, and collects trace evidence:

   ./bin/promptbeat eval \
     --config artifacts/llm-basic/generate/generated_redteam.yaml \
     --output-dir artifacts/llm-basic/eval
   

   Results land in artifacts/llm-basic/eval/evaluation_result.json. The JSON file contains every probe, response, judge verdict, and captured evidence.

5. Generate the report

   Convert the evaluation result into a human-readable HTML report:

   ./bin/promptbeat report \
     --eval-result artifacts/llm-basic/eval/evaluation_result.json \
     --output artifacts/llm-basic/report.html
   

   Open artifacts/llm-basic/report.html in a browser to review pass/fail results, risk-type breakdowns, and the evidence captured for each failing case.

What you’ll find in the report

The HTML report organizes results by risk type and scenario. For each evaluated probe you will see:

• Verdict — pass or fail, based on the judge strategy defined in the scenario.
• Risk type — the risk taxonomy tag the scenario maps to (for example t-002 for secret environment reads).
• Probe and response — the exact adversarial input sent to the target and the raw response returned.
• Failure evidence — for failing cases, the specific signal that triggered the failure, such as a matched secret string or a detected unsafe workaround pattern.
• Trace artifacts — when the target emits trace data (commands run, files written, tool calls made), those traces are captured alongside the response.

The raw evaluation_result.json artifact in artifacts/llm-basic/eval/ contains the full structured data if you need to feed results into a downstream pipeline or CI check.

The llm-basic example evaluates a single LLM provider and is designed to get you familiar with the workflow quickly. For a more complete walkthrough that runs attacks against a real coding-agent runtime with shell access and file-system capabilities, see the Codex Quickstart.

Next steps

Core Concepts

Understand targets, scenarios, seeds, subscriptions, and provider adapters in depth.

Agent Targets

See how Promptbeat models coding agents, browser agents, support agents, and more as first-class targets.

Datasets

Load seeds from HarmBench, JailbreakBench, ALERT, BeaverTails, and other catalog datasets via subscriptions.

Risk Taxonomy

Browse the risk type tags and attack categories used in scenario definitions.