# Promptbeat

## Docs

- [Coding Agent Attack Case Examples with Evidence](https://promptbeat.mintlify.app/coding-agent/attack-examples.md): Complete attack case examples in Promptbeat's six-field format: seed, generated probe, safe behavior, unsafe pattern, trace evidence, and judge signal for each case.
- [Coding Agent Attack Patterns and Probe Examples](https://promptbeat.mintlify.app/coding-agent/attack-patterns.md): The 8 coding-agent attack patterns Promptbeat uses, with sample probes, expected safe behavior, unsafe behavior patterns, and Promptfoo plugin IDs for each.
- [Broad Codex Safety Evaluation Results](https://promptbeat.mintlify.app/coding-agent/broad-codex-report.md): Baseline Codex coding-agent run: 32 sampled probes, 21 passed and 11 failed (65.6%), with risk-family breakdown, failure patterns, and 5 recommendations.
- [Coding Agent Safety Coverage: 8 Attack Surfaces](https://promptbeat.mintlify.app/coding-agent/coverage.md): Promptbeat covers 8 coding-agent attack surfaces: repo injection, terminal injection, secret reads, sandbox escape, network egress, procfs, and steganographic exfil.
- [Capability Sources: Promptfoo, Inspect, and Datasets](https://promptbeat.mintlify.app/concepts/capability-sources.md): Learn how Promptbeat coordinates Promptfoo plugins, dataset adapters, and Inspect environments — and how scenarios drive source selection automatically.
- [Promptbeat Configuration Files: How They Fit Together](https://promptbeat.mintlify.app/concepts/configuration-model.md): Understand how promptbeat.yaml, target.yaml, scenarios.yaml, seeds.yaml, and providers.yaml fit together to define a complete evaluation run.
- [Preflight Checks Before Running an Evaluation](https://promptbeat.mintlify.app/concepts/preflight-checks.md): Learn the preflight checks Promptbeat runs before generate and eval — what each check tests, how to read output, and how to fix common failures.
- [Promptbeat Risk Taxonomy: 10 Risk Categories](https://promptbeat.mintlify.app/concepts/risk-taxonomy.md): Explore Promptbeat's 10 risk categories, how they map to scenarios and evidence types, and how they drive judge selection and report grouping.
- [Targets, Scenarios, and Seeds: Core Abstractions](https://promptbeat.mintlify.app/concepts/target-scenario-seed.md): Learn how Promptbeat's three core abstractions — target, scenario, and seed — compose into a complete, reproducible AI security evaluation.
- [Dataset Catalog: Readiness and Risk Mapping](https://promptbeat.mintlify.app/datasets/catalog.md): All 15 supported datasets with readiness levels, field mappings, and risk taxonomy fit — choose the right seed source and understand provenance before your evaluation run.
- [Using HarmBench as a Seed Source in Promptbeat](https://promptbeat.mintlify.app/datasets/harmbench.md): Configure HarmBench local files, write the DatasetSpec YAML, load seeds with Python, map categories to Promptbeat risk types, and run eval commands end-to-end.
- [Dataset-Driven Evaluation: Seeds from Catalogs](https://promptbeat.mintlify.app/datasets/overview.md): Load curated datasets as attack seeds in two modes: direct eval or generator-steered probes, with full dataset provenance tracked through every generated case and report.
- [Run Your First Codex Agent Security Evaluation with Promptbeat](https://promptbeat.mintlify.app/getting-started/codex-quickstart.md): Use openai:codex-sdk to run a full generate/eval/report loop against a real Codex coding-agent target and capture secret-exfiltration trace evidence.
- [What Is Promptbeat? Scenario-Driven AI Security Testing](https://promptbeat.mintlify.app/getting-started/overview.md): Promptbeat is a scenario-driven AI security toolkit that red-teams real agent targets using generated attack probes, live execution, and trace evidence.
- [Get Started with Promptbeat: Install, Eval, and Report](https://promptbeat.mintlify.app/getting-started/quickstart.md): Install Promptbeat, validate a config, generate adversarial probes, run the evaluation, and open your first HTML safety report in under ten minutes.
- [Promptbeat: Scenario-Driven AI Security Testing](https://promptbeat.mintlify.app/introduction.md): Promptbeat helps you red-team real AI agents and LLMs with scenario-driven attack generation, trace-aware judging, and reproducible reports.
- [Promptbeat Go API Service: Seed Expansion via REST](https://promptbeat.mintlify.app/reference/api-service.md): Call Promptbeat over REST. Three endpoints cover sync preview, async batch tasks, and status polling. Run the service locally via Docker on port 8080.
- [Promptbeat CLI: Complete Four-Command Pipeline Guide](https://promptbeat.mintlify.app/reference/cli.md): Run validate, generate, eval, and report from the CLI using --config, --provider-file, --generator-provider, --count, --output-dir, and --output.
- [Promptbeat YAML Reference: All Four Config File Types](https://promptbeat.mintlify.app/reference/yaml.md): Minimal YAML for all four Promptbeat config types: project config, scenario, seed, and provider. Starting points for any new evaluation project.
- [Comprehensive Evaluation Reports in Promptbeat](https://promptbeat.mintlify.app/reports/comprehensive-reports.md): What a Promptbeat comprehensive report contains: four key questions, required sections, artifact table, example summary, and the command to generate an HTML report.
- [Report JSON Schema: Fields and Evidence Levels](https://promptbeat.mintlify.app/reports/report-schema.md): The Promptbeat evaluation result JSON schema: top-level fields, case record, assertion record, evidence levels, aggregation dimensions, and unsafe workaround classification.
- [Evaluation Result Artifacts: Paths and Contents](https://promptbeat.mintlify.app/reports/result-artifacts.md): Promptbeat artifact paths for generate, eval, and report stages: what each file contains, the default directory structure, and version control guidance for each artifact type.
- [Target Lab Architecture: Real Environments for Agents](https://promptbeat.mintlify.app/target-lab/architecture.md): Target Lab separates the Agent App from the Environment Adapter so you can evaluate agents needing a real filesystem, terminal, or network workspace.
- [Inspect Integration and Target Lab Environment Adapters](https://promptbeat.mintlify.app/target-lab/inspect-and-adapters.md): Connect Promptbeat to Inspect for benchmark execution. Learn the adapter model, Terminal-Bench integration, and Promptfoo bridge for generated test cases.
- [Agent Target Configuration Examples for Promptbeat](https://promptbeat.mintlify.app/targets/agent-configuration-examples.md): Complete YAML walkthroughs for Codex SDK and HTTP agent targets, including scenario, seed, provider, generate, and eval run commands.
- [Supported AI Agent Target Types in Promptbeat Evals](https://promptbeat.mintlify.app/targets/agent-targets.md): Promptbeat supports LLM providers, HTTP agents, Codex SDK, Claude Code, OpenClaw, and an adapter pattern for any agent runtime you already run.
- [Codex SDK Provider for Promptbeat: openai:codex-sdk](https://promptbeat.mintlify.app/targets/codex-sdk.md): Configure the openai:codex-sdk provider: required fields, baseline model config, credential setup, and production hardening tips for Promptbeat evals.
- [Provider Files: Connect Any Agent or LLM to Promptbeat](https://promptbeat.mintlify.app/targets/provider-files.md): Provider files define the execution contract for each target. Learn path handling, credential management, and multi-provider comparison in Promptbeat.
- [Typical Agent Application Targets and Readiness Status](https://promptbeat.mintlify.app/targets/typical-agent-apps.md): Readiness levels for every supported agent app type — from runnable Codex targets to adapter-pattern browser, support, data, and DevOps agents.