# PentestBench | OpenHack > PentestBench v1 is a white-box pentesting benchmark maintained by OpenHack. It evaluates models and harnesses on 17 modern web applications with 761 planted vulnerabilities, including vulnerability-chain scenarios. Published NOV 16, 2026. ## Overview PentestBench evaluates how well different models and harnesses find web vulnerabilities in real-life scenarios in modern frameworks. The corpus includes vulnerability-chain scenarios, where one weakness can enable another or increase its impact. The corpus spans Next.js, Express, Fastify, SvelteKit, Django, Flask, FastAPI, Ruby on Rails, Laravel, Spring Boot, and Gin. Each tool has access to the application source code; the vulnerability catalog stays sealed. Results compare detection coverage, report precision, and catalog F1. White box pentest benchmarks only. Black-box pentesting benchmarks are coming soon. ## Leaderboard First full pass. Results are ranked by catalog F1, with higher values indicating better performance. Each scanner and model configuration is scored separately across all 17 applications. These are results for the complete model and harness configuration, rather than model capability in isolation. | Rank | Scanner / harness | Model / configuration | Matched | Reports | Recall | Precision | Catalog F1 | Evidence | | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- | | 1 | OpenHack v2 Harness | Mythos 5.2 | 277 | 521 | 36.40% | 53.17% | 43.21% | Runtime proof + review | | 2 | Claude Code /security-review | Mythos 5.2 + Haiku | 200 | 319 | 26.28% | 62.70% | 37.04% | Unverified reports | | 3 | Claude Code /security-review | Opus 5.5 | 182 | 264 | 23.92% | 68.94% | 35.51% | Unverified reports | | 4 | Claude Security | Opus 5.5 + Sonnet | 310 | 1015 | 40.74% | 30.54% | 34.91% | Code review | | 5 | Claude Security | Mythos 5.2 + Sonnet | 327 | 1246 | 42.97% | 26.24% | 32.59% | Code review | | 6 | Semgrep CE 1.185.0 | security-audit rules | 19 | 50 | 2.50% | 38.00% | 4.69% | Unverified static hits | Matched means planted vulnerabilities found. Reports means total reported findings. - Recall: Planted bugs found, out of 761. Formula: matched ÷ 761. - Precision: Share of reports that matched a planted vulnerability. Formula: matched ÷ reports. - Catalog F1: Harmonic mean of recall and precision; the ranking metric. Formula: 2 × matched ÷ (761 + reports). The formulas yield fractions; the table displays percentages. ## Application corpus We plan to open-source the 17-application corpus used in PentestBench. It contains 761 planted vulnerabilities, and every evaluated tool receives the same application source code. | Application | Planted vulnerabilities | | --- | ---: | | ArtVault | 45 | | BoardSync | 45 | | BudgetBee | 45 | | Casa | 50 | | ClinicADB | 45 | | DataPulse | 45 | | Devils Advocate | 34 | | EduStream | 46 | | FreightFlow | 45 | | MedFlow | 45 | | MediVault | 49 | | Proplist | 43 | | RentEase | 44 | | ShipDash | 45 | | TeamForge | 46 | | TrailGuard | 45 | | VaultPay | 44 | | Total | 761 | ## FAQ ### Why create this when others exist? We created PentestBench to accurately benchmark [OpenHack](https://openhack.com), the always-on AI security engineer we’re building. We wanted an evaluation centered on finding vulnerabilities in complete, modern web applications, including vulnerability chains, rather than only isolated snippets or individual functions. This is the first version of the PentestBench corpus. Future versions will include applications deployed in cloud environments and evaluate vulnerability reachability and the ability to identify false positives—capabilities a real AI security engineer needs. ### What does the benchmark evaluate? PentestBench measures vulnerability discovery across 17 applications with 761 planted vulnerabilities. Recall measures how many planted vulnerabilities a tool finds; precision measures how many of its reports match the catalog. Catalog F1 balances the two. Results reflect each model and harness configuration working together, rather than model capability in isolation. ### Does the benchmark cover vulnerability chaining? Yes. The corpus includes documented scenarios where vulnerabilities combine to create greater impact, such as a weak session-signing key enabling an authorization bypass, or cross-site scripting combined with readable session cookies enabling session theft. The leaderboard scores individual vulnerability matches through recall, precision, and catalog F1. ### Why white-box pentesting? We want to assess how well tools use application source code to identify security weaknesses in modern web applications. Every evaluated tool receives the source code, while the reference vulnerability catalog stays sealed. This provides a common setting for comparing their findings. ### Will there be black-box pentesting benchmarks? Black-box pentesting results are coming soon. ### Why are there no OpenAI models? We wanted to compare the best models from OpenAI, but we’re still awaiting Daybreak Red access from OpenAI to use GPT-6 Astra for this benchmark. ## Citation OpenHack. PentestBench v1. 2026. Published NOV 16, 2026. 17 applications, 761 planted vulnerabilities, 6 scanner configurations. BibTeX is available at [/pentestbench-v1.bib](/pentestbench-v1.bib). ## Resources - [PentestBench](/): Benchmark overview and public results. - [Leaderboard](/#leaderboard): First full pass results, ranked by catalog F1. - [Application corpus](/#dataset): Applications and planted vulnerability counts. - [FAQ](/#faq): Purpose, evaluation scope, vulnerability chaining, and future plans. - [Full benchmark text](/llms-full.txt): Self-contained overview, results, dataset, and FAQ. - [Citation](/pentestbench-v1.bib): BibTeX citation for PentestBench v1. - [OpenHack](https://openhack.com): The always-on AI security engineer.