Introducing TMGoat: an open benchmark and dojo for threat modeling

Today we're releasing TMGoat — an open-source benchmark that scores threat-modeling tools, and a dojo that trains threat-modeling engineers. It's 30 realistic systems across 10 sectors, each seeded with deliberate design flaws, a hidden answer key, and a scoring harness that tells you — for the first time, objectively — whether an analysis was actually right.

A demo isn't evidence

Every threat-modeling tool ships with a sample architecture and a confident-looking report. Almost none of them can answer the only question that matters: was the report correct? Did it find the race condition, the fail-open control, the over-trusted channel — or just the obvious boxes any checklist would flag? Without a known answer to compare against, a threat model is a plausible essay, not a measured result.

Security has solved this pattern before. WebGoat made web-app vulnerabilities teachable; TerraGoat did it for infrastructure. TMGoat does it for threat modeling — deliberately vulnerable systems with a graded key, so a tool (or a person) can be scored instead of merely demonstrated.

What's in the corpus

TMGoat is 30 fixtures spanning finance, healthcare, government, energy/ICS, SaaS, retail, manufacturing/IIoT, telecom, transport, and defense. Between them they hide 239 planted design threats — including 10 multi-step attack chains — and exercise all 12 intake formats a real modeler runs into: draw.io and Lucidchart diagrams, PNG architecture images, Terraform / CloudFormation / Bicep / Kubernetes / Docker Compose, OpenAPI specs, MTMT and Open Threat Model files, Confluence exports, GitHub repos, and plain prose.

Crucially, the expected output is a first-class artifact. Every practice fixture ships a reference threat model — expert-authored, with each planted flaw, its mitigation mapped to a standard (NIST 800-53, PCI DSS, IEC 62443…), and documented false-positive guidance so noise gets penalized, not rewarded.

Two tracks, one corpus

The same 30 systems serve two audiences:

• ▸Tools (benchmark). Run your threat-modeling tool over the inputs, export findings, and score them with the harness. Publish recall and precision against a common yardstick — including tools built on automated and agentic AI.
• ▸Engineers (dojo). Open a challenge, model it yourself, lean on graduated hints if you're stuck, then self-score and study the reference. It's a gym for the skill, not just a test for the software.

Both are graded on the same three belts: Bronze for finding every planted threat, Silver for also catching the subtle chained one, and Gold for mapping mitigations to standards and stating residual risk.

Difficulty is subtlety, not size

The most important idea in TMGoat: a 6-component app can hide a nastier flaw than a 30-component one. So difficulty isn't measured by box count — it's tagged along three independent axes, and the hardest fixtures are the subtle ones, not the big ones.

A tool that just runs STRIDE on each box will pass the easy tier and quietly miss the fail-open control, the over-trusted channel, and — in the adversarial fixtures — the architecture doc that disagrees with the deployed code. Those design-level threats are where the points, and the real risk, live.

How we keep the score honest

A benchmark is only worth its integrity, so we built two safeguards in from day one. First, contamination resistance: the 20 easy and moderate fixtures publish full solutions for learning, but the 10 difficult fixtures are a held-out tier — public inputs, private keys, scored through a submission flow so no tool is graded on answers it could have memorized.

Second, stated limitations. The corpus is synthetic and deliberately cleaner than a messy production estate; the bundled scorer is a reproducible lexical matcher, not an oracle; and the ground truth is expert judgment, not universal truth. We wrote all of that down in LIMITATIONS.md rather than let you discover it later.

Why Virantis is publishing a yardstick

We'll say the obvious part plainly: Virantis builds a commercial continuous threat-modeling product, so we have an interest here. That's exactly why we made the benchmark open, tool-neutral, and scored by a fixed harness rather than by us. We would rather compete on an open yardstick than a private demo — and we'd rather the whole field could measure progress than argue about screenshots. The held-out tier grades everyone, us included, against keys nobody has seen.

FAQ

← Back to the blog or read the threat modeling guide.