The Defensive Brain: Why Claude + Foundation-Sec-8B + RAG
An AI agent that analyzes security events is only as good as the model behind it and the knowledge it can access. We chose both by real benchmarks — and learned one insight that saves a lot of money.
The problem: not every model is good at cyber
It's easy to assume that "the smartest model" is the best at everything. In cyber, that's not true. Incident analysis requires an iteration loop: write a query, read the results, correct, and try again. Some models excel at this and some fall apart — and it's not always the "strongest" model.
What we compared — benchmarks, not hype
We relied on two primary sources: Microsoft's CTI-REALM (50 tasks against real attack simulations) and Meta/CrowdStrike's CyberSOCEval (609 expert-validated cases). Two counter-intuitive insights emerged:
- Claude leads on agentic work — thanks to the iterate-and-refine loop, where it reached 0.86–0.92 while competitors dropped below 0.50.
- Expensive reasoning mode doesn't improve cyber analysis — unlike math and code. Models weren't trained to "think" about security analysis, so paying for reasoning-high on triage is a waste.
What we chose — smart routing, not a single model
| Model | Role | Why |
|---|---|---|
| Claude (Sonnet/Opus) | Agentic triage + incident-report writing | Ranked 1–3 on CTI-REALM |
| Foundation-Sec-8B (Cisco) | The cheap-and-private worker: CVE→CWE, ATT&CK questions | ~70B quality at 8B size, runs on our infra |
| Workers AI (at the edge) | Embeddings + default triage | Free, stays within the Cloudflare tenant |
Routing saves money and preserves privacy: most of the work runs on free/open models, and only the hardest cases are routed to paid Claude.
And why RAG — and not fine-tuning
Cyber knowledge changes every day: new CVEs, new attack techniques, new IOCs. You can't retrain a model on that. So the foundation is RAG — we embed the public knowledge bases (MITRE ATT&CK, D3FEND, Sigma, threat feeds) in a vector store, and the agent retrieves the relevant technique, the matching countermeasure, and the detection rule in real time.
Fine-tuning is reserved for just two narrow transforms where RAG hits its ceiling: generating Sigma rules and mapping a technique to ATT&CK — where a small, tuned open model beats even the giant models.
The four anchors — and all of them are commercially legal
We deliberately chose open sources that may be embedded in a paid product: MITRE ATT&CK + D3FEND (technique↔countermeasure), Sigma (detection rules, DRL license), and Atomic Red Team (MIT) as the example generator. We watched for licensing landmines: some academic datasets are permitted for research only and may not be embedded in a product — so we keep them separate.
Privacy: a "private mode" for sensitive clients
A business's logs are sensitive data. So the default is to run on Workers AI (the data stays within the Cloudflare tenant), and before any hop to an external API there's a PII-scrubbing step. The most sensitive clients get a "private mode" pinned to the Foundation-Sec-8B we host ourselves — and the data doesn't leave our boundary at all.
How we keep improving this — continuous quality control
An AI model that performs actions is also an attack surface. So: (1) deterministic guardrails outside the LLM — a policy that intercepts destructive actions, so injection into the LLM cannot expand damage. (2) A nightly red-team against our agent (prompt-injection/jailbreak) that runs regression tests and fails the deploy if a defense breaks. (3) The model ranking is re-evaluated against new benchmarks — the routing updates when a better model appears. The choice is alive.
This article describes AI architecture and selection considerations. Benchmark numbers are accurate as of 2026 and should be verified against the source. There are no API keys, secrets, or internal prompts here.
← All posts