CVE → MITRE ATT&CK techniques dataset

CVE → MITRE ATT&CK techniques dataset#

This page documents the methodology behind the CIRCL/vulnerability-attack-techniques dataset and the design decisions that led to it. The goal (tracked in VulnTrain issue #6) is to train a model that suggests MITRE ATT&CK techniques from a vulnerability description: CVSS tells you how bad a vulnerability is, CWE tells you what kind of flaw it is, ATT&CK tells defenders what adversary behavior to expect and detect. Very few public models cover that gap.

Findings at a glance#

The experiment series behind this page spans two papers (the original classifier + LLM-expansion study, and the metadata/label-semantics follow-up). The main points to remember, each detailed with full protocol and numbers in its dated section below:

  • The task is label-quality bound, not data bound. LLM-assisted label expansion at ≈ 0.39 labeler agreement brings no reliable gain. Growing the curated gold set is the one intervention with measured payoff — every scaling curve still rises at the current size (1,207 CVEs).

  • Free metadata does not pay. The CVSS vector is null, affected products (CPE) and the CVE2CAPEC-derived candidates are mildly harmful, and concatenating all signals lands below the description-only baseline. The one conditional positive is the curated gold CWE (+0.030 recall@5 pooled over ten paired runs), concentrated entirely on the KEV-curated rows; substituting a predicted CWE recovers less than half of that, and none of it consistently.

  • The CWE → CAPEC → ATT&CK derivation chain is closed for this task: its candidate sets cover only 3.3 % of the analyst-chosen techniques on the held-out split, so it fails both as a label source and as a re-ranking prior.

  • Structural training signals hurt at this scale. Splitting the head across the CTID exploitation/impact roles (bucket multi-task) or adding a free tactic-level auxiliary loss both degrade the union task — at ~1,000 examples every signal competes for the same supervision budget.

Which model to use (head-to-head, five paired seeds, same session):

  • …-classification-roberta-base (classification head) — best top-5 ranking on the 53 trained techniques (recall@5 0.667 ± 0.015). The deployment default: description-only, flattened union, with the gold CWE appended when a curated assignment exists.

  • …-attack-technique-biencoder (label-semantics bi-encoder) — slightly lower recall@5 (0.643 ± 0.019) but the largest consistent rare-technique gain measured on this task (macro-F1 0.212 vs 0.176, +21 % relative) and open-vocabulary ranking over all 222 active parent techniques (recall@5 0.515, 2.3× a generic zero-shot sentence embedder). Prefer it when rare-technique coverage matters more than top-5 sharpness. Caveat: on techniques absent from training it ranks worse than an off-the-shelf embedder (recall@5 0.12 vs 0.24) — below the vocabulary floor, only curated examples help.

Workflow at a glance#

        flowchart TD
    classDef data fill:#e8f0fe,stroke:#4285f4,color:#000;
    classDef tool fill:#fff4e5,stroke:#f9a825,color:#000;
    classDef gate fill:#fde7e9,stroke:#d93025,color:#000;
    classDef out fill:#e6f4ea,stroke:#188038,color:#000;

    subgraph P1["Phase 1 — build the gold dataset"]
        CTID["MITRE CTID gold mappings<br/>attack_to_cve + KEV Explorer"]:::data
        STIX["Enterprise ATT&CK STIX<br/>catalog + revoked-by"]:::data
        SCORES["CIRCL/vulnerability-scores<br/>descriptions"]:::data
        C2C["CVE2CAPEC<br/>weak derived labels"]:::data
        GEN["attack-generation<br/>normalize IDs · join · merge"]:::tool
        GOLD[("CIRCL/vulnerability-attack-techniques<br/>1,207 gold rows")]:::out
        CTID --> GEN
        STIX --> GEN
        SCORES --> GEN
        C2C -. techniques_derived .-> GEN
        GEN --> GOLD
    end

    subgraph P2A["Phase 2 — train & evaluate"]
        TRAIN["train-attack-classification<br/>roberta-base · multi-label BCE"]:::tool
        MODEL[("…-classification-roberta-base")]:::out
        BASE["zero-shot similarity baseline<br/>SMET-style"]:::tool
        EVAL{"beats baseline?<br/>recall@k / MRR"}:::gate
        TRAIN --> MODEL --> EVAL
        BASE --> EVAL
    end

    subgraph P2B["Phase 2 — LLM label expansion"]
        VAL{"validate: LLM vs gold<br/>agreement good enough?"}:::gate
        EXP["expand: label sampled CVEs<br/>Ollama (Qwen) or Claude"]:::tool
        LLMDS[("…-llm-&lt;model&gt;<br/>label_sources = [llm]")]:::out
        STOP["stronger model,<br/>or keep gold-only"]:::gate
        VAL -- yes --> EXP --> LLMDS
        VAL -- no --> STOP
    end

    GOLD --> TRAIN
    GOLD --> VAL
    LLMDS --> MERGE["merge gold + LLM labels"]:::tool
    GOLD --> MERGE
    MERGE -. retrain .-> TRAIN
    EVAL -- passes --> DONE(["published model"]):::out

The full pipeline, in the order the commands are meant to be run (each step is detailed in its own section below):

# 1. Build the curated dataset from the MITRE CTID gold mappings
vulntrain-dataset-attack-generation --output-dir ./attack-dataset       # dry run, local only
vulntrain-dataset-attack-generation --push --repo-id CIRCL/vulnerability-attack-techniques

# 2. Train the multi-label classifier (GPU recommended)
vulntrain-train-attack-classification --base-model roberta-base \
  --repo-id CIRCL/vulnerability-attack-technique-classification-roberta-base

# 3. Evaluate: the trained model must beat the zero-shot similarity baseline
vulntrain-validate-attack-classification --method similarity
vulntrain-validate-attack-classification --method classifier \
  --model CIRCL/vulnerability-attack-technique-classification-roberta-base

# 4. Grow the dataset with LLM-assisted labeling — local Ollama model (no
#    API key) or Claude; validate agreement against the gold set BEFORE expanding
vulntrain-dataset-attack-llm-labeling --mode validate --backend ollama --model qwen3.5:122b
vulntrain-dataset-attack-llm-labeling --mode expand --backend ollama --model qwen3.5:122b \
  --sample-n 300 --push --agreement-note "f1_micro 0.392 on the 121-CVE gold test split"

# 5. Retrain on the gold + LLM union (LLM rows go into train only) and re-run step 3
vulntrain-train-attack-classification --base-model roberta-base \
  --extra-dataset-id CIRCL/vulnerability-attack-techniques-llm-ollama-qwen3.5-122b \
  --repo-id CIRCL/vulnerability-attack-technique-classification-pilot

Steps 1–3 are done and published. Step 4’s model selection is done (qwen3.5:122b, f1_micro 0.392 agreement — see the benchmark below), and the step-5 pilot (300-CVE expansion + union retrain) is complete: across five seeds it gives a small but consistent ranking gain (recall@3 +0.038, recall@5 +0.030) though no rare-technique improvement — a single-run version had misleadingly shown a degradation (see “Pilot expansion experiment” below). Source files (CTID mappings, ATT&CK STIX data, CVE2CAPEC databases) are cached in ~/.cache/vulntrain.

Candidate label sources, and what we measured#

There is no large ground-truth CVE → ATT&CK dataset. The candidate sources fall into two categories: small and hand-curated, or large and automatically derived. We evaluated both before deciding.

Gold sources: the MITRE CTID mappings (used as training labels)#

The MITRE Center for Threat-Informed Defense (CTID) produced two mapping efforts, both following the “Mapping ATT&CK to CVE for Impact” methodology, which assigns each CVE up to three kinds of techniques:

  • Exploitation technique — how the vulnerability is exploited (e.g. T1190 Exploit Public-Facing Application).

  • Primary impact — what the exploitation directly yields (e.g. T1059 Command and Scripting Interpreter).

  • Secondary impact — what the attacker can do afterwards.

The two sources:

Source

CVEs

ATT&CK version

Notes

attack_to_cve (2021)

~840

v9 era

CSV, the original project

Mappings Explorer KEV mappings

~420

16.1

JSON, CISA Known Exploited Vulnerabilities, includes per-mapping justification comments

Both are Apache-2.0 licensed. Together they cover 1,228 distinct CVEs (1,207 rows after normalization and description joining) — small, but every label was written by an analyst. The resulting technique distribution matches what one would expect from real-world exploitation: T1190 (348), T1059 (262), T1203 (213), T1068 (189), with 192 distinct techniques of which 66 have at least 5 examples (57 once sub-techniques are collapsed to their parent, which is what the trainer uses as its label vocabulary).

Derived source: CVE2CAPEC (included, but not as training labels)#

CVE2CAPEC (Galeax, GPLv3) maintains a daily-updated database chaining CVE → CWE → CAPEC → ATT&CK through the official cross-framework mappings. It is an impressive piece of automation with near-complete coverage, and it is referenced from issue #6, so we analyzed whether its technique labels could serve as training targets.

Measurements on its CVE-2024.jsonl database file (39,156 CVEs):

  • 88.3% of CVEs receive at least one technique — coverage is excellent.

  • But the fan-out is huge: the median CVE gets between 4 and 20 techniques, and 7,381 CVEs (19%) get 20 or more.

  • The most frequent technique overall is T1574.007 (Path Interception by PATH Environment Variable), tagged on 53% of all labeled CVEs — followed by T1574.006, T1562.003 and T1134.001, all around 50%. These frequencies bear no relation to how vulnerabilities are actually exploited; they are artifacts of the CWE → CAPEC → ATT&CK table expansion, where one generic CWE fans out into dozens of CAPECs and techniques.

  • Spot check: CVE-2024-21732, an XSS-family CVE (CWE-79), maps to 48 CAPECs and to techniques T1027 (Obfuscated Files or Information) and T1574.006/.007 (Hijack Execution Flow) — nothing related to XSS or drive-by exploitation.

Conclusion: training on these labels would teach the model the noise of the mapping tables rather than adversary behavior. The derived techniques are still valuable, so the dataset keeps them in a clearly separated techniques_derived column, useful as:

  1. a candidate prior at inference time (only suggest techniques compatible with the CWE chain) — measured 2026-08-06 and rejected: see “Derived-prior re-ranking” below;

  2. a baseline that any trained model must beat;

  3. a comparison column for studying where the deterministic chain diverges from analyst judgment.

Other sources considered#

  • BRON (MIT Lincoln Laboratory): same CWE → CAPEC → ATT&CK chain as CVE2CAPEC, same noise profile.

  • TRAM (CTID): maps threat reports to ATT&CK, not CVE descriptions — a different text distribution.

  • Academic work: CVE2ATT&CK (Grigorescu et al., 2022) fine-tuned BERT on ~1,800 CVEs and 31 techniques; SMET (Abdeen et al., ACSAC 2023) deliberately avoided supervised classification because of label scarcity and used semantic similarity against ATT&CK technique descriptions instead. That SMET-style similarity ranking is exactly what vulntrain-validate-attack-classification --method similarity implements as the zero-shot baseline (see Training below).

Pipeline#

vulntrain/datasets/attack_guesser_dataset.py performs the following steps:

  1. Fetch the two CTID mapping files.

  2. Normalize every technique ID against the current enterprise ATT&CK STIX data (attack-stix-data): techniques revoked since 2021 are remapped to their successor via the STIX revoked-by relationships (e.g. T1562 Impair Defenses → T1685 Disable or Modify Tools, revoked in v19); deprecated techniques have no successor and are dropped with a warning. Mobile and ICS techniques (T1404, T0855, …) present in a handful of 2021 mappings are also dropped — the dataset targets the enterprise ATT&CK domain only, which costs about 20 mobile-focused CVEs. The ATT&CK version used is recorded in the attack_version column.

  3. Merge the two sources per CVE (union of technique sets, provenance kept in label_sources).

  4. Join descriptions (and CPE lists) from CIRCL/vulnerability-scores; CVEs missing there fall back on the raw record.

  5. Fetch the raw CVE record for every CVE from the Vulnerability-Lookup API (disk-cached in ~/.cache/vulntrain/vuln-records/) and extract the structured metadata columns (v2): the CVSS vector (highest available version, CNA container preferred over ADP within a version — the CISA ADP Vulnrichment container supplies vectors and CWEs for many older CVEs), the CWE assignments, and the affected vendor/product pairs.

  6. Attach the CVE2CAPEC derived techniques as techniques_derived (skippable with --skip-cve2capec).

  7. Split 90/10 into train/test and optionally push to the Hub.

Note that the KEV mappings URL points to a dated release directory; pass --kev-mappings-url (or update the constant) when CTID publishes mappings for a newer ATT&CK release.

Dataset schema#

Column

Type

Description

id

str

CVE identifier

title

str

Vulnerability title

description

str

English vulnerability description (model input)

exploitation_techniques

list[str]

CTID exploitation technique(s)

primary_impact

list[str]

CTID primary impact technique(s)

secondary_impact

list[str]

CTID secondary impact technique(s)

techniques

list[str]

Union of all curated techniques (training target)

techniques_derived

list[str]

CVE2CAPEC weak labels — not for training

label_sources

list[str]

ctid_cve and/or ctid_kev

attack_version

str

Enterprise ATT&CK version the IDs are normalized to

cvss_vector

str

CVSS vector string, highest available version (v2, empty if none)

cvss_version

str

Version of cvss_vector (4.0/3.1/3.0/2.0, v2)

cwes

list[str]

CWE assignments, e.g. CWE-502 Deserialization of Untrusted Data (v2)

affected_products

list[str]

vendor product pairs from the CVE record (v2)

cpes

list[str]

CPE identifiers (from vulnerability-scores, raw-record fallback) (v2)

Columns marked (v2) were added on 2026-08-06 for the metadata-ablation follow-up work; v1 columns are unchanged (the extension is strictly additive: identical rows and splits, verified against the published v1). Coverage differs sharply by label source — stratify by label_sources when using these columns as inputs:

Subset

CVEs

cvss_vector

cwes

affected_products

cpes

all

1,207

72.0%

84.3%

67.4%

93.2%

ctid_kev

392

100%

100%

79.8%

79.1%

ctid_cve

788

57.1%

76.0%

62.2%

100%

both

27

100%

100%

40.7%

100%

The KEV subset owes its 100% CVSS/CWE coverage to the CISA ADP Vulnrichment container; the 2021 ctid_cve subset’s lower coverage is the source-truth limit for old CVEs (many have no metrics in either the CNA or ADP containers). CVSS versions among the 869 vectors: 677 × v3.1, 173 × v3.0, 18 × v4.0, 1 × v2.0.

Known limitations#

  • Size: ~1,200 CVEs supports a proof-of-concept, not a production model. LLM-assisted label expansion (see below), validated against this gold set, addresses this.

  • Selection bias: both CTID sets over-represent exploited-in-the-wild vulnerabilities (the KEV set by construction), so the technique distribution is skewed toward remote exploitation of servers compared to the full CVE corpus.

  • Version drift: the 2021 mappings were made against ATT&CK v9; normalization fixes revoked IDs but cannot retroactively add sub-techniques an analyst working today might have chosen.

  • Inherent task ceiling: a CVE description describes a flaw, while ATT&CK describes attacker behavior around it — even human annotators disagree on such mappings. Any model trained on this data should be presented as suggesting candidate techniques for analyst review, not as an authoritative mapping.

Training (Phase 2)#

The trainer is implemented in vulntrain/trainers/attack_guesser.py (vulntrain-train-attack-classification):

  • The task is multi-label (a CVE legitimately maps to several techniques): the model trains on the techniques column with a sigmoid head and binary cross-entropy loss, unlike the single-label CWE trainer. Per-label positive weights (--class-weights) counter class imbalance.

  • Sub-techniques are collapsed to their parent technique at training time (the same trick as the CWE ancestor mapping), and the label vocabulary is restricted to techniques with at least --min-examples (default 5) training examples.

  • Evaluation reports micro/macro F1 at the 0.5 threshold plus recall@3/recall@5, the metrics that matter for suggesting candidate techniques to an analyst.

  • The weak techniques_derived column is intentionally ignored by the trainer.

The fine-tuned classifier and a zero-shot similarity baseline (SMET-style — rank techniques by cosine similarity between the description embedding and the official ATT&CK technique name+description, no training involved) are evaluated with the same protocol by vulntrain-validate-attack-classification (vulntrain/validators/attack_guesser.py): both report the same recall@k and MRR on the same test split and label vocabulary, so the numbers are directly comparable. The fine-tuned model has to beat the zero-shot baseline to justify existing.

Model results (Phase 2)#

The first trained model, CIRCL/vulnerability-attack-technique-classification-roberta-base (roberta-base, 57-technique vocabulary), roughly doubles the zero-shot baseline on every ranking metric:

Metric

Zero-shot baseline

Fine-tuned model

recall@3

0.257

0.482

recall@5

0.322

0.686

recall@10

0.491

0.842

MRR

0.397

0.620

So the supervised approach is justified even on ~1,100 training examples. The remaining weakness is rare-technique performance (macro-F1 0.20), which is what label expansion targets.

Note (2026-07-17). The table above records the first trained model (pre-correction protocol, 57-technique vocabulary). The checkpoint published at that repo ID has since been retrained under the corrected protocol (seed 42, --val-split 0.1, 53-technique vocabulary); its expected metrics are the corrected-protocol numbers of record below.

LLM-assisted label expansion (Phase 2)#

vulntrain/datasets/attack_llm_labeler.py (vulntrain-dataset-attack-llm-labeling) grows the training set beyond the ~1,200 curated CVEs by having an LLM label additional CVEs with the same CTID methodology (exploitation technique / primary impact / secondary impact), so the output stays schema-compatible with the gold set.

Two backends, selected with --backend:

  • ollama (no API key, no per-token cost): labels with a local model served by an Ollama instance — e.g. Qwen — using Ollama structured outputs. Set --model (default qwen3; e.g. qwen3:32b) and, if the server is not local, --ollama-url.

  • anthropic: labels with Claude via the Anthropic API. Requires an API key exported as ANTHROPIC_API_KEY (create one at platform.claude.com; note that a Claude Max subscription does not include API access — it is billed separately).

The system prompt is identical for every CVE — the methodology, the full active enterprise ATT&CK technique catalog (from the STIX data), and a set of diverse few-shot examples drawn from the gold set — so both backends’ prompt-prefix caching keeps all but the first request cheap. The model returns a structured mapping (constrained to the label schema on both backends); hallucinated or out-of-catalog technique IDs are dropped, and the Ollama backend retries on malformed output.

The validate gate matters most with a local model: it tells you objectively whether the chosen Ollama model agrees with the analysts well enough to trust, or whether the gap justifies paying for the API.

Raising agreement. By default each CVE is labeled in a single constrained call. Because the JSON-schema grammar forces the model to emit the answer immediately, a thinking model (e.g. Qwen) cannot reason first — which tends to depress recall. Pass --reason for a two-step pass: an unconstrained analysis (the model reasons freely) followed by a constrained extraction of the technique IDs from that analysis. It roughly doubles the per-CVE time, so compare it against the single-call baseline on a small --limit before committing to a full run. Any change to the model, the prompt, or --reason invalidates a previous agreement number — re-run validate to re-baseline.

Model-selection benchmark. We measured LLM-vs-gold agreement on a held-out slice of the CTID gold set, at the parent-technique granularity the trainer uses, to pick the best model to expand with. We use the trained classifier’s own agreement with gold (f1_micro ~0.41) as a reference level — the intuition being that a labeler below it adds labels noisier than the model’s own predictions. This is a reference, not a hard gate: the seed sweep below shows labels at 0.39 agreement still help ranking in aggregate, so the figure contextualises the labeler rather than accepting or rejecting it.

Backend / model

Prompt & mode

Sample

Precision

Recall

f1_micro

Notes

ollama / qwen3.6:35b

conservative, single-call

30

0.429

0.248

0.314

original baseline

ollama / qwen3.6:35b

assertive, single-call

30

0.442

0.271

0.336

prompt helps marginally

ollama / qwen3.6:35b

assertive, --reason

30

0.395

0.214

0.278

worse; reasoning pass times out, drops CVEs

supervised classifier

(trained on gold)

121

—

—

~0.41

reference level

ollama / qwen3.5:122b

assertive, single-call

30

0.509

0.429

0.465

optimistic on the small slice

ollama / qwen3.5:122b

assertive, single-call

121

0.431

0.360

0.392

full split — the reliable figure

Few-shot examples: 8, identical across rows. The 35B rows and the first 122B row share one 30-CVE slice; the final row is the full 121-CVE test split. Three findings emerge:

  1. Model capacity, not prompt engineering, is the binding constraint. On the 30-CVE slice the assertive prompt lifted the 35B by only +0.02 f1, while moving to the 122B lifted it by +0.13, almost entirely by fixing recall (0.27 → 0.43). The smaller model under-predicts — it agrees when it commits, but stays silent too often. Capacity is what buys the commitment.

  2. Two-step --reason did not pay off on a mid-size thinking model. On the same 30 CVEs it scored below single-call (0.278 vs 0.336), and the unconstrained reasoning pass on the 35B repeatedly exceeded the Ollama timeout, dropping whole CVEs to empty labels. It may still help a larger box with a longer timeout, but it is not a substitute for model size.

  3. Small validation slices are optimistic — always confirm on the full split. The 122B scored 0.465 on 30 CVEs but 0.392 on the full 121, a 0.07 f1 drop driven mostly by recall (0.43 → 0.36). At n=30 the agreement metric has enough variance to mislead a go/no-go decision, so the full-split number is the one of record.

Selected expansion model: qwen3.5:122b (single-call, assertive prompt), at f1_micro 0.392 on the full test split. This sits marginally below the classifier’s own ~0.41 agreement — so on the benchmark alone the LLM is not clearly better than the trained model at reproducing gold. That made it a best-case candidate to test rather than a sure thing; the seed sweep below is what actually decides it. Its agreement is within the range commonly reported for inter-analyst agreement on technique-level ATT&CK CVE mappings, and it is used for provenance-tiered expansion (new CVEs, label_sources=["llm"]), never as a silent replacement for gold labels. Throughput is ~1.4 min/CVE on our GPU server (2× NVIDIA H100 NVL, Ollama): the 1000-CVE scaling batch took 23h03m and kept 984 CVEs, pushed as CIRCL/vulnerability-attack-techniques-llm-scaling.

Validate before trusting expansion. Run the validate mode first: it labels a held-out slice of the gold set and reports agreement (precision/recall/F1 at the parent-technique level) between the model and the analysts.

# Local model via Ollama (no API key):
vulntrain-dataset-attack-llm-labeling --mode validate --backend ollama --model qwen3.6:35b

# Or Claude via the Anthropic API:
export ANTHROPIC_API_KEY=sk-ant-...
vulntrain-dataset-attack-llm-labeling --mode validate --backend anthropic

Only if that agreement is comparable to inter-analyst agreement on ATT&CK mappings should you scale up. The expand mode then labels a sample of CVEs (from CIRCL/vulnerability-scores by default, excluding gold CVEs) and writes a dataset with label_source = ["llm"] plus the backend/model ID and its justification comment per row:

vulntrain-dataset-attack-llm-labeling --mode expand --backend ollama --model qwen3.6:35b \
  --sample-n 2000 --push --repo-id CIRCL/vulnerability-attack-techniques-llm \
  --agreement-note "f1_micro 0.61 on the 121-CVE test split"

Each run appends the backend/model slug to --repo-id (so the example above pushes to …-llm-ollama-qwen3.6-35b) and writes a dataset card recording the labeling model, the CVE count, and — via --agreement-note — the validation score. This keeps multiple test runs (one per model) distinguishable rather than overwriting one another; the exact model is also stored per row in the llm_model column. Pass --no-model-suffix to push to --repo-id verbatim.

Keep the LLM-labeled rows in a separate provenance tier: merge them with the gold set for training, but always retain the label_sources column so consumers can filter back to gold-only, and publish the measured validation agreement on the expanded dataset card (the --agreement-note flag does exactly this) so the labels’ quality is documented rather than assumed.

Retraining on the gold + LLM union#

The trainer merges the two provenance tiers through --extra-dataset-id:

vulntrain-train-attack-classification --base-model roberta-base \
  --extra-dataset-id CIRCL/vulnerability-attack-techniques-llm-ollama-qwen3.5-122b \
  --repo-id CIRCL/vulnerability-attack-technique-classification-pilot

The extra rows are concatenated into the train split only; the gold test split is left untouched. This is the crucial part of the experimental design: the yardstick stays gold-only, so the union model’s test metrics are directly comparable to the gold-only model’s. Use a distinct --repo-id (e.g. a -pilot suffix) so the experiment never overwrites the production gold-only model.

Pilot expansion experiment#

To decide whether LLM expansion is worth scaling, we run a small, measurable pilot rather than committing to a full expansion up front.

Design.

  1. expand 300 new non-gold CVEs with the selected model (qwen3.5:122b, single-call, assertive prompt), recording the 0.392 full-split agreement on the dataset card via --agreement-note.

  2. Retrain the classifier on the gold-train + LLM union with --extra-dataset-id, evaluating on the untouched gold test split.

  3. Compare against the gold-only baseline on the same test split.

Success criterion. The pilot succeeds if recall rises — especially recall_at_5 and f1_macro (which weights rare techniques equally) — without f1_micro collapsing. The hypothesis under test is that LLM labels, even at 0.39 agreement, add coverage of rare techniques that the ~1,200-CVE gold set under-represents. A flat or worse result means expansion does not pay off at this agreement level, and the gold-only model stays the product.

Important — this experiment produced three successive verdicts, and only the last survives. The single-run pilot (seed 42) said expansion degrades the model. A five-seed sweep said it helps (small consistent ranking gain). An independent replication plus an expansion-size scaling sweep showed both were artifacts: expansion at ~0.39 agreement gives no reliable gain at any size from 100 to 984 rows, and degrades rare-technique macro-F1 at scale. The root cause of the churn was evaluation noise from best-checkpoint selection on the small test split (now fixed in the trainer — see below). The earlier tables are kept as cautionary examples.

Single-run pilot (seed 42) — misleading. Both models were trained with identical code, seed (42), and hyper-parameters; the only difference is the 297 LLM-labeled rows folded into training (train split only; gold test untouched). The gold-only figures are a matched re-run under the current code (f1_micro 0.407; the old 0.42 was a slightly different configuration).

Metric

Gold-only (seed 42)

Gold + LLM (seed 42)

Δ

f1_micro

0.407

0.395

−0.012

f1_macro

0.185

0.164

−0.021

recall_micro

0.625

0.626

+0.001

recall_at_3

0.546

0.491

−0.055

recall_at_5

0.683

0.633

−0.050

Taken alone this says expansion hurts. That conclusion did not survive — though the reason turned out to be subtler than first thought (see the mechanism below).

Five-seed sweep (seeds 42–46) — appeared to reverse it; second cautionary example. Mean ± std across seeds; Δ is the mean of the paired per-seed differences. “Consistent” marks |Δ| > 2·SEM (see --seed on the trainer and aggregate_sweep.py). At the time this was adopted as the number of record; the replication below showed the gold-only column had drawn low.

Metric

Gold-only

Gold + LLM

Δ (paired)

recall_at_3

0.506 ± 0.019

0.544 ± 0.023

+0.038

consistent ↑

recall_at_5

0.641 ± 0.019

0.670 ± 0.033

+0.030

consistent ↑

f1_micro

0.405 ± 0.019

0.424 ± 0.010

+0.020

consistent ↑

f1_macro

0.177 ± 0.012

0.173 ± 0.017

−0.004

within noise

recall_micro

0.651 ± 0.013

0.636 ± 0.007

−0.015

consistent ↓

Scaling sweep + replication (2026-07-15) — the result of record. A fresh, independently sampled batch of 1,000 CVEs was labeled with the same configuration (984 kept, pushed as CIRCL/vulnerability-attack-techniques-llm-scaling), and the trainer’s --extra-max-rows folded in the first N rows (nested subsets), five seeds per size, against a matched gold-only baseline from the same session (scaling_sweep.py):

N extra rows

recall_at_3

recall_at_5

f1_micro

f1_macro

0

0.531 ± 0.025

0.682 ± 0.021

0.418 ± 0.016

0.189 ± 0.014

100

0.534 ± 0.018

0.655 ± 0.025

0.408 ± 0.022

0.170 ± 0.013

300

0.523 ± 0.016

0.671 ± 0.011

0.412 ± 0.014

0.175 ± 0.006

600

0.536 ± 0.024

0.656 ± 0.031

0.416 ± 0.026

0.175 ± 0.004

984

0.532 ± 0.018

0.651 ± 0.013

0.437 ± 0.013

0.150 ± 0.007

No size reproduces the five-seed gain: recall@3 is flat, recall@5 declines at full size (−0.031, 2.8 SEM), f1_macro degrades markedly (−0.039, 5.6 SEM — the strongest effect in the sweep), and only f1_micro edges up at 984 (+0.019, borderline 2.1 SEM). Re-running the original 297-row union in the same session reproduced its recall@5 almost exactly (0.670 ± 0.007 vs 0.670 ± 0.033 in the old sweep) — but against the matched baseline of 0.682 ± 0.021, not 0.641. The five-seed “gain” was a low-drawn gold baseline, not a property of the union model.

The mechanism — evaluation noise from checkpoint selection. The trainer picked the best of 40 per-epoch checkpoints by macro-F1 on the 119-example test split. Per-epoch evals of a split that small wobble hugely (recall@5 spans 0.62–0.70 within one run), so the reported metric is an argmax over 40 noisy evaluations, and ordinary GPU nondeterminism changes which epoch wins: three runs of the identical gold-only configuration (same seed, data, code, hardware) reported recall@5 of 0.636, 0.683, and 0.685 — a 0.048 spread from evaluation noise alone. This defeats single runs and the paired 2·SEM test. Fixed in the trainer: --val-split (default 0.1) carves a gold-only validation split for checkpoint selection so the test split is evaluated exactly once, and --deterministic (transformers full_determinism) makes fixed-seed runs bit-reproducible — note it sets CUDA_LAUNCH_BLOCKING=1, which deadlocks multi-GPU DataParallel (observed on 2× H100: zero steps in eight hours; the trainer now refuses to start with more than one visible GPU) and slows single-GPU training. Reserve it for single-GPU archival runs (CUDA_VISIBLE_DEVICES=0); a multi-seed sweep already averages over run-to-run noise and does not need it. Numbers above predate the fix (mild select-on-test optimism, same on both sides of every comparison).

Numbers of record (corrected protocol, 2026-07-16). Gold-only, --val-split 0.1 (972 train / 106 validation / 118 test), five seeds (42–46):

recall@5

recall@3

micro-F1

macro-F1

0.673 ± 0.019

0.536 ± 0.032

0.410 ± 0.006

0.177 ± 0.014

As predicted, about one point below the pre-correction values (0.682 / 0.531 / 0.418 / 0.189) and within noise of them — an upper bound on the select-on-test optimism that also absorbs the 10% of training data ceded to the validation split. Micro-F1’s run-to-run std drops from ±0.016 to ±0.006: selecting checkpoints on a dedicated split removes variance, not just bias.

The verdict under the corrected instrument (2026-07-16). The decisive contrast re-run under the corrected protocol — the same five seeds, gold-only vs the 297-row union vs all 984 LLM rows:

training data

recall@5

recall@3

micro-F1

macro-F1

gold only

0.673 ± 0.019

0.536 ± 0.032

0.410 ± 0.006

0.177 ± 0.014

gold + 297 LLM rows

0.655 ± 0.027

0.511 ± 0.023

0.404 ± 0.012

0.169 ± 0.009

gold + 984 LLM rows

0.651 ± 0.022

0.534 ± 0.012

0.427 ± 0.028

0.151 ± 0.014

The null verdict is confirmed with the cleaner instrument: no ranking gain at either size (at 297 rows every metric sits at or below gold-only — no trace of the once-“consistent” gain), the macro-F1 degradation at scale is confirmed (−0.026, ≈2.9 SEM), and the borderline micro-F1 uptick at 984 persists unresolved (+0.017, ≈1.3 SEM).

Gold labels scale; LLM labels do not (2026-07-16). The natural control: train on nested subsets of the gold train split (--train-fraction, label vocabulary and test set frozen to the full-gold ones, five seeds each):

gold rows

recall@5

recall@3

micro-F1

macro-F1

243

0.556 ± 0.034

0.384 ± 0.045

0.328 ± 0.013

0.114 ± 0.018

484

0.623 ± 0.023

0.475 ± 0.019

0.383 ± 0.014

0.153 ± 0.017

725

0.656 ± 0.037

0.507 ± 0.015

0.395 ± 0.028

0.173 ± 0.035

972

0.673 ± 0.019

0.536 ± 0.032

0.410 ± 0.006

0.177 ± 0.014

Every metric rises monotonically with gold size, and the curve has not saturated. The contrast with expansion is stark: the last ~250 gold rows add +0.017 recall@5, while 297 LLM rows added to the same full gold set subtract 0.018. The classifier is label-quality bound, not data bound — growing the curated set is the one intervention with measured payoff.

Protocol warning. A naive version of this experiment — rebuilding the label vocabulary from each subset — inverts the macro-F1 trend (0.282 at 243 rows): smaller train sets yield smaller vocabularies, an easier averaging set, and (through the in-vocabulary test filter) an easier test set. --train-fraction therefore freezes the vocabulary and test set to the full-gold ones.

Base-model robustness check (2026-07-17). The same contrast re-run on answerdotai/ModernBERT-base — ten runs, corrected protocol, five seeds × {gold-only, gold + 984 LLM rows} — deliberately slimmed from a full encoder grid to a robustness check:

encoder

training data

recall@5

recall@3

micro-F1

macro-F1

roberta-base

gold only

0.673 ± 0.019

0.536 ± 0.032

0.410 ± 0.006

0.177 ± 0.014

roberta-base

gold + 984 LLM rows

0.651 ± 0.022

0.534 ± 0.012

0.427 ± 0.028

0.151 ± 0.014

ModernBERT-base

gold only

0.614 ± 0.018

0.503 ± 0.017

0.416 ± 0.020

0.180 ± 0.014

ModernBERT-base

gold + 984 LLM rows

0.635 ± 0.029

0.524 ± 0.023

0.450 ± 0.020

0.152 ± 0.015

Three take-aways. (1) The newer encoder does not raise the ceiling: micro/macro-F1 are a wash and both ranking metrics are clearly lower (recall@5 −0.059, ≈5 SEM) — roberta-base stays the released default. (2) The expansion verdict replicates on a second encoder: macro-F1 again degrades (0.180 → 0.152, ≈3.1 SEM) and recall@5 again shows no reliable gain (+0.021, ≈1.4 SEM). (3) The micro-F1 uptick that was borderline on roberta-base resolves here (+0.034, ≈2.7 SEM) — LLM rows concentrated on head techniques buy example-weighted F1 while eroding the tail, consistent with the mechanism above.

Decision. Expansion at ~0.39 agreement is not worth folding in: no reliable ranking gain at any size, and a real rare-technique cost at scale. The gold-only model stays the product. The result also vindicates the “reference level” heuristic — a labeler agreeing below the classifier’s own accuracy (0.392 < 0.407) added nothing the model didn’t already know.

The methodological lesson matters as much as the metrics: multi-seed reporting was necessary but not sufficient — the five-seed comparison passed its own consistency criterion and was still wrong. In this regime you also need a selection split that is not the test split, deterministic (or repeated) runs, and replication on an independent sample before believing a small effect.

Derived-prior re-ranking (negative result, 2026-08-06)#

The one remaining untested use of the CVE2CAPEC techniques_derived column was as an inference-time candidate prior: re-rank the trained classifier’s scores toward the techniques the CWE → CAPEC → ATT&CK chain considers compatible. The validator now implements this as --prior boost (add --prior-alpha to the logit of every derived candidate) and --prior mask (candidates rank strictly first — the α → ∞ limit), with --split validation reconstructing the trainer’s checkpoint-selection carve-out so α is tuned without touching the test split:

vulntrain-validate-attack-classification --method classifier \
  --model CIRCL/vulnerability-attack-technique-classification-roberta-base \
  --split validation --prior boost --prior-alpha 0.25 0.5 1 2 4 8

The prior only hurts, at every strength. On the validation sweep the ranking metrics degrade monotonically with α (MRR 0.597 → 0.590 at α=0.25 → 0.287 at α=4); the tuned choice is therefore “no prior”. On the test split (evaluated once, for the record): boost at the mildest α=0.25 is flat-to-worse on every metric (recall@3 0.518 → 0.509, MRR 0.616 → 0.608), and mask is catastrophic (recall@5 0.644 → 0.494, MRR 0.616 → 0.340).

The mechanism is coverage, not weighting. The derived candidate sets almost never contain the analyst-chosen techniques: at the parent level they cover 3.3% of the gold techniques on test (6.7% on train) — the hard ceiling of any mask-style filter. The most frequent derived parents (T1134, T1562, T1574 — the table-expansion artifacts quantified above) are entirely disjoint from the most frequent gold parents (T1190, T1059, T1203, T1068). The chain’s noise is not random scatter around the right answer; it is systematically pointed at different techniques than analyst judgment, so no re-weighting can rescue it. This closes the last proposed use of the derived labels beyond documentation: the techniques_derived column remains in the dataset for transparency and as a comparison column only.

Metadata ablation: CWE pays, nothing else does (2026-08-07)#

The dataset v2 metadata columns exist to answer one question: does any structured signal add technique information the description doesn’t already carry? The trainer’s --metadata cvss|cwe|products|derived|all flag appends the verbalized signal(s) to the input text (constant section headers, explicit unknown for missing values — see vulntrain/attack_metadata.py); the chosen signals are recorded in the model config (metadata_inputs), and the validator and single-CVE inference rebuild the exact training-time input automatically.

Six arms × five seeds (42–46), corrected protocol, roberta-base, 2× H100 (papers sweep scripts; logs archived with the paper):

input

recall@5

recall@3

micro-F1

macro-F1

description only

0.664 ± 0.017

0.521 ± 0.011

0.408 ± 0.011

0.182 ± 0.018

+ CVSS vector

0.666 ± 0.026

0.529 ± 0.032

0.405 ± 0.018

0.181 ± 0.020

+ CWE (gold)

0.710 ± 0.009

0.546 ± 0.013

0.427 ± 0.011

0.199 ± 0.008

+ affected products

0.646 ± 0.016

0.517 ± 0.021

0.396 ± 0.012

0.166 ± 0.011

+ derived candidates

0.642 ± 0.021

0.523 ± 0.023

0.398 ± 0.013

0.168 ± 0.009

all signals

0.659 ± 0.027

0.506 ± 0.023

0.390 ± 0.009

0.176 ± 0.015

CWE is the only signal that pays — and it pays on every metric. The paired per-seed deltas vs the description-only baseline clear the 2·SEM consistency threshold on all four metrics (recall@5 +0.046, recall@3 +0.025, micro-F1 +0.018, macro-F1 +0.017), and the CWE arm also has the smallest seed spread of any arm: the weakness class doesn’t just add information, it stabilizes what the model learns from free text. CVSS is a null result (+0.001 recall@5 — severity is already recoverable from the description). Affected products and the derived candidate IDs are mildly harmful; the latter confirms the E1 re-ranking verdict from the input side. Concatenating everything (all) destroys the CWE gain (recall@3 −0.015, micro-F1 −0.018, both consistent): at ~1,000 training examples, input serialization is signal selection, not accumulation.

Because gold-CWE coverage differs by label source (100% on ctid_kev vs 76.0% on the 2021 ctid_cve rows), the validator’s --stratify flag reports metrics per label_sources group to check where the gain concentrates.

Stratified replication (2026-08-07). The grid above retained no checkpoints, so the stratified analysis retrained all 30 runs (same seeds and hyperparameters). Two findings:

  1. The CWE gain attenuates under retraining. With GPU nondeterminism as the only difference, the overall paired delta shrinks from +0.046 (consistent on all four metrics) to +0.014 recall@5, with only macro-F1 (+0.012) still clearing 2·SEM — seed spread understates run-to-run variance. The qualitative ordering is unchanged in both grids: CWE is the only arm ever consistently positive, products the only one consistently harmful in both.

  2. The gain concentrates where CWE coverage is complete. Per-stratum paired deltas (recall@5): +0.053 on ctid_kev (consistent; 100% gold-CWE coverage, curated alongside the KEV process) vs −0.003 on ctid_cve (76% coverage, ordinary NVD assignments). Missing-token dilution alone can’t explain the split (76% coverage of a uniform +0.053 effect would still show ≈ +0.040); the description-only baseline is also much weaker on KEV rows (0.614 vs 0.715 recall@5), so headroom and/or CWE assignment quality drive the difference. The KEV stratum is small (n=36): location established, magnitude indicative.

CWE-presence re-evaluation (2026-08-08). The stratified sweep kept its 30 final models, so a second, inference-only pass re-scored them with --stratify extended to gold-CWE-presence cross-strata (per label_sources group, per presence, and their intersection). This rules out coverage dilution directly: on the 64 ctid_cve rows that do carry a gold CWE, the cwe arm gains nothing (+0.004 recall@5, null on every metric), while the same models gain +0.053 on ctid_kev. The gain is a property of the KEV stratum — curated (CISA-enrichment-grade) CWE assignments and/or more headroom (the description-only baseline is 0.614 there vs 0.700 on the CWE-covered ctid_cve rows) — not of token availability. The CWE-absent ctid_cve rows are the easy ones (desc 0.774 recall@5, n=17). Pooled over both grids (10 paired runs), the honest overall effect of gold CWE is +0.030 recall@5, +0.011 micro-F1, +0.015 macro-F1 (all consistent), +0.010 recall@3 (not).

Cascade arm: predicted CWE (2026-08-11). With coverage ruled out, a cascade arm separates the two remaining explanations on the KEV rows: if headroom drives the gain, an NVD-grade predicted CWE should still help there; if curation quality drives it, the cascade gain should vanish. Dataset v2.1 adds a cwes_predicted column (top-1 of CIRCL/cwe-parent-vulnerability-classification-roberta-base over title+description; 100% coverage by construction, 27.3% ancestor-level top-1 agreement with gold), and the trainer gained --metadata cwe_predicted — verbalized under the same CWE: header as gold, so signal quality is the only difference; mutually exclusive with cwe and excluded from all. Three arms × five seeds, all retrained fresh in one session (the attenuation lesson):

input (recall@5)

overall (n=121)

ctid_cve (n=81)

ctid_kev (n=36)

description only

0.664 ± 0.010

0.701 ± 0.015

0.587 ± 0.042

+ CWE (gold)

+0.026

+0.003

+0.082*

+ CWE (predicted)

−0.006

−0.027

+0.038

The cascade is closed: the KEV gain is substantially curation quality, not headroom. Gold CWE replicates its KEV gain a third time, strongest yet (+0.082 recall@5, +0.034 micro-F1, +0.014 macro-F1, all consistent) — the one ablation finding that survives every retraining. The predicted CWE recovers less than half of it (+0.038, not consistent) and clears 2·SEM on no metric in no stratum; overall it is null (−0.006 recall@5). At current CWE-classifier accuracy, feeding predicted CWEs costs nothing but buys nothing — the deployment configuration stays description-only, with gold CWE appended when a curated assignment exists. (Note the mild leakage direction: the CWE guesser’s training data likely includes NVD descriptions+CWEs of these test CVEs, which can only have flattered the cascade arm — the negative verdict survives it.) The description-only baseline also moved between sessions again (0.682 → 0.664 overall recall@5, same seeds and code): a third replication of run-to-run variance exceeding seed spread.

Label semantics: bi-encoder vs classification head (2026-08-12)#

A per-label classification head cannot score a technique it was never trained on — the 53-technique vocabulary floor is structural. The label-semantics alternative (vulntrain-train-attack-biencoder) embeds the CVE text and each technique’s official STIX name+description with one shared roberta-base encoder; the score is a learnable affine over the cosine, trained with the same weighted BCE, carve-out, and frozen vocabulary as the head. Any technique with an official description can then be ranked. Head-to-head (five paired seeds, fresh desc baseline, n=118):

model

recall@5

MRR

micro-F1

macro-F1

classification head

0.667 ± 0.015

0.632

0.409

0.176

bi-encoder

0.643 ± 0.019

0.626

0.419

0.212

paired Δ

−0.024*

−0.006

+0.010

+0.037*

In-vocabulary, the bi-encoder is the best architectural gain measured on this task: the largest consistent macro-F1 improvement of the experiment series (+21% relative on the binding-constraint metric), at a small head-class ranking cost concentrated entirely on the 17 CWE-absent test rows. Open-vocabulary (ranking over all 222 active parent techniques, --candidates full), it holds 0.515 ± 0.020 recall@5 — 2.3× a generic zero-shot MiniLM reference (0.227).

Below the vocabulary floor, the zero-shot promise does not survive fine-tuning. A five-fold label-holdout evaluation (--holdout-techniques: each fold retrains with ~11 vocabulary techniques withheld — no labeled examples, no technique text — and scores only the held-out gold over 222 candidates):

model (recall@5 on held-out gold)

fold 0

fold 1

fold 2

fold 3

fold 4

mean ± std

bi-encoder (holdout-trained)

0.057

0.010

0.321

0.000

0.229

0.123 ± 0.144

MiniLM (zero-shot, same gold)

0.211

0.078

0.369

0.333

0.195

0.237 ± 0.117

The fine-tuned encoder ranks unseen techniques worse than an off-the-shelf embedder in 4 of 5 folds, while staying healthy on its trained labels (0.544 ± 0.037 in-vocab recall@5, open candidates): fine-tuning specializes the embedding space toward the trained technique texts and away from unseen ones. Nothing evaluated is operationally usable below the floor (the best, generic MiniLM, gets 0.24 recall@5 over 222 candidates) — the tail below the vocabulary floor is data-bound, and only more curated examples reach it.

A seed-42 bi-encoder is published as CIRCL/vulnerability-attack-technique-biencoder (DOI: 10.57967/hf/9968), shipping the training-time technique texts (technique_texts.json) and scoring calibration (config.biencoder) alongside the weights; the model card carries a usage snippet and the zero-shot caveat. Prefer it over the classification head when rare-technique coverage matters more than top-5 sharpness.

Bucket-aware training: role structure costs union accuracy (2026-08-12)#

The CTID methodology assigns each technique a role — exploitation technique, primary impact, secondary impact — that the flattened union target discards. Two trainer arms tested whether that structure helps (3 arms × 5 paired seeds, fresh same-session baseline):

  • --bucket-multitask widens the head to 4×53 blocks (the three named roles plus an uncategorized slot for the ~27% of training occurrences the source mapping leaves unbucketed); the union task (max over buckets) drives checkpoint selection and reported metrics, so the paired comparison is exact.

  • --tactic-aux 0.5 adds an auxiliary tactic-level head (15 tactics in the current STIX bundle) supervised for free from the gold techniques’ kill-chain phases, dropped at save time.

arm (Δ vs union baseline)

recall@5

recall@3

MRR

micro-F1

macro-F1

union baseline (absolute)

0.668 ± 0.018

0.539

0.642

0.413

0.182

bucket multi-task

−0.076*

−0.083*

−0.044*

−0.065*

−0.072*

tactic auxiliary

−0.018

−0.065*

−0.061*

−0.036*

−0.030*

Both arms lose; the bucket arm loses on every metric consistently. Splitting ~2,500 positive training occurrences across four role blocks starves each of them — at ~1,000 examples the union task needs all the supervision on one head. The tactic gradient competes rather than regularizes (λ untuned; measured harm at the one pre-registered weight). The structure itself is learnable, though: scored against role-restricted gold, the bucket model ranks the exploitation technique at 0.781 recall@5, primary impact at 0.682, secondary impact at 0.527 — role-conditioned prediction (which the union model cannot express) works, at a union-accuracy price a deployment likely won’t pay. The flattened-union, description-only classifier remains the deployment configuration.

Inspecting a single CVE#

vulntrain-infer-attack-classification runs one description (or one CVE looked up in the gold dataset, with its gold techniques shown) through a trained checkpoint:

vulntrain-infer-attack-classification --cve CVE-2021-44077 \
  --model CIRCL/vulnerability-attack-technique-classification-roberta-base

CVE-2021-44077 (test split)
gold techniques: T1003, T1027, T1047, T1070, T1087, T1136, T1140, T1190, ...
rank  technique   prob  pred  gold  name
   1  T1133       0.72   *          External Remote Services
   2  T1190       0.72   *     +    Exploit Public-Facing Application
   3  T1059       0.70   *          Command and Scripting Interpreter
   4  T1005       0.63   *          Data from Local System
   5  T1505       0.58   *     +    Server Software Component

Running the same CVE through the published negative-result checkpoint ...-roberta-base-llm-expanded (gold + 984 LLM rows; released for reproducibility of the expansion experiment, not for use) shows the expansion mechanism on one example: the head call T1190 gains confidence (0.72 → 0.75) while the rest of the gold mapping deflates — T1505 drops below the 0.5 prediction threshold, T1003 falls from rank 8 to 17, and the tail technique T1136 (5 training examples) from rank 18 to 32.

Published artifacts#

DOIs generated via Hugging Face / DataCite on 2026-07-17.

Still to do#

  • ~~Retrain and republish the released checkpoint~~ — done 2026-07-17 (corrected protocol, seed 42; the -llm-expanded companion published at the same time). Optionally one single-GPU --deterministic archival run.

  • Raise labeler agreement before any further expansion attempt — stronger model, human-reviewed silver labels, or high-confidence slots only — aimed at the long tail that f1_macro shows is the binding constraint.

  • Stratify any future expansion sample by CWE so it isn’t dominated by the most common weakness classes (XSS, SQLi); the current expand mode samples CVEs without stratification.

  • Grow the gold set directly (more CTID-style curated mappings) — the gold scaling curve shows every metric still rising at 972 training rows, so this is the intervention with measured payoff.

  • ~~Base-model comparison~~ — done as a slimmed robustness check (ModernBERT-base, 2026-07-17, table above): roberta-base stays the default, the expansion verdict replicates. A full encoder search (roberta-large, SecureBERT) remains optional future work.