CNVD Severity Classifier: Improvements Report

CNVD Severity Classifier: Improvements Report#

Technical report on changes made to the CNVD severity trainer (classify_severity_cnvd.py) following the independent analysis in VulnTrain#19.

Context#

An external review of the CIRCL/vulnerability-severity-classification-chinese-macbert-base model identified several issues: data leakage in the train/test split, poor Low-class recall, keyword dependency, and a suboptimal published checkpoint. The reported headline accuracy of 77.83% was inflated by ~1.7pp due to leakage.

Changes#

Data leakage fix#

Problem: CNVD reuses boilerplate descriptions across different vulnerability IDs. The original train_test_split split by row index, allowing 1,587 identical descriptions (15.6% of the test set) to appear in both splits.

Fix: New deduplicate_split() function groups all entries by description text and assigns entire groups to one split. No description appears in both train and test.

Impact: The old model evaluated on the deduplicated test set scores 85.2% (inflated — it was trained on data overlapping this test set). A model retrained on the deduplicated split scores 76.8%, matching the independently measured unleaked accuracy of 76.6%.

Class weighting experiments#

Problem: The Low class (~9% of data) had only ~41% recall on unleaked data, with 60% of Low entries misclassified as Medium.

Four loss strategies were tested on the deduplicated split:

Mode

Low recall

Medium recall

High recall

Overall acc

Uniform (none)

0.4099

0.8165

0.7809

0.7677

Sqrt-dampened

0.4902

0.7481

0.8056

0.7457

Balanced

0.6084

0.7024

0.8099

0.7323

Focal (gamma=2)

0.6328

0.6441

0.8349

0.7110

Conclusion: Every weighting strategy that improved Low recall caused disproportionate Medium recall loss. The Low/Medium vocabulary overlap in CNVD descriptions makes this a data-level limitation, not a loss-function problem. The trainer defaults to uniform loss.

A --class-weights flag (none, sqrt, balanced, focal) was added for future experimentation.

Per-class metrics#

compute_metrics now reports precision, recall, and F1 per class (Low/Medium/High) alongside overall accuracy and macro F1 at each evaluation epoch.

Best model checkpoint selection#

  • metric_for_best_model set to accuracy (was defaulting to eval_loss)

  • save_total_limit increased from 2 to 3 to prevent the best checkpoint from being pruned

CodeCarbon tracker scoping#

The @track_emissions decorator wrapped the entire train() function, including push_to_hub(). The codecarbon background thread never stopped during the upload. Replaced with an explicit EmissionsTracker start/stop scoped to trainer.train() only. Also removed push_to_hub=True from TrainingArguments (it caused trainer.train() to upload internally before returning). The same fix was applied to classify_severity.py.

Dynamic model card#

The model card is now a template (model_card_cnvd_severity.md) populated with actual eval metrics from trainer.evaluate() after each training run. Documents per-class metrics, training configuration, and known limitations.

Known limitations documented#

The model card now documents:

  • Low severity recall (~41%): ~60% of Low entries are misclassified as Medium due to vocabulary overlap. All weighting strategies degrade Medium recall disproportionately.

  • Keyword dependency: the model biases toward a vulnerability type’s typical severity. Accuracy drops from ~89% to ~55% on atypical-severity entries.

  • Negation blindness: “does NOT allow RCE” still predicts High with high confidence.

  • CVE overlap: 81% of CNVD entries have a CVE equivalent. The model primarily adds value for the ~19% CNVD-only entries.

These findings align with independent results from CyberScale Phase 1 (Point 29), which plateaued at ~62% band accuracy on a 4-class CVSS classifier using ModernBERT-base with similar approaches (CWE enrichment, multi-task heads, CPE features — none moving the needle beyond ~2pp).

Dataset improvements#

CVE cross-references#

The extract_cnvd function now extracts the cve_id field from cves.cve.cveNumber in the raw CNVD JSON. This enables users to cross-reference CNVD entries with their CVE equivalents and filter CNVD-only entries.

Dataset card#

A dataset card (dataset_card_cnvd.md) was added documenting:

  • Field descriptions including the new cve_id column

  • CVE overlap rate: 81% overall (68-69% in 2020-2021, 91-97% after 2022)

  • Severity distribution: High ~36%, Medium ~55%, Low ~9%

  • Coverage decline: 94% of reserved IDs published in 2015 → 4% in 2023 (post-RMSV regulations, September 2021)

  • Warning about duplicate descriptions and train/test split leakage

Comparison validator#

A dedicated validator (validators/severity_cnvd.py) was added to evaluate the old and new models side by side on the same deduplicated test set. It reports per-class precision/recall/F1, confusion matrices, and a summary delta table.

Commits#

Commit

Description

6352273

Data leakage fix, class-weighted loss, per-class metrics, best model selection

b1679fb

CNVD severity model comparison validator

1fdee05

Sqrt-dampened class weights

a81f90d

Scoped codecarbon tracker (CNVD trainer)

65d3d88

Removed push_to_hub from TrainingArguments

b7a2d6d

Scoped codecarbon tracker (severity trainer)

ed2c230

--class-weights flag (none/sqrt/balanced)

f1cd426

Focal loss option

9fa0f86

Default to uniform loss

7920361

Model card

5b2866c

Dynamic model card from eval metrics

30f1872

CVE cross-references and dataset card

References#