Independent defensive-security research
Can AI reliably distinguish a new security problem from a duplicate, recurrence, or already-remediated incident?
Defensive Drift is building a reproducible benchmark and evaluation pipeline for security-drift reconciliation across configuration evidence, historical incidents, scanner findings, logs, and remediation records.
Research integrity: no performance claims are published until they are measured against the frozen benchmark and linked to reproducible run artifacts.
Research question
Evidence-grounded defensive triage, not autonomous remediation.
Modern security evidence is fragmented across configuration snapshots, infrastructure-as-code, scanners, logs, deployment history, prior incidents, and operational notes. The project measures whether AI can reconcile those artifacts without hiding genuinely new or unresolved problems inside a false duplicate.
The benchmark requires each method to classify an observation as NEW, DUPLICATE, RECURRENCE, RELATED_BUT_DISTINCT, or INSUFFICIENT_EVIDENCE, while grounding its conclusion in the supplied evidence.
The historical drift corpus is intentionally heterogeneous. The current raw/confirmed-incidents/ stream contains curated ChatGPT drift records, while older and other-model drift evidence may exist elsewhere and may use different templates. Template conformity is not an evidence-validity requirement; normalization happens only in derived private research artifacts.
Frozen measurement contract
Success means learning where AI helps — and where it fails.
Duplicate precision & recall
Measures both unsafe over-merging and alert-fatigue-causing misses.
Novel-issue recall
Measures whether new or materially distinct conditions remain visible for review.
Dangerous false-duplicate rate
Primary security-safety metric for incorrect closure-like classifications.
Evidence grounding
Checks whether cited evidence actually exists and materially supports the conclusion.
Remediation accuracy
Tests whether models correctly distinguish detected, mitigated, remediated, recurred, unresolved, and unknown states.
Cost, latency & consistency
Compares practical operating cost, response time, and repeated-trial stability.
Accelerated 35-day pre-grant sprint
Five milestone gates. Dates are deadlines, not pacing targets.
Research design frozen
Taxonomy, ground-truth rules, metrics, sanitization boundaries, and experiment protocol were frozen before final benchmark/model evaluation.
Benchmark v0.1 frozen
Discover the heterogeneous cross-model evidence corpus, preserve source provenance, normalize only in the private workspace, and build a 100-case human-adjudicated v0.1 benchmark with a 150–200 expansion range when scientifically useful.
Conventional baselines complete
Run normalized lexical, TF-IDF/similarity, and embedding baselines on the frozen benchmark.
AI evaluation complete
Compare OpenAI and practical open-weight approaches, including repeated high-risk trials, cost, latency, and failure analysis.
Grant-ready evidence package
Publish methodology, preliminary results, reproducibility artifacts, failure catalog, research brief, and the final grant evidence matrix.
Current status
M2: cross-model benchmark construction underway.
M1 is complete. The active work is discovering and inventorying valid drift evidence across models, repositories, storage conventions, and historical template families; preserving provenance; and preparing derived private records for human adjudication. Measured benchmark and model results will appear here only after the corresponding methodology gates are complete.
Public benefit
Built to be inspected, reproduced, and challenged.
Defensive Drift is intended to release a safe public benchmark subset, schemas, evaluator code, methodology, measured results, representative failure cases, and reproducibility guidance. Private operational records are not automatically public research data; public artifacts must pass explicit sanitization and confidentiality review.