Add reproducible evaluation and provenance research

This commit is contained in:
2026-09-17 10:40:29 -05:00
parent d9fc3ca18b
commit 68f02d035f
53 changed files with 1432 additions and 14 deletions
+11 -7
View File
@@ -8,22 +8,26 @@
- [x] JSON and SARIF reports
- [x] Unit tests and contributor guidance
- [x] Threat model and research brief
- [ ] Publish schema files for JSON and rule metadata
- [ ] Add deterministic corpus manifests
- [x] Publish schema files for scan reports, corpus manifests, and evaluation results
- [x] Add deterministic corpus manifests and content-derived corpus identity
- [x] Publish a versioned rule catalog with standards references
- [x] Add stable finding fingerprints across JSON and SARIF
## Evaluation harness
- [ ] Define positive, negative, and ambiguous fixture classes
- [ ] Add per-rule precision and recall measurement
- [x] Define initial positive and negative fixture classes
- [x] Add per-rule precision and recall measurement
- [ ] Add runtime and peak-memory benchmarks
- [ ] Add report reproducibility checks
- [x] Add report reproducibility checks
- [ ] Establish release-blocking redaction tests
## Artifact expansion
- [ ] SPDX and CycloneDX SBOM parsing
- [ ] Model card and dataset card schemas
- [ ] Agent tool-definition and permission analysis
- [x] Initial CycloneDX ML-BOM model-card and digest checks
- [x] Initial agent tool-definition and permission analysis
- [x] Initial SLSA provenance v1 binding checks
- [ ] Model card and dataset card validation profiles
- [ ] Infrastructure-as-code provenance checks
- [ ] Signed artifact and attestation verification