Add reproducible evaluation and provenance research

2026-09-17 10:40:29 -05:00
parent d9fc3ca18b
commit 68f02d035f
53 changed files with 1432 additions and 14 deletions
@@ -0,0 +1,31 @@
+# Deterministic Foundation Corpus v1
+
+## Purpose
+
+This corpus verifies that AAISA's deterministic foundation rules behave consistently on small, public, synthetic artifacts. It is a conformance and regression suite. It is not representative of production prevalence, adversary diversity, or naturally occurring false-positive rates.
+
+## Contents
+
+The 12 cases cover prompt injection, dependency pinning, container immutability, agent tool authorization, SLSA provenance, and CycloneDX ML-BOM evidence. Each supported condition has one positive and one negative fixture. Secret-detection fixtures remain in unit tests where values are assembled in memory so a credential-shaped string is not committed to the repository.
+
+## Reproduction
+
+```bash
+PYTHONPATH=src python -m aaisa evaluate research/corpus-v1/manifest.json --output evaluation.json
+```
+
+The result includes a digest derived from the manifest bytes, every fixture's relative path, and every fixture's SHA-256 digest. Repeated evaluation with the same checkout and tool version must produce the same parsed JSON object.
+
+## Ground-truth policy
+
+Expected findings are exact counts by rule identifier. An unlisted observed rule is a false positive for that case. A missing expected result is a false negative. Multiple findings from one rule are counted separately when they represent independent unsafe capabilities.
+
+## Limitations
+
+- Fixtures are intentionally small and syntactically simple.
+- The corpus does not measure runtime or peak memory.
+- It does not establish statistically meaningful precision or recall.
+- It does not include multilingual, obfuscated, adversarially encoded, or large-scale artifacts.
+- It does not validate exploitability or system compromise.
+
+These gaps are tracked as public research issues and roadmap work.