Add reproducible evaluation and provenance research

This commit is contained in:
2026-09-17 10:40:29 -05:00
parent d9fc3ca18b
commit 68f02d035f
53 changed files with 1432 additions and 14 deletions
+29 -1
View File
@@ -4,7 +4,7 @@
AAISA prioritizes inspectability, deterministic evidence, safe handling of untrusted artifacts, and low barriers for research contributors. The initial implementation uses only the Python standard library and keeps discovery, normalization, policy evaluation, and reporting separate.
## Data flow
## Scan data flow
```text
approved path
@@ -31,6 +31,34 @@ A rule exposes an identifier, title, description, default severity, and an `eval
Future parser and analyzer plugins will require a versioned interface and explicit isolation policy before they are loaded dynamically.
## Evaluation data flow
```text
versioned manifest + immutable fixtures
|
v
path and schema validation
|
v
one isolated scan per test case
|
v
expected counts <-> observed counts
|
v
case verdicts + TP/FP/FN + precision/recall + corpus digest
```
Evaluation manifests cannot reference files outside their corpus directory. Every fixture contributes its relative path and SHA-256 digest to the reported corpus identity. Output is deterministic for the same tool version, manifest, and fixture bytes.
## Structured artifact rules
JSON analyzers parse content as data and never import or execute it. The foundation recognizes agent manifests, SLSA provenance statements, and CycloneDX ML-BOM documents through file classification plus document-level discriminators. A structured rule returns no result when the document is outside its supported format; malformed-format reporting will be developed as a separate policy so format errors are not confused with missing security evidence.
## Finding identity
Each finding receives an `aaisa/v1` SHA-256 fingerprint over the rule identifier, artifact path, location, and normalized message. JSON reports include it as `fingerprint`; SARIF reports include it under `partialFingerprints`. The fingerprint supports result correlation but is not a cryptographic signature or proof of report authenticity.
## Evidence model
Each artifact records a relative path, class, media type, byte size, and SHA-256 digest. Each finding records its rule, severity, message, artifact path, optional location, redacted evidence, remediation guidance, and tags. The report adds tool version, timestamps, root, summary counts, and nonfatal errors.