Add reproducible evaluation and provenance research

docs/ARCHITECTURE.md

@@ -4,7 +4,7 @@
 
 AAISA prioritizes inspectability, deterministic evidence, safe handling of untrusted artifacts, and low barriers for research contributors. The initial implementation uses only the Python standard library and keeps discovery, normalization, policy evaluation, and reporting separate.
 
-## Data flow
+## Scan data flow
 
 ```text
 approved path
@@ -31,6 +31,34 @@ A rule exposes an identifier, title, description, default severity, and an `eval
 
 Future parser and analyzer plugins will require a versioned interface and explicit isolation policy before they are loaded dynamically.
 
+## Evaluation data flow
+
+```text
+versioned manifest + immutable fixtures
+                 |
+                 v
+        path and schema validation
+                 |
+                 v
+     one isolated scan per test case
+                 |
+                 v
+ expected counts <-> observed counts
+                 |
+                 v
+ case verdicts + TP/FP/FN + precision/recall + corpus digest
+```
+
+Evaluation manifests cannot reference files outside their corpus directory. Every fixture contributes its relative path and SHA-256 digest to the reported corpus identity. Output is deterministic for the same tool version, manifest, and fixture bytes.
+
+## Structured artifact rules
+
+JSON analyzers parse content as data and never import or execute it. The foundation recognizes agent manifests, SLSA provenance statements, and CycloneDX ML-BOM documents through file classification plus document-level discriminators. A structured rule returns no result when the document is outside its supported format; malformed-format reporting will be developed as a separate policy so format errors are not confused with missing security evidence.
+
+## Finding identity
+
+Each finding receives an `aaisa/v1` SHA-256 fingerprint over the rule identifier, artifact path, location, and normalized message. JSON reports include it as `fingerprint`; SARIF reports include it under `partialFingerprints`. The fingerprint supports result correlation but is not a cryptographic signature or proof of report authenticity.
+
 ## Evidence model
 
 Each artifact records a relative path, class, media type, byte size, and SHA-256 digest. Each finding records its rule, severity, message, artifact path, optional location, redacted evidence, remediation guidance, and tags. The report adds tool version, timestamps, root, summary counts, and nonfatal errors.

docs/RESEARCH_NOTE_001_EVALUATION_AND_PROVENANCE.md

@@ -0,0 +1,47 @@
+# Research Note 001 Evaluation and Provenance Baseline
+
+## Conclusion
+
+AAISA now has a reproducible way to test deterministic artifact rules and three structured checks for high-value AI security evidence: agent tool authorization, SLSA provenance bindings, and CycloneDX machine-learning BOM completeness. The included 12-case corpus passes exactly, but it is a conformance baseline rather than evidence of field accuracy.
+
+## Research basis
+
+The design maps current primary guidance into testable artifact conditions:
+
+- [NIST AI 600-1](https://doi.org/10.6028/NIST.AI.600-1) treats generative-AI risk management as lifecycle work spanning governance, mapping, measurement, and management. AAISA represents findings and evaluations as versioned evidence rather than a one-time verdict.
+- [SLSA Provenance v1](https://slsa.dev/spec/v1.2/provenance) separates build definition from run details and binds subjects, builders, external parameters, and resolved dependencies. `AAISA-PROVENANCE-002` checks the minimum fields required for meaningful downstream comparison; it does not verify signatures or establish builder trust.
+- [CycloneDX 1.7](https://cyclonedx.org/docs/1.7/json/) represents `machine-learning-model` components and supports model-card content. `AAISA-MLBOM-001` checks stable identity, version, SHA-256 evidence, and presence of a model card; it is not a complete CycloneDX schema validator.
+- The [OWASP Agent Control Standard](https://genai.owasp.org/resource/agent-control-standard-acs/) calls for inspectable, traceable, instrumentable agents with enforceable runtime controls, while [OWASP Agentic AI Threats and Mitigations](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/) frames emerging risks around autonomy and tool use. `AAISA-AGENT-001` identifies three auditable authorization failures: wildcard permission, command execution without approval, and network access without a destination allowlist.
+- [MITRE ATLAS](https://atlas.mitre.org/) provides a living knowledge base for attacks against AI-enabled systems. The existing prompt-injection rule retains its mapping to `AML.T0051`.
+- [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html) remains the interchange target for static-analysis results. AAISA now adds a stable `aaisa/v1` partial fingerprint to support result correlation.
+
+## Evaluation method
+
+Each manifest case points to one artifact beneath the corpus directory and declares exact expected finding counts by rule. The evaluator rejects path escape, duplicate case identifiers, unknown rule identifiers, and nonpositive expected counts. Each artifact is scanned independently at information severity, preventing findings in one fixture from masking another.
+
+For each rule and case:
+
+- true positives are the minimum of expected and observed counts;
+- false positives are observed counts above expectation;
+- false negatives are expected counts not observed;
+- precision is `TP / (TP + FP)` when defined;
+- recall is `TP / (TP + FN)` when defined.
+
+The corpus digest covers the manifest bytes, ordered fixture paths, and fixture SHA-256 digests. Timestamps are excluded so parsed output is deterministic.
+
+## Initial result
+
+AAISA 0.2.0 evaluates all 12 corpus cases as passing with no scan errors. Positive and negative cases are included for six rule families. Rules without a safe committed positive fixture do not receive a recall claim.
+
+## Interpretation limits
+
+The initial result demonstrates internal consistency only. The sample is too small, balanced by construction, and synthetic. It cannot estimate real-world prevalence, operational false-positive burden, adversarial evasion, multilingual behavior, or analyst value. The structured checks validate presence and shape, not signatures, semantic accuracy, or trust.
+
+## Next experiments
+
+1. Build a reviewed ambiguous-case corpus with quoted attacks, documentation, and test fixtures.
+2. Add formal JSON-schema validation for CycloneDX and provenance envelopes.
+3. Add signature and policy verification as separate opt-in analyzers.
+4. Measure runtime and peak memory across corpus scales.
+5. Blind-label a public artifact sample with at least two reviewers and report agreement.
+6. Establish release-blocking redaction and deterministic-output gates.

docs/ROADMAP.md

@@ -8,22 +8,26 @@
 - [x] JSON and SARIF reports
 - [x] Unit tests and contributor guidance
 - [x] Threat model and research brief
-- [ ] Publish schema files for JSON and rule metadata
-- [ ] Add deterministic corpus manifests
+- [x] Publish schema files for scan reports, corpus manifests, and evaluation results
+- [x] Add deterministic corpus manifests and content-derived corpus identity
+- [x] Publish a versioned rule catalog with standards references
+- [x] Add stable finding fingerprints across JSON and SARIF
 
 ## Evaluation harness
 
-- [ ] Define positive, negative, and ambiguous fixture classes
-- [ ] Add per-rule precision and recall measurement
+- [x] Define initial positive and negative fixture classes
+- [x] Add per-rule precision and recall measurement
 - [ ] Add runtime and peak-memory benchmarks
-- [ ] Add report reproducibility checks
+- [x] Add report reproducibility checks
 - [ ] Establish release-blocking redaction tests
 
 ## Artifact expansion
 
 - [ ] SPDX and CycloneDX SBOM parsing
-- [ ] Model card and dataset card schemas
-- [ ] Agent tool-definition and permission analysis
+- [x] Initial CycloneDX ML-BOM model-card and digest checks
+- [x] Initial agent tool-definition and permission analysis
+- [x] Initial SLSA provenance v1 binding checks
+- [ ] Model card and dataset card validation profiles
 - [ ] Infrastructure-as-code provenance checks
 - [ ] Signed artifact and attestation verification
 

docs/RULE_AUTHORING.md

@@ -0,0 +1,61 @@
+# Rule Authoring Contract
+
+## Purpose
+
+AAISA rules convert immutable artifact evidence into reviewable findings. This contract keeps community rules deterministic, safe, explainable, and measurable.
+
+## Required metadata
+
+Every built-in rule defines:
+
+- a stable identifier in the form `AAISA-CATEGORY-NNN`;
+- a concise title and plain-language description;
+- a default severity with documented rationale;
+- supported artifact kinds;
+- relevant standards or frameworks;
+- primary-source reference URLs.
+
+Run `aaisa explain RULE_ID` to inspect the published metadata.
+
+## Evaluation behavior
+
+A rule receives one immutable `Artifact` and returns zero or more immutable `Finding` objects. It must produce the same findings for the same artifact and rule version. Rules cannot depend on wall-clock time, process environment, network responses, random values, or mutable global state.
+
+## Evidence requirements
+
+A finding must identify the rule, severity, artifact, message, and remediation. Include a line and column when they can be derived reliably. Evidence should name the detected condition without reproducing credentials, private content, or unnecessary source text.
+
+## Safety requirements
+
+A rule must not:
+
+- execute or import an artifact;
+- follow a link or open a path other than the supplied artifact;
+- make a network request;
+- change source content or configuration;
+- emit full credential values;
+- present absence of findings as proof of security;
+- use opaque scoring without exposing the underlying evidence.
+
+## Test requirements
+
+Each rule requires positive, negative, boundary, and redaction tests. Add at least one positive and one negative case to a versioned corpus when the artifact can be represented safely. Tests must cover malformed structures and nested data when the rule walks structured content.
+
+## Severity guidance
+
+- **Critical:** the artifact directly grants broad capability or exposes a credential with immediate compromise potential.
+- **High:** exploitation could cross a material trust boundary or break artifact integrity.
+- **Medium:** the condition weakens verification, provenance, or defense in depth.
+- **Low:** the condition is useful hardening evidence with limited direct impact.
+- **Info:** contextual evidence that requires no immediate remediation.
+
+Severity describes the detected condition, not the overall system.
+
+## Review checklist
+
+1. Confirm rule scope and primary references.
+2. Inspect all output fields for sensitive content.
+3. Run the full unit suite.
+4. Run the versioned evaluation corpus twice and compare parsed results.
+5. Scan the repository itself and explain intentional fixture findings.
+6. Document false positives, false negatives, and unsupported formats.

docs/decisions/ADR-0001-DETERMINISTIC-EVALUATION.md

@@ -0,0 +1,17 @@
+# ADR 0001 Deterministic Evaluation Before Probabilistic Analysis
+
+## Status
+
+Accepted for the 0.2.0 research baseline.
+
+## Context
+
+AAISA needs measurable rule quality before it introduces probabilistic or model-backed analyzers. Evaluations that depend on network services, mutable model versions, or hidden prompts would make early results difficult to reproduce and could expose sensitive artifacts.
+
+## Decision
+
+The first evaluation contract uses local synthetic fixtures, explicit expected rule counts, content-derived corpus identity, deterministic rule execution, and machine-readable TP, FP, and FN accounting. Probabilistic analyzers remain outside the default runtime until isolation, provenance, confidence, and reproducibility requirements are approved.
+
+## Consequences
+
+Contributors can reproduce baseline behavior without external services. Regression detection is straightforward and safe for public fixtures. The baseline cannot measure semantic attacks that require model behavior, and its metrics must not be generalized to production accuracy.