# Research Note 001 Evaluation and Provenance Baseline

## Conclusion

AAISA now has a reproducible way to test deterministic artifact rules and three structured checks for high-value AI security evidence: agent tool authorization, SLSA provenance bindings, and CycloneDX machine-learning BOM completeness. The included 12-case corpus passes exactly, but it is a conformance baseline rather than evidence of field accuracy.

## Research basis

The design maps current primary guidance into testable artifact conditions:

- [NIST AI 600-1](https://doi.org/10.6028/NIST.AI.600-1) treats generative-AI risk management as lifecycle work spanning governance, mapping, measurement, and management. AAISA represents findings and evaluations as versioned evidence rather than a one-time verdict.
- [SLSA Provenance v1](https://slsa.dev/spec/v1.2/provenance) separates build definition from run details and binds subjects, builders, external parameters, and resolved dependencies. `AAISA-PROVENANCE-002` checks the minimum fields required for meaningful downstream comparison; it does not verify signatures or establish builder trust.
- [CycloneDX 1.7](https://cyclonedx.org/docs/1.7/json/) represents `machine-learning-model` components and supports model-card content. `AAISA-MLBOM-001` checks stable identity, version, SHA-256 evidence, and presence of a model card; it is not a complete CycloneDX schema validator.
- The [OWASP Agent Control Standard](https://genai.owasp.org/resource/agent-control-standard-acs/) calls for inspectable, traceable, instrumentable agents with enforceable runtime controls, while [OWASP Agentic AI Threats and Mitigations](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/) frames emerging risks around autonomy and tool use. `AAISA-AGENT-001` identifies three auditable authorization failures: wildcard permission, command execution without approval, and network access without a destination allowlist.
- [MITRE ATLAS](https://atlas.mitre.org/) provides a living knowledge base for attacks against AI-enabled systems. The existing prompt-injection rule retains its mapping to `AML.T0051`.
- [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/os/sarif-v2.1.0-os.html) remains the interchange target for static-analysis results. AAISA now adds a stable `aaisa/v1` partial fingerprint to support result correlation.

## Evaluation method

Each manifest case points to one artifact beneath the corpus directory and declares exact expected finding counts by rule. The evaluator rejects path escape, duplicate case identifiers, unknown rule identifiers, and nonpositive expected counts. Each artifact is scanned independently at information severity, preventing findings in one fixture from masking another.

For each rule and case:

- true positives are the minimum of expected and observed counts;
- false positives are observed counts above expectation;
- false negatives are expected counts not observed;
- precision is `TP / (TP + FP)` when defined;
- recall is `TP / (TP + FN)` when defined.

The corpus digest covers the manifest bytes, ordered fixture paths, and fixture SHA-256 digests. Timestamps are excluded so parsed output is deterministic.

## Initial result

AAISA 0.2.0 evaluates all 12 corpus cases as passing with no scan errors. Positive and negative cases are included for six rule families. Rules without a safe committed positive fixture do not receive a recall claim.

## Interpretation limits

The initial result demonstrates internal consistency only. The sample is too small, balanced by construction, and synthetic. It cannot estimate real-world prevalence, operational false-positive burden, adversarial evasion, multilingual behavior, or analyst value. The structured checks validate presence and shape, not signatures, semantic accuracy, or trust.

## Next experiments

1. Build a reviewed ambiguous-case corpus with quoted attacks, documentation, and test fixtures.
2. Add formal JSON-schema validation for CycloneDX and provenance envelopes.
3. Add signature and policy verification as separate opt-in analyzers.
4. Measure runtime and peak memory across corpus scales.
5. Blind-label a public artifact sample with at least two reviewers and report agreement.
6. Establish release-blocking redaction and deterministic-output gates.