32 lines
1.8 KiB
Markdown
32 lines
1.8 KiB
Markdown
# Deterministic Foundation Corpus v1
|
|
|
|
## Purpose
|
|
|
|
This corpus verifies that AAISA's deterministic foundation rules behave consistently on small, public, synthetic artifacts. It is a conformance and regression suite. It is not representative of production prevalence, adversary diversity, or naturally occurring false-positive rates.
|
|
|
|
## Contents
|
|
|
|
The 12 cases cover prompt injection, dependency pinning, container immutability, agent tool authorization, SLSA provenance, and CycloneDX ML-BOM evidence. Each supported condition has one positive and one negative fixture. Secret-detection fixtures remain in unit tests where values are assembled in memory so a credential-shaped string is not committed to the repository.
|
|
|
|
## Reproduction
|
|
|
|
```bash
|
|
PYTHONPATH=src python -m aaisa evaluate research/corpus-v1/manifest.json --output evaluation.json
|
|
```
|
|
|
|
The result includes a digest derived from the manifest bytes, every fixture's relative path, and every fixture's SHA-256 digest. Repeated evaluation with the same checkout and tool version must produce the same parsed JSON object.
|
|
|
|
## Ground-truth policy
|
|
|
|
Expected findings are exact counts by rule identifier. An unlisted observed rule is a false positive for that case. A missing expected result is a false negative. Multiple findings from one rule are counted separately when they represent independent unsafe capabilities.
|
|
|
|
## Limitations
|
|
|
|
- Fixtures are intentionally small and syntactically simple.
|
|
- The corpus does not measure runtime or peak memory.
|
|
- It does not establish statistically meaningful precision or recall.
|
|
- It does not include multilingual, obfuscated, adversarially encoded, or large-scale artifacts.
|
|
- It does not validate exploitability or system compromise.
|
|
|
|
These gaps are tracked as public research issues and roadmap work.
|