Add reproducible evaluation and provenance research

This commit is contained in:
2026-09-17 10:40:29 -05:00
parent d9fc3ca18b
commit 68f02d035f
53 changed files with 1432 additions and 14 deletions
@@ -0,0 +1,17 @@
# ADR 0001 Deterministic Evaluation Before Probabilistic Analysis
## Status
Accepted for the 0.2.0 research baseline.
## Context
AAISA needs measurable rule quality before it introduces probabilistic or model-backed analyzers. Evaluations that depend on network services, mutable model versions, or hidden prompts would make early results difficult to reproduce and could expose sensitive artifacts.
## Decision
The first evaluation contract uses local synthetic fixtures, explicit expected rule counts, content-derived corpus identity, deterministic rule execution, and machine-readable TP, FP, and FN accounting. Probabilistic analyzers remain outside the default runtime until isolation, provenance, confidence, and reproducibility requirements are approved.
## Consequences
Contributors can reproduce baseline behavior without external services. Regression detection is straightforward and safe for public fixtures. The baseline cannot measure semantic attacks that require model behavior, and its metrics must not be generalized to production accuracy.