Add reproducible evaluation and provenance research
This commit is contained in:
@@ -0,0 +1,17 @@
|
||||
# ADR 0001 Deterministic Evaluation Before Probabilistic Analysis
|
||||
|
||||
## Status
|
||||
|
||||
Accepted for the 0.2.0 research baseline.
|
||||
|
||||
## Context
|
||||
|
||||
AAISA needs measurable rule quality before it introduces probabilistic or model-backed analyzers. Evaluations that depend on network services, mutable model versions, or hidden prompts would make early results difficult to reproduce and could expose sensitive artifacts.
|
||||
|
||||
## Decision
|
||||
|
||||
The first evaluation contract uses local synthetic fixtures, explicit expected rule counts, content-derived corpus identity, deterministic rule execution, and machine-readable TP, FP, and FN accounting. Probabilistic analyzers remain outside the default runtime until isolation, provenance, confidence, and reproducibility requirements are approved.
|
||||
|
||||
## Consequences
|
||||
|
||||
Contributors can reproduce baseline behavior without external services. Regression detection is straightforward and safe for public fixtures. The baseline cannot measure semantic attacks that require model behavior, and its metrics must not be generalized to production accuracy.
|
||||
Reference in New Issue
Block a user