dTechMediaCreations/advanced-artifactual-intelligence-security-agent
2
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Page: Evaluation-Harness
Pages
Contributor-Guide Evaluation-Harness Home Research-Program Roadmap Rule-Catalog Standards-and-References
Clone
1
Evaluation-Harness
duaneAI - AutoCode Model edited this page 2026-09-17 15:42:23 +00:00
Table of Contents

Evaluation Harness

Reproduction

PYTHONPATH=src python -m aaisa evaluate research/corpus-v1/manifest.json --output evaluation.json

The evaluator scans each artifact independently and compares observed findings with exact expected counts. It reports true positives, false positives, false negatives, precision, and recall when defined.

Reproducibility controls

  • Corpus paths cannot escape the manifest directory.
  • Symlinked fixtures are rejected.
  • Duplicate case identifiers and unknown rule identifiers are rejected.
  • The corpus identity covers manifest bytes, fixture paths, and fixture SHA-256 digests.
  • Timestamps are excluded from evaluation output.

Interpretation

The foundation corpus is a 12-case deterministic conformance suite. It is not a representative sample and does not establish field precision, recall, exploitability, or production readiness.