dTechMediaCreations/advanced-artifactual-intelligence-security-agent
2
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Add reproducible evaluation and provenance research #8

Merged
SysAdmin merged 1 commits from research/evaluation-harness-v1 into main 2026-09-17 15:42:28 +00:00
Conversation 1 Commits 1 Files Changed 53
+1432 -14
duaneAI commented 2026-09-17 15:42:26 +00:00
Member
Copy Link
Copy Source

Closes #1.

Research contribution

  • deterministic 12-case corpus evaluator with content-derived identity and TP/FP/FN metrics
  • machine-readable schemas and versioned rule catalog
  • structured agent authorization, SLSA provenance, and CycloneDX ML-BOM rules
  • stable JSON and SARIF finding fingerprints
  • 25 tests, clean packaging verification, research note, ADR, contributor contract, and version-controlled wiki source

Verification

  • python scripts/verify.py passes
  • 25 unit tests pass
  • 12/12 corpus cases pass with zero mismatches
  • installed CLI reports eight rules and AAISA 0.2.0
  • repository scan reports only eight intentional corpus findings and zero errors

Research limitations

The included corpus is a deterministic conformance suite, not a statistically representative benchmark. Structured rules validate presence and shape, not signatures, semantic truth, builder trust, or exploitability.

Closes #1. ## Research contribution - deterministic 12-case corpus evaluator with content-derived identity and TP/FP/FN metrics - machine-readable schemas and versioned rule catalog - structured agent authorization, SLSA provenance, and CycloneDX ML-BOM rules - stable JSON and SARIF finding fingerprints - 25 tests, clean packaging verification, research note, ADR, contributor contract, and version-controlled wiki source ## Verification - `python scripts/verify.py` passes - 25 unit tests pass - 12/12 corpus cases pass with zero mismatches - installed CLI reports eight rules and AAISA 0.2.0 - repository scan reports only eight intentional corpus findings and zero errors ## Research limitations The included corpus is a deterministic conformance suite, not a statistically representative benchmark. Structured rules validate presence and shape, not signatures, semantic truth, builder trust, or exploitability.
duaneAI added this to the Research Milestone 2 - Evaluation and Provenance milestone 2026-09-17 15:42:26 +00:00
duaneAI added the researchevaluationprovenanceagent-securitydocumentation labels 2026-09-17 15:42:26 +00:00
duaneAI added 1 commit 2026-09-17 15:42:27 +00:00
Add reproducible evaluation and provenance research 68f02d035f
SysAdmin commented 2026-09-17 15:42:27 +00:00
Owner
Copy Link
Copy Source

Administrator verification completed: contributor identity and write permission confirmed; commit attribution is correct; 25 tests pass; 12/12 evaluation cases pass; packaging and installed CLI checks pass; public artifacts contain no live credentials. Approved for merge.

Administrator verification completed: contributor identity and write permission confirmed; commit attribution is correct; 25 tests pass; 12/12 evaluation cases pass; packaging and installed CLI checks pass; public artifacts contain no live credentials. Approved for merge.
SysAdmin merged commit ca5d947957 into main 2026-09-17 15:42:28 +00:00
SysAdmin deleted branch research/evaluation-harness-v1 2026-09-17 15:42:28 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent-security
Agent tools, authorization, and trust boundaries
 benchmark
Corpora, fixtures, and performance measurement
 documentation
Research and contributor documentation
 evaluation
Evaluation harnesses, metrics, and reproducibility
 provenance
Artifact identity, attestations, and supply-chain evidence
 research
Research design or experimental work
No labels agent-security documentation evaluation provenance research
Milestone
No items
No Milestone Research Milestone 2 - Evaluation and Provenance
Projects
Clear projects
No projects
Assignees
Clear assignees
SysAdmin (dTech Administrator) duaneAI (duaneAI - AutoCode Model)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dTechMediaCreations/advanced-artifactual-intelligence-security-agent#8