Add reproducible evaluation and provenance research
This commit is contained in:
@@ -0,0 +1,61 @@
|
||||
# Rule Authoring Contract
|
||||
|
||||
## Purpose
|
||||
|
||||
AAISA rules convert immutable artifact evidence into reviewable findings. This contract keeps community rules deterministic, safe, explainable, and measurable.
|
||||
|
||||
## Required metadata
|
||||
|
||||
Every built-in rule defines:
|
||||
|
||||
- a stable identifier in the form `AAISA-CATEGORY-NNN`;
|
||||
- a concise title and plain-language description;
|
||||
- a default severity with documented rationale;
|
||||
- supported artifact kinds;
|
||||
- relevant standards or frameworks;
|
||||
- primary-source reference URLs.
|
||||
|
||||
Run `aaisa explain RULE_ID` to inspect the published metadata.
|
||||
|
||||
## Evaluation behavior
|
||||
|
||||
A rule receives one immutable `Artifact` and returns zero or more immutable `Finding` objects. It must produce the same findings for the same artifact and rule version. Rules cannot depend on wall-clock time, process environment, network responses, random values, or mutable global state.
|
||||
|
||||
## Evidence requirements
|
||||
|
||||
A finding must identify the rule, severity, artifact, message, and remediation. Include a line and column when they can be derived reliably. Evidence should name the detected condition without reproducing credentials, private content, or unnecessary source text.
|
||||
|
||||
## Safety requirements
|
||||
|
||||
A rule must not:
|
||||
|
||||
- execute or import an artifact;
|
||||
- follow a link or open a path other than the supplied artifact;
|
||||
- make a network request;
|
||||
- change source content or configuration;
|
||||
- emit full credential values;
|
||||
- present absence of findings as proof of security;
|
||||
- use opaque scoring without exposing the underlying evidence.
|
||||
|
||||
## Test requirements
|
||||
|
||||
Each rule requires positive, negative, boundary, and redaction tests. Add at least one positive and one negative case to a versioned corpus when the artifact can be represented safely. Tests must cover malformed structures and nested data when the rule walks structured content.
|
||||
|
||||
## Severity guidance
|
||||
|
||||
- **Critical:** the artifact directly grants broad capability or exposes a credential with immediate compromise potential.
|
||||
- **High:** exploitation could cross a material trust boundary or break artifact integrity.
|
||||
- **Medium:** the condition weakens verification, provenance, or defense in depth.
|
||||
- **Low:** the condition is useful hardening evidence with limited direct impact.
|
||||
- **Info:** contextual evidence that requires no immediate remediation.
|
||||
|
||||
Severity describes the detected condition, not the overall system.
|
||||
|
||||
## Review checklist
|
||||
|
||||
1. Confirm rule scope and primary references.
|
||||
2. Inspect all output fields for sensitive content.
|
||||
3. Run the full unit suite.
|
||||
4. Run the versioned evaluation corpus twice and compare parsed results.
|
||||
5. Scan the repository itself and explain intentional fixture findings.
|
||||
6. Document false positives, false negatives, and unsupported formats.
|
||||
Reference in New Issue
Block a user