Justin Sleight / Insight

Checklist / Evidence pipelines

<- Back to index

Shipping Compliance Evidence Continuously

Getting audit artifacts as code with pipelines, policy, and reproducible builds.

Objective

Create a repeatable, verifiable, and continuously updated stream of compliance evidence — generated by systems, not spreadsheets — that supports ITGC, SOC 2, ISO 27001, HIPAA, or internal control frameworks without manual collection cycles.

The goal: turn compliance from a snapshot event into a continuous pipeline.

1. Foundational Principles

Principle Implementation Focus
Evidence as Code Express compliance requirements in version-controlled, machine-verifiable formats (YAML, Terraform, OPA/Rego).
Immutable Proofs Artifacts must be timestamped, signed, and traceable back to a specific build, deployment, or configuration.
Continuous Traceability Every control assertion links directly to telemetry, config state, or pipeline logs.
Audit Readiness by Design Evidence is generated, not gathered. Collection occurs automatically through CI/CD, policy engines, or build metadata.

2. CI/CD Integration Controls

Embed controls directly into build and deployment pipelines so evidence is generated alongside every release.

Pipeline Instrumentation

Embed compliance checkpoints in build and deploy pipelines. Example: Jenkins, GitHub Actions, or GitLab pipelines validate security scans, dependency checks, and code signing before promotion.

Evidence Generation Hooks

Store pipeline run logs, artifact hashes, and approval records in a secure evidence repository (e.g., S3 with object lock, GCS with versioning). Tag pipeline output with a release identifier and environment target (build_id, git_commit, deploy_env).

Controlled Promotion Flow

Require artifact promotion only from signed, validated pipeline runs. Enforce separation of duties between build, approval, and deployment steps to align with SoD controls.

Pipeline Policy Enforcement

Use policy-as-code (OPA, Conftest, or Sentinel) to enforce controls such as:

  • Mandatory code review before merge
  • Dependency license compliance
  • Required test coverage thresholds
  • Security scan pass/fail gates

3. Policy-as-Code and Control Mapping

Control Domain Example Tooling Evidence Produced
Identity & Access Terraform + OPA rules on IAM policies policy.rego, access diff reports, least-privilege assertions
Infrastructure Config Terraform plan + drift detection Config snapshot, statefile hash, drift logs
Change Management CI pipeline metadata + approval log git log + signed merge request data
Vulnerability Management SCA/SAST/DAST integration Scan reports with build ID + timestamp
Backup & Recovery Automated restore verification jobs Restore test logs, checksum comparison
Logging & Monitoring SIEM ingestion validation Log coverage metrics, alert test outcomes

4. Artifact Repository Requirements

Centralize evidence in an append-only repository with clear taxonomy and retention policy.

Structure:

pgsql: /evidence/
    /builds/<build_id>/
        pipeline_summary.json
        artifact_hash.txt
        dependency_report.csv
    /infrastructure/
        terraform_state_snapshot.json
        policy_violations.log
    /access/
        iam_diff_YYYYMMDD.json
        approval_audit.log

Security Controls

  • Object immutability (write-once, append-only).
  • Access via least privilege (read-only for auditors, write via automation only).
  • Retention aligned to control framework timelines (e.g., 12–18 months).
  • Automated checksum validation and periodic integrity attestations.

5. Reproducible Builds and Deployment Evidence

Source Integrity

Require all build inputs to be checksum-verified or signed, with SBOM source references documented. Maintain the SBOM (Software Bill of Materials) as a version-controlled artifact.

Build Determinism

Document the reproducibility process to ensure identical inputs yield identical outputs (hash match). Use containerized or isolated build environments to reduce environmental drift.

Artifact Signatures

Digitally sign each build and deployment package. Validate signatures before deployment and archive verification logs as compliance artifacts.

Deployment Audit Chain

Record the target environment, approver, deployment hash, and timestamp for every release. Enforce multi-person approval for production rollouts to align with COBIT SoD and ISO 27001 A.12.1.2.

6. Continuous Control Validation

Control Type Validation Mechanism Evidence
Access Review Automated comparison of IAM policy vs. HR roster CSV diff + approval log
Patch Compliance Config management or MDM telemetry JSON compliance report
Vulnerability Status Scanner output with build metadata HTML or JSON report
Change Tracking Git and CI metadata correlation Commit log, issue ID, deployment ID
Backup Validation Scheduled restore job logs Timestamped verification report

7. Auditor Interaction Model

When audit season arrives:

  • Provide auditors read-only access to the evidence repository or evidence catalog API.
  • Share the evidence metadata index (evidence_index.yaml) listing source, timestamp, and hash for each control domain.
  • Demonstrate chain of custody from source system to pipeline to artifact to approval.
  • Enable independent revalidation so auditors can run the same query or rebuild to reproduce outcomes.

This converts audit testing from a trust exercise into a verification exercise.

8. Key Metrics for Continuous Compliance Health

Metric Target Rationale
Evidence generation coverage >95% of controls automated Gaps signal manual risk
Evidence aging <30 days Ensures data freshness
Artifact integrity verification rate 100% Confirms tamper-proof evidence
Drift detection alerts resolved <48h median Demonstrates continuous posture management
CI/CD policy violation rate <2% Tracks pre-deploy control effectiveness

Summary Reference

Continuous compliance means:

  • Pipelines generate the proof.
  • Policy defines the rules.
  • Evidence lives as code.

If an auditor can rebuild your environment and independently confirm every control assertion without opening a ticket — you’ve achieved the right level of maturity.