Logging and Audit Trails: Building for Compliance from Day One

Adding audit logging retroactively is one of the most painful engineering projects a team can take on. Every code path that touches sensitive data needs to be threaded with logging. Database schemas need new columns. Event pipelines need to be built. Old data is lost forever.

Building it from day one is barely additional work. Here is the model that scales from MVP to SOC 2 to SOC 2 + HIPAA + HITRUST without needing to be redone.

What audit logs are actually for

Three audiences. Each one wants different information from the same underlying data.

Compliance auditors. Who accessed what data, when, from where. Used during SOC 2, HIPAA, ISO 27001 audits to demonstrate access controls work as documented.

Security incident response. When something looks suspicious, the audit log is how you reconstruct what happened. Did the attacker access customer data? Which records? When?

Customer trust. Increasingly, enterprise customers want to export their own audit logs to their SIEM. Their security teams use this for their own monitoring and compliance.

The schema that works

Every audit log entry needs the same six fields:

• Actor. Who did the thing. User ID, service account ID, system process. Include the auth method (password, SSO, API key).
• Action. What they did. Use a controlled vocabulary, not free text. "user.login", "customer_record.read", "billing.update". Stable across versions.
• Resource. What they did it to. Type plus ID. "customer:cust_abc123".
• Outcome. Success or failure. If failure, the reason.
• Timestamp. ISO 8601 with timezone, microsecond precision.
• Context. IP, user agent, request ID, session ID. Whatever is useful for reconstructing the request.

Optional but valuable: the before/after state for mutations, the request ID linking the audit log to your application logs, a hash of the previous audit entry for tamper-evidence.

What to actually log

Not everything. Logging every read of every record drowns the signal in noise.

Always log:

• Authentication events (login, logout, MFA challenge, password change).
• Authorization changes (role granted, permission updated).
• Access to sensitive data (customer PII, financial records, PHI).
• Mutations to important state (customer record updates, billing changes, configuration changes).
• Administrative actions (impersonation, data export, account deletion).
• Failed authentication or authorization attempts.

Do not log every page view, every API call, every routine read. Those belong in application logs, not audit logs. The audit log is for the things that matter for compliance and security investigation.

Storage and retention

Audit logs go to a separate, append-only store. Not the same database as your application data. The two key properties are: hard to tamper with, easy to query for audit and incident purposes.

The pattern that works: write audit events to a queue (Kinesis, SQS, Kafka), have a consumer that writes them to an immutable store (S3 with object lock, dedicated audit log database with restricted write access). Index the events for query in something searchable (OpenSearch, ClickHouse, BigQuery).

Retention varies by framework:

• SOC 2: typically 1 year minimum, your auditor will define.
• HIPAA: 6 years for records related to PHI.
• PCI DSS: 1 year minimum, 3 months immediately accessible.
• GDPR: as long as needed for stated purpose, no fixed minimum.

The pragmatic answer for a multi-framework startup is 7 years to cover everything.

What to give customers

Enterprise customers will eventually ask for their audit logs. Plan for it. The minimum: an API or export that returns their audit events in a defined format (JSON or CSV), filterable by date range. The customer should not need to ask support to get their own audit log.

For larger enterprise customers, support real-time streaming to their SIEM. SCIM-style standards exist; pick one that fits your customer base.

Tamper evidence

For higher-trust environments, audit logs should be tamper-evident. The pragmatic approach: each entry includes a hash of the previous entry, forming a chain. Anyone with the chain can verify no entries were modified or deleted in the middle. This is cheap to implement and meaningful for security investigations.