Audit Logging

Beta

Audit logging is in beta. The event schema, configuration and query filters may change in a future release.

OpenChoreo keeps an audit trail of who did what, to which resource, from where, and whether it was allowed. Every state-modifying operation on the OpenChoreo API and the Observer API produces a record, whether the call arrives over REST, from the Backstage portal or occ, or as an MCP tool call.

The trail is a separate signal from logs. Records go to their own index or stream with their own retention, so they outlive operational logs and cannot be flooded by application output.

Overview

A record answers:

Question	Fields
Who	actor.id, actor.type, actor.issuer, actor.session_id, actor.entitlements
Did what	action (e.g. create_project), operation_id (e.g. CreateProject), category
To which resource	resource.type, resource.namespace, resource.environment, resource.project, resource.component, resource.name
With what outcome	result: success, failure or denied
From where	source_ip, user_agent, surface (rest or mcp), producer (openchoreo-api or observer)
Correlation	event_id, event_time, request_id

Every record carries one of three categories. management covers create, update and delete on platform resources. authorization covers changes to roles and role bindings. access covers reads that are worth recording on their own, which today means reads of the audit trail itself.

Other reads are not audited, and neither are requests that never authenticate. Those belong to the access log.

Prerequisites

• An observability plane installed and registered as a ClusterObservabilityPlane or ObservabilityPlane. See Getting Started or Multi-Cluster Connectivity.
• A logs module that collects and serves audit logs. See Logs Modules.

Already enabled on quick installs

k3d-install.sh --with-observability, the quick-start installer's --with-observability, and the guided k3d setup all turn audit logging on. Use this page for other installations, or to tune one of those.

Enabling Audit Logging

Enable collection first, so the first records have somewhere to land, then emission on the observability plane and the control plane. The commands use the OpenSearch logs module in its default namespace. For another module, see Logs Modules.

caution

--reuse-values is safe here because the chart version does not change. When upgrading a chart across versions, pass your full values file with the audit settings added instead.

Step 1: Collect audit records in the logs module

helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.6.0 \
  --reuse-values \
  --set auditLogs.enabled=true

This assumes log collection is already on, as set up when the observability plane was installed.

The upgrade updates the collection agent's configuration, but Fluent Bit reads it only at startup and is not restarted by the upgrade. Restart it so it picks up the audit rules:

kubectl rollout restart daemonset/fluent-bit -n openchoreo-observability-plane

Step 2: Emit audit records from the Observer

helm upgrade openchoreo-observability-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.3.0-rc.2 \
  --namespace openchoreo-observability-plane \
  --reuse-values \
  --set observer.audit.enabled=true

Step 3: Emit audit records from the control plane

Point observabilityPlaneRef at the observability plane whose logs module collects the trail. For a ClusterObservabilityPlane named default:

helm upgrade openchoreo-control-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.3.0-rc.2 \
  --namespace openchoreo-control-plane \
  --reuse-values \
  --set openchoreoApi.config.audit.enabled=true \
  --set openchoreoApi.config.audit.observabilityPlaneRef.kind=ClusterObservabilityPlane \
  --set openchoreoApi.config.audit.observabilityPlaneRef.name=default

For a namespace-scoped ObservabilityPlane, set kind=ObservabilityPlane and add openchoreoApi.config.audit.observabilityPlaneRef.namespace.

warning

openchoreo-api refuses to start when audit.enabled is true and the reference has no kind or name.

The plane itself is resolved only when a client asks where the trail lives. Name one that does not exist and the API starts anyway, logs a warning, and clients cannot find the trail. Records are still written, so nothing is lost as long as collection is on.

Step 4: Verify

Create and delete a project, then check the trail end to end:

# The API server writes the record
kubectl logs -n openchoreo-control-plane deployment/openchoreo-api -c api-server | grep AUDIT-LOG

# The Observer serves it back
occ auditlogs --since 1h

In the console, Audit Logs in the sidebar shows the same records to anyone with auditlogs:view.

Multi-Cluster Setups

openchoreo-api writes records in the control plane cluster and observer writes them in the observability plane cluster, so collection has to run in both:

Cluster	What to configure
Control plane	openchoreoApi.config.audit.* as in Step 3. Install the logs module with only its collection agent enabled, pointed at the observability plane, with audit collection on
Observability plane	observer.audit.enabled=true as in Step 2, and audit collection on the logs module as in Step 1
Data plane	Nothing. No audit producer runs here
Workflow plane	Nothing

The control plane cluster's exact settings depend on the module and on how the observability plane exposes its ingestion endpoint, so follow the module's own multi-cluster instructions from Logs Modules. Each record carries the openchoreo_cluster_instance of the cluster it was collected in.

Trusted Producers

A logs module accepts audit records only from producers you trust. The OpenSearch and OpenObserve modules express that as a container allowlist:

auditLogs:
  producers:
    - producer: openchoreo-api
      namespace: openchoreo-control-plane
      container: api-server
    - producer: observer
      namespace: openchoreo-observability-plane
      container: observer

Entries match the container log file name written by the kubelet, not anything in the log line. A workload outside the list can print something shaped exactly like an audit record and it still lands in the ordinary container logs. Note the container is api-server, not openchoreo-api.

warning

Each entry grants a workload the right to write into the audit trail, so review additions as you would any other grant. Installing the control plane or observability plane into a non-default namespace means editing these entries, or collection stops silently: nothing errors, the trail just stays empty.

Configuring What Is Published

Every audited operation is published by default. Policies let you drop noise, such as a CI service account that updates workloads hundreds of times a day, or record only a chosen few.

The control plane (openchoreoApi.config.audit) and the Observer (observer.audit) are configured separately with the same syntax:

Key	Default	Description
defaults.publish	true	Whether an operation that matches no policy is published
policies	[]	Ordered list of {match, set} rules. The first rule whose match fits the operation applies

set takes one key, publish. Every match field is a list: a field matches when any of its values does, and a rule matches when all of its non-empty fields do.

match field	Values	Checked at startup
categories	management, authorization	Yes
resources	Resource types, e.g. project, component, workload, releasebinding, authzrolebinding	Yes
operations	Operation IDs, e.g. CreateProject, UpdateComponent	Yes
actions	Action names, e.g. create_project, delete_project	Yes
surfaces	rest, mcp	Yes
actor_types	user, anonymous, and each subject type configured for the service (e.g. service_account)	Yes
results	success, failure, denied	Yes
actors	Actor IDs, the claim set by actor.idClaim	No
entitlements	Entitlement values such as group names. Matched against the values of every entitlement claim, whatever the claim name	No

Resource types, operation IDs and action names differ between the two services. A value the service never produces is rejected at startup, and the error lists what it accepts.

Policy keys use snake_case

Helm passes policies through untouched, so write actor_types, not actorTypes. Lists cannot be set with --set; use a values file or --set-json.

Examples

Drop successful workload updates from service accounts, keeping their failures and denials:

openchoreoApi:
  config:
    audit:
      policies:
        - match:
            actor_types: [service_account]
            resources: [workload]
            results: [success]
          set:
            publish: false

Record only authorization changes and anything that was refused:

openchoreoApi:
  config:
    audit:
      defaults:
        publish: false
      policies:
        - match:
            categories: [authorization]
          set:
            publish: true
        - match:
            results: [denied]
          set:
            publish: true

Rules Enforced at Startup

Break one of these and the service refuses to start, naming the rule:

• A rule must set publish.
• A rule with an empty match cannot set publish: false, since it would silence everything.
• actors and entitlements can raise publishing to true but never suppress it. Hiding one person's actions is what an audit trail must not allow.
• set.category is rejected. The category comes from the operation.
• match.categories takes management and authorization only.

Allowlist mode fails closed

With defaults.publish: false, only what a publish: true rule matches is recorded. Selector values are validated, but a rule that is valid and narrower than you intended drops events silently. Check the trail after changing policies.

Actor Identity

actor.id holds one token claim, sub by default. Set actor.idClaim on both services to record something more readable, such as an email claim your identity provider issues:

# Control plane
openchoreoApi:
  config:
    audit:
      actor:
        idClaim: email

# Observability plane
observer:
  audit:
    actor:
      idClaim: email

The same value is matched by policies[].match.actors and by occ auditlogs --actor.

An ID is unique only within its issuer, so with more than one identity provider configured, filter on both (occ auditlogs --actor <id> --issuer <issuer>). Client credentials tokens carry no session, which leaves actor.session_id empty for service accounts.

Access to the Audit Trail

Reading the trail requires the auditlogs:view action at cluster scope. The portal, occ auditlogs and the query_audit_logs MCP tool all enforce it.

Among the default roles, only admin and platform-engineer have it. Namespace, project and component filters narrow results but grant nothing: managing a project does not confer the right to read its audit trail.

For a read-only auditor, bind a ClusterAuthzRole carrying the action:

apiVersion: openchoreo.dev/v1alpha1
kind: ClusterAuthzRole
metadata:
  name: auditor
spec:
  actions:
    - "auditlogs:view"

See Custom Roles and Bindings to bind it to a group, or Customizing Bootstrap Roles and Bindings to create it at install time.

Reads of the trail are themselves recorded, with category access and action read_audit_log.

Viewing the Audit Trail

Client	How
Backstage portal	Audit Logs in the sidebar
CLI	occ auditlogs. See the CLI reference
MCP	query_audit_logs on the Observability Plane MCP server. See MCP Servers
API	POST /api/v1alpha1/audit-logs/query on the Observer

No client needs to be told which Observer holds the trail. They ask the control plane, which resolves observabilityPlaneRef.

A query covers at most 366 days and returns newest first. Values within a filter are OR-ed, separate filters are AND-ed.

# Refused requests in the last 7 days
occ auditlogs --since 7d --result denied

# Everything one user changed in a namespace
occ auditlogs --actor alice@example.com --namespace acme-corp --category management

Logs Modules

Collecting, storing and serving the trail is the logs module's job, so the destination, the retention and the way a remote cluster ships records are all module-specific. These modules support audit logs:

Module	Audit destination	Retention setting	Default
observability-logs-opensearch	Daily audit-logs-YYYY-MM-DD indices	openSearchSetup.dataRetentionTime.auditLogs	365d
observability-logs-openobserve	audit_logs stream	openObserveSetup.auditLogsRetentionDays	365

Both take the auditLogs.* values used on this page, and both can ship records to a separate backend through auditLogs.output.*, which is how several observability planes can report into one audit store. Their own documentation covers the collection agent, multi-cluster ingestion and credentials.

A module without audit support answers audit queries with 501 Not Implemented. See Building a Module for what support involves.

Configuration Reference

Control Plane (openchoreoApi.config.audit)

Value	Default	Description
enabled	false	Emit audit records from openchoreo-api
observabilityPlaneRef.kind	""	ClusterObservabilityPlane or ObservabilityPlane. Required when enabled is true
observabilityPlaneRef.name	""	Name of the plane. Required when enabled is true
observabilityPlaneRef.namespace	""	Namespace of an ObservabilityPlane. Not used for ClusterObservabilityPlane
defaults.publish	true	Publish operations that match no policy
policies	[]	Ordered publishing rules. See Configuring What Is Published
actor.idClaim	sub	Token claim recorded as actor.id

Observability Plane (observer.audit)

Value	Default	Description
enabled	false	Emit audit records from observer
defaults.publish	true	Publish operations that match no policy
policies	[]	Ordered publishing rules
actor.idClaim	sub	Token claim recorded as actor.id

Logs Module

Value	Default	Description
auditLogs.enabled	false	Route audit records to their own index or stream
auditLogs.producers	API and Observer	Trusted-producer allowlist. See Trusted Producers
auditLogs.output.*	unset	Ship audit records to a separate backend

Troubleshooting

openchoreo-api fails to start after enabling audit

observability_plane_ref.kind or observability_plane_ref.name is required, so set openchoreoApi.config.audit.observabilityPlaneRef. An error under audit.policies[N] means a policy breaks a startup rule, or names a match value the service never produces. The message lists what it accepts.

AUDIT-LOG lines are written, but the trail is empty

• Collection is not enabled where the producer runs. In a multi-cluster setup it has to run in the control plane cluster too.
• Fluent Bit was not restarted after audit collection was enabled with helm upgrade, so it still runs without the audit rules. Restart it as in Step 1, in every cluster where the module was upgraded.
• The producer allowlist still names the default namespaces. See Trusted Producers.
• On OpenSearch, an audit-logs-* index created before the module applied its index template gets dynamic mappings and answers nothing. See the module's documentation from Logs Modules.
• With auditLogs.output.host set, the audit credentials Secret is missing. The collection agent logs is used but not set.

occ auditlogs cannot find the trail

audit logging is not enabled on this OpenChoreo installation means openchoreoApi.config.audit.enabled is false. audit logging is enabled, but the control plane advertises no observer serving it means observabilityPlaneRef names a plane that does not exist. Check kubectl get clusterobservabilityplane, and look for Audit observability plane not found in the openchoreo-api logs.

403 or 501 when querying

403 means the caller lacks auditlogs:view at cluster scope; a namespace or project binding is not enough. 501 means the logs module in that observability plane does not serve audit queries.

Related Documentation

• Observability & Alerting: the observability plane and its modules
• Multi-Cluster Connectivity: connecting planes across clusters
• Authorization: roles, bindings and actions
• Building a Module: implementing audit support in a logs module
• CLI Reference: auditlogs
• MCP Servers