Notification Channels

Configure notification channels to receive alerts. Platform Engineers can configure notification channels per environment. The first notification channel created in an environment is marked as the default channel. The default channel is used by alert rules that don't specify a channel.

OpenChoreo currently supports the following notification channel types:

• Email: Sends alerts via SMTP email.
• Webhook: Sends alerts as HTTP POST requests to an external endpoint.

Providing Secret Values

SMTP credentials and secret-backed webhook headers come from SecretReferences. A channel's secretKeyRef names a SecretReference in the channel's namespace and one of its keys, and OpenChoreo syncs the values from the secret store of the environment's observability plane into the observability plane through an ExternalSecret. Values you update in the store reach the observer within the SecretReference's refresh interval, without changing the channel.

The observability plane needs a secretStoreRef pointing at its ClusterSecretStore. The k3d and quick-start installs set this up. Then create the secrets with Secret Management, targeting the observability plane:

occ secret create generic smtp-credentials \
  --namespace default \
  --target-plane ClusterObservabilityPlane/default \
  --from-literal=username=<smtp-username> \
  --from-literal=password=<smtp-password>

occ secret create generic webhook-token \
  --namespace default \
  --target-plane ClusterObservabilityPlane/default \
  --from-literal=token=<webhook-token>

A channel can only read the secret store key that the Secret Management API writes for the SecretReference, so a hand-written SecretReference must have the same name and type as the secret it points at. See Secret Values for the full rules.

Deprecated: Kubernetes Secrets

Channels that name a Kubernetes Secret instead of a SecretReference still work: the values are copied from the control plane to the observability plane. This is deprecated and will be removed in a future release. Create SecretReferences with the same names to move such channels over; the channel switches automatically.

Check that a channel is ready, and why not if it isn't:

kubectl get observabilityalertsnotificationchannels -n default
kubectl describe observabilityalertsnotificationchannel my-notification-channel -n default

The Ready and SecretsReady conditions report problems such as a missing SecretReference or key, an observability plane without a secret store, or an ExternalSecret that failed to sync.

Email Notification Channel Example

from must be a bare email address, because it is also the SMTP envelope sender. To show a display name such as Example Alerts <alerts@example.com> in recipients' inboxes, set the optional fromName field.

Email templates support CEL expressions for dynamic content. Available CEL variables for templates include: ${alertName}, ${alertSeverity}, ${alertDescription}, ${alertValue}, ${alertTimestamp}, ${alertThreshold}, ${alertType}, ${component}, ${project}, ${environment}, ${componentId}, ${projectId}, ${environmentId}, ${releaseLine}, ${releaseLineId}, and ${triggerAiRca}.

${releaseLine} is the release line the alert fired on. It is main for the main line of a component with release lines, and empty for a component without them. ${releaseLineId} is the line's UID, and empty for the main line.

apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityAlertsNotificationChannel
metadata:
  name: my-notification-channel
  namespace: default
spec:
  environment: development
  isEnvDefault: true
  type: email
  emailConfig:
    from: alerts@example.com
    fromName: Example Alerts # Optional sender display name
    to:
      - team@example.com
      - oncall@example.com
    smtp:
      host: smtp.example.com
      port: 587
      auth:
        username:
          secretKeyRef:
            name: smtp-credentials
            key: username
        password:
          secretKeyRef:
            name: smtp-credentials
            key: password
      tls:
        insecureSkipVerify: false
    template:
      subject: "[${alertSeverity}] ${alertName} Triggered"
      body: |
        Alert: ${alertName}
        Severity: ${alertSeverity}
        Time: ${alertTimestamp}
        Description: ${alertDescription}
        Component: ${component}
        Project: ${project}
        Environment: ${environment}

Email bodies are sent as text/plain by default. To send an HTML email, set template.contentType to text/html and use HTML markup, including inline CSS, in the body:

template:
  contentType: text/html
  subject: "[${alertSeverity}] ${alertName} Triggered"
  body: |
    <html>
      <body>
        <h2 style="color: #d9534f;">Alert: ${alertName}</h2>
        <p><strong>Severity:</strong> ${alertSeverity}</p>
        <p><strong>Description:</strong> ${alertDescription}</p>
      </body>
    </html>

Webhook Notification Channel Example

Use a webhook notification channel to deliver alerts to an HTTP endpoint (for example, a custom incident management system, notification system such as Slack, or a chatops bridge). The payloadTemplate can be templated using CEL expressions. If payloadTemplate is omitted, the full alert payload is sent as JSON.

Available CEL variables for templates include: ${alertName}, ${alertSeverity}, ${alertDescription}, ${alertValue}, ${alertTimestamp}, ${alertThreshold}, ${alertType}, ${component}, ${project}, ${environment}, ${componentId}, ${projectId}, ${environmentId}, ${releaseLine}, ${releaseLineId}, ${incidentEnabled}, and ${triggerAiRca}.

${releaseLine} and ${releaseLineId} name the release line the alert fired on, as for email templates.

apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityAlertsNotificationChannel
metadata:
  name: my-webhook-channel
  namespace: default
spec:
  environment: development
  isEnvDefault: false
  type: webhook
  webhookConfig:
    url: https://alerts.example.com/webhook
    headers:
      X-OpenChoreo-Source:
        value: observer
      Authorization:
        valueFrom:
          secretKeyRef:
            name: webhook-token
            key: token
    payloadTemplate: |
      {
        "alertName": "${alertName}",
        "alertSeverity": "${alertSeverity}",
        "alertTimestamp": "${alertTimestamp}",
        "alertDescription": "${alertDescription}"
      }