Upgrading OpenChoreo

This guide covers how to upgrade OpenChoreo components to newer versions.

Before You Upgrade

1. Review release notes for the target version
2. Backup critical data (see below)
3. Test in non-production environment first
4. Plan for downtime if required by the release notes

Backup Recommendations

Before upgrading, consider backing up:

• OpenChoreo CRDs and resources: Organizations, Projects, Components, etc.
• OpenSearch data (if using Observability Plane)
• Container registry data (if using Workflow Plane)

# Export OpenChoreo resources
kubectl get projects,components,dataplanes,workflowplanes -A -o yaml > openchoreo-backup.yaml

Upgrading CRDs

Helm does not upgrade CRDs

OpenChoreo ships its CustomResourceDefinitions in the crds/ directory of each Helm chart. By Helm's design, CRDs in this directory are installed on helm install but are never modified by helm upgrade. If a release introduces new fields, new CRDs, or removes existing ones, you must apply CRD changes manually before running helm upgrade.

Apply Updated CRDs

Only needed when CRDs have changed

This step is only required if the target release introduces CRD changes (new CRDs, schema/field changes, or removed CRDs) compared to the version you currently have installed. Check the release notes first — if no CRDs changed between your current version and the target version, skip this section and go straight to Upgrade Process.

For every plane you are about to upgrade, pull the target chart version and apply its CRDs with server-side apply:

# Example for the Control Plane — repeat for each plane being upgraded
helm pull oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 1.2.0-m.1 \
--untar --untardir ./charts

kubectl apply --server-side --force-conflicts \
-f ./charts/openchoreo-control-plane/crds/

Notes:

• Use kubectl apply --server-side. CRD manifests routinely exceed the 256 KB annotation limit of client-side apply and will fail with metadata.annotations: Too long.
• --force-conflicts is required because the original CRDs were installed by Helm and are owned by the helm field manager. The flag transfers ownership of the changed fields to kubectl.
• Repeat the helm pull + kubectl apply step for openchoreo-data-plane, openchoreo-workflow-plane, and openchoreo-observability-plane if you have them installed. Each chart ships only the CRDs it owns.
• On multi-cluster deployments, target each cluster with kubectl --context $CP_CONTEXT apply ... so CRDs land on the same cluster as the chart that uses them.

Removed CRDs

kubectl apply will not delete CRDs that were removed in a new release. Check the release notes for removed CRDs and delete them explicitly only after confirming no custom resources of that kind remain:

kubectl get <crd-name> -A
kubectl delete crd <crd-name>

Deleting a CRD also deletes every custom resource of that type, cluster-wide.

Upgrade Process

Single Cluster Deployment

Upgrade each plane in order. Make sure you have applied the updated CRDs for that plane (see Upgrading CRDs) before running helm upgrade.

Why --reset-then-reuse-values?

The commands below use --reset-then-reuse-values rather than --reuse-values. With plain --reuse-values, Helm reuses all values from the last release including the old chart's defaults which means new default values introduced in the target chart are never picked up. This often surfaces as nil-pointer errors during templating. --reset-then-reuse-values resets values to the new chart's defaults first, then layers your previously-set user values on top, so new defaults are honored while your overrides are preserved.

1. Control Plane

helm upgrade openchoreo-control-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 1.2.0-m.1 \
--namespace openchoreo-control-plane \
--reset-then-reuse-values

Wait for pods to be ready:

kubectl rollout status deployment -n openchoreo-control-plane

2. Data Plane

helm upgrade openchoreo-data-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
--version 1.2.0-m.1 \
--namespace openchoreo-data-plane \
--reset-then-reuse-values

3. Workflow Plane (if installed)

helm upgrade openchoreo-workflow-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
--version 1.2.0-m.1 \
--namespace openchoreo-workflow-plane \
--reset-then-reuse-values

4. Observability Plane (if installed)

Observer deployment strategy change may cause Helm upgrade failure

Observer deployment rolling strategy is conditionally being set based on the alertStoreBackend since v1.0.0. If you are using the sqlite as the alert store backend (which is the default), execute the following patch command before upgrading the observability plane to avoid Helm upgrade failures:

kubectl patch deployment observer -n openchoreo-observability-plane \
    -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'

helm upgrade openchoreo-observability-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
--version 1.2.0-m.1 \
--namespace openchoreo-observability-plane \
--reset-then-reuse-values

If you are using the default logs, metrics, and traces modules with the versions specified in OpenChoreo v1.0.0 documentation, you may also need to upgrade them to the versions compatible with OpenChoreo 1.2.0-m.1 as well.

• Observability Logs OpenSearch module: v0.3.11 -> v0.4.1

helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --reset-then-reuse-values \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set adapter.openSearchSecretName="opensearch-admin-credentials"

• Observability Traces OpenSearch module: v0.3.11 -> v0.4.1

helm upgrade --install observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
 --reset-then-reuse-values \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials"

Reindex old tracing data

Observability Traces OpenSearch module v0.4.0 uses a new index template that indexes resource.openchoreo.dev/namespace and uses it for queries. Since the old template did not index this field, traces query endpoints may return empty results after the upgrade. To fix this, reindex your old tracing data to the new index template:

curl -fsSL https://raw.githubusercontent.com/openchoreo/community-modules/refs/heads/main/observability-tracing-opensearch/scripts/upgrade-to-0-4-1.sh | bash

• Observability Metrics Prometheus module: v0.2.5 -> v0.6.1

helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.6.1 \
  --reset-then-reuse-values \
  --set adapter.image.tag=""

Multi Cluster Deployment

Follow the same order, but use --kube-context for each cluster:

helm upgrade openchoreo-control-plane ... --kube-context $CP_CONTEXT
helm upgrade openchoreo-data-plane ... --kube-context $DP_CONTEXT
helm upgrade openchoreo-workflow-plane ... --kube-context $WP_CONTEXT
helm upgrade openchoreo-observability-plane ... --kube-context $OP_CONTEXT

Data Plane upgrade from v1.0.0: stale kube-prometheus-stack values

The v1.0.0 multi-cluster Data Plane values file included kube-prometheus-stack configuration. This subchart was removed in later versions and metrics collection is now handled by the standalone community module.

When upgrading from v1.0.0, the stored Helm values still contain kube-prometheus-stack configuration. --reset-then-reuse-values will reapply these against the new chart schema, causing:

UPGRADE FAILED: values don't meet the specifications of the schema(s):
- at '': additional properties 'kube-prometheus-stack' not allowed

Check whether your stored values contain the stale key:

helm get values openchoreo-data-plane -n openchoreo-data-plane --kube-context $DP_CONTEXT

If the output includes kube-prometheus-stack, use --reset-values with your multi-cluster values file instead of --reset-then-reuse-values for the Data Plane upgrade:

helm upgrade openchoreo-data-plane ... --kube-context $DP_CONTEXT \
  --reset-values \
  --values <path-to-your-multi-cluster-dp-values-file>

--reset-values discards all previously stored values (including the stale kube-prometheus-stack key) and uses the new chart's defaults. The --values flag reapplies your multi-cluster overrides (e.g. clusterAgent.serverUrl, gateway.tls.enabled: false) which are required for the Data Plane to function correctly. Single-cluster deployments are not affected.

Verifying the Upgrade

# Check helm releases
helm list -A

# Check pod versions
kubectl get pods -n openchoreo-control-plane -o jsonpath='{.items[*].spec.containers[*].image}'

# Check CRD versions
kubectl get crds | grep openchoreo

Rollback

If issues occur, rollback to the previous release:

# List release history
helm history openchoreo-control-plane -n openchoreo-control-plane

# Rollback to previous revision
helm rollback openchoreo-control-plane -n openchoreo-control-plane

CRDs are not rolled back automatically

helm rollback does not revert CRD changes — Helm treats CRDs the same on rollback as it does on upgrade. If the new release introduced breaking CRD schema changes, re-apply the previous release's CRDs manually before rolling back the chart:

helm pull oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version <previous-version> --untar --untardir ./charts
kubectl apply --server-side --force-conflicts \
  -f ./charts/openchoreo-control-plane/crds/

Note that CRDs newly added in the failed release will remain in the cluster after rollback. They are inert if no controller reconciles them, but you can delete them explicitly once no custom resources of that kind exist.

Version Compatibility

OpenChoreo Version	Kubernetes Version	Helm Version
0.7.x	1.32+	3.12+
0.6.x	1.31+	3.12+
0.5.x	1.30+	3.12+

Getting Help

If you encounter issues during upgrade:

1. Check the FAQ
2. Review pod logs for errors
3. Open an issue on GitHub