Upgrading from v1.3 to v1.4

Follow the standard upgrade process: apply each plane's updated CRDs, then helm upgrade the control, data, workflow, and observability planes in order.

Images and charts are served from cr.openchoreo.dev

The v1.4 charts pull OpenChoreo images from cr.openchoreo.dev/openchoreo/<image> instead of ghcr.io/openchoreo/<image>, and the install and upgrade commands use oci://cr.openchoreo.dev/openchoreo/helm-charts. Image names and tags are unchanged.

Most installations need no action. Check the following before you upgrade:

• Admission policies and image allowlists. If a policy (for example Kyverno or OPA Gatekeeper) only admits images matching ghcr.io/openchoreo/*, add cr.openchoreo.dev/openchoreo/* to it. Otherwise the upgraded pods are rejected.
• Restricted egress. cr.openchoreo.dev redirects authentication to ghcr.io and layer downloads to pkg-containers.githubusercontent.com. If your nodes pull through an egress allowlist, allow all three hosts.
• Mirror registries. If you set global.imageRegistry to point at your own mirror, the chart replaces the registry host of every OpenChoreo image, so nothing changes for you.

To keep pulling OpenChoreo images from ghcr.io, set the registry override on each plane you upgrade:

helm upgrade openchoreo-control-plane oci://cr.openchoreo.dev/openchoreo/helm-charts/openchoreo-control-plane \
  --version <version> --namespace openchoreo-control-plane --reset-then-reuse-values \
  --set global.imageRegistry=ghcr.io

Release lines

v1.4 adds release lines, which run several versions of a component side by side. The control plane CRDs add the ReleaseLine kind and the releaseLine owner fields. The v1.4 controller manager doesn't start without the ReleaseLine CRD, so apply the CRDs before you helm upgrade the control plane. Existing components keep their names, labels, and release hashes, so helm upgrade creates no new ComponentReleases for them.

The steps in this section are needed only to use release lines.

Update the default component types and build workflows

The default component types and build workflows come from the getting-started samples, not from the Helm charts, so helm upgrade keeps their v1.3 versions. Their v1.4 versions support release lines:

• The service and web-application ClusterComponentTypes name each line's Service, HTTPRoutes, gateway paths, and hostnames from metadata.componentLineName. The worker and scheduled-task types are unchanged.
• The build ClusterWorkflows pass a run's release line to the build, and tag the line's images with it.
• The generate-workload ClusterWorkflowTemplate publishes a line's build to the line's Workload.

The v1.4 component types need the v1.4 control plane, so apply them after upgrading it. Apply the build workflows and the generate-workload template together. With only one of them updated, a build of a line publishes its image to the main line's Workload.

Applying them replaces any changes you made to them after installing. If you changed them, make the same changes to the v1.4 versions.

A changed component type changes the release hash of every component that uses it. So for each service and web-application component with autoDeploy, applying the types creates a new ComponentRelease and binds it to the first environment. The main line renders the same objects as before, so its workloads are not redeployed, and later environments keep their releases until you promote them.

On the control plane:

kubectl apply \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/component-types/service.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/component-types/webapp.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/ci-workflows/dockerfile-builder.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/ci-workflows/paketo-buildpacks-builder.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/ci-workflows/gcp-buildpacks-builder.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/ci-workflows/ballerina-buildpack-builder.yaml

On the workflow plane, apply the generate-workload template the way you installed it. For an installation from Run in Your Environment, with CP_BASE_DOMAIN set as during the installation:

curl -fsSL https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/workflow-templates/generate-workload.yaml \
| sed "s#https://host.k3d.internal:8080/oauth2/token#https://thunder.${CP_BASE_DOMAIN}/oauth2/token#g" \
| sed "s#http://host.k3d.internal:8080#https://api.${CP_BASE_DOMAIN}#g" \
| kubectl apply -f -

For an installation from Run Locally on K3d:

kubectl apply -f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/samples/getting-started/workflow-templates/generate-workload-k3d.yaml

Update your own component types, traits, and workflows

Component types, traits, and build workflows that you created need the same kind of changes before the components that use them can add release lines. Release Lines for Component Types describes the changes. Components without release lines don't need them.

If your components use component types or traits from the community modules, such as the micro-integrator and agent sandbox component types, the keda-scaling trait, or the API and AI gateway traits of the kgateway, Envoy Gateway, NGINX Gateway Fabric, WSO2 API Platform, and agentgateway modules, update those modules to their latest versions after upgrading OpenChoreo and before adding release lines. The latest versions need the v1.4 control plane, and with the earlier versions most lines stop with a ReleaseLineNameCollision error. The earlier API traits of the kgateway, Envoy Gateway, and NGINX Gateway Fabric modules don't stop a line. The line deploys, but its policies attach to the main line's HTTPRoute, so the line's API has no authentication or rate limiting.

Update custom authorization roles

The control plane upgrade adds the releaseline:view, releaseline:create, releaseline:update, and releaseline:delete actions to the default roles. Add them to the roles you created, and to roles you define in openchoreoApi.config.security.authorization.bootstrap.roles. Grant releaseline:view where a role views components, and all four where it manages them. See Authorization.

Update the community modules

Filtering logs, events, metrics, traces, and costs by release line needs module versions that support release lines. With module versions that don't support release lines, such as those from the OpenChoreo v1.3 documentation, the main line's data stays available, and a query of one line or of every line answers 501 Not Implemented. Once a component has lines, the main line's events also include the events of the Deployments and Services of its other lines. Upgrade the modules after upgrading OpenChoreo.

• Observability Logs OpenSearch module: v0.7.0 -> v0.8.0

helm upgrade --install observability-logs-opensearch \
  oci://cr.openchoreo.dev/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.8.0 \
  --reset-then-reuse-values

Then add the release line fields to the existing event and audit indices, so that records written to them from the day of the upgrade can be filtered by release line. If you set auditLogs.indexPrefix, run the script with AUDIT_INDEX_PREFIX=<your-prefix> bash instead.

curl -fsSL https://raw.githubusercontent.com/openchoreo/community-modules/observability-logs-opensearch-0.8.0/observability-logs-opensearch/scripts/upgrade-to-0-8.sh | bash

• Observability Tracing OpenSearch module: v0.8.0 -> v0.9.0

helm upgrade --install observability-tracing-opensearch \
  oci://cr.openchoreo.dev/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.9.0 \
  --reset-then-reuse-values

Then add the release line fields to the existing trace indices:

curl -fsSL https://raw.githubusercontent.com/openchoreo/community-modules/observability-tracing-opensearch-0.9.0/observability-tracing-opensearch/scripts/upgrade-to-0-9.sh | bash

• Observability Metrics Prometheus module: v0.8.0 -> v0.9.0

helm upgrade --install observability-metrics-prometheus \
  oci://cr.openchoreo.dev/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.9.0 \
  --reset-then-reuse-values

• FinOps OpenCost module: v0.2.0 -> v0.3.0

helm upgrade --install finops-opencost \
  oci://cr.openchoreo.dev/openchoreo/helm-charts/finops-opencost \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.3.0 \
  --reset-then-reuse-values

Plane namespaces are deleted with their namespace

The Secret API stores each secret on its target plane in a namespace named openchoreo-kv-<namespace>, and pushes the value to the plane's secret store. Workflows run in workflows-<namespace> on the workflow plane, which also holds the git secrets created through the git secrets API. In v1.3, deleting a namespace left these plane namespaces behind on every plane, together with their Kubernetes Secrets and the values in the secret store.

In v1.4, OpenChoreo labels each plane namespace with the namespace it belongs to, and deletes it when you delete that namespace. External Secrets Operator then removes the values from the secret store. This also changes the following:

• Deleting a namespace deletes its workflows-<namespace> namespace, including its workflow runs and any resources you created in it yourself, such as Secrets that your workflow templates read.
• Deleting a DataPlane, WorkflowPlane, or ObservabilityPlane, or one of the cluster-scoped kinds, waits until no secret targets the plane. The plane's Finalizing condition shows DeletionBlocked and how many secrets remain. Delete those secrets first.
• If a namespace has its own plane and that plane's cluster can't be reached while the namespace is deleted, the namespace stays Terminating until the plane can be reached again. The plane's Finalizing condition shows the reason. If the cluster is gone for good, remove the plane's cleanup finalizer. If the cluster comes back later, run the cleanup script below against it to remove the plane namespaces left there.
• Deleting a git secret also removes its value from the secret store.

Clean up plane namespaces from earlier versions

Plane namespaces created before v1.4 don't have the labels, so OpenChoreo doesn't delete them. Some may already be left over from namespaces you deleted. After you upgrade the control plane, run the cleanup script once. It needs the kubectl context of the control plane and of each data, workflow, and observability plane cluster. For every openchoreo-kv-<namespace> and workflows-<namespace> namespace on those clusters, the script:

• Deletes the plane namespace if <namespace> no longer exists on the control plane, which removes its secret values from the secret store.
• Adds the labels if <namespace> exists, so OpenChoreo deletes the plane namespace when you delete <namespace>.
• Turns on store cleanup for every PushSecret in the plane namespace that doesn't have it, such as a git secret created before v1.4. Without it, deleting the PushSecret leaves its value in the secret store.

Download the script:

curl -fsSLo cleanup-plane-namespaces.sh https://raw.githubusercontent.com/openchoreo/openchoreo/release-v1.4.0-m.2/install/upgrade/cleanup-plane-namespaces.sh

Pass the right contexts

--cp-context must point at the control plane that the plane clusters belong to, and each --plane-context at one of its plane clusters. The script treats every namespace it can't find through --cp-context as deleted. With a wrong --cp-context, --apply deletes the plane namespaces of namespaces that still exist, together with their secret values in the secret store.

To confirm the control plane context, list its OpenChoreo namespaces. They should be the namespaces you use:

kubectl --context <control-plane-context> get namespaces -l openchoreo.dev/control-plane=true

Run it without --apply first. It prints what it would do and changes nothing:

bash cleanup-plane-namespaces.sh \
  --cp-context <control-plane-context> \
  --plane-context <data-plane-context> \
  --plane-context <workflow-plane-context> \
  --plane-context <observability-plane-context>

For a single-cluster installation, pass the same context to both options:

bash cleanup-plane-namespaces.sh --cp-context <context> --plane-context <context>

Each line of the output names an action:

Action	Meaning
delete	<namespace> no longer exists, so the plane namespace is deleted.
label	<namespace> exists, so the plane namespace gets the labels.
keep	The plane namespace already has the labels.
skip	The plane namespace is already being deleted, or you excluded it with --skip.
enable store cleanup on PushSecret <name>	A PushSecret without store cleanup, such as a git secret created before v1.4, gets it before its plane namespace is handled.

Check the delete and label lines before you apply them. On a cluster shared with other software, a namespace that only follows the same naming pattern can show up too, and labeling it marks it for deletion with the OpenChoreo namespace of the same name. Leave such a namespace out with --skip <name>, repeated for each. --skip applies to every plane context in the run. If the same name is an OpenChoreo plane namespace on another cluster, run the script once per plane cluster and pass --skip only in the run for the cluster with the unrelated namespace.

Then run the same command with --apply, keeping any --skip options:

bash cleanup-plane-namespaces.sh \
  --cp-context <control-plane-context> \
  --plane-context <data-plane-context> \
  --plane-context <workflow-plane-context> \
  --plane-context <observability-plane-context> \
  --apply

Running the script again is safe. The plane namespaces it already handled show keep or skip.