Version: v0.5.x

Multi Cluster Setup

This guide walks you through step-by-step instructions for deploying OpenChoreo across multiple k3d clusters. This deploys a Control Plane, a Data Plane, and optional Build and Observability Planes in separate clusters for better isolation and to mimic production architecture.

Prerequisites

Docker – Just have it installed on your machine, and you're good to go.
- We recommend using Docker Engine version 26.0+.
- Allocate at least 8 GB RAM and 4 CPU cores to Docker (or the VM running Docker).
- Important: Ensure your Docker VM has sufficient inotify limits. Set fs.inotify.max_user_watches=524288 and fs.inotify.max_user_instances=512 to prevent k3d cluster creation from hanging.
k3d v5.8+ installed
kubectl v1.32+ installed
Helm v3.12+ installed (helm v4 is not supported yet)

Verify Prerequisites

Before proceeding, verify that all tools are installed and meet the minimum version requirements:

# Check Docker (should be v20.10+)
docker --version

# Check k3d (should be v5.8+)
k3d --version

# Check kubectl (should be v1.32+)
kubectl version --client

# Check Helm (should be v3.12+)
helm version --short

Make sure Docker is running:

docker info

note

If you're using Colima, set the K3D_FIX_DNS=0 environment variable when creating clusters to avoid DNS issues. See k3d-io/k3d#1449 for more details.

Quick Setup

This multi-cluster setup deploys OpenChoreo components across separate clusters for better isolation and scalability:

Control Plane Cluster: Hosts the OpenChoreo API server and controllers
Data Plane Cluster: Hosts application workloads and runtime components
Build Plane Cluster (Optional): Hosts CI/CD capabilities using Argo Workflows
Observability Plane Cluster (Optional): Hosts monitoring and logging infrastructure

1. Setup the Control Plane

Create the Control Plane Cluster

Create a dedicated k3d cluster for the control plane components:

curl -s https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/config-cp.yaml | k3d cluster create --config=-

This will:

Create a cluster named "openchoreo-cp"
Set up control plane with k3d
Configure port mappings: localhost:8080 (HTTP), localhost:8443 (HTTPS)
Expose Kubernetes API on port 6550
Set kubectl context to "k3d-openchoreo-cp"

Install OpenChoreo Control Plane

Install the OpenChoreo control plane using Helm. This will create the openchoreo-control-plane namespace automatically:

helm install openchoreo-control-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 0.5.0 \
--kube-context k3d-openchoreo-cp \
--namespace openchoreo-control-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/values-cp.yaml

Wait for the installation to complete and verify all pods are running:

kubectl get pods -n openchoreo-control-plane --context k3d-openchoreo-cp

You should see pods for:

controller-manager (Running)
cert-manager-* (3 pods, all Running)

2. Setup the Data Plane

Create the Data Plane Cluster

Create a dedicated k3d cluster for the data plane components:

curl -s https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/config-dp.yaml | k3d cluster create --config=-

This will:

Create a cluster named "openchoreo-dp"
Set up data plane with k3d
Configure port mappings: localhost:9080 (HTTP), localhost:9443 (HTTPS) - for deployed workloads
Expose Kubernetes API on port 6551
Set kubectl context to "k3d-openchoreo-dp"

Install OpenChoreo Data Plane

Install the OpenChoreo data plane using Helm. This will create the openchoreo-data-plane namespace automatically:

helm install openchoreo-data-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
--version 0.5.0 \
--kube-context k3d-openchoreo-dp \
--namespace openchoreo-data-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/values-dp.yaml

Wait for dataplane components to be ready:

kubectl get pods -n openchoreo-data-plane --context=k3d-openchoreo-dp

You should see pods for:

envoy-gateway-* (Running)
external-secrets-* (3 pods, all Running)
fluent-bit-* (Running on each node)
gateway-external-* (Running)

Configure DataPlane

curl -s https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/add-data-plane.sh | bash -s -- \
--control-plane-context k3d-openchoreo-cp \
--target-context k3d-openchoreo-dp \
--server https://host.k3d.internal:6551

This script creates a DataPlane resource in the default namespace of the control plane cluster.

Verify the DataPlane was created:

kubectl get dataplane -n default --context k3d-openchoreo-cp

3. Setup the Build Plane (Optional)

The Build Plane is required if you plan to use OpenChoreo's internal CI capabilities. If you're only deploying pre-built container images, you can skip this step.

Create the Build Plane Cluster

Create a dedicated k3d cluster for the build plane components:

curl -s https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/config-bp.yaml | k3d cluster create --config=-

This will:

Create a cluster named "openchoreo-bp"
Set up build plane with k3d
Configure port mappings: localhost:10081 (Argo Workflows UI), localhost:10082 (Container Registry)
Expose Kubernetes API on port 6552
Set kubectl context to "k3d-openchoreo-bp"

Install OpenChoreo Build Plane

Install the OpenChoreo build plane using Helm for CI/CD capabilities. This will create the openchoreo-build-plane namespace automatically:

helm install openchoreo-build-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-build-plane \
--version 0.5.0 \
--kube-context k3d-openchoreo-bp \
--namespace openchoreo-build-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/values-bp.yaml

Wait for the build plane components to be ready:

kubectl get pods -n openchoreo-build-plane --context k3d-openchoreo-bp

You should see pods for:

argo-server-* (Running)
argo-workflow-controller-* (Running)
registry-* (Running)

Configure BuildPlane

curl -s https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/add-build-plane.sh | bash -s -- \
--control-plane-context k3d-openchoreo-cp \
--target-context k3d-openchoreo-bp \
--server https://host.k3d.internal:6552

This script creates a BuildPlane resource in the default namespace of the control plane cluster.

Verify that the BuildPlane was created:

kubectl get buildplane -n default --context k3d-openchoreo-cp

4. Setup the Observability Plane (Optional)

Install the OpenChoreo observability plane for monitoring and logging capabilities across all clusters.

Create the Observability Plane Cluster

Create a dedicated k3d cluster for the observability plane components:

curl -s https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/config-op.yaml | k3d cluster create --config=-

This will:

Create a cluster named "openchoreo-op"
Set up observability plane with k3d
Configure port mappings: localhost:11081 (OpenSearch Dashboard), localhost:11082 (OpenSearch API)
Expose Kubernetes API on port 6553
Set kubectl context to "k3d-openchoreo-op"

Install OpenChoreo Observability Plane

Install the OpenChoreo observability plane using Helm. This will create the openchoreo-observability-plane namespace automatically:

helm install openchoreo-observability-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
--version 0.5.0 \
--kube-context k3d-openchoreo-op \
--namespace openchoreo-observability-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.5/install/k3d/multi-cluster/values-op.yaml

Wait for the observability plane components to be ready:

kubectl get pods -n openchoreo-observability-plane --context k3d-openchoreo-op

You should see pods for:

observer-* (Running) - Log processing service
opensearch-master-0 (Running) - Log storage backend
opensearch-dashboards-* (Running) - Visualization dashboard
opensearch-cluster-setup-* (Completed) - One-time setup job

note

The OpenSearch dashboard pod may take several minutes to start.

Verify that all pods are ready:

kubectl wait --for=condition=Ready pod --all -n openchoreo-observability-plane --timeout=600s --context k3d-openchoreo-op

Configure Cross-Cluster Observability

Configure the build plane and data plane to send logs to the observability plane. The host and port should be accessible from the data/build plane clusters:

# Configure Build Plane FluentBit
helm upgrade build-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-build-plane \
--version 0.5.0 \
--namespace openchoreo-build-plane \
--set fluentBit.config.opensearch.host="openchoreo-op-control-plane" \
--set fluentBit.config.opensearch.port=30920 \
--kube-context kind-openchoreo-bp \
--set fluentBit.enabled=true \
--set global.defaultResources.registry.local.pushEndpoint="openchoreo-dp-control-plane:30003" \
--set global.defaultResources.registry.local.pullEndpoint="localhost:30003"

# Configure Data Plane FluentBit
helm upgrade data-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
--version 0.5.0 \
--namespace openchoreo-data-plane \
--set fluentBit.config.opensearch.host="openchoreo-op-control-plane" \
--set fluentBit.config.opensearch.port=30920 \
--set cert-manager.enabled=false \
--set cert-manager.crds.enabled=false \
--kube-context kind-openchoreo-dp

Important Security Note: The observability plane collects data from outside clusters without encryption in this setup. For production environments, we recommend implementing proper TLS encryption and network security measures.

After updating the FluentBit configuration, restart the FluentBit pods to apply the new settings:

# Restart FluentBit pods in Build Plane
kubectl rollout restart daemonset/fluent-bit -n openchoreo-build-plane --context kind-openchoreo-bp

# Restart FluentBit pods in Data Plane
kubectl rollout restart daemonset/fluent-bit -n openchoreo-data-plane --context kind-openchoreo-dp

# Verify FluentBit pods are running
kubectl get pods -n openchoreo-build-plane --context kind-openchoreo-bp | grep fluent
kubectl get pods -n openchoreo-data-plane --context kind-openchoreo-dp | grep fluent

Verify FluentBit is sending logs to OpenSearch:

# Check if kubernetes indices are being created
kubectl exec -n openchoreo-observability-plane opensearch-master-0 --context k3d-openchoreo-op -- curl -s "http://localhost:9200/_cat/indices?v" | grep kubernetes

# Check recent log count
kubectl exec -n openchoreo-observability-plane opensearch-master-0 --context k3d-openchoreo-op -- curl -s "http://localhost:9200/kubernetes-*/_count" | jq '.count'

If the indices exist and the count is greater than 0, FluentBit is successfully collecting and storing logs.

Configure Observer Integration

Configure the DataPlane and BuildPlane to use the observer service. For multi-cluster setup, we need to expose the observer service via NodePort for cross-cluster communication.

First, expose the observer service with a NodePort:

# Patch the observer service to use NodePort
kubectl patch svc observer -n openchoreo-observability-plane --type='json' \
  -p='[{"op": "replace", "path": "/spec/type", "value": "NodePort"}, {"op": "add", "path": "/spec/ports/0/nodePort", "value": 30880}]' \
  --context kind-openchoreo-op

Then configure the DataPlane and BuildPlane to use the observer service via NodePort:

# Configure DataPlane to use observer service via NodePort
kubectl patch dataplane default -n default --type merge \
  -p '{"spec":{"observer":{"url":"http://openchoreo-op-control-plane:30880","authentication":{"basicAuth":{"username":"dummy","password":"dummy"}}}}}' \
  --context kind-openchoreo-cp

# Configure BuildPlane to use observer service via NodePort
kubectl patch buildplane default -n default --type merge \
  -p '{"spec":{"observer":{"url":"http://openchoreo-op-control-plane:30880","authentication":{"basicAuth":{"username":"dummy","password":"dummy"}}}}}' \
  --context kind-openchoreo-cp

This configuration enables:

Application logs to appear in Backstage portal
Enhanced logging and monitoring across build and data planes
Integration with the observability plane for comprehensive platform monitoring
Centralized log publishing and access through the observer service

Verify the observer configuration:

# Check DataPlane observer config
kubectl get dataplane default -n default -o jsonpath='{.spec.observer}' --context kind-openchoreo-cp | jq '.'

# Check BuildPlane observer config
kubectl get buildplane default -n default -o jsonpath='{.spec.observer}' --context kind-openchoreo-cp | jq '.'

Verification

Switch to Control Plane Context

Set your default kubectl context to the control plane for easier management:

kubectl config use-context k3d-openchoreo-cp

Check that default OpenChoreo resources were created:

# Check default organization and project (on control plane)
kubectl get organizations,projects,environments -A

# Check default component types (on control plane)
kubectl get componenttypes -n default

# Check all OpenChoreo CRDs (on control plane)
kubectl get crds | grep openchoreo

# Check gateway resources (on data plane)
kubectl get gateway,httproute -n openchoreo-data-plane --context k3d-openchoreo-dp

Verify Data Plane and Build Plane Registration

# Verify DataPlane is registered and ready
kubectl get dataplane -n default
kubectl describe dataplane -n default | grep -A 5 "Status:"

# Verify BuildPlane is registered and ready (if installed)
if kubectl get buildplane -n default &>/dev/null; then
  echo "BuildPlane found:"
  kubectl get buildplane -n default
  kubectl describe buildplane -n default | grep -A 5 "Status:"
else
  echo "BuildPlane not installed - skipping verification"
fi

Check that all components are running:

# Check control plane cluster
kubectl cluster-info
kubectl get pods -n openchoreo-control-plane

# Check data plane cluster
kubectl cluster-info --context k3d-openchoreo-dp
kubectl get pods -n openchoreo-data-plane --context k3d-openchoreo-dp
kubectl get nodes --context k3d-openchoreo-dp

# Check build plane cluster (if installed)
kubectl cluster-info --context k3d-openchoreo-bp
kubectl get pods -n openchoreo-build-plane --context k3d-openchoreo-bp

# Check observability plane cluster (if installed)
kubectl cluster-info --context k3d-openchoreo-op
kubectl get pods -n openchoreo-observability-plane --context k3d-openchoreo-op

Troubleshooting

Cluster Creation Issues

If you encounter issues creating k3d clusters:

Ensure you have sufficient CPU and memory allocated to Docker (or the VM running Docker). We recommend at least 8 GB RAM and 4 CPU cores.
If using Colima, make sure to set K3D_FIX_DNS=0 environment variable when creating clusters.
Check Docker is running: docker info

If k3d cluster creation gets stuck or hangs, you may need to increase inotify limits in your VM:

# For Colima users
colima ssh
sudo sysctl -w fs.inotify.max_user_watches=524288
sudo sysctl -w fs.inotify.max_user_instances=512
exit

# Then retry creating the cluster

Next Steps

After completing this multi-cluster setup you can:

Deploy your first component to get started with OpenChoreo
Test the GCP microservices demo to see multi-component applications in action across clusters
Deploy additional sample applications from the OpenChoreo samples
Experiment with cross-cluster deployments and observe how components interact across the distributed platform

Cleaning Up

To completely remove the multi-cluster installation:

# Delete all k3d clusters
k3d cluster delete openchoreo-cp
k3d cluster delete openchoreo-dp
k3d cluster delete openchoreo-bp  # if installed
k3d cluster delete openchoreo-op  # if installed

Prerequisites​

Verify Prerequisites​

Quick Setup​

1. Setup the Control Plane​

Create the Control Plane Cluster​

Install OpenChoreo Control Plane​

2. Setup the Data Plane​

Create the Data Plane Cluster​

Install OpenChoreo Data Plane​

Configure DataPlane​

3. Setup the Build Plane (Optional)​

Create the Build Plane Cluster​

Install OpenChoreo Build Plane​

Configure BuildPlane​

4. Setup the Observability Plane (Optional)​

Create the Observability Plane Cluster​

Install OpenChoreo Observability Plane​

Configure Cross-Cluster Observability​

Configure Observer Integration​

Verification​

Switch to Control Plane Context​

Check that default OpenChoreo resources were created:​

Verify Data Plane and Build Plane Registration​

Check that all components are running:​

Troubleshooting​

Cluster Creation Issues​

Next Steps​

Cleaning Up​

Prerequisites

Verify Prerequisites

Quick Setup

1. Setup the Control Plane

Create the Control Plane Cluster

Install OpenChoreo Control Plane

2. Setup the Data Plane

Create the Data Plane Cluster

Install OpenChoreo Data Plane

Configure DataPlane

3. Setup the Build Plane (Optional)

Create the Build Plane Cluster

Install OpenChoreo Build Plane

Configure BuildPlane

4. Setup the Observability Plane (Optional)

Create the Observability Plane Cluster

Install OpenChoreo Observability Plane

Configure Cross-Cluster Observability

Configure Observer Integration

Verification

Switch to Control Plane Context

Check that default OpenChoreo resources were created:

Verify Data Plane and Build Plane Registration

Check that all components are running:

Troubleshooting

Cluster Creation Issues

Next Steps

Cleaning Up