Skip to main content
Version: v0.17.x

Run OpenChoreo on K3d Locally

This guide walks you through setting up OpenChoreo on your machine with k3d. You will install each plane one at a time, and after each one you will do something real with it: log in, deploy a service, or trigger a build.

OpenChoreo has four planes:

  • Control Plane runs the API, console, identity provider, and controllers.
  • Data Plane runs your workloads and routes traffic to them.
  • Build Plane builds container images from source using Argo Workflows.
  • Observability Plane collects logs and metrics from all other planes.

By the end you will have all four running in a single k3d cluster.

What you will get:

  • A working OpenChoreo installation on localhost
  • A deployed web app you can open in your browser
  • A source-to-image build pipeline
  • Log collection and querying

Prerequisites​

ToolVersionPurpose
Dockerv26+ (8 GB RAM, 4 CPU)Container runtime
k3dv5.8+Local Kubernetes clusters
kubectlv1.32+Kubernetes CLI
Helmv3.12+Package manager

Verify everything is installed:

docker --version
k3d --version
kubectl version --client
helm version --short

Verify container runtime is running:

docker info > /dev/null

Step 1: Create the Cluster​

Colima users

If you are using Colima as your container runtime, prefix the cluster create command with K3D_FIX_DNS=0 to avoid DNS resolution issues inside the cluster.

curl -fsSL https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/single-cluster/config.yaml | k3d cluster create --config=-

This creates a cluster named openchoreo. Your kubectl context is now k3d-openchoreo.

Step 2: Install Prerequisites​

These are third-party components that OpenChoreo depends on. None of them are OpenChoreo-specific, they are standard Kubernetes building blocks.

This runs all the prerequisite commands from the Step-by-Step tab sequentially in a single script. If you want to understand what each component does and why it is needed, switch to the Step-by-Step tab instead.

The script installs the following components:

  • Gateway API CRDs — Kubernetes-native ingress and routing definitions
  • cert-manager — Automated TLS certificate management
  • External Secrets Operator — Syncs secrets from external providers into Kubernetes
  • kgateway — Gateway API implementation that handles traffic routing
  • OpenBao — Secret backend (open-source Vault fork) with a ClusterSecretStore
curl -fsSL https://openchoreo.dev/docs/v0.17.x/getting-started/try-it-out/on-k3d-locally/k3d-prerequisites.sh | bash

Step 3: Setup Control Plane​

The control plane is the brain of OpenChoreo. It runs the API server, the web console, the identity provider, and the controllers that reconcile your resources.

Install Thunder (Identity Provider)​

Thunder handles authentication and OAuth flows. The values file includes bootstrap scripts that run on first startup and configure the organization, users, groups, and OAuth applications automatically.

helm upgrade --install thunder oci://ghcr.io/asgardeo/helm-charts/thunder \
--namespace thunder \
--create-namespace \
--version 0.24.0 \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/common/values-thunder.yaml

Wait for Thunder to be ready:

kubectl wait -n thunder \
  --for=condition=available --timeout=300s deployment -l app.kubernetes.io/name=thunder

Backstage Secrets​

The web console (Backstage) needs a backend secret for session signing and an OAuth client secret to authenticate with Thunder. This pulls values from the ClusterSecretStore created earlier:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: backstage-secrets
  namespace: openchoreo-control-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: backstage-secrets
  data:
  - secretKey: backend-secret
    remoteRef:
      key: backstage-backend-secret
      property: value
  - secretKey: client-secret
    remoteRef:
      key: backstage-client-secret
      property: value
  - secretKey: jenkins-api-key
    remoteRef:
      key: backstage-jenkins-api-key
      property: value
EOF

Install the Control Plane​

helm upgrade --install openchoreo-control-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 0.17.0 \
--namespace openchoreo-control-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/single-cluster/values-cp.yaml

Wait for all deployments to come up:

kubectl wait -n openchoreo-control-plane \
  --for=condition=available --timeout=300s deployment --all

What Got Installed​

Here is what is now running in and around the control plane:

  • controller-manager reconciles OpenChoreo resources (Projects, Components, Environments, etc.)
  • openchoreo-api is the REST API the console and CLI talk to
  • backstage is the web console
  • cluster-gateway accepts WebSocket connections from agents in remote planes
  • gateway (managed by kgateway) routes external traffic to services

In the thunder namespace:

  • thunder handles authentication and OAuth flows
Thunder Admin Console

You can browse and modify the bootstrapped identity configuration (users, groups, OAuth applications) in the Thunder admin console at http://thunder.openchoreo.localhost:8080/develop using admin / admin. For details on what the bootstrap configured, see the On Your Environment guide.

Step 4: Install Default Resources​

OpenChoreo needs some base resources before you can deploy anything: a project, environments, component types, and a deployment pipeline. These define what kinds of things you can build and where they run.

kubectl apply -f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/samples/getting-started/all.yaml && \
kubectl label namespace default openchoreo.dev/controlplane-namespace=true

What was created:

  • Project: default
  • Environments: development, staging, production
  • DeploymentPipeline: default (development -> staging -> production)
  • ComponentTypes: service, web-application, scheduled-task, worker
  • Workflows: docker, google-cloud-buildpacks, ballerina-buildpack, react
  • Traits: api-configuration, observability-alert-rule

Step 5: Setup Data Plane​

The data plane is where your workloads actually run. It has its own gateway for routing traffic, and a cluster-agent that connects back to the control plane to receive deployment instructions.

Namespace and Certificates​

Each plane needs a copy of the cluster-gateway CA certificate so its agent can establish a trusted connection to the control plane. We read it directly from the cert-manager Secret that the control plane installation created.

kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

# Wait for cert-manager to issue the cluster-gateway CA
kubectl wait -n openchoreo-control-plane \
  --for=condition=Ready certificate/cluster-gateway-ca --timeout=120s

# Copy the CA directly from the cert-manager Secret into a ConfigMap the agent can mount
CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

Install the Data Plane​

helm upgrade --install openchoreo-data-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
--version 0.17.0 \
--namespace openchoreo-data-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/single-cluster/values-dp.yaml

Register the Data Plane​

The DataPlane resource tells the control plane about this data plane. It includes the agent's CA certificate (so the control plane trusts its WebSocket connection) and the gateway's public address (so the control plane knows how to route traffic to workloads).

AGENT_CA=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: DataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$AGENT_CA" | sed 's/^/        /')
  secretStoreRef:
    name: default
  gateway:
    ingress:
      external:
        http:
          host: openchoreoapis.localhost
          listenerName: http
          port: 19080
        name: gateway-default
        namespace: openchoreo-data-plane
EOF

The cluster-agent in the data plane establishes an outbound WebSocket connection to the control plane's cluster-gateway. The control plane sends deployment instructions over this connection. No inbound ports need to be opened on the data plane.

Try it: Log in to OpenChoreo​

Open http://openchoreo.localhost:8080 in your browser.

Log in with the default credentials:

UsernamePassword
admin@openchoreo.devAdmin@123

You should see the OpenChoreo console. The control plane is working.

Try it: Deploy the React Starter App​

kubectl apply -f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/samples/from-image/react-starter-web-app/react-starter.yaml

Wait for the deployment to come up:

kubectl wait --for=condition=available deployment \
  -l openchoreo.dev/component=react-starter -A --timeout=120s

Get the application URL:

HOSTNAME=$(kubectl get httproute -A -l openchoreo.dev/component=react-starter \
  -o jsonpath='{.items[0].spec.hostnames[0]}')
echo "http://${HOSTNAME}:19080"

Open that URL in your browser. You should see the React starter application running.

The data plane is routing traffic to your workload through the gateway.

Step 6: Setup Build Plane (Optional)​

The build plane takes source code, builds a container image, pushes it to a registry, and tells the control plane about the new image. It uses Argo Workflows to run build pipelines.

Namespace and Certificates​

Same as the data plane. Copy the cluster-gateway CA from the cert-manager Secret so the build plane's agent can connect to the control plane:

kubectl create namespace openchoreo-build-plane --dry-run=client -o yaml | kubectl apply -f -

# Copy the CA directly from the cert-manager Secret
CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-build-plane --dry-run=client -o yaml | kubectl apply -f -

Container Registry​

Builds need somewhere to push images. For local dev, a simple in-cluster Docker registry works:

helm repo add twuni https://twuni.github.io/docker-registry.helm && helm repo update && \
helm install registry twuni/docker-registry \
--namespace openchoreo-build-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/single-cluster/values-registry.yaml

Install the Build Plane​

helm upgrade --install openchoreo-build-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-build-plane \
--version 0.17.0 \
--namespace openchoreo-build-plane \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/single-cluster/values-bp.yaml

Install Workflow Templates​

Build pipelines are defined as ClusterWorkflowTemplates. Each build workflow (docker, react, etc.) is composed from smaller shared templates: a checkout step (controls how source code is cloned), build coordinator templates (docker, react, ballerina-buildpack, google-cloud-buildpacks), and a publish step (controls where built images get pushed). For k3d, the publish step targets the local registry at host.k3d.internal:10082. In a real environment you would point it at ECR, GAR, GHCR, or whatever registry you use.

kubectl apply \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/samples/getting-started/workflow-templates/checkout-source.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/samples/getting-started/workflow-templates.yaml \
-f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/samples/getting-started/workflow-templates/publish-image-k3d.yaml

Register the Build Plane​

AGENT_CA=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-build-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: BuildPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$AGENT_CA" | sed 's/^/        /')
  secretStoreRef:
    name: default
EOF

Try it: Build from Source​

Apply a sample component that builds a Go service from source:

kubectl apply -f https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/samples/from-source/services/go-docker-greeter/greeting-service.yaml

Watch the build progress:

kubectl get workflow -n openchoreo-ci-default --watch

You can also open the Argo Workflows UI at http://localhost:10081 to see the build pipeline visually.

After the build completes, wait for the deployment:

kubectl wait --for=condition=available deployment \
  -l openchoreo.dev/component=greeting-service -A --timeout=300s

Resolve the hostname and path, then call the service:

HOSTNAME=$(kubectl get httproute -A -l openchoreo.dev/component=greeting-service \
  -o jsonpath='{.items[0].spec.hostnames[0]}')
PATH_PREFIX=$(kubectl get httproute -A -l openchoreo.dev/component=greeting-service \
  -o jsonpath='{.items[0].spec.rules[0].matches[0].path.value}')

curl "http://${HOSTNAME}:19080${PATH_PREFIX}/greeter/greet"

OpenChoreo built your code, pushed the image to the local registry, and deployed it to the data plane.

Step 7: Setup Observability Plane (Optional)​

OpenChoreo follows a modular architecture. The observability plane consists of system services plus various observability modules that you can install to get observability features. E.g. If you require observability logs features, you may install a logs module.

Namespace and Certificates​

kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

# Copy the CA directly from the cert-manager Secret
CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

OpenSearch Credentials​

note

This step is required only if you use a logs or tracing module based on OpenSearch

The Observer API needs credentials to connect to OpenSearch. This pulls values from the ClusterSecretStore created earlier:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-opensearch-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-opensearch-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
EOF

Generate a machine ID​

Fluent Bit (the log collector) needs /etc/machine-id to identify the node. k3d containers don't have one by default, so generate it:

docker exec k3d-openchoreo-server-0 sh -c \
  "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id"

Install the Observability Plane​

System services​

Execute the following command to install observability plane's system services

helm upgrade --install openchoreo-observability-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
--version 0.17.0 \
--namespace openchoreo-observability-plane \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/release-v0.17/install/k3d/single-cluster/values-op.yaml \
--timeout 25m

Observability Modules​

We will now install Logs, Metrics, and Tracing modules based on OpenSearch and Prometheus in the observability plane.

If you would like to install different observability modules, visit the Modules catalog to explore all available modules. Select the ones that match your observability requirements and follow the installation instructions for each module.

Create OpenSearch secret

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
EOF

Install OpenSearch based logs module

helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.3.2 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials"

# Enable log collection
helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.3.2 \
  --reuse-values \
  --set fluent-bit.enabled=true

Install Prometheus based metrics module

helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.2.2

Install OpenSearch based tracing module

helm upgrade --install observability-tracing-opensearch \
  oci://ghcr.io/openchoreo/charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.3.3 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials"

Register the Observability Plane​

AGENT_CA=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$AGENT_CA" | sed 's/^/        /')
  observerURL: http://observer.openchoreo.localhost:11080
EOF

Tell the data plane (and build plane, if installed) where to send their telemetry:

kubectl patch dataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ObservabilityPlane","name":"default"}}}'

# If you installed the build plane:
kubectl patch buildplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ObservabilityPlane","name":"default"}}}'

Cleanup​

Delete the cluster and everything in it:

k3d cluster delete openchoreo

Next Steps​