Perch Agent

The Perch Agent is an AI-powered, chat assistant for OpenChoreo. Perch is read-only by contract. It can list, get, and query — it cannot create, update, delete, or trigger anything. Write-style requests are politely declined.

Prerequisites

Before enabling the Perch Agent, ensure the following:

• OpenChoreo Control Plane installed (this is where Perch runs).
• OpenChoreo Observability Plane installed if you want runtime-debug answers (Perch's "what's going wrong in my logs?" path calls the Observer MCP).
• Backstage deployed and reachable — Perch is a backend; users interact with it through Backstage's chat drawer.
• An LLM API key from OpenAI (support for other providers coming soon)
• (Optional) The SRE Agent deployed if you want Tier-2 deep analysis from inside a chat.

note

Perch streams responses over HTTP/ndjson and can issue several MCP tool calls per turn. Choose a model with reasonable streaming + tool-use quality (e.g. openai:gpt-5).

Enabling the Perch Agent

Step 1: Create the Perch Agent Secret

The Perch Agent requires a Kubernetes Secret in the openchoreo-control-plane namespace with the following key:

Key	Description
PERCH_LLM_API_KEY	Your LLM provider API key

The Secret name is referenced from the chart via perchAgent.llm.secretName, so pick any name (the deployment fails fast at install time if the value is unset).

You can create this secret using any method you prefer. If you followed the Try It Out on k3d locally guide:

kubectl exec -n openbao openbao-0 -- \
  env BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root \
  bao kv put secret/perch-llm-api-key value="<YOUR_LLM_API_KEY>"

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: perch-agent-secret
  namespace: openchoreo-control-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: perch-agent-secret
  data:
  - secretKey: PERCH_LLM_API_KEY
    remoteRef:
      key: perch-llm-api-key
      property: value
EOF

If you are not using OpenBao + External Secrets, a plain Kubernetes Secret works just as well:

kubectl create secret generic perch-agent-secret \
  -n openchoreo-control-plane \
  --from-literal=PERCH_LLM_API_KEY="<YOUR_LLM_API_KEY>"

Step 2: Upgrade the Control Plane

Enable the Perch Agent and point it at your secret + model. The --reuse-values flag preserves your existing configuration.

helm upgrade --install openchoreo-control-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 0.0.0-latest-dev \
--namespace openchoreo-control-plane \
--reuse-values \
--set perchAgent.enabled=true \
--set perchAgent.llm.secretName=perch-agent-secret \
--set perchAgent.llm.modelName=openai:gpt-5

Supported Models

Any provider:model string accepted by LangChain's init_chat_model works. Common picks:

• openai:gpt-5 — base model, good default for development.
• openai:gpt-5.2-pro — better tool-use reasoning, recommended for production.
• openai:gpt-5.4 — latest, highest-quality option.

If your Observer or SRE Agent live outside the cluster, also point Perch at them:

--set perchAgent.config.observerApiUrl=http://observer.example.com:8080
--set perchAgent.config.rcaAgentApiUrl=http://rca-agent.example.com:11080

When rcaAgentApiUrl is empty, Perch simply does not register the SRE Agent MCP — chat keeps working with just the OpenChoreo + Observability tools.

Step 3: Wire Backstage to call Perch

Backstage ships with the Perch chat drawer behind a feature flag. The control-plane chart exposes it as backstage.features.assistant:

helm upgrade --install openchoreo-control-plane oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 0.0.0-latest-dev \
--namespace openchoreo-control-plane \
--reuse-values \
--set backstage.features.assistant.enabled=true

Value	Default	Notes
backstage.features.assistant.enabled	false	Toggles the "Ask Perch" affordance on the Logs tab and the chat drawer. Renders as OPENCHOREO_FEATURES_ASSISTANT_ENABLED.
backstage.features.assistant.perchAgentUrl	http://perch-agent.openchoreo-control-plane.svc.cluster.local:8080	Cluster-internal Service URL the Backstage forwarder calls. Renders as OPENCHOREO_PERCH_AGENT_URL.

If Perch runs in a different cluster or namespace, override the URL:

--set backstage.features.assistant.perchAgentUrl=http://perch-agent.example.com:8080

After this, restart the Backstage pod so the new env vars take effect:

kubectl -n openchoreo-control-plane rollout restart deployment/backstage

Step 4: Verify the installation

Check that the Perch Agent pod is running:

kubectl get pods -n openchoreo-control-plane -l app.kubernetes.io/component=perch-agent

Hit the health endpoint over a port-forward:

kubectl -n openchoreo-control-plane port-forward svc/perch-agent 8088:8080 &
curl -sf http://localhost:8088/health   # {"status":"healthy"}

Open Backstage in a browser, navigate to any component's Logs tab, and you should see an "Ask Perch" affordance — click it to send your first chat. The drawer streams tokens as the model produces them.

If the drawer never appears, the most common cause is the Backstage feature flag — check the running pod's env:

kubectl -n openchoreo-control-plane get deployment backstage \
  -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep -i 'assistant\|perch'

You should see both OPENCHOREO_FEATURES_ASSISTANT_ENABLED=true and OPENCHOREO_PERCH_AGENT_URL=....

Exposing Perch externally via HTTPRoute

By default, Perch is only reachable in-cluster. To expose /api/v1alpha1/perch-agent/* through the OpenChoreo Gateway, set a hostname:

helm upgrade --install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --namespace openchoreo-control-plane \
  --reuse-values \
  --set 'perchAgent.http.hostnames[0]=perch.openchoreo.example.com'

The chart renders an HTTPRoute for /chat (with backendRequest: 450s to allow long LLM turns) and /warmup. A streaming TrafficPolicy (streamIdle: 24h, request: 0s) is also applied so the ndjson stream is not torn down mid-turn by the gateway.

Authentication and Authorization (External IdP)

By default, OpenChoreo configures Thunder as the identity provider for Perch with a pre-configured OAuth client (openchoreo-backstage-client) bound to the backstage-catalog-reader role — which is granted the perch:invoke action. Chat requests originate from Backstage with the end user's JWT; Perch validates the JWT against the Thunder JWKS and forwards it verbatim to the MCP servers, so the per-tool authorization is end-user scoped, not service-account scoped.

If you are using an external identity provider, configure the JWKS / issuer / audience on Perch via the standard control-plane security values:

security:
  oidc:
    jwksUrl: "<your-idp-jwks-url>"
    issuer: "<your-idp-issuer>"
  jwt:
    audience: "<your-perch-audience>"

For service accounts that need to call /chat programmatically (rather than via Backstage forwarding a user JWT), create an OAuth client that supports client_credentials and grant it the perch:invoke action:

openchoreoApi:
  config:
    security:
      authorization:
        bootstrap:
          mappings:
            - name: perch-service-binding
              roleRef:
                name: backstage-catalog-reader # or a dedicated role that has perch:invoke
              entitlement:
                claim: sub
                value: "<your-client-id>"
              effect: allow

See Identity Provider Configuration for detailed setup.

Tuning capacity and behaviour

Value	Default	What it does
perchAgent.replicas	1	Increase for HA / higher chat throughput. Perch is stateless — adding replicas is safe.
perchAgent.config.maxConcurrentChats	20	Per-pod concurrent-chat semaphore. Bound to your LLM provider's per-minute rate limit; raising it past your provider quota will surface as 429s.
perchAgent.config.authzTimeoutSeconds	30	Hard timeout for the per-request authz check.
perchAgent.config.logLevel	INFO	Set to DEBUG for verbose MCP / tool-call logging during incident investigation.
perchAgent.cors.allowedOrigins	[]	Only relevant if you hit /chat directly from a browser bypassing the Backstage forwarder.
perchAgent.extraEnvs	[]	Inject additional env vars without editing the chart.

Bring-your-own auth-config

Perch ships with a default /app/auth-config.yaml baked into the image that defines its action catalog (perch:invoke) and subject-type rules. To override, create a ConfigMap with your replacement file and point the chart at it:

kubectl create configmap perch-auth-config \
  -n openchoreo-control-plane \
  --from-file=auth-config.yaml=./my-auth-config.yaml

helm upgrade --install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --namespace openchoreo-control-plane \
  --reuse-values \
  --set perchAgent.authConfigConfigMap=perch-auth-config

The chart will mount the ConfigMap at /etc/openchoreo/auth-config.yaml and set AUTH_CONFIG_PATH accordingly.