Zitadel

Zitadel is the identity/auth provider (IAM) — organizations, users, and machine service accounts. Runs as 2+ replicas for HA, backed by Postgres (CNPG). Assumes PostgreSQL is already running. This doc covers two deployment modes: externally-routed (public domain, TLS terminated upstream) and internal-only (no external route, plain HTTP inside the cluster) — pick whichever matches your use case, or run both for different purposes.

Overview

2 replicas minimum for HA, spread across nodes via pod anti-affinity
Backed by CNPG Postgres — either the direct service or the PgBouncer pooler
An init container blocks pod startup until Postgres is reachable
/zitadel-data is an EmptyDir — anything written there (like the PAT) does not survive a pod restart

Architecture

Zitadel pods (2+, anti-affinity across nodes)
   ↓ init container: wait-for-postgres (nc -z check before starting)
   ↓
Postgres (direct: postgres-cluster-rw, or pooled: postgres-pooler-rw)
   ↓
CNPG cluster (see PostgreSQL doc)

First run only (start-from-init):
   creates org + admin user + machine user → writes PAT to /zitadel-data (EmptyDir)

Key concepts

`MACHINE` vs `LOGINCLIENT` first-instance users

Zitadel's first-instance bootstrap can create two different kinds of service users automatically:

	`ORG_MACHINE_*`	`ORG_LOGINCLIENT_*`
Role granted	`IAM_OWNER`	`IAM_LOGIN_CLIENT`
Used by	Your own automation / scripts	A separate Login V2 UI container
Needed here?	Yes	No — this setup doesn't run Login V2 (`ZITADEL_DEFAULTINSTANCE_FEATURES_LOGINV2_REQUIRED: "false"`)

Only configure the MACHINE block — LOGINCLIENT env vars are unnecessary overhead if Login V2 isn't running.

`--tlsMode` and `ExternalSecure` — this is what differs between the two modes

--tlsMode external tells Zitadel that TLS termination happens upstream (an ingress/gateway in front), so Zitadel itself serves plain HTTP internally. Per Zitadel's own CLI docs, --tlsMode external also overwrites ExternalSecure and TLS.Enabled internally — it sets ExternalSecure: true and TLS.Enabled: false automatically. You do not need to separately set ZITADEL_EXTERNALSECURE when using --tlsMode external.

For the internal-only mode, there's no upstream TLS termination at all, so --tlsMode disabled is used instead, and ZITADEL_EXTERNALSECURE: "false" is set explicitly (since there's no --tlsMode external to set it automatically).

Prerequisites

PostgreSQL (CNPG) already running
The real postgres superuser password (not a made-up admin username — see the ⚠️ callout below)
If using the PgBouncer pooler: this normally just works with no extra steps — see the troubleshooting section below only if you actually hit a missing-role or missing-function error

Two deployment modes

Setting	External (public domain)	Internal only
Namespace (example)	`production`	`zitadel`
`ZITADEL_EXTERNALDOMAIN`	`iam.playtelly.castis.io`	`zitadel.zitadel.svc.cluster.local`
`--tlsMode`	`external`	`disabled`
`--externalPort`	`443`	`8080`
`ZITADEL_EXTERNALSECURE`	not set — `--tlsMode external` sets this automatically	`"false"` — set explicitly, nothing else sets it

Everything else (database connection, first-instance settings, HA structure, probes) is identical between the two — only the handful of rows above differ. Broken down as explicit checklists:

External mode — set: - ZITADEL_EXTERNALDOMAIN → your real public domain - --tlsMode external - --externalPort 443

External mode — do NOT set: - ZITADEL_EXTERNALSECURE — leave it unset; --tlsMode external sets it automatically, and setting it yourself is redundant at best

Internal mode — set: - ZITADEL_EXTERNALDOMAIN → the internal cluster DNS name, e.g. <service>.<namespace>.svc.cluster.local - --tlsMode disabled - --externalPort → same as --port (e.g. both 8080) — there's no separate public port - ZITADEL_EXTERNALSECURE: "false" — set this explicitly

Internal mode — do NOT set: - --tlsMode external — there's no upstream TLS termination in this mode, so this would misrepresent the connection as secure when it isn't

⚠️ Must configure before applying

What	Where	Why it matters
`ZITADEL_MASTERKEY`	`01-secret.yaml`	Generate a real key for your environment — don't reuse an example value from any doc or repo
`ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME`	`02-configmap.yaml`	Must be `postgres` — the actual CNPG superuser. A made-up name here (e.g. a custom "admin" username) will never exist in Postgres and fails with a generic `password authentication failed` error that looks like a wrong-password bug, not a wrong-username one
`ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD`	`01-secret.yaml`	Must match the real `postgres` role's actual password in Postgres — not whatever was typed into the Secret. Editing a Kubernetes Secret does not change or create anything inside Postgres itself
`ZITADEL_DATABASE_POSTGRES_HOST`	`02-configmap.yaml`	Host only, no port — e.g. `postgres-cluster-rw.database.svc.cluster.local`, not `...cluster.local:5432`. DNS lookups don't include a port; combining them causes a hostname resolution failure
`ZITADEL_DATABASE_POSTGRES_PORT`	`02-configmap.yaml`	Set separately from host, typically `"5432"`
`ZITADEL_EXTERNALDOMAIN` / `--tlsMode` / `--externalPort` / `ZITADEL_EXTERNALSECURE`	`02-configmap.yaml` + `03-deployment.yaml` args	Depends entirely on which mode you're deploying — see the table above
`ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME` / `PASSWORD` / `ORG_NAME`	`01-secret.yaml` / `02-configmap.yaml`	Your real bootstrap admin identity — only used on the very first `start-from-init` run
Container args: `start-from-init` vs `start`	`03-deployment.yaml` → `args`	`start-from-init` only for the very first run; switch to `start` afterward — see Deploy below
Image tag	`03-deployment.yaml` → `image:`	Pin an exact version (e.g. `v4.15.1`)

Do not change these — required exactly as documented

Setting	Why
`securityContext` (`runAsUser`/`runAsGroup`/`fsGroup: 1000`, `runAsNonRoot: true`)	Standard non-root hardening for the Zitadel image
`wait-for-postgres` init container	Blocks the Zitadel container from starting until Postgres actually accepts connections on port 5432 — without it, pods can crash-loop against a Postgres that isn't ready yet
`podAntiAffinity` across `kubernetes.io/hostname`	Spreads replicas across nodes — required for actual HA, not just replica count
`ORG_MACHINE_` env vars (not `ORG_LOGINCLIENT_`)	This deployment doesn't run the separate Login V2 UI, so only the `MACHINE` service-user block is needed
`readinessProbe`/`livenessProbe`/`startupProbe` paths (`/debug/ready`, `/debug/healthz`)	Zitadel-specific health endpoints — the generous `startupProbe` (up to ~5 min) accounts for first-instance bootstrap taking longer than a normal restart
`/zitadel-data` as `EmptyDir`	Works for now, but means the PAT/masterkey artifacts must be extracted immediately after first-instance setup — see below. Moving this to a PVC is a reasonable future improvement if anything needs to read those files across restarts

Deploy

This is the safe order for standing Zitadel up from scratch (or rebuilding it later) — skipping steps or reordering them is what causes the issues in the Troubleshooting section below.

Phase 1 — first-time bootstrap (1 replica)

kubectl apply -f 01-secret.yaml
kubectl apply -f 02-configmap.yaml
kubectl apply -f 03-deployment.yaml   # replicas: 1, args: start-from-init
kubectl apply -f 04-service.yaml

kubectl get pods -n <namespace> -w

Wait for the pod to reach 1/1 Running — this run performs the actual first-instance bootstrap (org, admin user, machine user) and writes the PAT.

Extract and save the PAT and masterkey immediately — /zitadel-data is an EmptyDir and won't survive a pod restart:

kubectl debug -it <pod> -n <namespace> --image=busybox --target=zitadel -- sh
ps aux                                       # find the zitadel PID (usually 1)
cat /proc/1/root/zitadel-data/zitadel.pat

Do not proceed to Phase 2 until the PAT is copied out somewhere durable.

Phase 2 — scale to HA

Only after the PAT is safely saved, make these edits together, then re-apply:

# In 02-configmap.yaml: switch ZITADEL_DATABASE_POSTGRES_HOST to the pooler
#   (only if you want pooled connections — direct is fine to keep otherwise)
# In 03-deployment.yaml: replicas 1 → 2+, args start-from-init → start

kubectl apply -f 02-configmap.yaml
kubectl apply -f 03-deployment.yaml
kubectl rollout status deployment/zitadel -n <namespace>

# Optional, now that replicas >= 2:
kubectl apply -f 05-pdb.yaml
kubectl apply -f 06-hpa.yaml

Why this order: running start-from-init on more than one replica risks a race on the setup routine, and switching to the pooler is one more variable worth isolating from the initial bootstrap — if the pooler's role/function happen to be missing (see the troubleshooting note below), it's easier to spot and fix against an already-working instance than during first-time setup. Doing the fragile one-time steps in isolation on a single pod against the simplest connection path first, then introducing the pooler and extra replicas once the instance is already known-good, keeps the two concerns separate.

Phase 3 — routing (External mode only)

Skip this entirely if running Internal-only mode — there's no external route to set up. If running External mode, apply the private route first (private node/context):

kubectl apply -f apisix/private/httproute.yml

Then switch context to the public node and apply the public route files:

kubectl apply -f apisix/public/01-certificate.yaml
kubectl apply -f apisix/public/02-apisixtls.yaml
kubectl apply -f apisix/public/03-apisixroute.yaml
kubectl apply -f apisix/public/04-apisixupstream.yaml

See Routing below for what each of these actually does.

Routing (External mode only)

Same pattern as authapi's routing: private and public are two separate APISIX installations connected over Tailscale, not a Kubernetes namespace split. This section only applies to External mode — Internal mode has no external route at all.

Private route (`apisix/private/httproute.yml`)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: zitadel-internal
  namespace: production
spec:
  parentRefs:
    - name: apisix-gateway
      namespace: apisix
  hostnames:
    - iam.playtelly.castis.io
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: zitadel
          port: 8080

Routes all traffic (PathPrefix: /) for iam.playtelly.castis.io on the private APISIX environment straight to the zitadel Service on port 8080 — the same port Zitadel's container listens on internally (--port 8080), separate from the externally-advertised --externalPort 443.

Public route (`apisix/public/01`–`04`)

Same 4-file legacy ApisixRoute shape as authapi's public route — Certificate, ApisixTls, ApisixRoute, ApisixUpstream. I don't have the actual contents of Zitadel's four files here, only the filenames from the file tree. Based on authapi's verified pattern, the shape would be:

01-certificate.yaml — cert-manager Certificate for iam.playtelly.castis.io
02-apisixtls.yaml — ApisixTls, binding that certificate to APISIX
03-apisixroute.yaml — an ApisixRoute matching iam.playtelly.castis.io, referencing an upstream (likely named zitadel-upstream), probably with the same redirect (HTTP→HTTPS) and proxy-rewrite (X-Forwarded-* headers) plugins authapi's public route uses
04-apisixupstream.yaml — an ApisixUpstream with externalNodes pointing at the private APISIX environment's Tailscale address (authapi's points at 100.64.0.2:80 — Zitadel's may be the same address, since that's the private environment's address, not something per-service)

⚠️ This is inferred from authapi's verified pattern, not Zitadel's actual files — share the real 01–04 contents and I'll replace this with the verified version, same as was done for authapi.

Files to apply (reference)

What each file from the commands above actually contains:

File	Purpose
`01-secret.yaml`	`zitadel-secret` — masterkey + DB/admin passwords
`02-configmap.yaml`	`zitadel-config` — DB connection, first-instance settings, external domain
`03-deployment.yaml`	Deployment — replica count and args change between Phase 1 and Phase 2, see above
`04-service.yaml`	ClusterIP Service
`05-pdb.yaml`	PodDisruptionBudget — only meaningful once `replicas >= 2`
`06-hpa.yaml`	HorizontalPodAutoscaler (optional)
`apisix/private/httproute.yml`	Gateway API route, private APISIX environment (External mode only)
`apisix/public/01-certificate.yaml`	cert-manager `Certificate` (External mode only)
`apisix/public/02-apisixtls.yaml`	`ApisixTls` (External mode only)
`apisix/public/03-apisixroute.yaml`	Legacy `ApisixRoute`, public-facing (External mode only)
`apisix/public/04-apisixupstream.yaml`	`ApisixUpstream`, forwards to private environment over Tailscale (External mode only)

Extracting the PAT from a minimal image

The official Zitadel image ships with only the zitadel binary — no shell, no coreutils, by design. kubectl exec ... cat or kubectl cp both fail with executable file not found in $PATH. The commands for this are shown in Phase 1 of Deploy above — this section just explains why that approach is necessary and how it works.

The workaround borrows a shell via a shared process namespace (kubectl debug --target), rather than anything built into the Zitadel image itself. This works because --target shares the Linux process namespace with the zitadel container — /proc/<pid>/root/ exposes that process's filesystem view to the debug container, even though they're separate container images.

If the pooler role/function is missing (troubleshooting, not a required step)

CNPG's pooler integration is supposed to automatically create a cnpg_pooler_pgbouncer role and a user_search lookup function in Postgres when you use the PgBouncer pooler. Most of the time this just works and there's nothing to do here. This section only applies if you actually hit one of the two errors below — it's not something to run preemptively.

-- If you see: role "cnpg_pooler_pgbouncer" does not exist
CREATE ROLE cnpg_pooler_pgbouncer WITH LOGIN;

-- If you see: function public.user_search(unknown) does not exist
-- (run against the postgres database)
CREATE OR REPLACE FUNCTION public.user_search(uname TEXT)
RETURNS TABLE (usename name, passwd text)
LANGUAGE sql SECURITY DEFINER
AS $$
    SELECT usename, passwd FROM pg_shadow WHERE usename = $1;
$$;

REVOKE ALL ON FUNCTION public.user_search(text) FROM public;
GRANT EXECUTE ON FUNCTION public.user_search(text) TO cnpg_pooler_pgbouncer;

No password needed for the role — authentication is via TLS client certificate, which pg_hba.conf handles correctly even when the role/function creation step above didn't run.

Troubleshooting

Symptom	Cause	Fix
ConfigMap fails to parse / applies incorrectly	Shell-style `KEY=VALUE` lines mixed into a YAML `data:` block, or an unquoted value containing a colon (e.g. a date) misparsed as a nested mapping	Use proper `KEY: "value"` YAML syntax throughout; quote any value containing a colon
`hostname resolving error: lookup ...:5432: no such host`	Host and port combined into one `ZITADEL_DATABASE_POSTGRES_HOST` value	Split into separate `HOST` and `PORT` variables
`password authentication failed for user "<made-up-name>"`	Admin username set to something that doesn't exist as a real Postgres role — Postgres returns the same generic error for "wrong password" and "role doesn't exist" (deliberate, prevents username enumeration)	Confirm real roles with `\du` inside Postgres; point the admin username at `postgres` (the actual superuser), not a custom name
`role "cnpg_pooler_pgbouncer" does not exist`	CNPG's pooler auto-provisioning partially failed (see gotcha above)	Create the role manually
`function public.user_search(unknown) does not exist`	Same partial pooler provisioning gap, different piece	Create the lookup function manually, against the `postgres` database
`kubectl exec ... cat` / `kubectl cp` fail: `executable file not found`	Minimal Zitadel image has no shell/coreutils	Use `kubectl debug --target` with busybox (see extraction technique above)
Concurrent setup issues when scaling replicas	Every pod runs `start-from-init` if that's still the configured arg, re-executing the full bootstrap routine on each one	Switch to `start` once the instance already exists — see the setup sequence above

Diagnostic commands

kubectl describe pod <pod> -n <namespace>                     # init container state, events
kubectl logs <pod> -n <namespace> -c zitadel --previous        # pre-restart crash logs
kubectl exec -n <db-namespace> <postgres-pod> -- psql -U postgres -c "\du"   # actual roles vs assumed
kubectl logs -n cnpg-system <operator-pod>                     # CNPG reconciliation activity
kubectl logs -n <db-namespace> <postgres-pod>                  # server-side auth logs — shows exact SQLSTATE + pg_hba.conf line matched
kubectl get pooler -n <db-namespace> -o yaml                    # pooler resource spec/status

Outstanding follow-ups worth tracking

Persist the PAT and masterkey somewhere durable — currently only ever extracted to terminal output from an EmptyDir
Apply the same fixes (host/port split, correct admin username, pooler role/function) to any other environment using Zitadel
Split start-from-init from start into separate Job vs. Deployment, matching the official CNPG/Zitadel Helm chart pattern, to remove any chance of the concurrent-setup race entirely
If ReplicaSet/ConfigMap drift shows up without a matching manual kubectl apply, check whether a GitOps tool (ArgoCD) is syncing from a repo — confirm and document the actual source of truth