Authapi

Sits between client apps and Zitadel: issues JWTs, manages cookies, and brokers OAuth2 flows. Has its own authapi Postgres database and talks to coreapi over an internal pre-shared key. Depends on PostgreSQL and Zitadel already being set up.

Overview

Backend auth service, deployed with Kustomize (base/ + overlays/prod/)
Connects directly to Postgres (not via the pooler) as postgres
Talks to Zitadel via OAuth2 + a service-account PAT
Issues access/refresh JWTs, sets a cookie scoped to .castis.io
Requires database seeding before first deploy — see below

Architecture

Internet
   ↓ id.castis.io (HTTPS)
Public APISIX  — redirects HTTP→HTTPS, sets X-Forwarded-* headers
   ↓ over Tailscale (100.64.0.2:80)
Private APISIX  — HTTPRoute matches id.castis.io
   ↓
authapi (production namespace)
   ├─→ postgres-cluster-rw.database.svc.cluster.local  (authapi database, direct — no pooler)
   ├─→ Zitadel  (OAuth2 app + service-account PAT)
   └─→ coreapi  (internal call, authenticated via shared PSK)

Client apps (tellyid, admin.telltboard, playout, spatio, shaka) reach authapi through the path above; CORS/OAuth2 config on authapi itself still governs which of those origins/redirects are actually allowed.

Repository structure (Kustomize)

services/access/authapi/
├── base/
│   ├── deployment.yml
│   ├── kustomization.yaml
│   └── service.yml
└── overlays/
    └── prod/
        ├── apisix/
        │   ├── private/
        │   │   └── httproute.yaml       # Gateway API route, Tailscale-private APISIX env
        │   └── public/
        │       ├── 01-certificate.yaml
        │       ├── 02-apisixtls.yaml
        │       ├── 03-apisixroute.yaml
        │       └── 04-apisixupstream.yaml
        ├── 002_invites_sessions.sql   # DB seed
        ├── 03-init-authapi.sql        # DB seed
        ├── hpa.yml
        ├── kustomization.yaml
        └── secret.yml                 # env vars — documented in full below

Prerequisites

PostgreSQL (CNPG) already running
Zitadel already running, with an OAuth2 app and a service-account PAT already generated (see that doc's PAT extraction section)
The authapi database created in Postgres, then seeded — see below

Database setup — create + seed before first deploy

Create the authapi database in Postgres, if it doesn't already exist:

kubectl exec -n database postgres-cluster-1 -- \
  psql -U postgres -c "CREATE DATABASE authapi;"

Run the two seed files against it, in this order:

kubectl exec -i -n database postgres-cluster-1 -- psql -U postgres -d authapi < overlays/prod/002_invites_sessions.sql

kubectl exec -i -n database postgres-cluster-1 -- psql -U postgres -d authapi < overlays/prod/03-init-authapi.sql

Do this once, before the first deploy — not something to repeat on every rollout.

Environment variables (`secret.yml`)

None of the actual secret values from your example are repeated here (some were already partially redacted) — this documents what each one is for and how to get or generate a real value.

Server

Key	Example	Notes
`AUTHAPI_SERVER_PORT`	`3002`	Port the container listens on
`AUTHAPI_COOKIE_DOMAIN`	`.castis.io`	Must be the parent domain shared across every client app below — a cookie scoped to `.castis.io` is valid for all its subdomains

CORS

Key	Notes
`AUTHAPI_CORS_ORIGIN`	Comma-separated list of allowed origins. Must exactly match your real production frontend domains — no wildcard here, since these are credentialed (cookie-bearing) requests
`ALLOWED_REDIRECT_URIS`	OAuth2 redirect target(s) allowed after login completes

PostgreSQL

Key	Notes
`AUTHAPI_DB_HOST` / `AUTHAPI_DB_PORT`	Already correctly split into two variables (see the Zitadel doc for why combining host:port breaks DNS resolution)
`AUTHAPI_DB_USER`	`postgres` in the example — the real Postgres superuser. Works, but means authapi's day-to-day traffic runs with superuser privileges rather than a scoped app user. Worth deciding deliberately rather than by default — see the note below
`AUTHAPI_DB_PASSWORD`	⚠️ Empty in the example — set it to the same `postgres` password already configured when PostgreSQL was set up (since `AUTHAPI_DB_USER` is `postgres`) — not a new/separate password
`AUTHAPI_DB_NAME`	`authapi` — must already exist (see Database setup above)
`AUTHAPI_DB_SSLMODE`	`disable` — fine for in-cluster traffic
`AUTHAPI_DB_MAX_CONNS`	`10` — per-pod connection cap. Since this connects directly (not via PgBouncer), multiply by replica count to know actual load on Postgres — e.g. 3 replicas × 10 = 30 connections reserved

Using postgres (the superuser) for an app's ongoing connection works, but a dedicated authapi role with only the permissions it needs is the tighter option long-term — same tradeoff discussed in the PostgreSQL doc's superuser note. Not fixing this now, just flagging it as a deliberate choice to revisit rather than an oversight.

Zitadel — OAuth2 App

Key	Notes
`ZITADEL_DOMAIN`	The real Zitadel external domain — what this looks like depends on which deployment mode Zitadel is running in (public domain vs. internal-only)

Zitadel — Service Account

Key	Notes
`ZITADEL_PAT`	The PAT generated during Zitadel's first-instance bootstrap — see the PAT extraction section in the Zitadel doc. Treat as a credential: rotate it if it's ever exposed

JWT

Key	Notes
`AUTHAPI_JWT_ACCESS_SECRET`	Symmetric secret — generate fresh, see below
`AUTHAPI_JWT_REFRESH_SECRET`	Symmetric secret — generate fresh, see below
`AUTHAPI_JWT_RSA_PRIVATE_KEY`	Base64-encoded RSA private key — different generation method, see below
`AUTHAPI_JWT_ACCESS_EXPIRY`	`900` — seconds (15 min)
`AUTHAPI_JWT_REFRESH_EXPIRY`	`168h` — 7 days
`AUTHAPI_JWT_ISSUER`	`authapi` — must match whatever validates these tokens downstream
`AUTHAPI_COOKIE_SECURE`	`true` — requires HTTPS; fine as long as TLS is terminated somewhere upstream (APISIX)

Misc / coreapi integration

Key	Notes
`AUTHAPI_ZITADEL_DEFAULT_ORG`	Must match the org name from Zitadel's first-instance bootstrap (`ZITADEL_FIRSTINSTANCE_ORG_NAME`)
`AUTHAPI_ZITADEL_SYSADMIN_USER`	Must match Zitadel's `ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME`
`COREAPI_AUTHAPI_PSK`	Shared secret between authapi and coreapi — generate fresh (see below), and configure the identical value on coreapi's side once that service is deployed
`COREAPI_BASE_URL`	Internal cluster DNS + port — must match coreapi's actual Service once it exists

Generating the secrets

Key	Command
`AUTHAPI_JWT_ACCESS_SECRET`	`openssl rand -base64 32`
`AUTHAPI_JWT_REFRESH_SECRET`	`openssl rand -base64 32`
`COREAPI_AUTHAPI_PSK`	`openssl rand -base64 32`
`AUTHAPI_JWT_RSA_PRIVATE_KEY`	`openssl genrsa -out authapi_jwt.pem 2048` then `base64 -w0 authapi_jwt.pem` — paste that base64 output as the value

Generate all four fresh per environment — never reuse a value between staging and production, or across services.

⚠️ Must configure before applying

What	Why
`AUTHAPI_DB_PASSWORD`	Empty in the example — set to the same `postgres` password already used for PostgreSQL, not a new value
`ZITADEL_DOMAIN` / `ZITADEL_PAT`	Placeholders in the example — must come from your actual Zitadel instance
All four generated secrets above	Example values were partially redacted — generate real ones, don't reuse anything from this doc or any example
`AUTHAPI_CORS_ORIGIN` / `ALLOWED_REDIRECT_URIS`	Must match your real frontend domains exactly
`COREAPI_BASE_URL` / `COREAPI_AUTHAPI_PSK`	Must match coreapi's real Service address and an identical PSK value once coreapi is deployed

Autoscaling (`hpa.yml`)

Scales between 2 and 4 replicas on CPU or memory utilization, whichever hits first — target 70% for both
Scale-up is fast: no stabilization delay, adds 1 pod per 30s if still over target
Scale-down is deliberately slower: a 5-minute (300s) stabilization window, then removes at most 1 pod per 60s — avoids flapping (scaling down right after a brief traffic dip, then immediately needing to scale back up)

Routing — private + public APISIX environments

private and public here are two separate APISIX installations connected over Tailscale, not a Kubernetes namespace split — which is why they're applied separately in Deploy below, against different nodes/contexts.

Private route (`apisix/private/httproute.yaml`)

This uses Gateway API (HTTPRoute), not the legacy ApisixRoute CRD — meaning the private APISIX environment's ingress-controller here is on ≥2.0.0 (see the version distinction in the Prerequisite doc):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: authapi-route
  namespace: production
spec:
  parentRefs:
    - name: apisix-gateway
      namespace: apisix
  hostnames:
    - id.castis.io
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: authapi
          port: 3002

Routes all traffic (PathPrefix: /) for id.castis.io on the private APISIX environment straight to the authapi Service on port 3002 — no path rewriting or extra plugins.

Public route (`apisix/public/01`–`04`)

Legacy ApisixRoute CRD across 4 files — Certificate, ApisixTls, ApisixRoute, ApisixUpstream. The public route doesn't point at a Kubernetes Service directly — it forwards over Tailscale to the private APISIX environment, which is what actually reaches authapi. See the note after the YAML below.

01-certificate.yaml — cert-manager Certificate for id.castis.io
02-apisixtls.yaml — ApisixTls, binding that certificate to APISIX

03-apisixroute.yaml:

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: authapi-public
  namespace: production
spec:
  http:
    - name: authapi-public
      match:
        hosts:
          - id.castis.io
        paths:
          - "/*"
      upstreams:
        - name: authapi-upstream
          weight: 100
      plugins:
        - name: redirect
          enable: true
          config:
            http_to_https: true
        - name: proxy-rewrite
          enable: true
          config:
            headers:
              set:
                X-Forwarded-Proto: "https"
                X-Forwarded-Host: "id.castis.io"

04-apisixupstream.yaml:

apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: authapi-upstream
  namespace: production
spec:
  externalNodes:
    - type: Domain
      name: "100.64.0.2"
      port: 80
      weight: 1
  scheme: http
  loadbalancer:
    type: roundrobin

How this actually reaches authapi: 100.64.0.2 is a Tailscale IP (the 100.64.0.0/10 CGNAT range Tailscale uses) — this ApisixUpstream isn't pointing at a Kubernetes Service at all, it's pointing at the private APISIX environment's Tailscale address. So the real path is: public APISIX (id.castis.io, HTTPS, redirects HTTP→HTTPS) → over Tailscale to the private APISIX environment on port 80 → the private HTTPRoute above (also matching id.castis.io) → the authapi Service. The proxy-rewrite plugin's X-Forwarded-* headers exist so authapi (behind two proxy hops by the time a request arrives) still sees the original scheme/host correctly.

01-certificate.yaml and 02-apisixtls.yaml content isn't shown here — only 03 and 04 were provided. Share those two as well if you want them documented too.

Deploy (Kustomize)

# From services/access/authapi/ — this targets the private node/context
kubectl apply -k overlays/prod

This applies base/ (Deployment, Service) merged with the prod overlay (Secret, HPA) in one shot — that's the point of the Kustomize base/overlay split.

Then apply the private route separately (same node/context as above, still private side):

kubectl apply -f overlays/prod/apisix/private/httproute.yaml

The public APISIX files are a separate APISIX installation reached over Tailscale — switch context and apply those separately, on the public node:

# Switch kubectl context to the public node first, then:
kubectl apply -f overlays/prod/apisix/public/01-certificate.yaml
kubectl apply -f overlays/prod/apisix/public/02-apisixtls.yaml
kubectl apply -f overlays/prod/apisix/public/03-apisixroute.yaml
kubectl apply -f overlays/prod/apisix/public/04-apisixupstream.yaml

Verify:

kubectl get pods -n production -l app=authapi
kubectl logs -n production -l app=authapi