Skip to content

Infrastructure Setup

This is the entry point for the production data/messaging/upload layer. Each service below is its own document — this page just explains what each one does and in what order to set them up. All four assume the 3-node HA k3s cluster (and Longhorn, and APISIX) from the cluster Prerequisite doc already exist.

Services

1. PostgreSQL — CloudNativePG

The primary relational database. Runs as 1 primary + 2 replicas via CloudNativePG, with a PgBouncer pooler in front so apps never connect to Postgres directly. Uses local-path-retain storage, not Longhorn — CNPG already replicates the data itself.

Start here first — most other services either store their own state independently or don't depend on Postgres being up, but it's the most foundational piece if your apps need a database on day one.

2. SeaweedFS — Internal S3 storage

Distributed, S3-compatible object storage (master/volume/filer/S3-gateway), internal-only — no external route, no -externalUrl. Apps reach it purely via in-cluster DNS over plain HTTP. The playtelly bucket lives here.

Set up before tusd — tusd streams uploads directly into this, so the bucket needs to exist first.

3. VerneMQ — Shared MQTT broker

One 3-pod HA MQTT broker shared by both production and staging apps (separated by username + topic prefix, not separate clusters). Also covers the public APISIX WebSocket route for external MQTT-over-WebSocket clients.

Independent of the other three — can be set up any time after the cluster Prerequisite doc is done.

4. tusd — Resumable file uploads

Handles resumable/chunked uploads (tus protocol) and streams them straight into SeaweedFS, notifying coreapi via webhook on completion.

Set up last — it depends on SeaweedFS (Section 2 above) already having the playtelly bucket created.

5. Zitadel — Identity/auth (IAM)

Identity provider — organizations, users, machine service accounts. Runs 2+ replicas for HA, backed by Postgres. Covers both an externally-routed deployment and an internal-only one.

Depends on PostgreSQL — needs the CNPG cluster (and optionally its PgBouncer pooler) already running before its first-instance bootstrap can succeed.

6. NATS JetStream — Messaging/streaming

3-node Raft cluster for pub/sub and persistent streams, across a mixed-architecture cluster (amd64 + arm64). Requires a custom multi-platform image since the base image lacks the nats CLI.

Independent of the other five — no shared dependencies, can be set up any time.

Suggested order

1. PostgreSQL     ← foundational, no dependencies on the others
2. SeaweedFS      ← needed before tusd
3. VerneMQ        ← independent, any time
4. tusd           ← depends on SeaweedFS's bucket existing
5. Zitadel        ← depends on PostgreSQL
6. NATS JetStream ← independent, any time