NATS JetStream

NATS JetStream is the messaging/streaming layer — a 3-node Raft cluster for pub/sub and persistent streams. Runs across a mixed-architecture cluster here (2× amd64 + 1× arm64), which is why a custom multi-platform image is required. Independent of the other services in this set — no dependency on Postgres, SeaweedFS, etc.

Overview

3 replicas — odd number required for Raft quorum (2 of 3 must agree)
Custom image: base nats:2.14.3-alpine doesn't include the nats CLI, so a custom image bakes it in — built multi-arch since the cluster mixes amd64 and arm64 nodes
Storage: local-path, not Longhorn — JetStream already replicates stream data itself via Raft
Server clustering does not automatically replicate stream data — each stream needs replicas set explicitly at creation, or it lives on just 1 node despite the cluster being healthy (see "Creating streams" below — this is the step most likely to be missed)

Architecture

Client
   ↓
nats-cluster-client Service (ClusterIP, load-balances across all 3 pods)
   ↓
nats-cluster-0 / nats-cluster-1 / nats-cluster-2  (StatefulSet, 1 per node)
   ↑ peer discovery via headless Service (publishNotReadyAddresses: true)
   ↓
local-path-retain-expand StorageClass → 5Gi PVC per node (jetstream-data)

The headless Service (nats-cluster) is only for peer-to-peer cluster routing between the 3 pods themselves — clients connect via the separate nats-cluster-client Service instead.

Key decisions

Storage: `local-path`, not Longhorn

JetStream already replicates stream data across all 3 nodes at the application layer via Raft. Stacking Longhorn's volume-level replication on top would mean 3× (NATS) × 3× (Longhorn) = 9 copies of the same data for no real extra safety — same reasoning as why Postgres/CNPG in this doc set also skips Longhorn. If a node's disk is lost, JetStream just re-syncs that node's data from the other 2 healthy members.

local-path-retain-expand StorageClass: reclaimPolicy: Retain (a deleted PVC orphans the volume instead of wiping it — cheap insurance against an accidental kubectl delete) and allowVolumeExpansion: true (accepted by the API, but rancher.io/local-path doesn't actually implement true online resize under the hood — practically, growing storage later means delete PVC → let it recreate → let JetStream re-sync from peers, not a live resize).

Custom multi-arch image

The base image only ships the nats-server binary, not the natscli tool needed for kubectl exec ... -- nats ... commands. The Dockerfile adds it at build time using buildx's auto-populated TARGETARCH build arg, so the same Dockerfile produces a correct binary for both amd64 and arm64:

docker buildx create --use   # one-time
docker buildx build --platform linux/amd64,linux/arm64 \
  -t registry.nexus.castis.io/nats-with-cli:2.14.3 \
  --push .

If your cluster is single-architecture, you can drop arm64 from --platform — but if there's any chance of a mixed cluster later, building both now costs nothing extra and avoids re-doing this later.

Prerequisites

3+ node cluster (mixed-architecture is fine — see the custom image above)
nexus-registry-secret pull secret already provisioned in the messaging namespace
docker buildx available wherever the custom image gets built

Files to apply

File	Purpose
`00-storageclass.yaml`	`local-path-retain-expand` StorageClass
`01-configmap.yaml`	`nats.conf` — cluster + JetStream config
`02-services.yaml`	Headless Service (peer discovery) + client-facing Service (load-balanced)
`03-statefulset.yaml`	The 3-pod StatefulSet

⚠️ Must configure before applying

What	Where	Why it matters
Image tag	`03-statefulset.yaml` → `image:`	Pin the exact built version (e.g. `2.14.3`) — not the floating `alpine` tag, which can silently upgrade on the next pod restart
Cluster routes list	`01-configmap.yaml` → `nats.conf` → `cluster.routes`	Must list exactly `nats-cluster-0`, `-1`, `-2` (matching `replicas: 3` and the `messaging` namespace) — a mismatch here means nodes can't find each other
Storage size	`03-statefulset.yaml` → `volumeClaimTemplates`	`5Gi` here is a starting point sized for staging-level traffic — revisit once real production retention/throughput numbers are known
Registry/image path	`03-statefulset.yaml` → `image:`	Must point at wherever the custom multi-arch image was actually pushed

Do not change these — required exactly as documented

Setting	Why it can't be adjusted casually
`publishNotReadyAddresses: true` on the headless Service	Without it: pods discover peers via DNS, but a headless Service only publishes DNS records for pods already `Ready` — and pods can't become `Ready` without first reaching their peers. This breaks that circular deadlock
`imagePullPolicy: Always`	`IfNotPresent` (the default) only checks by tag name, not digest — on a mixed-arch cluster this can silently keep reusing a stale single-arch image layer on a node after a multi-arch rebuild, causing `exec format error`
`replicas: 3` (odd number)	Raft quorum math — 2 of 3 must agree. Even counts don't add fault tolerance, same logic as etcd elsewhere in this doc set
`podManagementPolicy: Parallel`	All 3 pods need to start simultaneously and see each other to form the Raft cluster — the StatefulSet default (sequential) would stall waiting on a pod that itself is waiting on peers
`topologySpreadConstraints` with `whenUnsatisfiable: DoNotSchedule`	Actually spreads pods across nodes — without this, a single node failure could take out enough replicas to lose quorum even with `replicas: 3`
`pid_file: "/tmp/nats.pid"` in `nats.conf` (and the matching `preStop` path)	The default pidfile directory doesn't exist in this container; the `preStop` hook's `ldm=` path must point at the same file for graceful shutdown to work

Deploy

# 1. Build and push the custom multi-arch image (see above) — one time per version

# 2. Apply manifests
kubectl apply -f 00-storageclass.yaml
kubectl apply -f 01-configmap.yaml
kubectl apply -f 02-services.yaml
kubectl apply -f 03-statefulset.yaml

# 3. Watch it come up
kubectl get pods -n messaging -w

Verify:

kubectl exec -it nats-cluster-0 -n messaging -- wget -qO- http://localhost:8222/healthz
# {"status":"ok"}

kubectl exec -it nats-cluster-0 -n messaging -- wget -qO- http://localhost:8222/jsz
# "meta_cluster": { "cluster_size": 3, "leader": "nats-cluster-1", "pending": 0 }

All 3 pods should show 1/1 Running, no restarts, and a leader elected.

⚠️ Critical next step — creating streams with explicit replication

Server clustering alone does not replicate stream data. The cluster being healthy above only means the 3 servers can see each other — an individual stream still needs replicas set explicitly when it's created, or that stream's data lives on just 1 node.

This is a client-side action (any code connecting via a NATS SDK), done once per stream, typically in an app's bootstrap step — not a server config setting and not a per-message setting.

CLI example:

kubectl exec -it nats-cluster-0 -n messaging -- \
  nats stream add ORDERS --subjects="orders.*" --replicas=3 --storage=file \
  --server=localhost:4222