VerneMQ

One VerneMQ HA cluster (3 pods) in the vernemq namespace, shared by both production and staging apps via topic-prefix separation — not two separate clusters. Assumes the 3-node HA k3s cluster and Longhorn from the infrastructure doc already exist.

Overview

3 pods, one per node, spread across castis-desktop, nuc-a, nuc-b
mac-mini (arm64) is excluded — VerneMQ has no ARM64 image
Storage: longhorn-retain, 3Gi per pod, 2 Longhorn replicas each
Shared by prod and staging apps, separated by MQTT username + topic prefix

Architecture

k3s Cluster
├── namespace: vernemq
│   ├── vernemq-0  → castis-desktop (amd64)  → PVC: 3Gi (longhorn-retain)
│   ├── vernemq-1  → nuc-a         (amd64)  → PVC: 3Gi (longhorn-retain)
│   └── vernemq-2  → nuc-b         (amd64)  → PVC: 3Gi (longhorn-retain)
│
├── namespace: production  → apps connect via vernemq.vernemq.svc.cluster.local:1883
└── namespace: staging     → apps connect via vernemq.vernemq.svc.cluster.local:1883

mac-mini (arm64) → excluded via nodeSelector

Prerequisites

3-node HA k3s cluster with Longhorn (longhorn-retain StorageClass)
Access to your Nexus registry (or Docker Hub directly, if allowed)
The CoreDNS fix below, if on Ubuntu nodes using systemd-resolved

One-time cluster fix — CoreDNS override for the Kubernetes API

Why: Ubuntu nodes running systemd-resolved cause k3s to write 127.0.0.1 into CoreDNS's NodeHosts instead of the correct 10.43.0.1 (the Kubernetes API's real service IP). VerneMQ needs to reach the API for peer discovery — with the wrong IP, that fails with Connection refused.

kubectl apply -f - << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  kubernetes.override: |
    template IN A kubernetes.default.svc.cluster.local {
      match "^kubernetes\.default\.svc\.cluster\.local\.$"
      answer "{{ .Name }} 60 IN A 10.43.0.1"
      fallthrough
    }
EOF
kubectl rollout restart deployment coredns -n kube-system

⚠️ Use template, not hosts — the hosts plugin is already used once in the main Corefile, and a second hosts block crashes CoreDNS. This is a one-time, cluster-wide fix (survives restarts via etcd) — not something to repeat per-app.

Nexus Registry

# Mirror the image (one time per version)
docker pull vernemq/vernemq:2.1.2-alpine
docker tag vernemq/vernemq:2.1.2-alpine registry.nexus.castis.io/vernemq/vernemq:2.1.2-alpine
docker push registry.nexus.castis.io/vernemq/vernemq:2.1.2-alpine

The image pull secret (nexus-registry-secret) is already maintained as its own YAML file elsewhere in the repo — apply that file in the vernemq namespace rather than creating it imperatively (no need to duplicate it here, and this way it stays managed the same way as everywhere else it's used).

Files to apply

File	Purpose
`nexus-registry-secret.yaml`	Image pull secret — your existing file, applied into the `vernemq` namespace
`01-rbac.yaml`	ServiceAccount + Role (`pods` + `statefulsets`) + RoleBinding
`02-service.yaml`	ClusterIP (MQTT/WS) + headless service for Erlang clustering between pods
`03-statefulset.yaml`	3-pod HA StatefulSet

No separate namespace.yaml — the namespace is created imperatively in Step 1 below, same as the other infra components in this doc.

⚠️ Must configure before applying

What	Where	Why it matters
Image tag	`03-statefulset.yaml` → `image:`	Must be the `-alpine` variant, already mirrored to your registry — Debian-based tags crash with a cuttlefish config error on Kubernetes
`nexus-registry-secret.yaml`	applied into the `vernemq` namespace before Step 3 below	If missing, pods sit in `ImagePullBackOff`
`DOCKER_VERNEMQ_USER_prod_app` / `staging_app` / `admin`	`03-statefulset.yaml` → env	Placeholder passwords — change before applying, not after

⚠️ Do not change these — required exactly as documented

Setting	Why it can't be adjusted casually
`securityContext` (uid/gid 10000)	VerneMQ runs as uid 10000 in the container; without this the PVC mounts as root and VerneMQ can't write its own config
RBAC — both `pods` and `statefulsets`	Missing `statefulsets` causes a `403 Forbidden` during cluster formation — VerneMQ queries it for expected replica count
`nodeSelector: kubernetes.io/arch: amd64`	No ARM64 image exists — without this, any pod scheduled on an arm64 node (e.g. a Mac mini) fails with `exec format error`
`DOCKER_VERNEMQ_LISTENER__TCP__LOCALHOST` env var	Without it, the startup script auto-generates a duplicate listener key and cuttlefish crashes
Manual manifests (no Helm)	The official Helm chart doesn't set `MY_POD_NAMESPACE`, causing a double-dot Erlang node name (`vernemq-0..vernemq...`) regardless of the CoreDNS fix

Deploy

# 1. Namespace (imperative — no separate yaml)
kubectl create namespace vernemq

# 2. Image pull secret — your existing yaml, not created imperatively
kubectl apply -f nexus-registry-secret.yaml -n vernemq

# 3. RBAC, services, statefulset
kubectl apply -f vernemq/01-rbac.yaml
kubectl apply -f vernemq/02-service.yaml
kubectl apply -f vernemq/03-statefulset.yaml

# 4. Watch it come up
kubectl get pods -n vernemq -w

# 5. Verify cluster formed
kubectl exec -n vernemq vernemq-0 -- vmq-admin cluster show

Connecting from apps

	Value
MQTT host	`vernemq.vernemq.svc.cluster.local`
MQTT port	`1883`
WebSocket host	`vernemq.vernemq.svc.cluster.local`
WebSocket port	`8080`

Environment	Username	Topic prefix
Production	`prod_app`	`/prod/#`
Staging	`staging_app`	`/staging/#`

Exposing VerneMQ externally — APISIX route (public, WebSocket)

The two host names above are internal-only, for apps already inside the cluster. For clients connecting over MQTT-over-WebSocket from outside the cluster, a route needs to go through the public APISIX environment.

This production cluster uses the legacy ApisixRoute CRD (not Gateway API) — same distinction called out in the cluster/APISIX infrastructure doc. public and private here are two separate APISIX environments connected over Tailscale, not a Kubernetes namespace split.

Repo layout:

apisix/
├── private/
│   └── apisixroute.yaml       # internal-only routes, reached over Tailscale
└── public/
    ├── apisixroute.yml        # public-facing routes — the MQTT WebSocket route goes here
    ├── apisixtls.yml
    ├── apisixupstream.yml
    └── certificate.yml

apisix/public/apisixroute.yml:

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: mqtt
  namespace: vernemq
spec:
  http:
    - name: mqtt
      match:
        hosts:
          - mqtt.castis.io
        paths:
          - "/*"
      websocket: true
      backends:
        - serviceName: vernemq
          servicePort: 8080
      timeout:
        connect: 120s
        send: 120s
        read: 120s

websocket: true is required — without it, APISIX handles this as a normal HTTP route and the WebSocket upgrade handshake from MQTT clients fails. The long 120s timeouts matter too: MQTT-over-WebSocket connections are long-lived (clients stay connected, not request/response), so the default shorter timeouts would drop idle connections that are still legitimately open.

Frontend app config (external WebSocket clients)

For a browser/frontend app connecting through the public route above, the connection details are split into three env vars:

VITE_MQTT_HOST: mqtt.castis.io
VITE_MQTT_PORT: "443"
VITE_MQTT_PATH: "/mqtt"

Concatenated, that's the full WebSocket URL the MQTT client actually connects to:

wss://mqtt.castis.io:443/mqtt

wss:// (not ws://) because port 443 implies TLS — this connects to APISIX's HTTPS listener, which terminates TLS and proxies the WebSocket upgrade through to vernemq:8080 internally, per the ApisixRoute above.

Useful commands

kubectl exec -n vernemq vernemq-0 -- vmq-admin cluster show   # cluster status
kubectl exec -n vernemq vernemq-0 -- vmq-admin session show   # connected clients
kubectl exec -n vernemq vernemq-0 -- vmq-admin listener show  # listeners
kubectl logs -n vernemq vernemq-0 -f                          # logs
kubectl rollout restart statefulset/vernemq -n vernemq        # restart

Troubleshooting

Symptom	Cause	Fix
`Error generating config with cuttlefish`	Debian-based image instead of `-alpine`	Confirm the image tag ends in `-alpine`
Cuttlefish still crashes with the right image	Missing `securityContext`	Add `fsGroup`/`runAsUser: 10000`
Crash on startup, missing `/vernemq/log`	No writable log volume	Ensure the `logs` `emptyDir` and its mount at `/vernemq/log` are present
Duplicate listener cuttlefish crash	Startup script auto-generates a listener key that collides	Set `DOCKER_VERNEMQ_LISTENER__TCP__LOCALHOST` explicitly
`exec format error` on one node	That node is ARM64	Confirm `nodeSelector: kubernetes.io/arch: amd64`
Node name has a double dot (`vernemq-0..vernemq...`) / `vmq-admin cluster show` fails / API connection refused on 443	CoreDNS resolving the K8s API to `127.0.0.1`	Apply the CoreDNS override above
CoreDNS crashes: `plugin/hosts: this plugin can only be used once per Server Block`	Used `hosts` instead of `template` in `coredns-custom`	Switch to the `template` plugin
`403 Forbidden` on the statefulsets API	RBAC Role missing `statefulsets`	Add it to `01-rbac.yaml`
`Cannot join a cluster, as local node is non-empty`	Stale data on PVC from a prior standalone run	`kubectl exec ... rm -rf /vernemq/data/*` then force-delete the pod
PVC stuck `Terminating`	`longhorn-retain`'s finalizer waiting on replica cleanup	Patch out finalizers on the PVC/PV as a last resort, then delete the PV