Prerequisite

This document covers the infrastructure setup end-to-end: HA cluster concepts and bootstrap (multiple control-plane nodes + optional workers), Longhorn storage, an HA APISIX gateway, and Gateway API wiring in place. Routing (HTTPRoute) and TLS (ClusterIssuer / Certificate) are deliberately left out of this pass — infrastructure first, application routing next.

---

0. Background — How a K3s HA Cluster Works

A few concepts worth understanding before touching any commands — they explain why the steps later in this doc are structured the way they are (odd node counts, why some nodes need more disk speed than others, why taints/tolerations rarely matter in a small homelab-style cluster, etc.).

Node roles

Role	Meaning
control-plane	Manages the cluster — runs the API server, scheduler, controller-manager
etcd	Hosts a member of the distributed etcd database
master	Legacy alias for control-plane — same meaning

A node can hold more than one role at once. In k3s's default "stacked" mode, control-plane and etcd run on the same nodes (as opposed to "external etcd," where they're split onto separate machines — a large-production pattern, not needed here).

etcd and quorum

etcd is the cluster's database — it stores definitions and state (pod specs, services, secrets, configmaps, node status, RBAC, etc.), never actual application data, container images, or PV contents. It stays small even in large clusters (tens to a few hundred MB) for exactly that reason.

etcd uses Raft consensus: before any change is committed, a strict majority of members must agree. That majority is quorum:

quorum = (N / 2) + 1   (rounded down)

3 nodes → quorum 2 → tolerates 1 failure
4 nodes → quorum 3 → tolerates 1 failure   (same as 3 — wasted node)
5 nodes → quorum 3 → tolerates 2 failures
7 nodes → quorum 4 → tolerates 3 failures

This is why odd node counts are the rule for etcd members — even counts buy you nothing extra. Minimum viable HA is 3; beyond 7 isn't recommended, since more members means more network chatter per write.

The four control-plane components

Component	Role
kube-apiserver	The only thing that talks to etcd directly; every kubectl command passes through it
kube-scheduler	Decides which node a new pod runs on
kube-controller-manager	Watches desired vs. actual state and reconciles drift (e.g. "3 replicas wanted, 2 running → create 1 more")
cloud-controller-manager	Only relevant on managed cloud (AWS/GCP/Azure); not present in this bare-metal setup

Control-plane vs. worker

Control-plane = the brain  → decides what runs and where, doesn't run your apps
Worker node   = the muscle → actually runs your containers

By default, k3s taints control-plane nodes with node-role.kubernetes.io/control-plane:NoSchedule, so ordinary pods won't land there unless a matching toleration is set. Exception that matters for this guide: if a cluster has only control-plane nodes and no separate workers, k3s automatically removes that taint — so on a small 3-node all-control-plane cluster, pods (including Longhorn, APISIX, cert-manager) schedule freely everywhere with no extra YAML needed.

Persistent storage lives on the node, not in etcd

etcd only stores a PV's definition. The actual bytes (database files, uploads, logs) live on whichever node's disk the pod was scheduled to. This is exactly the problem Longhorn (Step 4) solves: with plain local storage, if the node holding the data dies, the data doesn't move with the pod. Longhorn replicates the actual volume contents across multiple nodes so a pod can be rescheduled anywhere and still see its data.

Practical takeaway for this guide

Because Longhorn's replica placement and the APISIX etcd HA upgrade (Step 6.2) both use hard anti-affinity (no two replicas/pods on the same node), and because k3s itself only gets HA benefits from 3+ control-plane+etcd nodes, the node-count requirement is really one requirement, not three: get to 3 physical/virtual nodes, and k3s HA, Longhorn replication, and the APISIX etcd upgrade all become meaningfully fault-tolerant at the same time.

---

1. Install K3s (HA bootstrap, control-plane & worker nodes)

Plan for 3 control-plane nodes, not 1. Everything in Section 0 above boils down to this: k3s only gets real HA when etcd has 3+ members, and etcd only runs on nodes with the control-plane role — plain workers don't count toward that number. If you're building this for real (not just a single-node dev box), bootstrap with 3 control-plane nodes from the start; it's much less disruptive than converting a single-node cluster later.

1.1 — First control-plane node (bootstrap)

Run this on the first node only — --cluster-init is what bootstraps the embedded etcd cluster:

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --tls-san <LOAD_BALANCER_IP_OR_DNS> \
  --tls-san <NODE1_IP> \
  --tls-san <NODE2_IP> \
  --tls-san <NODE3_IP>

- --cluster-init — starts embedded etcd instead of the default SQLite backend, required for any multi-server setup - --tls-san — adds each address you might use to reach the API server into the server cert's Subject Alternative Names, so kubectl doesn't reject the connection later over a cert mismatch. Add every IP/DNS name you'll ever hit the API on (a load balancer VIP, each node's own IP) — SANs are cheap to add now, painful to add after the cert is already issued.

Fetch the join token (needed for every additional node):

sudo cat /var/lib/rancher/k3s/server/node-token

Verify:

sudo k3s kubectl get node

1.2 — Registering another control-plane node (required for HA — skip on a single node)

Run this on each additional control-plane node, pointing back at the first node and using the token from 1.1:

curl -sfL https://get.k3s.io | sh -s - server \
  --server https://<FIRST_NODE_IP>:6443 \
  --token <NODE_TOKEN_FROM_1.1> \
  --tls-san <LOAD_BALANCER_IP_OR_DNS>

This node joins the existing etcd cluster as a new member and starts running its own apiserver/scheduler/controller-manager. Do this twice more to reach 3 total control-plane nodes (or 4 more times to reach 5, if that's the target from Section 0's quorum table).

This is the same command whether you're doing the initial 3-node bootstrap or adding a 4th/5th control-plane node to an already-running cluster months later — there's no separate "add node" command, just re-run this against the running cluster's current server IP and current token.

📌 The same "skip if single-node, required for HA" rule applies throughout this doc — Longhorn (Step 4) and the APISIX etcd upgrade (Step 6.2) both need this same 3-node minimum to actually be HA.

1.3 — Registering a worker node (optional)

Workers don't run etcd or the control-plane components — they only run kubelet/kube-proxy and your actual workloads. Use this if you want to add raw compute capacity without adding more etcd members (e.g. you already have 3 control-plane nodes for quorum and just want more room to schedule pods):

curl -sfL https://get.k3s.io | \
  K3S_URL=https://<SERVER_IP>:6443 \
  K3S_TOKEN=<TOKEN_FROM_1.1> \
  sh -

Note this is K3S_URL=... sh - (agent mode), not sh -s - server — that distinction is what makes it a worker instead of another control-plane node.

📌 Worker nodes count toward the same 3-node HA minimum (Longhorn, APISIX), just not toward k3s's own etcd quorum.

1.4 — Verify the full cluster

kubectl get nodes

Expect to see roles like:

NAME     STATUS   ROLES                       AGE   VERSION
node-1   Ready    control-plane,etcd,master    5m    v1.33.3+k3s1
node-2   Ready    control-plane,etcd,master    3m    v1.33.3+k3s1
node-3   Ready    control-plane,etcd,master    2m    v1.33.3+k3s1

A worker node (if added) shows no control-plane/etcd/master roles.

What you get automatically on any k3s node: - kubectl, crictl, ctr - k3s-killall.sh, k3s-uninstall.sh - Kubeconfig at /etc/rancher/k3s/k3s.yaml (on server nodes) - Local Path Provisioner (default storage class — Longhorn replaces this for anything that needs to survive a node failure, see Step 4)

Uninstalling a node

# On a control-plane/server node
/usr/local/bin/k3s-uninstall.sh

# On a worker/agent node
/usr/local/bin/k3s-agent-uninstall.sh

1.5 — Configure kubectl

By default kubectl only works via sudo k3s kubectl ... on the node itself. This step points a plain kubectl at the cluster (on the control-plane node, or wherever you're managing the cluster from):

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
source ~/.bashrc

kubectl get nodes

If you're managing the cluster from a machine other than a control-plane node, copy that same k3s.yaml there instead, and change server: https://127.0.0.1:6443 inside it to the control-plane node's actual IP (or your load balancer VIP, if you set one up via --tls-san in 1.1).

---

2. Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

---

3. Disable Traefik, keep svclb

K3s ships Traefik as the default ingress and binds it to host ports 80/443. Since APISIX will own those ports, disable Traefik but keep svclb — it's what binds the LoadBalancer service to the node's physical interface.

sudo nano /etc/rancher/k3s/config.yaml

disable:
  - traefik

sudo systemctl restart k3s

# Confirm
kubectl get pods -n kube-system | grep traefik   # should be empty
kubectl get pods -n kube-system | grep svclb      # should be running

Without svclb, a LoadBalancer-type service gets EXTERNAL-IP: <pending> and nothing listens on ports 80/443 on the host at all.

---

4. Install Longhorn

Longhorn provides the distributed block storage that etcd's HA setup (Step 6.2) will sit on. Install it before touching APISIX, since the longhorn-etcd storage class referenced later depends on it existing first.

Requires at least 3 nodes for real HA. Longhorn's numberOfReplicas: 3 (used below) only gives you genuine fault tolerance if there are 3 separate nodes for those replicas to land on. On a single-node cluster, Longhorn will still create 3 replicas, but all of them sit on the same disk — replication in name only, since losing that one node loses all copies. If you're still on 1 node, this step is fine to do now (it's a lot easier to set up correctly from the start), just know the HA guarantee doesn't kick in until node 2 and node 3 join the cluster (Steps 1.2/1.3 — control-plane or worker nodes both count here).

Prerequisites (per node)

Longhorn needs open-iscsi on every node — this is what actually attaches volumes to pods, and is required regardless of cluster size:

sudo apt update
sudo apt install -y open-iscsi
sudo systemctl enable --now iscsid

Install via Helm

helm repo add longhorn https://charts.longhorn.io
helm repo update

kubectl create namespace longhorn-system

helm install longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --set defaultSettings.defaultDataPath="/var/lib/longhorn"

Verify

kubectl get pods -n longhorn-system -w

Wait until all pods (longhorn-manager, longhorn-driver-deployer, longhorn-ui, csi-*, engine-image-*, instance-manager-*) are Running.

kubectl get storageclass

You should now see a default longhorn class alongside K3s's built-in local-path.

Create a dedicated longhorn-etcd storage class

Rather than using the generic longhorn class for etcd, create a dedicated one so etcd's replica count / retention can be tuned independently of other workloads that might use Longhorn later:

nano longhorn-etcd-sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-etcd
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fsType: "ext4"

kubectl apply -f longhorn-etcd-sc.yaml
kubectl get storageclass longhorn-etcd

reclaimPolicy: Retain here is intentional — if an etcd PVC is accidentally deleted, the underlying Longhorn volume is kept rather than wiped, giving you a chance to recover it manually.

---

5. Install Cert-Manager (with Gateway API support)

Cert-manager itself is infrastructure (the actual ClusterIssuer / Certificate objects that request real certs come later, once routing is in place). Installing it now, though, so the Gateway API flag is set from the start.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

kubectl wait --namespace cert-manager \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/instance=cert-manager \
  --timeout=120s

kubectl get pods -n cert-manager

Expected:

cert-manager-xxx           1/1 Running
cert-manager-cainjector    1/1 Running
cert-manager-webhook       1/1 Running

Enable the Gateway API HTTP01 solver (required, one-time)

By default cert-manager does not support the Gateway API HTTP01 solver. Without this, any future challenge using a Gateway-based solver would stay pending with gateway api is not enabled:

kubectl patch deployment -n cert-manager cert-manager \
  --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--enable-gateway-api"}]'

kubectl rollout status deployment -n cert-manager cert-manager

# Verify
kubectl get deployment -n cert-manager cert-manager -o yaml | grep enable-gateway-api

---

6. APISIX — Complete Setup

This section covers APISIX end-to-end as one topic: the initial install, the HA upgrade, and the Gateway API plumbing that routing will build on later.

⚠️ Read this first — APISIX version determines which routing model you use.

APISIX ingress-controller version	Routing model	CRDs used
< 2.0.0	Legacy CRDs	ApisixRoute, ApisixUpstream, ApisixTls
≥ 2.0.0	Kubernetes Gateway API	GatewayClass, GatewayProxy, Gateway, HTTPRoute

Check which one you're on before following 6.3 below:

helm list -n ingress-apisix
kubectl get pods -n ingress-apisix -l app.kubernetes.io/name=apisix-ingress-controller -o jsonpath='{.items[0].spec.containers[0].image}'

This section is written for ingress-controller ≥ 2.0.0 (Gateway API). If you're still on an older version, the GatewayClass/GatewayProxy/Gateway pieces in 6.3 don't apply — use ApisixRoute instead. The two models are not interchangeable, and mixing them on the same ingress-controller install will cause routes to silently fail to sync.

6.1 — Initial install

helm repo add apisix https://charts.apiseven.com
helm repo update

kubectl create namespace ingress-apisix

helm install apisix apisix/apisix -n ingress-apisix \
  --set gateway.type=LoadBalancer \
  --set gateway.tls.enabled=true \
  --set apisix.ssl.enabled=true \
  --set ingress-controller.enabled=true \
  --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
  --set ingress-controller.gatewayProxy.createDefault=false \
  --set ingress-controller.apisix.adminService.namespace=ingress-apisix

Verify the gateway service got a LoadBalancer IP and both ports:

kubectl get svc -n ingress-apisix apisix-gateway
# EXTERNAL-IP should be your node's LAN IP, e.g. 172.17.200.154   80:xxxxx/TCP,443:xxxxx/TCP

If it came up as NodePort instead, patch it:

kubectl patch svc -n ingress-apisix apisix-gateway \
  -p '{"spec": {"type": "LoadBalancer"}}'

If port 443 is missing from the service, add it:

kubectl patch svc -n ingress-apisix apisix-gateway \
  -p '{"spec": {"ports": [
    {"name": "apisix-gateway",     "port": 80,  "targetPort": 9080, "protocol": "TCP"},
    {"name": "apisix-gateway-tls", "port": 443, "targetPort": 9443, "protocol": "TCP"}
  ]}}'

At this point APISIX is running with a single-node etcd and no HA. That's fine to confirm things work end-to-end before layering on the HA config below.

---

6.2 — Upgrade to HA (3× replicas + Longhorn etcd) — required for HA, skip on a single node

Once the base install is verified and Longhorn (Step 4) is in place, upgrade to a highly-available configuration: APISIX gateway spread across 3 pods (soft anti-affinity), and etcd running as a real 3-node cluster backed by the longhorn-etcd storage class (hard anti-affinity so no two etcd pods land on the same node).

Requires at least 3 nodes. etcd.podAntiAffinityPreset=hard means the scheduler will refuse to place two etcd pods on the same node — with etcd.replicaCount=3 on a single-node (or 2-node) cluster, the 2nd and 3rd etcd pods will simply stay stuck Pending forever since there's nowhere else to schedule them. Same logic applies more loosely to apisix.podAntiAffinityPreset=soft for the gateway pods — soft won't block scheduling on fewer nodes, but you won't get real pod-level fault tolerance until each replica has its own node to land on. This is the same 3-node requirement flagged back in Steps 1.2/1.3 — confirm you're actually there before running this:

kubectl get nodes

helm upgrade apisix apisix/apisix \
  -n ingress-apisix \
  --set apisix.ssl.enabled=true \
  --set apisix.replicaCount=3 \
  --set apisix.podAntiAffinityPreset=soft \
  --set "apisix.admin.allow.ipList[0]=127.0.0.1/24" \
  --set "apisix.admin.allow.ipList[1]=10.42.0.0/16" \
  --set etcd.enabled=true \
  --set etcd.image.registry=quay.io \
  --set etcd.image.repository=coreos/etcd \
  --set etcd.image.tag=v3.5.9 \
  --set etcd.replicaCount=3 \
  --set etcd.persistence.enabled=true \
  --set etcd.persistence.size=1Gi \
  --set etcd.persistence.storageClass=longhorn-etcd \
  --set etcd.livenessProbe.enabled=false \
  --set etcd.readinessProbe.enabled=false \
  --set etcd.podAntiAffinityPreset=hard \
  --set gateway.type=LoadBalancer \
  --set ingress-controller.enabled=false \
  --set global.security.allowInsecureImages=true \
  --no-hooks \
  --reuse-values

What each flag is doing

Flag	Why
apisix.replicaCount=3	3 APISIX gateway pods instead of 1 — survives a pod restart with zero downtime
apisix.podAntiAffinityPreset=soft	Prefers spreading APISIX pods across nodes, but won't block scheduling if it can't (soft, not hard)
apisix.admin.allow.ipList	Restricts who can hit the admin API (9180) — only localhost and the pod CIDR (10.42.0.0/16)
etcd.replicaCount=3	Real 3-node etcd cluster (quorum-based), not a single point of failure
etcd.image.registry/repository/tag	Pins etcd to quay.io/coreos/etcd:v3.5.9 explicitly instead of the chart default, for a known-good version
etcd.persistence.storageClass=longhorn-etcd	etcd data survives pod restarts/rescheduling — uses the dedicated Longhorn storage class from Step 4 rather than the k3s default local-path (which is node-local and would break etcd if a pod moved nodes)
etcd.livenessProbe/readinessProbe.enabled=false	Disables the chart's built-in health probes — commonly needed because the default probe commands don't match this etcd image/version and cause false-positive restarts
etcd.podAntiAffinityPreset=hard	Hard requirement — etcd pods must not share a node. If they did, losing one node could take out enough etcd members to lose quorum
ingress-controller.enabled=false	The ingress-controller subchart is turned off here — either already deployed separately, or intentionally decoupled from this particular upgrade so it isn't re-templated/restarted as a side effect
global.security.allowInsecureImages=true	Required by the Bitnami-derived chart when you override image.registry/repository/tag away from its verified defaults (like the quay.io/coreos/etcd override above)
--no-hooks	Skips Helm chart hooks (e.g. pre-upgrade jobs) — usually done when a hook has already run or would conflict with an in-place HA migration
--reuse-values	Keeps every value from the previous release that isn't explicitly overridden here — critical, otherwise this upgrade would silently reset ssl.enabled, gateway.type, TLS settings, etc. back to chart defaults

⚠️ etcd.persistence.storageClass=longhorn-etcd must already exist (Step 4) before running this upgrade. If it doesn't, the etcd PVCs will stay Pending forever. Check with:

kubectl get storageclass longhorn-etcd

Watch the rollout:

kubectl get pods -n ingress-apisix -w

Confirm etcd is backed by Longhorn and spread across nodes:

kubectl get pvc -n ingress-apisix
kubectl get pods -n ingress-apisix -o wide | grep etcd

---

6.3 — Gateway API resources

Because this staging cluster runs an ingress-controller ≥ 2.0.0, routing will eventually be configured with Gateway API, not ApisixRoute. This pass sets up the one-time cluster plumbing only — no listeners for real hostnames or TLS yet, and no HTTPRoutes. Those come in the next phase once actual services need to be exposed.

GatewayClass (cluster-scoped)        ← one-time
    │  parametersRef →
    ▼
GatewayProxy (ingress-apisix namespace)   ← one-time
    connects the ingress controller to the APISIX admin API

Gateway (ingress-apisix namespace)   ← created now, listeners added later
    gatewayClassName → GatewayClass

Important: GatewayProxy and Gateway must live in the same namespace (ingress-apisix). If they're split across namespaces, the parametersRef lookup fails silently and kubectl get gateway shows PROGRAMMED: Unknown forever.

6.3.1 — GatewayProxy (one-time)

# Find the admin service ClusterIP first
kubectl get svc -n ingress-apisix apisix-admin

# 00-gatewayproxy.yaml
apiVersion: apisix.apache.org/v1alpha1
kind: GatewayProxy
metadata:
  name: apisix-proxy
  namespace: ingress-apisix
spec:
  provider:
    type: ControlPlane
    controlPlane:
      endpoints:
        - http://apisix-admin.ingress-apisix.svc.cluster.local:9180
      auth:
        type: AdminKey
        adminKey:
          value: <your-admin-key>

6.3.2 — GatewayClass (one-time)

# 01-gatewayclass.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: apisix
spec:
  controllerName: apisix.apache.org/apisix-ingress-controller
  parametersRef:
    group: apisix.apache.org
    kind: GatewayProxy
    name: apisix-proxy
    namespace: ingress-apisix

6.3.3 — Gateway (bare shell for now, listeners added per-domain later)

# 02-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: apisix-gateway
  namespace: ingress-apisix
spec:
  gatewayClassName: apisix
  infrastructure:
    parametersRef:
      group: apisix.apache.org
      kind: GatewayProxy
      name: apisix-proxy
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All

Apply the three:

kubectl apply -f 00-gatewayproxy.yaml
kubectl apply -f 01-gatewayclass.yaml
kubectl apply -f 02-gateway.yaml

HTTPS listeners with tls.certificateRefs will be added to this Gateway once ClusterIssuer/Certificate objects exist — that's covered in the routing/TLS follow-up, not this infrastructure pass.

---

7. Install ArgoCD

ArgoCD is the last piece of the recommended stack order — it watches a Git repo and continuously syncs its manifests into the cluster, so from here on changes to any of the infra above (or to application deployments) can be made by committing YAML rather than running kubectl apply by hand.

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify:

kubectl get pods -n argocd

Expected pods:

argocd-application-controller-0   Running
argocd-applicationset-controller  Running
argocd-dex-server                 Running
argocd-notifications-controller   Running
argocd-redis                      Running
argocd-repo-server                Running
argocd-server                     Running

Get the initial admin password

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

This secret only exists until the password is changed for the first time — change it after first login (argocd account update-password via the CLI, or through the UI under User Info).

Access the UI (no routing/TLS yet, so port-forward for now)

Since this pass doesn't include HTTPRoute/Certificate (Section 6.3 note), the simplest way to reach ArgoCD right now is a port-forward rather than a real hostname:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Then open https://localhost:8080 and log in as admin with the password above. A proper argocd.<domain> route can be added the same way any other service will be, once the routing/TLS follow-up doc is applied.

The basic GitOps pattern going forward

Git repo
  └── apps/
       ├── some-service/
       │    ├── deployment.yaml
       │    ├── service.yaml
       │    └── httproute.yaml
       └── another-service/

An Application resource in ArgoCD points at a path in that repo and keeps the cluster in sync with it:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: some-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<you>/<repo>.git
    targetRevision: main
    path: apps/some-service
  destination:
    server: https://kubernetes.default.svc
    namespace: staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

This is intentionally just the entry point here — a full ArgoCD app-of-apps / repo-structure convention is a separate topic once actual services start being deployed through it, rather than part of this infrastructure pass.

---

8. Verify the infrastructure

# Node & core add-ons
kubectl get nodes
kubectl get pods -n kube-system | grep -E "traefik|svclb"

# Longhorn
kubectl get pods -n longhorn-system
kubectl get storageclass

# Cert-manager
kubectl get pods -n cert-manager
kubectl get deployment -n cert-manager cert-manager -o yaml | grep enable-gateway-api

# APISIX (HA)
kubectl get pods -n ingress-apisix -o wide
kubectl get svc -n ingress-apisix apisix-gateway
kubectl get pvc -n ingress-apisix

# Gateway API plumbing
kubectl get gatewayproxy -n ingress-apisix
kubectl get gatewayclass apisix        # ACCEPTED = True
kubectl get gateway -n ingress-apisix  # PROGRAMMED = True

# ArgoCD
kubectl get pods -n argocd

If everything above is green, the infrastructure layer is done — routing and TLS for individual services is the next phase.

---

9. Troubleshooting

kubectl get gateway shows PROGRAMMED: Unknown - Confirm GatewayProxy and Gateway are both in ingress-apisix. - Confirm GatewayClass.spec.controllerName is exactly apisix.apache.org/apisix-ingress-controller.

no GatewayProxy configs provided in controller logs

kubectl get gatewayproxy -n ingress-apisix
kubectl logs -n ingress-apisix \
  $(kubectl get pod -n ingress-apisix -l app.kubernetes.io/name=apisix-ingress-controller -o jsonpath='{.items[0].metadata.name}') \
  -c manager --tail=20

Also check the ingress-controller configmap for init_sync_delay — some chart versions delay the first sync by 20 minutes:

kubectl edit configmap -n ingress-apisix apisix-ingress-config
# init_sync_delay: 20m → init_sync_delay: 0s
kubectl rollout restart deployment -n ingress-apisix apisix-ingress-controller

Longhorn pods stuck Pending / CrashLoopBackOff - Confirm open-iscsi is installed and iscsid is running on every node (Step 4 prerequisite) — this is the most common cause.

sudo systemctl status iscsid

etcd PVCs stuck Pending after the HA upgrade (Step 6.2)

kubectl get pvc -n ingress-apisix
kubectl get storageclass longhorn-etcd

If longhorn-etcd doesn't exist, revisit Step 4 — Longhorn and the storage class must exist before the HA upgrade.

etcd pods (not PVCs) stuck Pending after the HA upgrade (Step 6.2)

kubectl describe pod <etcd-pod-name> -n ingress-apisix | grep -A5 Events
# Look for: "0/N nodes are available: N node(s) didn't match pod anti-affinity rules"

This means there are fewer nodes than etcd replicas. etcd.podAntiAffinityPreset=hard requires one node per etcd pod — with replicaCount=3 you need 3 nodes. Either add more nodes to the cluster, or reduce etcd.replicaCount to match what you have (and accept reduced fault tolerance until more nodes join).

HTTPS connection refused on 443

kubectl get pods -n kube-system | grep svclb
kubectl patch svc -n ingress-apisix apisix-gateway -p '{"spec": {"type": "LoadBalancer"}}'
sudo systemctl restart k3s