SeaweedFS

This covers SeaweedFS as an internal-only service — no external route, no -externalUrl, no public DNS. Apps reach it strictly via the in-cluster Kubernetes DNS name over plain HTTP. If external access is ever needed later, that's a separate follow-up (same pattern as the routing/TLS follow-up for APISIX) — not part of this pass.

---

Overview

A production-grade SeaweedFS cluster on Kubernetes: 3-node HA, data replication, and an S3-compatible API — reachable only from inside the cluster.

---

Architecture

nuc-a      → master-0 + volume-0
nuc-b      → master-1 + volume-1 + s3-gateway
mac-mini   → master-2 + volume-2 + filer-0

Components

Component	Replicas	Role
Master	3	Leader election, cluster metadata, volume assignment
Volume	3	Actual file storage, one per node
Filer	1	Directory structure, file metadata, S3 path mapping
S3 Gateway	1	S3-compatible API entry point — internal only

Storage

PVC	Size	Replicas	Total
master-data	1Gi	× 3	3Gi
volume-data	50Gi	× 3	150Gi
filer-data	5Gi	× 1	5Gi

Usable storage = 75Gi (replication=001 stores every file on 2 nodes).

---

Prerequisites

• Kubernetes cluster with 3 nodes (this guide uses nuc-a, nuc-b, mac-mini as example names)
• kubectl configured and connected to the cluster
• local-path provisioner available on all nodes
• No APISIX/ingress config needed — this deployment stays internal

Verify cluster is ready

kubectl get nodes
kubectl top nodes

---

Repository Path

playtelly-iac/services/infrastructure/seaweedfs/prod/
├── 00-namespace.yaml
├── 00b-storageclass.yaml
├── 01-configmap-s3.yaml
├── 02-master-service.yaml
├── 03-master-statefulset.yaml
├── 04-volume-service.yaml
├── 05-volume-statefulset.yaml
├── 06-filer-service.yaml
├── 07-filer-statefulset.yaml
├── 08-s3-service.yaml
├── 09-s3-deployment.yaml
├── 10-init-job.yaml
└── kustomization.yaml

File	Purpose
00-namespace.yaml	seaweedfs namespace
00b-storageclass.yaml	local-path-retain StorageClass with Retain policy
01-configmap-s3.yaml	S3 identity config — ⚠️ update credentials before applying
02-master-service.yaml	Headless service for master pods with publishNotReadyAddresses
03-master-statefulset.yaml	3 master pods with Parallel management and quorum
04-volume-service.yaml	Headless service for volume pods with publishNotReadyAddresses
05-volume-statefulset.yaml	3 volume pods, 50Gi PVC each
06-filer-service.yaml	Headless service for filer pod with publishNotReadyAddresses
07-filer-statefulset.yaml	1 filer pod pinned to mac-mini
08-s3-service.yaml	ClusterIP service for S3 gateway on port 8333 — internal only, no LoadBalancer/NodePort
09-s3-deployment.yaml	1 S3 gateway pod — plain internal endpoint, no -externalUrl flag
10-init-job.yaml	One-shot job to create the playtelly bucket
kustomization.yaml	Deploys everything in one command

---

Installation Steps

Step 1 — Update credentials

Edit playtelly-iac/services/infrastructure/seaweedfs/prod/01-configmap-s3.yaml and replace with real credentials:

{
  "identities": [
    {
      "name": "admin",
      "credentials": [
        {
          "accessKey": "your-real-access-key",
          "secretKey": "your-real-secret-key"
        }
      ]
    }
  ]
}

Step 2 — Label the target nodes

kubectl label node nuc-a seaweedfs=true
kubectl label node nuc-b seaweedfs=true
kubectl label node mac-mini seaweedfs=true

Verify:

kubectl get nodes --show-labels | grep seaweedfs

Step 3 — Deploy

cd playtelly-iac/services/infrastructure/seaweedfs/prod
kubectl apply -k .

Step 4 — Watch pods come up

kubectl get pods -n seaweedfs -w

Expected output when ready:

NAME                           READY   STATUS    NODE
seaweedfs-master-0             1/1     Running   mac-mini
seaweedfs-master-1             1/1     Running   nuc-a
seaweedfs-master-2             1/1     Running   nuc-b
seaweedfs-volume-0             1/1     Running   mac-mini
seaweedfs-volume-1             1/1     Running   nuc-a
seaweedfs-volume-2             1/1     Running   nuc-b
seaweedfs-filer-0              1/1     Running   mac-mini
seaweedfs-s3-xxx               1/1     Running   nuc-b

Masters use podManagementPolicy: Parallel — all 3 start simultaneously to form quorum.

Step 5 — Create the playtelly bucket

Run this from inside the cluster (no external route exists to hit from outside):

kubectl run curl-test --image=curlimages/curl --rm -it \
  --restart=Never -n seaweedfs -- \
  curl -s -X PUT http://seaweedfs-s3.seaweedfs.svc.cluster.local:8333/playtelly \
  --user "admin:password" \
  --aws-sigv4 "aws:amz:us-east-1:s3"

Expected: HTTP 200 OK

Note this uses http://, not https:// — SSL is intentionally off for this internal-only deployment. There's no APISIX route or TLS termination in front of it.

---

Verification

Check master quorum

kubectl exec -n seaweedfs seaweedfs-master-0 -- \
  wget -qO- http://localhost:9333/cluster/status

Expected:

{"IsLeader":true,"Leader":"seaweedfs-master-0...","Peers":[...]}

Check volume servers registered

kubectl exec -n seaweedfs seaweedfs-master-0 -- \
  wget -qO- http://localhost:9333/dir/status

Expected: 3 DataNodes with replication 001

Upload and download test (from inside the cluster)

kubectl run curl-test --image=curlimages/curl --rm -it \
  --restart=Never -n seaweedfs -- sh

Then, inside that shell:

# Upload
curl -s -X PUT http://seaweedfs-s3.seaweedfs.svc.cluster.local:8333/playtelly/test.txt \
  --user "admin:password" \
  --aws-sigv4 "aws:amz:us-east-1:s3" \
  -d "hello production"

# Download
curl -s http://seaweedfs-s3.seaweedfs.svc.cluster.local:8333/playtelly/test.txt \
  --user "admin:password" \
  --aws-sigv4 "aws:amz:us-east-1:s3"

Expected: hello production

Browse files via Filer UI

kubectl port-forward svc/seaweedfs-filer -n seaweedfs 8888:8888

Then open http://localhost:8888

---

App Configuration

Since this deployment is internal only, apps running in the same cluster connect via the Kubernetes service DNS name, over plain HTTP:

MINIO_ENDPOINT=seaweedfs-s3.seaweedfs.svc.cluster.local:8333
MINIO_ACCESS_KEY=admin
MINIO_SECRET_KEY=password
MINIO_BUCKET=playtelly
MINIO_USE_SSL=false

⚠️ Apps must run inside the cluster to reach this endpoint — there is no external route. An app running outside the cluster (e.g. on your local machine, or a separate VM not on this k3s network) cannot reach seaweedfs-s3.seaweedfs.svc.cluster.local. If that's ever needed, it's a separate exercise: adding an APISIX route + -externalUrl on the S3 gateway, the same way the earlier draft of this doc described.

---

Resource Usage

Node	Pods	Memory Request	Memory Limit
nuc-a	master-1 + volume-1	320Mi	640Mi
nuc-b	master-2 + volume-2 + s3	448Mi	896Mi
mac-mini	master-0 + volume-0 + filer-0	576Mi	1.1Gi
Total	8 pods	~1.3Gi	~2.6Gi

---

Known Issues & Fixes

Masters stuck — no quorum

Cause: StatefulSet creates pods sequentially by default. Masters need all 3 running simultaneously to elect a leader. Fix: podManagementPolicy: Parallel in the master StatefulSet.

DNS NXDOMAIN for headless service

Cause: Kubernetes only adds pods to DNS when Ready. Masters aren't Ready until quorum forms — deadlock. Fix: publishNotReadyAddresses: true on all headless services.

S3 gateway crash — unknown flag

Cause: -filer.maxMB flag was removed in SeaweedFS 4.x. Fix: Remove -filer.maxMB from the S3 gateway args.

---

Scaling Up

# Add more filer replicas
kubectl scale statefulset seaweedfs-filer -n seaweedfs --replicas=2

# Add more S3 gateway replicas
kubectl scale deployment seaweedfs-s3 -n seaweedfs --replicas=2

---

Troubleshooting

# Check all pods
kubectl get pods -n seaweedfs -o wide

# Check master logs
kubectl logs -n seaweedfs seaweedfs-master-0

# Check volume logs
kubectl logs -n seaweedfs seaweedfs-volume-0

# Check filer logs
kubectl logs -n seaweedfs seaweedfs-filer-0

# Check S3 gateway logs
kubectl logs -n seaweedfs -l app.kubernetes.io/component=s3

# Check running config on S3 gateway
kubectl exec -n seaweedfs -l app.kubernetes.io/component=s3 -- ps aux | grep weed

---

Restart Procedures

S3 Gateway and Filer — safe to rollout restart

Stateless components, can restart all at once:

kubectl rollout restart deployment seaweedfs-s3 -n seaweedfs
kubectl rollout restart statefulset seaweedfs-filer -n seaweedfs

Master — delete one at a time

Masters need quorum (2 out of 3 must be alive). Restarting all at once loses quorum and stops writes:

kubectl delete pod seaweedfs-master-0 -n seaweedfs
kubectl get pods -n seaweedfs -w   # wait for 1/1
kubectl delete pod seaweedfs-master-1 -n seaweedfs
kubectl get pods -n seaweedfs -w   # wait for 1/1
kubectl delete pod seaweedfs-master-2 -n seaweedfs
kubectl get pods -n seaweedfs -w   # wait for 1/1

Volume — delete one at a time

Volumes use replication=001 (2 copies). Restarting all at once makes files temporarily unavailable:

kubectl delete pod seaweedfs-volume-0 -n seaweedfs
kubectl get pods -n seaweedfs -w   # wait for 1/1
kubectl delete pod seaweedfs-volume-1 -n seaweedfs
kubectl get pods -n seaweedfs -w   # wait for 1/1
kubectl delete pod seaweedfs-volume-2 -n seaweedfs
kubectl get pods -n seaweedfs -w   # wait for 1/1

Safe restart order

1. S3 Gateway   ← rollout restart, safest
2. Filer        ← rollout restart, safe
3. Volume       ← delete one at a time
4. Master       ← delete one at a time, last

⚠️ Never use rollout restart on master or volume StatefulSets — it restarts all pods simultaneously and will cause quorum loss or data unavailability.

---