Distribution Domain

Manages the CDN infrastructure layer — Castis Streamer nodes, cproxy cache nodes, SLB (Castis ELB) nodes, and the colorbars UDP broadcaster. Provides registration, health checking, proxy passthrough, UDP stream probing, colorbars stream lifecycle management, runtime cproxy origin switching, and SSH-push installation of streamer and cproxy binaries onto blank nodes.

Handles:

• Streamer node registration and health (ping, status)
• Streamer proxy passthrough — forward any request to a streamer's API
• UDP multicast probe — detect whether a stream is live on a given group:port
• Streamer installation — SSH-push provisioning onto a blank Linux node
• Cache (cproxy) node registration and health (ping, status)
• Cache origin switching — update URLs on existing key (SwitchOrigin)
• Cache origins replace — add/remove/reorder keys (PutOrigins)
• Cache proxy passthrough
• cproxy installation — SSH-push provisioning, same path as streamer via BinaryProfile
• SLB (Castis ELB) node registration and health (ping, status)
• SLB proxy passthrough
• Colorbars broadcaster — manage ffmpeg UDP streams via hosted Fiber controller
• Streamer node deletion (guarded — pending origin safety check)
• GSLB node management (planned)
• ELB installation via SSH push (planned — natural next BinaryProfile, see pkg/installer)

---

Domain Structure

domain/distribution/
├── config.go              package distribution  — Config struct + ConfigFromEnv()
│                                                   reads: COLORBARS_HOST, COLORBARS_PORT,
│                                                   INSTALLER_ASSETS_DIR
├── models.go              package distribution  — Streamer, Cache, SLB structs + status consts
│                                                   Streamer AND Cache both carry InstallStatus,
│                                                   InstallStage, InstallError, InstallStartedAt
│                                                   (shared InstallStatus type, no duplicate enum)
├── helper.go               package distribution  — ApplyNodeFilters()
├── validation.go          package distribution  — ParseID()
│
├── handlers/
│   ├── handler.go         package handlers      — Handler struct + NewHandler(db, cfg)
│   │                                              colorbarNode() helper
│   ├── handler_streamer.go package handlers     — GetStreamers, GetStreamerByID,
│   │                                              CreateStreamer, UpdateStreamer,
│   │                                              DeleteStreamer, PingStreamer,
│   │                                              ProxyStreamerRequest, ProbeUDP
│   ├── handler_install.go package handlers      — InstallStreamer, GetInstallStatus,
│   │                                              UninstallStreamer, InstallCache,
│   │                                              GetCacheInstallStatus, UninstallCache
│   ├── handler_cache.go   package handlers      — GetCaches, GetCacheByID,
│   │                                              CreateCache, UpdateCache, DeleteCache,
│   │                                              PingCache, SwitchOrigin, PutOrigins,
│   │                                              ProxyCacheRequest
│   ├── handler_slb.go     package handlers      — GetSLBs, GetSLBByID,
│   │                                              CreateSLB, UpdateSLB, DeleteSLB,
│   │                                              PingSLB, ProxySLBRequest
│   ├── handler_cache_origins.go package handlers — ListOriginKeys, CreateOriginKey,
│   │                                              UpdateOriginKey, DeleteOriginKey,
│   │                                              AddStreamerToKey, RemoveStreamerFromKey,
│   │                                              SyncOriginKeys, GetStreamerChannels,
│   │                                              GetStreamerCaches
│   └── handler_colorbars.go package handlers   — GetColorbarStatus, CreateColorbarStream,
│                                                  UpdateColorbarStream, StartColorbarStream,
│                                                  StopColorbarStream, DeleteColorbarStream,
│                                                  ProxyColorbarRequest
│
└── routes/
    ├── wire.go            package routes        — Setup() — only entry point main.go calls
    └── routes.go          package routes        — registerRoutes() — internal

pkg/
├── StreamerClient/        — Castis Streamer HTTP client (see below)
├── CProxyClient/          — cproxy HTTP client (models.go + client.go)
│     ProxyRequest, PutOriginURLs, PutOrigins, PurgeOrigin, PurgeContent, GetTraffic
├── ElbClient/             — Castis ELB HTTP client (used by handler_slb.go)
├── ColorbarClient/        — colorbars Fiber API client
└── installer/             — SSH-push binary provisioning, profile-driven (see pkg/installer/index.md)
      types.go      — InstallRequest + BinaryProfile + StreamerProfile/CproxyProfile
      debug.go, ssh.go, template.go, install.go, uninstall.go

---

Data Model

erDiagram
    Streamer {
        int id
        string name
        string host
        string publicHost "browser-accessible hostname, distinct from host"
        int httpPort
        int apiPort
        bool hasGpu
        string role "ingest | cproxy — see Streamer Role below"
        string status
        datetime lastPingedAt
        string installStatus
        string installStage
        string installError
        datetime installStartedAt
    }

    Cache {
        int id
        string name
        string host
        string publicHost
        int httpPort
        int apiPort
        string status
        datetime lastPingedAt
        string installStatus
        string installStage
        string installError
        datetime installStartedAt
    }

    SLB {
        int id
        string name
        string host
        string publicHost
        int proxyPort
        int apiPort
        string status
        datetime lastPingedAt
    }

    Channel {
        int id
        string name
    }

    ChannelOrigin {
        int channelId
        int streamerId
        string ingestUrl
        string streamId
        string playbackUrl
    }
    Streamer ||--o{ ChannelOrigin : "hosts"
    Channel  ||--o{ ChannelOrigin : "has"


    CacheOriginKey {
        int id
        int cacheId
        string originKey
        string urlPattern
        string urlRewriteMatch
        string urlRewriteReplace
        string ttl
        int priority
    }

    CacheOriginStreamer {
        int id
        int originKeyId
        int streamerId
        int priority
    }
    Cache    ||--o{ CacheOriginKey      : "has keys"
    CacheOriginKey ||--o{ CacheOriginStreamer : "has streamers"
    Streamer ||--o{ CacheOriginStreamer  : "assigned to"

Streamer Role

Streamer.Role is "ingest" (default) or "cproxy":

• ingest — a full origin-streamer, capable of UDP/multicast contribution ingest.
• cproxy — cache/shield only, no live contribution ingest capability.

This is enforced, not just descriptive: playback's CreateChannel handler rejects a UDP-mode ChannelOrigin whose target streamer has Role: "cproxy" ("streamer %q has role %q, udp ingest requires role %q"), per-origin, silently recorded in that origin's failures[] entry rather than as an upfront validation error. Registering a streamer node with the wrong role is a common cause of "channel created but this origin failed" when the streamer/host itself is reachable and healthy.

Colorbars streams are not persisted — they live in the colorbars container's memory and reset on restart.

Install credentials (sshHost, sshUser, sshPassword/sshPrivateKey) are never persisted — they exist only in the request body for the duration of a single install/uninstall call, for both Streamer and Cache installs alike.

---

Architecture

CoreAPI (Fiber)
  │
  ├── /distribution/streamers   → Castis Streamer nodes (DB-backed)
  │     └── /probe              → UDP multicast detection
  │     └── /install            → SSH-push provisioning (StreamerProfile)
  │     └── /proxy/*            → passthrough to Streamer API
  │
  ├── /distribution/caches      → cproxy nodes (DB-backed)
  │     └── /origins            → full origins array replace (PUT)
  │     └── /origins/:key/switch → hot-switch URLs on existing key (POST)
  │     └── /install            → SSH-push provisioning (CproxyProfile)
  │     └── /proxy/*            → passthrough to cproxy API
  │
  ├── /distribution/slbs        → Castis ELB nodes (DB-backed)
  │     └── /proxy/*            → passthrough to ELB API
  │
  └── /distribution/colorbars   → colorbars container (stateless proxy)
        └── /streams            → ffmpeg UDP stream lifecycle
        └── /proxy/*            → passthrough to colorbars Fiber API

playtelly_internal (Docker bridge)
  ├── CoreAPI          → streamer_bkk:18081, streamer_ntb:18081
  ├── CoreAPI          → cproxy_bkk_1:8081
  ├── CoreAPI          → elb_bkk:8100
  ├── CoreAPI          → colorbars:9999
  └── CoreAPI          → (SSH) any registered node's :22 — installer only,
                          same path regardless of which BinaryProfile is used

playtelly_cdn_net (Docker bridge — 172.28.0.0/16)
  ├── colorbars        → sends UDP multicast (e.g. udp://239.0.0.8:4444)
  ├── streamer_bkk     → joins multicast group, ingests stream
  └── streamer_ntb     → joins multicast group, ingests stream

---

StreamerClient Package

pkg/StreamerClient — the HTTP client distribution handlers use to talk to a Castis Streamer node's API (http://{host}:{apiPort}/api/...). Purely transport — no SSH, no installation logic. Distinct from pkg/installer, which provisions the binary onto a node before this client ever has anything to talk to.

Files: client.go, models.go

Types (models.go):

type StreamerNode struct {
    Host     string
    APIPort  int
    HTTPPort int
}

type StreamerResponse struct {
    StatusCode int
    Body       json.RawMessage
}

Functions (client.go):

Function	Description
BuildStreamID(name string) string	Slugifies a channel name into a Castis stream ID — lowercases, replaces spaces with underscores, appends .stream. e.g. "BBC World" → "bbc_world.stream"
CreateStream(streamer, streamID, config) (*StreamerResponse, error)	POST /api/streams/{streamID} with a raw JSON config body
DeleteStream(streamer, streamID) (*StreamerResponse, error)	DELETE /api/streams/{streamID}
GetStreams(streamer) (*StreamerResponse, error)	GET /api/streams — list all streams on the node
GetTraffic(streamer) (*StreamerResponse, error)	GET /api/traffic
PlaybackURL(streamer, streamID) string	Builds the public playback URL, e.g. http://host:httpPort/{streamID}/manifest.mpd
BuildPayload(ingestURL, ingestConfig, transcodeConfig) (json.RawMessage, error)	Merges an ingest URL and transcoder config into a single stream-creation payload — unmarshals ingestConfig, overwrites url, attaches transcoders from transcodeConfig, re-marshals
ProxyRequest(streamer, method, path string, body []byte) (*StreamerResponse, error)	Forwards any HTTP request verbatim to the node. Used as the implementation behind ALL /streamers/:id/proxy/*, and internally by ProbeUDP for status polling

Internal helpers: doGet(url), readResponse(resp) — the latter logs the raw response body, and wraps it as json.RawMessage if valid JSON, or as a JSON-encoded string if the streamer returned something non-JSON (e.g. an HTML error page) so the response shape stays consistent regardless of what the upstream node actually returns.

Client config: package-level http.Client with a 10s timeout — no per-call timeout override currently exposed.

Logging: every CreateStream and ProxyRequest call logs the outgoing method/URL and the response status/body via the standard log package — useful for tracing what CoreAPI actually sent to a streamer node, but verbose; no log-level gating currently exists in this package.

---

Installation — Streamer and cproxy

SSH-push provisioning of a Castis binary onto a blank Linux node — no agent, credentials supplied fresh per request, never persisted. A single profile-driven Install()/Uninstall() in pkg/installer handles both Streamer and cproxy, parameterized by a BinaryProfile (StreamerProfile / CproxyProfile). See pkg/installer for the full mechanics (stage breakdown per profile, the BinaryProfile shape, credential handling, asset layout, known limitations).

Routes — streamer:

POST   /distribution/streamers/:id/install           start install (async, 202)
GET    /distribution/streamers/:id/install/status     poll current stage/status
DELETE /distribution/streamers/:id/install             uninstall (?purge=true|false)

Routes — cproxy:

POST   /distribution/caches/:id/install               start install (async, 202)
GET    /distribution/caches/:id/install/status         poll current stage/status

Cache uninstall is implemented but not routed

UninstallCache in handler_install.go is fully implemented — identical shape to UninstallStreamer, nothing broken about it — but DELETE /distribution/caches/:id/install is commented out in routes.go with no explanatory comment. Previous versions of this doc claimed this route "mirrors the streamer routes exactly" and was live; it currently is not. Needs a decision: re-enable the route, or confirm it's intentionally disabled and note why.

curl -X POST http://localhost:3000/api/v1/distribution/streamers/7/install \
  -H "Content-Type: application/json" \
  -d '{"sshHost":"node_x","sshPort":22,"sshUser":"deploy","sshPassword":"deploy123","httpPort":18080,"apiPort":18081}'

curl -X POST http://localhost:3000/api/v1/distribution/caches/2/install \
  -H "Content-Type: application/json" \
  -d '{"sshHost":"node_x","sshPort":22,"sshUser":"deploy","sshPassword":"deploy123","httpPort":8080,"apiPort":8081}'

installStatus lifecycle (identical for both row types): not_installed → pending → installing → installed | failed. Reinstall reuses the same endpoint — the flow is idempotent (dependency install no-ops on existing packages; the starting stage stops any already-running process before relaunching).

Stage sequence differs slightly per profile — StreamerProfile includes uploading_media and configuring_libs (it has bundled sample media and three lib subdirectories needing ldconfig); CproxyProfile skips both, since its tarball has no lib/ directory and no sample asset is involved. Both are profile-driven gates in the shared Install(), not separate code paths.

---

cproxy Origin Management

cproxy origins define how incoming paths are routed to streamer nodes. Two endpoints handle different levels of control:

Cache Origin Key Management

Origin keys are named routing slots in cproxy — e.g. food, ads, live-ch1. Each key has its own urlPattern, optional URL rewriter, TTL, and an ordered list of streamers in its origin-urls.

DB is the source of truth. cproxy runtime config is derived from DB and pushed via sync.

Flow: 1. Create key → POST /caches/:id/keys 2. Assign streamers → POST /caches/:id/keys/:keyId/streamers 3. Sync to cproxy → POST /caches/:id/keys/sync

Topology queries: - GET /streamers/:id/channels — channels hosted on this streamer - GET /streamers/:id/caches — cproxy keys this streamer feeds into

SwitchOrigin — update URLs on an existing key

POST /caches/:id/origins/:originKey/switch
Body: { "urls": ["http://streamer_bkk:18080"] }

GET current config → find key → replace originUrls → PUT full array back. Preserves all other fields. Fails if the key doesn't exist. Purges stale cache automatically after switch.

PutOrigins — full replace (add/remove/reorder keys)

PUT /caches/:id/origins
Body: [ ...full origins array... ]

Passes the array straight to PUT /api/config/origins on cproxy. Caller owns the full desired state — GET current origins first if you need to preserve existing keys.

# get current origins to use as base
curl -s http://localhost:3000/api/v1/distribution/caches/1/proxy/api/config | jq .origins

# push modified array (e.g. add 'jap' key)
curl -X PUT http://localhost:3000/api/v1/distribution/caches/1/origins \
  -H "Content-Type: application/json" \
  -d '[...]'

cproxy bind mount workaround

PUT /api/config/origins backs up cproxy.yml before writing. Docker bind mounts block the rename (device or resource busy) causing the entire operation to fail.

Fix: mount config to a temp path, copy at startup:

volumes:
  - ./cdn/cproxy-bkk-1/cproxy.yml:/tmp/cproxy-init.yml
command: bash -c "cp /tmp/cproxy-init.yml /castis/bin/cproxy/cproxy.yml && ./cproxy"

Version requirement: PUT /api/config/origins requires cproxy v1.1.5.rc1+.

---

SLB — Server Load Balancer

CoreAPI uses the term SLB (Server Load Balancer) as a vendor-agnostic abstraction. The underlying technology is the Castis ELB (Edge Load Balancer) — pkg/ElbClient wraps its API.

The ELB sits between clients and cproxy nodes, distributing traffic using round-robin or consistent hashing. It monitors backend node health via WebSocket traffic streams.

client → elb_bkk:8090 → cproxy_bkk_1:8080 → streamer_bkk:18080

Ports: - :8090 — proxy port (client-facing) - :8100 — API port (management, used by pkg/ElbClient)

Health check: GET /api/version via pkg/ElbClient.GetVersion() — ELB has no /api/ping.

Seeded node: elb-bkk → elb_bkk:8100 registered on startup via pkg/seed/slb.go.

Note: ELB installation via SSH push is not yet built, but would follow the exact same pattern — a third BinaryProfile (ElbProfile) plugged into the existing Install()/Uninstall(), no new orchestration code needed.

---

Colorbars — UDP Multicast Broadcaster

The colorbars container is a Go Fiber application that manages ffmpeg subprocesses. Each subprocess broadcasts SMPTE colorbars + 1kHz sine tone as MPEG-TS over UDP multicast.

Why multicast: A single ffmpeg send reaches all streamer nodes simultaneously — no per-streamer connection needed. Each streamer joins the multicast group address and receives the same stream.

Port constraint: Each stream must use a unique port. Two streams on the same port — even different multicast IPs — will interfere because the Castis streamer binds on port only when joining a group.

Stream lifecycle:

Create (POST /colorbars/streams)
  → idle in memory, no ffmpeg process

Start (POST /colorbars/streams/:id/start)
  → spawns ffmpeg subprocess
  → ffmpeg sends to udp://239.x.x.x:PORT?pkt_size=1316&localaddr=0.0.0.0&ttl=5

Stop (POST /colorbars/streams/:id/stop)
  → kills ffmpeg process, stream stays in memory

Delete (DELETE /colorbars/streams/:id)
  → stops ffmpeg if running, removes from memory

Patch (PATCH /colorbars/streams/:id)
  → updates config
  → if running: kills and restarts ffmpeg with new config (hot-restart)

Direct access (dev): The colorbars UI is served at http://localhost:9999 — lets you create, start, stop, and delete streams without going through CoreAPI.

---

UDP Probe

The probe endpoint lets CoreAPI verify whether a UDP multicast stream is actually flowing on a given group:port, by asking a streamer node to attempt ingestion.

sequenceDiagram
    participant Client
    participant CoreAPI
    participant Streamer

    Client->>CoreAPI: POST /distribution/streamers/:id/probe
    Note over Client,CoreAPI: body: { "udp": "239.0.0.8", "port": 4444 }

    CoreAPI->>Streamer: POST /api/streams/probe_239_0_0_8_4444.stream
    Note over CoreAPI,Streamer: {"url": "udp://239.0.0.8:4444"}

    loop every 1s for up to 6s
        CoreAPI->>Streamer: GET /api/streams/probe_239_0_0_8_4444.stream
        Streamer-->>CoreAPI: { "status": "Running" | "Waiting" }
    end

    CoreAPI->>Streamer: DELETE /api/streams/probe_239_0_0_8_4444.stream
    CoreAPI-->>Client: { "detected": true | false }

Running = packets are flowing. Waiting = socket open but no data arrived within 6s.

---

Environment Variables

Variable	Default	Description
COLORBARS_HOST	colorbars	Docker service name or IP of the colorbars container
COLORBARS_PORT	9999	Fiber API port on the colorbars container
INSTALLER_ASSETS_DIR	/castis/installer-assets	Directory containing streamer.tar.gz, streamer.yml.tmpl, kitchen.mp4, cproxy.tar.gz, cproxy.yml.tmpl — bind-mounted, sibling to the CoreAPI repo on the host

---

Seeding

All distribution nodes are seeded on startup. Existing records (matched by name) are skipped — safe to run on every boot.

pkg/seed/streamer.go  — Streamer nodes (local compose + staging)
pkg/seed/cache.go     — Cache nodes (cproxy)
pkg/seed/slb.go       — SLB nodes (Castis ELB)

Local compose nodes:

// streamers
{ Name: "streamer-bkk", Host: "streamer_bkk", HTTPPort: 18080, APIPort: 18081 }
{ Name: "streamer-ntb", Host: "streamer_ntb", HTTPPort: 18080, APIPort: 18081 }

// caches
{ Name: "cproxy-bkk-1", Host: "cproxy_bkk_1", HTTPPort: 8080, APIPort: 8081 }

// slbs
{ Name: "elb-bkk", Host: "elb_bkk", ProxyPort: 8090, APIPort: 8100 }

Installation-test nodes (node_x, container name streamer_x) are not seeded — deliberately blank Rocky9 containers with nothing registered until install is run against them. A single fixture node now commonly hosts both a streamer and a cproxy install simultaneously, on different port pairs, registered manually for testing.

---

Changelog

See changelog.md

---

API Routes