High availability and scaling
Components and their failure behaviour
Section titled “Components and their failure behaviour”| Component | HA approach | If it fails |
|---|---|---|
| server | 3+ replicas behind a load balancer, spread across zones | Remaining replicas serve; live streams reconnect and resume |
| scheduler | 2+ replicas; coordinates through Postgres leases | One replica holds each responsibility; another takes over when a lease expires |
| worker | 2+ replicas consuming JetStream work queues | Queue redelivers to a surviving worker |
| Postgres | Managed Multi-AZ primary with automatic failover (plus optional read replicas) | Writes pause during failover; the app reconnects. This is your most important dependency |
| NATS JetStream | 3-node cluster, nats.replicas: 3, persistent volumes |
A single node loss is tolerated |
| Object storage | Managed (S3 or equivalent) | Live log fan-out continues via NATS; durable writes retry |
| Valkey | Replicated managed service | Sessions may drop; users sign in again. Rate limits reset |
| Runners | Many runners per pool, across failure domains | Leases expire, jobs become runner_lost, retried if retry.when includes it |
Reference configuration
Section titled “Reference configuration”The Helm chart’s ci/ha-values.yaml is the production-shaped starting point: three server replicas with hard anti-affinity and zone spread, autoscaling for all three components, PodDisruptionBudgets, managed Postgres, NATS, object store and Valkey, ingress with long timeouts, NetworkPolicies and a ServiceMonitor. See Helm.
Load balancing and streams
Section titled “Load balancing and streams”Use a load balancer that supports long-lived HTTP/2 streams and does not buffer responses, and that passes WebSocket upgrades (the runner control stream, /api/v1/runner/connect). Set idle timeouts above your longest quiet period (an hour is common). Slipway does not need sticky sessions: sessions are in Valkey, and streams resume with a sequence number. See Streaming.
Scaling guidance
Section titled “Scaling guidance”| Symptom | Scale |
|---|---|
| High API latency, high CPU on server | More server replicas (HPA on CPU) |
| Jobs queue even though runners are idle | Scheduler replicas, then check Postgres |
| Notifications or webhooks delayed | Worker replicas |
| Slow queries, connection saturation | Postgres size and connection pooling (database.max_conns, PgBouncer in transaction mode where compatible with RLS settings per transaction) |
| Log ingestion backlog | NATS resources, object-store throughput |
| Queue wait for builds | More runners, or a Kubernetes executor with an autoscaler |
The scale path is: stateless components scale out; Postgres scales up with read replicas and monthly partitioning of high-volume tables (log_chunks, run events); JetStream is clustered; and cells per region give data residency.
Multi-region and data residency
Section titled “Multi-region and data residency”Slipway Cloud runs regional cells (EU, US). Self-hosted, run one control plane per residency boundary. Runners can be anywhere; they only need outbound access to their control plane.
Maintenance
Section titled “Maintenance”PodDisruptionBudgets keep at least one replica of each component available during node drains. Drain runner nodes by setting runners to draining, or give runner pods a long terminationGracePeriodSeconds (the chart uses 3600) so they finish jobs.