Canary verification

Canary verification is how Slipway decides, with evidence, whether to continue a rollout. During a canary (and optionally during blue/green) the engine queries a metric provider at regular intervals, evaluates each metric against bounds or a baseline, scores the interval and either advances or rolls back.

Available on Team and above.

Providers

A provider is an observability integration: Prometheus, Datadog, New Relic, Amazon CloudWatch, Google Cloud Monitoring and Azure Monitor. Prometheus ships as the reference provider. You name the provider in the verification config and credentials come from the integration: provider is the integration’s provider (prometheus, grafana, datadog …) or its name when the organisation has one such metrics integration, or the integration id (int_…) to pick one of several.

A complete example

strategy: canary
strategy_config:
  steps:
    - { weight: 5,   pause: 5m }
    - { weight: 25,  pause: 10m }
    - { weight: 100 }
  verify:
    provider: prometheus
    interval: 1m              # time between analyses (default 1m)
    intervals: 5              # number of analyses per step (default 5)
    initial_delay: 1m         # let the canary warm up
    metrics:
      - name: error-rate
        kind: error_rate
        query: sum(rate(http_requests_total{service="api",version="canary",status=~"5.."}[2m]))
        total_query: sum(rate(http_requests_total{service="api",version="canary"}[2m]))
        baseline_query: sum(rate(http_requests_total{service="api",version="stable",status=~"5.."}[2m]))
        baseline_total_query: sum(rate(http_requests_total{service="api",version="stable"}[2m]))
        max: 0.005              # never above 0.5%
        max_ratio: 1.5          # and no more than 1.5x the stable version
        critical: true
      - name: p99-latency
        kind: relative
        query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="api",version="canary"}[2m])) by (le))
        baseline_query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="api",version="stable"}[2m])) by (le))
        max_increase_pct: 20
        significance: 0.05
    max_consecutive_failures: 2

Metric kinds

Kind	Compares	Typical use
absolute	The canary’s aggregated value to fixed min / max	Saturation, queue depth, “no panics”
relative	The canary to a baseline (the stable version), by max_ratio or max_increase_pct	Latency, memory
error_rate	query / total_query (errors over requests), to bounds or a baseline	HTTP 5xx rate

Other per-metric fields:

Field	Meaning
aggregation	How samples in an interval window combine: avg, sum, min, max, last, p50, p90, p95, p99, rate
direction	Which way is bad: higher_is_worse (default), lower_is_worse, either
abs_tolerance	Ignore differences smaller than this when the baseline is tiny
significance	When above 0 (for example 0.05), a Mann-Whitney U test is run between canary and baseline samples, and a relative metric only fails when the difference is statistically significant
min_samples	Minimum canary samples required for a verdict (default 3)
critical	A failing critical metric fails the whole interval regardless of score
weight	Scales the metric in the interval score (default 1)

Intervals, scores and verdicts

Each analysis interval produces a score from 0 to 100 from the weighted metric results. An interval passes when its score is at least pass_score (default 100, meaning every metric must pass).

Setting	Default	Effect
interval	1m	Time between analyses
intervals	5	Analyses performed at each canary step
window	= interval	Lookback window per analysis
step	15s	Sample resolution requested from the provider
max_consecutive_failures	2	Failed intervals in a row that abort verification (rollback)
max_failures	0 (unlimited)	Cap on total failed intervals
inconclusive_is_failure	false	Treat intervals without enough data as failures
allow_inconclusive	false	Let a verification that never had data pass. By default a deployment that cannot be verified fails verification

The result is recorded as a verification run on the deployment (per-interval results and the verdict) and shown in the deployment view, with the metric charts.

What happens on failure

With auto_rollback on (the default), a failed verification rolls the deployment back: traffic returns to the stable version and the deployment finishes as rolled_back. Subscribed notification and incident integrations fire on deployment.rolled_back.

Tips

• Compare to a baseline running at the same time whenever you can. Absolute thresholds break when traffic changes; relative ones track it.
• Start with few, high-signal metrics. Error rate and tail latency catch most bad releases.
• Use significance for noisy metrics so a single spike does not roll back a healthy release.
• Set initial_delay so cold caches and JIT warmup do not fail the first interval.
• Dry-run the queries against a past deployment in your metrics tool before relying on them. A query that returns no data produces an inconclusive interval, which fails by default.
• Alert on the verifier, too. Treat “verification could not run” as an incident signal.