Observability

The Scaleway provider is built to be operated from its signals. Every Scaleway API call, every gRPC request, and every background loop is instrumented; liveness and readiness are separate probes; and requests are logged through a panic-recovering interceptor chain.

Observability lives on a separate HTTP port from the gRPC server. The gRPC contract (CapacityProvider + grpc.health.v1 health + reflection) is served on --addr (:9000); /metrics, /healthz, and /readyz are served on --metrics-addr (:9090). Set --metrics-addr "" to disable the HTTP server entirely (the gRPC health service stays up regardless).

Metrics catalogue

Metrics are registered on an isolated Prometheus registry (not the global default), exposed at GET /metrics on --metrics-addr. Every series is prefixed bigfleet_scaleway_. The Go runtime and process collectors are also registered, so you get go_* and process_* for free.

Scaleway API

The Scaleway client is wrapped by a transparent metrics decorator, so every API call the provider makes is counted and timed.

Metric	Type	Labels	What it tells you
bigfleet_scaleway_api_calls_total	counter	op, outcome	Scaleway API call volume, split by operation (CreateServer, DeleteServer, DescribeManaged, EnsureRunning, ReapOrphanVolumes, Configure, Drain, Pricing, Catalogue) and success/error. The first place to look when creates or drains are failing.
bigfleet_scaleway_api_duration_seconds	histogram	op	Scaleway API latency by operation. Watch the CreateServer tail on Elastic Metal — physical commissioning is slow.

gRPC

Metric	Type	Labels	What it tells you
bigfleet_scaleway_grpc_requests_total	counter	method, code	CapacityProvider RPCs by method and gRPC status code. A spike of FailedPrecondition means a zombie shard is being fenced — alert on it.
bigfleet_scaleway_grpc_request_duration_seconds	histogram	method	Per-RPC latency. The mutating RPCs are async, so these should stay sub-millisecond.
bigfleet_scaleway_panics_total	counter	—	Recovered panics in gRPC handlers (should stay 0).

Background loops

Metric	Type	Labels	What it tells you
bigfleet_scaleway_reconcile_total	counter	outcome	Background Scaleway→inventory reconcile runs by outcome.
bigfleet_scaleway_price_refresh_total	counter	outcome	Background price-refresh runs by outcome.

Health vs readiness

Two distinct probes, served on --metrics-addr:

• /healthz (liveness) — always 200 ok once the HTTP server is up. Wire it to a liveness probe; a failure means the process is wedged and should be restarted.
• /readyz (readiness) — 200 ready once the gRPC server is serving; 503 not ready during startup and shutdown. A best-effort price/spec warm-up runs before readiness is signalled, but readiness is not gated on its success — a catalogue outage or pricing-auth hiccup leaves the provider ready and serving (it falls back to the pinned price/spec tables). Wire it to a readiness probe so BigFleet only dials the Service once the provider can serve. On SIGTERM it flips to not ready first, so traffic drains before the gRPC server stops.

The standard gRPC health service (grpc.health.v1) is also registered on --addr, reporting SERVING/NOT_SERVING for gRPC-native health checking.

Logging

Structured slog text on stderr. Every RPC is logged through the interceptor chain with its method, gRPC code, and duration; the mutating RPCs log at INFO, reads at DEBUG, and any error at WARN. Neither the API key, the bootstrap secret (nor any derived per-machine token), nor the opaque bootstrap blob is ever logged. A panic in a handler is recovered, counted (bigfleet_scaleway_panics_total), logged at ERROR, and turned into codes.Internal — it never crashes the process.

Sample Prometheus scrape

The chart’s Service carries prometheus.io/scrape annotations on the metrics port, so a standard annotation-based scrape config picks it up. A static scrape:

scrape_configs:
  - job_name: bigfleet-scaleway
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

Useful alerts: a non-zero rate of bigfleet_scaleway_grpc_requests_total{code="FailedPrecondition"} (zombie-shard fencing), a rising bigfleet_scaleway_api_calls_total{outcome="error"} (Scaleway API trouble), or any bigfleet_scaleway_panics_total increase.