Observability

The libvirt provider is built to be operated from its signals. Every libvirt API call, every gRPC request, and the background reconcile loop are instrumented; liveness and readiness are separate probes; and requests are logged through a panic-recovering interceptor chain.

Observability lives on a separate HTTP port from the gRPC server. The gRPC contract (CapacityProvider + grpc.health.v1 health + reflection) is served on --addr (:9000); /metrics, /healthz, and /readyz are served on --metrics-addr (:9090). Set --metrics-addr "" to disable the HTTP server entirely (the gRPC health service stays up regardless).

Metrics catalogue

Metrics are registered on an isolated Prometheus registry (not the global default), exposed at GET /metrics on --metrics-addr. Every series is prefixed bigfleet_libvirt_. The Go runtime and process collectors are also registered, so you get go_* and process_* for free.

libvirt API

The libvirt client is wrapped by a transparent metrics decorator, so every API call the provider makes is counted and timed.

Metric	Type	Labels	What it tells you
bigfleet_libvirt_api_calls_total	counter	op, outcome	libvirt API call volume, split by operation (CreateDomain, DeleteDomain, DescribeManaged, EnsureRunning, Configure, Drain) and success/error. The first place to look when creates or drains are failing.
bigfleet_libvirt_api_duration_seconds	histogram	op	libvirt API latency by operation.

gRPC

Metric	Type	Labels	What it tells you
bigfleet_libvirt_grpc_requests_total	counter	method, code	CapacityProvider RPCs by method and gRPC status code. A spike of FailedPrecondition means a zombie shard is being fenced — alert on it.
bigfleet_libvirt_grpc_request_duration_seconds	histogram	method	Per-RPC latency. The mutating RPCs are async, so these should stay sub-millisecond.
bigfleet_libvirt_panics_total	counter	—	Recovered panics in gRPC handlers (should stay 0).

Background loop

Metric	Type	Labels	What it tells you
bigfleet_libvirt_reconcile_total	counter	outcome	Background libvirt→inventory reconcile runs by outcome.

Health vs readiness

Two distinct probes, served on --metrics-addr:

• /healthz (liveness) — always 200 ok once the HTTP server is up. Wire it to a liveness probe; a failure means the process is wedged and should be restarted.
• /readyz (readiness) — 200 ready only after the gRPC server is serving; 503 not ready during startup and shutdown. Wire it to a readiness probe so BigFleet only dials the Service once the provider can serve. On SIGTERM it flips to not ready first, so traffic drains before the gRPC server stops.

The standard gRPC health service (grpc.health.v1) is also registered on --addr, reporting SERVING/NOT_SERVING for gRPC-native health checking.

Logging

Structured slog text on stderr. Every RPC is logged through the interceptor chain with its method, gRPC code, and duration; the mutating RPCs log at INFO, reads at DEBUG, and any error at WARN. Neither any libvirt credential nor the opaque bootstrap blob is ever logged. A panic in a handler is recovered, counted (bigfleet_libvirt_panics_total), logged at ERROR, and turned into codes.Internal — it never crashes the process.

Sample Prometheus scrape

The chart’s Service carries prometheus.io/scrape annotations on the metrics port, so a standard annotation-based scrape config picks it up. A static scrape:

scrape_configs:
  - job_name: bigfleet-libvirt
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"

Useful alerts: a non-zero rate of bigfleet_libvirt_grpc_requests_total{code="FailedPrecondition"} (zombie-shard fencing), a rising bigfleet_libvirt_api_calls_total{outcome="error"} (libvirt trouble — host unreachable, pool/network missing), or any bigfleet_libvirt_panics_total increase.