Security

The Latitude.sh provider sits on the trust boundary between BigFleet’s control plane and your Latitude.sh project: it accepts lifecycle RPCs over the network and turns them into server deploys, deprovisions, power actions, and SSH commands. This page covers the four things an operator must get right — the gRPC transport (mTLS), the API token the process holds, the SSH-bootstrap trust model, and how the process is exposed.

Transport: mTLS on the gRPC port

The CapacityProvider gRPC service, the grpc.health.v1 health service, and (optionally) reflection all share --addr (default :9000). Secure it with the TLS flags:

./latitude --provider latitude-ash --token "$LATITUDESH_API_TOKEN" --project proj_x \
           --tls-cert server.pem --tls-key server-key.pem \
           --tls-ca client-ca.pem

The flags compose into three modes (logged at startup as the security field):

Mode	Flags	Behaviour
insecure	none of --tls-cert/--tls-key	Plaintext. Acceptable only for trusted in-cluster traffic or the fake backend.
TLS	--tls-cert + --tls-key	Server presents a cert; clients are not authenticated.
mTLS	--tls-cert + --tls-key + --tls-ca	Server presents a cert and requires a client cert chaining to --tls-ca. Use this in production.

Notes from the implementation, so you do not fight the validation:

• --tls-cert and --tls-key are required together — supplying only one is a startup error.
• --tls-ca without a cert/key is rejected; a CA only makes sense once the server itself has a cert.
• When --tls-ca is set, client auth is RequireAndVerifyClientCert: a missing or untrusted client certificate is refused at the TLS layer, before any RPC handler runs.
• The server pins TLS 1.3 (MinVersion). Make sure BigFleet’s client and any debugging tooling can negotiate 1.3.
• A bad keypair or an unparseable CA bundle fails the process at startup rather than degrading silently, so a misconfigured cert can never come up insecure.

The TLS keypair and CA are read once, at startup, so to roll a certificate, restart the process after the new PEM is in place (a Deployment rollout). The persisted --state file is the restart path, so a rolling restart is safe.

The API token

Authorisation to Latitude is a single project-scoped API token plus the project id/slug — there is no IAM, role, or instance profile. Consequences for security:

• The token can deploy and deprovision every server in the project. Keep the project scoped to BigFleet-managed capacity so the token’s blast radius is only what this provider owns.
• Store it as a Kubernetes Secret mounted as LATITUDESH_API_TOKEN, never in args, an image, or values. The full minting / storage / rotation flow is on the Credentials page.
• The token is never logged. Use a distinct, named token per deployment so it can be rotated and audited independently.

The SSH bootstrap trust model

Latitude.sh has no in-guest command API, so Configure and Drain reach the server over SSH. The cluster-join secret rides in the bootstrap_blob delivered at Configure, so this channel must be authenticated in both directions.

• The provider connects as --ssh-user (default root) with the private key from --ssh-key. Use a dedicated key for the provider, stored as its own Secret, not an operator’s personal key. The provider registers the matching public key with Latitude and authorises it on every server it deploys.
• The cluster-join bootstrap blob is delivered to <bootstrap-hook>.blob and the hook is run as <bootstrap-hook> <cluster-id> — the blob is opaque and the provider never parses it.

The join secret is never in user_data

Latitude user_data is consumed once at first boot and stored as a Latitude UserData resource — so it is the wrong place for a per-cluster secret (a slot’s target cluster is not even known at deploy). The provider therefore never puts the cluster-join secret in user_data/cloud-init. user_data carries only generic, non-secret pre-binding material: your --base-user-data plus the injected SSH host key (below). The join secret is delivered later, over SSH, at Configure.

Host-key verification

The provider verifies the server’s SSH host key before delivering the bootstrap payload, so an on-path (MITM) attacker cannot impersonate a freshly deployed server and capture the cluster-join material in the Configure blob. It never uses ssh.InsecureIgnoreHostKey and never disables TLS verification — a host whose key does not match the pin fails closed. The model:

• Injected, pinned key (default, for servers the provider deploys). At Create the provider mints a fresh ed25519 host keypair, injects the private key into the server via first-boot user_data cloud-init (ssh_keys: — merged with your --base-user-data as a MIME multipart archive), and pins the public key’s SHA-256 fingerprint in the provider-owned substrate index (--substrate-state). Every later Configure/Drain connection checks the presented host key against that pin and aborts on mismatch (a possible MITM). Because the key is known before the first connection — and the pin is persisted, so it survives a restart — there is no trust-on-first-use window for servers the provider deployed.
• Trust-on-first-use (fallback, only for servers with no pin). A server the provider did not deploy (an orphan it adopted), or one whose pin was lost with the substrate index, has no pinned fingerprint. The first connection records the observed host key and pins it; all later connections are verified against it. The residual risk is confined to that single first connection, and it is logged at WARN. Persist --substrate-state (on the same volume as --state) so this fallback is not re-entered on every restart.

For defence in depth, run the SSH path over a private/management network the control plane trusts. If your --base-user-data sets its own ssh_keys host-key block, avoid conflicting with the injected one — the provider’s injected key is what it pins and verifies against.

Exposure

Run the provider with replicas: 1 per site, reachable only by BigFleet:

• Keep the gRPC --addr on a ClusterIP Service inside the mesh/namespace, not a LoadBalancer. If you terminate TLS at a mesh sidecar instead of in the provider, leave the provider insecure but ensure the port is never reachable outside the mesh.
• The metrics/health port (--metrics-addr) serves no secrets, but scope it to your Prometheus and kubelet probes all the same.
• The pod runs non-root (uid 65532) on a read-only root filesystem with all capabilities dropped (the chart’s hardened defaults match the distroless image).