Observing an ephemeral AI inference fleet

Router inventoryTrack worker heartbeats

HTTP discoveryRefresh ephemeral targets

Fleet dashboardInspect GPU and inference signals

Five targets were UP in the rollout snapshot. This delivery added visibility; paging rules were a separate follow-up.

Contribution at a glance

My responsibility

Dynamic target inventory, Prometheus discovery integration and fleet dashboards.

Scope

Ephemeral inference workers; discovery refreshed every 30 seconds.

Collaboration & boundaries

I connected the inventory and monitoring layers. Paging rules were separate follow-up work.

Outcome

Five targets were UP in the rollout snapshot; worker redeployments no longer required a manually maintained static list.

The problem

Rented GPU pods receive new provider addresses after redeployment. A static Prometheus target list breaks as those addresses change, and an operator-only web view does not provide the history or shared visibility needed for diagnosis.

My contribution

I exposed a dynamic target inventory from the router, connected it to Prometheus HTTP service discovery and built a Grafana view of the inference fleet. The router already tracked worker heartbeats, so it was the right place to describe the active topology.

How discovery worked

Prometheus polled the inventory over HTTPS every 30 seconds. The list included the router, active workers and inference pods; stale workers disappeared automatically. The inventory endpoint used bearer authentication, and worker metrics were protected before being advertised for scraping.

The dashboard brought together active streams, worker utilization and inference tokens per second. I kept high-cardinality stream identifiers out of router-level gauges and retained detailed stream metrics at the worker layer.

Results and measurement

The rollout verified five UP targets: one router, two workers and two inference pods. The dashboard showed live stream and inference data. These counts describe the validation snapshot, not the maximum size of the platform.

The operational boundary

This delivery established observability, with alert thresholds and on-call ownership left as a separate decision. The inference engine's aggregate metrics endpoint retained a documented authentication exception; the discovery integration did not make every endpoint private by itself.

The dashboard definition was versioned, but import remained a manual step. Automatic provisioning was a further improvement, not a capability claimed for this rollout.

Why it mattered

The useful abstraction was the fleet, not an individual rented pod. Reusing the router's inventory made that abstraction available to the monitoring system without adding a second manually maintained source of truth.

Related work: serving multimodal models across engines and clouds.

Decisions and next steps

A retrospective view of the approach and its limits. The next improvement is a proposal, not a delivered result.

Alternative and rationale

Static targets are simple but decay after redeployments. Router-owned inventory follows the component that already knows active workers.

Trade-off

Discovery depends on the inventory endpoint; high-cardinality stream labels can make fleet-level metrics expensive.

What I would improve next

Next improvement: add discovery freshness checks and test alert behavior when inventory or a worker disappears.