Self-hosting Gemma 3 12B with vLLM on serverless GPUs
What it took to run a 12B open-weight model in production on rented GPUs: a three-provider fallback router with measured SLAs, cold starts cut from ten minutes to three, and the driver, scheduler and shell bugs that ate my afternoons.
I run platform and reliability engineering for a regulated online-safety product. One of our workloads needed a large language model served in-house instead of through a hosted API, for cost control and to keep inference on infrastructure we own the behaviour of. I chose Gemma 3 12B, quantized to GGUF Q4 (about 6.9 GB on disk), served with vLLM 0.15.1.
What follows is the engineering log: the bugs that cost me the most time, how I cut cold starts, the router that removed the single point of failure, and the numbers I measured. I have left out anything tied to the employer's internal naming. Everything here is the technique.
Three bugs that cost me the most time
None of these printed a useful error at the surface. That is the real lesson: on rented GPUs, your startup path has to fail loudly and early, because the layers underneath will not do it for you.
Zero workers provisioned, for hours, with no error
Serverless GPU endpoints select hardware by VRAM tier, not by the GPU's marketing name. I asked for NVIDIA GeForce RTX 4090 and the scheduler matched nothing, so it quietly provisioned zero workers and never told me why. The fix was to pass the tier id alongside the name (for the 24 GB Ada class, ADA_24). The moment the tier was correct, workers appeared.
CUDA Error 804: forward compatibility on non-supported hardware
Workers crash-looped with a CUDA init failure. The cause was a version mismatch I had assumed away: the vLLM base image ships CUDA 12.8, which needs an NVIDIA driver of at least 570. The serverless hosts I landed on ran driver 550.x, which caps at CUDA 12.4. The container could not initialize the GPU at all. The takeaway is simple and easy to forget: match your CUDA toolkit to the host driver, do not assume the newest image runs everywhere.
A silent crash with no logs at all
Workers died on startup with an empty console. The cause was an exit 1 inside a config script that the startup script pulled in with source. Because the file was sourced, that exit killed the whole shell before a single diagnostic line could run. I moved the validation out of the sourced file into a guarded startup step, and the logs came back immediately.
Cold starts: ten minutes to three
The first version installed vLLM with pip on every worker start and pulled the model weights at boot. Cold starts ran seven to twelve minutes. That is fine for a demo and useless in production, because every scale-up under load makes a user wait.
What moved the number:
- Pre-bake the environment into the image. vLLM and its patches go in at build time, not on every cold start. This alone removed the five to ten minute
pip installfrom the critical path. - Cache weights on a network volume. A cold worker attaches an already-populated volume instead of downloading 6.9 GB again.
- Load GGUF directly. Using the GGUF load-format skips a safetensors conversion step at startup.
- Pre-compile and trim the boot. Pre-compiling Python bytecode and masking boot services that add nothing (snap, unattended upgrades, the apt timers) shaved the rest.
What did not work: Hyperdisk
I checked whether a faster disk would speed up model loading. Hyperdisk Extreme cannot be a boot disk, only a data disk, which would mean a separate disk plus a mount script and would defeat the pre-baked image. Hyperdisk Balanced can boot, but the image-build pipeline cannot create images from it. The expected gain was single-digit seconds. Not worth the complexity, so I wrote it down and moved on.
The fallback router
Serving from a single GPU provider is a single point of failure. A provider outage or a GPU supply constraint takes the whole capability down, and each provider has a different latency profile that a single endpoint cannot exploit. So I put a small router in front.
It is a roughly 150-line FastAPI reverse proxy that speaks the OpenAI API (/v1/chat/completions, /v1/models). It routes through a provider chain: serverless GPUs first, a managed serverless GPU second, a dedicated cluster third. Each hop has a mock-failure flag, so I can test failover without waiting for a real outage. When every provider fails, it returns HTTP 503 with per-provider diagnostics instead of a blank error, which is the difference between a two-minute triage and a one-hour one.
The latencies are measured, not estimated, and they are what the router uses to reason about which hop a request took:
| Hop | Hardware | Latency | Role |
|---|---|---|---|
| Serverless GPUs | Ada / Ampere, 6 types | 568 ms | Primary, cost-optimized burst |
| Managed serverless GPU | NVIDIA L4 | 1.9 s warm, 41 s cold | Fallback 1, scales to zero |
| Dedicated cluster | L40S / A100 SXM 80 GB | 7.7 s | Fallback 2, high-throughput |
Multi-GPU and multi-datacenter
GPU supply on rented serverless is not guaranteed. On a bad day the GPU type you want is simply not available in the region you asked for. Two things made this survivable.
Profile-driven GPU selection. I replaced a hardcoded if-else over GPU names with one table that maps each GPU type to its tuned config: VRAM, max model length, dtype, kv-cache dtype, fp8 support and tensor-parallel settings. One flag selects the whole profile. Six GPU types are deployable (RTX 4090, RTX 5090, A6000, RTX 6000 Ada, L40S and A100 SXM), and switching between them is a single parameter rather than a code change.
Datacenter failover. The deploy tries datacenters in an ordered list and moves to the next one when it hits a supply constraint, with exponential backoff. Each datacenter has its own cached weights volume, so a failover does not re-download the model.
What I would tell someone starting this
- Pin your CUDA toolkit to the host driver, not to the newest base image. The newest image is the one that crash-loops on older hosts.
- Make startup fail loudly. A sourced
exitor a silently-unmatched GPU tier will cost you an afternoon of staring at an empty log. - Treat cold start as a first-class SLO. Pre-bake everything deterministic and cache the weights; do not assemble the environment on every scale-up.
- Put a router in front before you think you need it. Single-provider GPU serving is a single point of failure, and GPU supply is the part you do not control.