Own the baseline, rent the peak: notes from a hybrid inference test
We put a small on-prem GPU and a rented cloud GPU behind one endpoint, killed the on-prem GPU in the middle of a load test, and lost 0 of 600 requests.
Hybrid inference
Most inference load is a steady baseline with spikes on top. Buying GPUs for the peak wastes money. Buying for the baseline drops requests during spikes. Hybrid inference does both: owned GPUs carry the steady load, rented GPUs take the spikes, and clients send every request to one address.
The setup
One consumer GPU a few generations old on-prem, and one datacenter GPU rented by the hour, both serving the same small open-weight model. We used test prompts only.
- Gateway: the single entry point, in the on-prem Kubernetes cluster.
- Pool: both GPUs. Each request goes to the one with the shortest queue.
- Private link: an encrypted path that joins the cloud GPU to the pool without exposing it to the internet.
Everything in the request path is open source.
Results
| Metric | On-prem GPU | Cloud GPU |
|---|---|---|
| Time to first token | 40 ms | 88 ms |
| Output speed, one request | about 80 tokens/s | about 140 tokens/s |
Most of the cloud GPU's extra 48 ms is the network round trip. After the first token, it streams nearly twice as fast.
- Load: 48 concurrent requests split 47 on-prem and 49 cloud, with 0 errors.
- Failover: at 2 requests per second, we force-killed the on-prem GPU. 600 of 600 requests succeeded, and the on-prem GPU rejoined on its own after about 2 minutes.
- Cost: about five dollars of rented GPU time.
Lessons
Route on load, not on health
Failover moves traffic when a GPU is down. Overflow moves it when a GPU is busy. Our first version only did failover: under a burst, the gateway refused requests while the cloud GPU sat idle, because a busy GPU still passes its health check. Routing on queue length fixed it.
Capacity is the real risk
Getting one datacenter GPU took about ten hours: free capacity appeared four times overnight, and other customers took it each time. Reserve capacity in advance and use more than one provider. How far overflow can go depends on your risk tolerance:
- Your own GPUs for the steady load.
- A GPU rented from any cloud with capacity, running your model over a private link. Lower risk if the provider supports attestation.
- A hosted model API, as a last resort. Capacity is almost always there, but the provider sees the prompts.
Test where it runs
Every real problem appeared only on the on-prem cluster, never on the local test cluster, and each one would have blocked a deploy.
Open problems
- Gateway limit: at about 64 concurrent requests, the gateway hits a connection limit. Up to 48, nothing failed.
- Long streams: an answer in progress on a GPU that dies is still lost.
- Cold starts: a replacement GPU took about two minutes to load the model. Storing it on the node ahead of time should remove most of that wait.
- Automatic burst: rented GPUs were added by hand. Next, on-prem queue length starts and stops them, under a budget cap.