Production LLM Inference on Google Cloud

The first version of an LLM endpoint is usually satisfying. A model server starts, curl returns a plausible answer, and the GPU looks busy. Then real traffic arrives.

One user sends a 30,000-token prompt while fifty others are waiting. A new replica spends several minutes obtaining a GPU and loading weights. Round-robin routing sends a repeated system prompt to the only pod that does not have it cached. A rollout replaces the last healthy replica before its successor is ready. The service is technically up, but the first token arrives after the user has left.

None of these are model-quality problems. They come from treating LLM inference like an ordinary stateless web service.

This article designs a production endpoint for an open-weight model on Google Cloud. The concrete path uses GKE, vLLM, and GKE Inference Gateway, but the reasoning begins with workload and failure boundaries rather than product names. Vertex AI remains a valid managed alternative; the choice between them should follow from how much control the serving system actually needs.

Start With a Workload Contract

“Serve a 32-billion-parameter model” is not a capacity plan. The same model behaves very differently under code completion, multi-turn chat, document summarization, and offline generation.

Assume the initial service has these requirements:

These are design inputs, not performance claims. A benchmark may show that the model and available accelerators cannot meet them. In that case, the honest options are a smaller or quantized model, more replicas, looser limits, or a revised SLO.

Two latency measures must remain separate:

• Time to first token (TTFT) includes queueing and prefill. It captures how long a user waits before the response begins.
• Inter-token latency (ITL) measures gaps during decode. It captures whether the response streams smoothly after it begins.

An average request latency can hide both a terrible queue and a stuttering decode. Requests per second is also a weak capacity measure when prompt and completion lengths vary by an order of magnitude.

Managed Endpoint or GKE?

Google Cloud offers several sensible ways to serve a model. The useful comparison is not “simple versus advanced”; it is which operational responsibilities the application actually needs to own.

Vertex AI is the better answer when the team wants managed model deployment, autoscaling, endpoint security, and monitoring without owning a cluster. It can run custom vLLM containers on accelerators, and it can source model artifacts from Cloud Storage. The vLLM server’s OpenAI protocol is not itself the Vertex prediction protocol, so Google’s documented custom-container path uses raw inference requests when appropriate.

GKE earns its cost when inference behavior is part of the product: prefix-aware routing, custom batching, model-specific queue policies, dynamic adapters, novel decoding methods, or a topology that must be benchmarked and changed independently. For the rest of this design, assume those needs are real and choose GKE deliberately.

The Serving Architecture

Clients reach stable or canary vLLM InferencePool replicas through an Application Load Balancer, GKE Gateway, and GKE Inference Gateway. The outer gateway handles authentication, quotas, and request limits. The inference-aware gateway owns model selection, a bounded queue, priority, load shedding, and affinity for reusable prefixes, KV-cache state, or LoRA adapters.

The release path is separate. CI tests and benchmarks an immutable server image from Artifact Registry together with versioned weights and a checksum manifest in Cloud Storage, then promotes that exact release through a canary. Managed Service for Prometheus and Cloud Monitoring collect TTFT, inter-token latency, queue depth, KV-cache pressure, token counts, GPU utilization, errors, and cost signals from the serving pools.

The data plane is intentionally short. Interactive streaming traffic does not pass through Pub/Sub: a queue that breaks the client connection also breaks token streaming and cancellation. If the product has an asynchronous batch API, that can be a separate admission path backed by a durable queue and a different SLO.

The control plane never overwrites a model directory or reuses an image tag such as latest. A deployment should identify a model revision, tokenizer revision, serving-image digest, quantization, and configuration as one immutable release.

For example:

{
  "release": "qwen3-32b-fp8-2026-08-23.1",
  "model_revision": "4f9c...",
  "tokenizer_revision": "4f9c...",
  "image_digest": "sha256:88d1...",
  "weight_manifest": "gs://llm-models/qwen3-32b-fp8/manifest.json",
  "max_model_len": 16384,
  "tensor_parallel_size": 2,
  "sampling_defaults_version": 5
}

This identity belongs in request logs and metrics. “The Qwen endpoint was slow yesterday” is not a debuggable incident report.

Memory Before Throughput

The first placement question is whether the model and a useful KV cache fit on the chosen accelerator topology.

For $P$ parameters stored at $b_w$ bits per parameter, raw weight memory is approximately

A 32B model needs roughly 64 GB for BF16 weights or 32 GB for one-byte weights before runtime overhead, scales, activations, CUDA graphs, temporary buffers, and fragmentation are counted. A label such as “FP8” also does not guarantee that every tensor occupies exactly one byte per parameter in memory.

For a conventional transformer KV cache, a useful approximation is

where $L$ is the layer count, $H_{KV}$ the number of KV heads, $d_h$ the head dimension, $T$ the total cached tokens across active sequences, and $b_{KV}$ the bytes per cached element. The leading 2 accounts for keys and values.

The actual per-device value depends on how the engine shards KV heads and model weights. The complete budget is closer to

This is why blindly exposing a model’s maximum supported context length can destroy concurrency. If the product needs at most 16K tokens, configuring 128K reserves or plans for a capability nobody uses. Set the service limit from the workload, then measure how many concurrent sequences fit without approaching out-of-memory behavior.

Tensor parallelism solves a memory problem by splitting model work across devices, but every layer may now pay collective-communication costs. The fastest topology is not necessarily the one with the most GPUs. Benchmark one, two, and larger device groups with the same model precision, prompt distribution, and latency target.

Weight Loading Is on the Request Path Eventually

An autoscaler can create a pod quickly and still fail the user because the pod spends several minutes pulling an image and reading tens of gigabytes of weights.

Keep the container image and weights separate. Store the immutable image in Artifact Registry and the versioned model artifact in Cloud Storage or another deliberately selected serving store. Verify a checksum manifest before the readiness probe passes.

Current GKE guidance recommends the Run:ai Model Streamer for recent vLLM releases to stream weights directly from Cloud Storage. Hyperdisk ML can provide high-throughput, read-only access to multiple serving pods. Cloud Storage FUSE is another integration option, but a convenient filesystem mount is not automatically the fastest startup path. Measure cold loading with the exact weight layout and replica fan-out.

A pod is not ready merely because the HTTP process has bound a port. Readiness should require:

1. all weight shards loaded;
2. distributed workers joined and collectives initialized;
3. KV-cache allocation completed;
4. a small inference health request succeeded;
5. the model release in memory matches the declared release.

The startup probe needs enough time for a legitimate cold load. The readiness probe needs to fail quickly when the server cannot accept traffic. They solve different problems.

Continuous Batching Is the Local Scheduler

Ordinary web servers batch requests that arrive together, run them, and wait for the whole batch to finish. LLM requests have variable prompt and output lengths, so static batches strand capacity behind the longest sequence.

An engine such as vLLM uses continuous batching: completed sequences leave and waiting sequences enter between decode iterations. The scheduler balances several competing resources:

• token slots processed in the next iteration;
• KV-cache blocks held by active sequences;
• prefill work from new prompts;
• decode work from existing streams;
• latency deadlines and request priorities.

Large batches improve accelerator utilization but can damage ITL. Aggressive prefills improve admission throughput but can stall ongoing decodes. Chunked prefill limits how much one long prompt monopolizes an iteration, at the cost of more scheduling complexity.

Tune the model server against a latency-throughput curve, not a single maximum-throughput run. The useful operating point is just before queueing and KV pressure cause tail latency to bend sharply upward.

Routing Must Understand Inference State

Round robin assumes interchangeable stateless replicas. LLM pods are neither.

One pod may have 90% of its KV cache occupied, another may have a long queue, and a third may already cache the 6,000-token prefix in the incoming request. Sending the request to the “next” pod can waste prefill compute and worsen both TTFT and cluster throughput.

GKE Inference Gateway uses an endpoint picker alongside the L7 gateway. It can consider:

• pending queue depth;
• KV-cache utilization;
• the length of a matching cached prefix;
• affinity for a requested LoRA adapter;
• model identity and request priority.

Prefix-aware routing is more useful than generic session affinity. Two unrelated sessions can share a system prompt or retrieved document prefix, while two turns from one session may no longer fit on the same healthy replica. Cache affinity is a scored preference, not permission to overload a hot pod.

For workloads with highly variable prompt and output lengths, queue depth alone is a poor estimate of future latency. GKE also documents predicted latency-based routing that estimates TTFT and TPOT from the request and live server state. As of 23 August 2026, this feature is in Preview, requires a warm-up period, and is not recommended for strict production SLAs without thorough testing. Its homogeneous-pool requirement also matters: predictions assume the pods use the same hardware, model, and serving configuration. A heterogeneous pool needs a routing policy that recognizes those different performance regimes.

The endpoint picker is part of the availability path. Run it redundantly, monitor its decision latency, and test its failure behavior. A clever router that is down is worse than a simple router that is alive.

Admission Control Comes Before Autoscaling

Accelerator capacity cannot react instantly. A bounded service therefore decides which work it can accept now.

At the gateway, validate or compute:

• tenant and authentication identity;
• model and adapter name;
• input token count, not only request bytes;
• maximum output-token request;
• deadline and priority class;
• per-tenant concurrent-request and token budgets;
• whether the request is interactive or asynchronous.

The worst request is not always the largest prompt. A short prompt asking for 20,000 output tokens can occupy a decode slot for a long time. Admission cost should include both input and requested output:

where the weights reflect measured prefill and decode pressure. This score can feed quotas and priority queues.

Bound the queue. Once a request cannot begin within its deadline, return an overload response—normally 429 with a meaningful retry policy—instead of holding it until the user times out. Load shedding protects the requests already streaming and gives autoscaling time to catch up.

If a client disconnects, propagate cancellation to the model server. Continuing to decode an answer nobody will read spends the scarcest resource in the system.

Autoscaling Has Two Clocks

There are two distinct scaling events:

1. the Horizontal Pod Autoscaler asks for another model-server pod;
2. GKE obtains a suitable accelerator node if no existing node can place it.

The second can dominate. Capacity might be unavailable, node initialization takes time, and the new pod still has to load weights. Scaling from zero is a poor default for an interactive SLO.

CPU utilization is also the wrong primary signal. The CPU can remain quiet while the GPU KV cache is full and requests are piling up. GKE’s current inference guidance recommends signals from the server, such as:

• vllm:num_requests_waiting for throughput-oriented backlog control;
• vllm:num_requests_running when active batch pressure tracks latency;
• vllm:gpu_cache_usage_perc as an early warning of KV-cache saturation.

The exact thresholds must come from the latency curve for one replica. A useful policy combines a fast queue signal with a slower stability rule so that one burst does not create pods that become ready after the burst has ended.

Keep a minimum warm replica count, plus enough spare capacity for the largest failure the SLO promises to absorb. For scarce production GPUs, quota is not capacity. Use an appropriate reservation strategy and verify that the GKE node pool consumes the intended zonal reservation. On-demand provisioning alone does not guarantee that another accelerator will exist during an incident.

Scale-down is a protocol too:

1. remove the pod from new endpoint selections;
2. stop admitting new sequences;
3. allow bounded streams to drain;
4. cancel or hand off only according to an explicit client contract;
5. terminate after a grace deadline.

Killing a streaming pod at the first HPA scale-down signal turns efficiency into visible errors.

Capacity and Cost From Measured Tokens

Let request arrival rate be $\lambda$ requests per second, mean generated length be $\mathbb{E}[O]$, and one replica sustain $\theta_o$ output tokens per second at the chosen latency target. With target utilization $\rho<1$, a decode-first estimate is

Run a separate calculation for prefill using input-token throughput. Provision to the larger requirement, then validate with mixed traffic. Means alone are insufficient when length distributions are heavy-tailed, so the load test must preserve p95 and p99 prompts and completions.

If one replica costs $C_h$ per hour and produces $\theta_o$ useful output tokens per second, its accelerator-only cost per million output tokens at that operating point is

Add gateway, CPU, storage, network, observability, idle headroom, and failed or cancelled generation. Report input and output token economics separately when prefill is material.

A more honest numerator uses goodput: tokens from requests that met their TTFT and ITL objectives. An overloaded server can advertise excellent raw throughput while most users receive late responses.

Speculative Decoding Fits Inside the Model Server

Speculative decoding is an optional optimization in the decode path, not a replacement for routing or autoscaling. A draft model proposes tokens and the target verifies them in a block, reducing serial target calls when enough proposals are accepted.

It also consumes memory for draft weights and another cache, changes the model server’s scheduling profile, and can lose value under large batches. Benchmark the complete server with the same concurrency and prompts used for the baseline.

My Prolepsis project provides configurable Hugging Face and vLLM implementations for Qwen draft-target pairs, along with retained prompts, responses, metrics, and limitations. The accompanying Speculative Decoding article derives the acceptance rule and latency model. In a GKE deployment, Prolepsis would live inside each serving replica; the Inference Gateway would still choose the replica and enforce queue policy.

When to Separate Prefill and Decode

Prefill and decode stress hardware differently. Prefill processes many known prompt tokens in parallel and tends to be compute-heavy. Decode advances one position at a time and often becomes limited by memory bandwidth and KV-cache movement.

At sufficient scale, separate prefill and decode pools can be sized and optimized independently. A prefill worker processes the prompt and transfers the generated KV state to a decode worker. Prefill capacity follows prompt-token demand, while decode capacity follows active sequences and their cache footprint.

This is not a free win. The KV state must move between workers, the scheduler must pair compatible model versions and parallel layouts, and failure recovery crosses another network boundary. For a moderate single-model deployment, colocated prefill and decode are usually the better baseline.

Disaggregate only after measurements show persistent interference that separate pools can fix. Google documents disaggregated serving on GKE for supported TPU stacks; GPU implementations and model servers have their own maturity and topology constraints. “Disaggregated” is an architecture to validate, not a flag that guarantees speed.

Rollouts Without Corrupting the Cache Boundary

Never replace weights inside a live serving pod. Start a separate canary InferencePool with a new immutable release and wait for full model readiness.

A safe progression is:

1. offline correctness and safety evaluation;
2. cold-start and single-replica load test;
3. shadow traffic with responses discarded or reviewed under the data policy;
4. a small weighted canary;
5. staged traffic increases with automatic rollback thresholds;
6. drain the old pool only after the new pool has stable headroom.

The stable and canary pools must be visible as separate metric dimensions. Compare TTFT, ITL, error rate, token rate, output length, cache hit rate, and safety outcomes. Stochastic responses are not expected to match word for word. Use deterministic golden requests for regression checks and task-specific evaluation for quality.

KV caches and LoRA adapters belong to a particular base model and serving configuration. A router must never assume cache compatibility across a version boundary. Weighted HTTP routing chooses a pool first; cache-aware endpoint selection happens within that compatible pool.

Kubernetes rolling updates can deadlock when every old replica occupies scarce GPUs and the new replica requires surge capacity. Reserve rollout headroom or deploy the canary into a separately provisioned pool. maxSurge: 1 cannot manufacture a GPU.

Streaming Changes Retry Semantics

Before the first response token, a gateway can often retry a failed request on another healthy replica if the deadline permits. After tokens have reached the client, blindly restarting can duplicate or contradict the visible answer.

Use a request ID for accounting and tracing, but do not pretend it makes generation resumable. Exact resumption requires a protocol that records the committed token prefix, sampling state, model release, and compatible KV state—or it requires regenerating and reconciling output. Most APIs instead expose a clear stream failure and let the caller decide whether to start a new request.

Billing follows the same boundary. Record admitted, generated, transmitted, and cancelled tokens separately. A client should not be charged twice because a gateway retried before any output, and the platform needs a policy for tokens computed after disconnection.

Failure Table

Failure drills should include a long live stream, not only short health requests. Delete a pod, exhaust its KV cache, block weight storage, remove an endpoint-picker replica, and send a burst above admission capacity. Observe whether the system fails in the way the table claims.

Security Without Giving the Model a Cloud Account

The inference server needs little authority. With Workload Identity Federation for GKE, give its Kubernetes service account read access only to the exact model-artifact location and any required secrets. It does not need project-wide editor permissions, database credentials, or agent tool permissions.

Terminate TLS at the managed gateway, authenticate callers, enforce tenant quotas, and restrict network paths to the model pods. Store registry or model-hub credentials in Secret Manager or a Kubernetes secret sourced from it; never bake them into the image.

Prompt injection is not solved by placing the model in a private subnet. If the endpoint feeds an agent, keep tool authorization outside the model and validate structured actions against deterministic policy. Google Cloud’s Model Armor can inspect prompts and responses in the gateway path, including content and data-leakage controls. Treat it as another production dependency: measure its latency, regional behavior, data handling, availability, and false-positive cost.

Do not log prompts and completions by default merely because they are useful for debugging. Logs often become a less protected copy of confidential input. Prefer request IDs, token counts, model release, timing, sampling parameters, and policy outcomes. Sample or retain content only under an explicit access and deletion policy.

Observability That Can Explain a Slow Token

Collect metrics at the gateway, router, model server, accelerator, and client boundary. Managed Service for Prometheus can collect metrics from vLLM and the llm-d components behind GKE Inference Gateway, while Cloud Monitoring provides dashboards and alerting.

At minimum, retain distributions—not only averages—for:

• end-to-end TTFT and model-server TTFT;
• ITL or TPOT;
• gateway and model queue time;
• prompt and generated token counts;
• input and output tokens per second;
• active, waiting, rejected, cancelled, and failed requests;
• KV-cache utilization and prefix-cache hit rate;
• GPU duty cycle, memory use, errors, and throttling;
• model load and pod readiness time;
• endpoint-picker decision time and selected reason;
• goodput and cost per million tokens by model release.

Stratify latency by prompt length, output length, tenant class, cache hit, sampling mode, model version, and replica. A p95 TTFT graph without prompt length can jump simply because traffic changed from chat to document summarization.

Trace one request through admission, routing, queueing, prefill, decode, safety checks, and stream completion. The trace should contain sizes and timings, not raw sensitive text.

Load Testing the System It Will Become

A credible load test uses an open-loop arrival process so the generator continues sending traffic when the service slows. A closed loop, where each virtual user waits for a response before sending the next request, reduces offered load during an incident and can hide queue collapse.

The workload should preserve:

• the production prompt and completion length distributions;
• realistic prefix repetition and multi-turn conversations;
• sampling settings;
• tenant mix and priority classes;
• streaming clients that sometimes disconnect;
• bursts, quiet periods, and sustained saturation;
• cold and warm cache phases.

Run at increasing offered load and plot TTFT, ITL, error rate, goodput, queue depth, and KV utilization. The saturation point is where another unit of offered load mostly produces waiting rather than useful tokens. Operate below it with enough headroom for a replica failure.

Then repeat the test during a canary, a node loss, an autoscale event, and a cold model load. Peak tokens per second in a quiet cluster is a kernel benchmark. Production capacity is what remains while the system is changing and failing.

A Practical Build Order

The architecture can be delivered without attempting every optimization at once:

1. Benchmark one immutable model release on one accelerator topology.
2. Deploy two warm replicas behind a simple bounded admission layer.
3. Add GKE Inference Gateway and verify load- and prefix-aware routing.
4. Export server, gateway, and GPU metrics; define TTFT and ITL SLOs.
5. Autoscale on queue or KV pressure while retaining minimum warm capacity.
6. Establish a separate canary pool and automated rollback.
7. Exercise pod, node, storage, router, and overload failures.
8. Only then test quantization, speculative decoding, dynamic LoRA, or prefill/decode disaggregation one at a time.

Each step should leave behind a benchmark and an operational invariant. Complexity that cannot demonstrate an SLO or cost improvement does not belong in the hot path.

Closing Thought

An LLM serving system is a scheduler wrapped around a large, stateful cache. The model matters, but so do the requests waiting beside it, the prefixes already resident on each pod, the GPUs that may not be available when asked for, and the half-finished streams that cannot be retried invisibly.

Google Cloud provides managed endpoints when the team wants to hand off most of that machinery, and GKE when the machinery itself needs to be designed. Choosing GKE is only justified if that control is used carefully: measure the workload, make admission explicit, route with inference state, keep warm capacity, and treat every rollout as a capacity event.

The endpoint is ready for production when it can say what happens to the next request during overload, a node loss, and a bad release—not when the first curl succeeds.

References

• Google Cloud, About AI/ML model inference on GKE.
• Google Cloud, Inference best practices on GKE.
• Google Cloud, About GKE Inference Gateway.
• Google Cloud, Serve an LLM with GKE Inference Gateway.
• Google Cloud, Predicted latency-based routing with GKE Inference Gateway.
• Google Cloud, Configure autoscaling for LLM workloads on GPUs.
• Google Cloud, GKE Inference Quickstart performance and cost analysis.
• Google Cloud, Deploy open models with a custom vLLM container on Vertex AI.
• Google Cloud, Cloud Storage FUSE CSI driver for GKE.
• Google Cloud, Accelerate model loading on GKE with Run:ai Model Streamer.
• Google Cloud, Accelerate AI/ML data loading with Hyperdisk ML.
• Google Cloud, Accelerator consumption options for GKE.
• Google Cloud, AI workload security best practices on GKE.
• Google Cloud, Configure Model Armor with GKE Inference Gateway.
• Google Cloud, Disaggregated LLM serving on multi-host TPUs.
• Google Cloud, Collect llm-d metrics with Managed Service for Prometheus.