Benchmarking LLM Serving Stacks: Production Patterns and Realistic Load Testing

If you've ever deployed a large language model and seen it crumble the moment more than five users hit the endpoint, you know that "it works on my machine" is a dangerous lie in AI infrastructure. Most teams make the mistake of measuring performance in a vacuum-sending a few prompts and celebrating the speed. But in the real world, LLM serving stacks face erratic burst patterns, varying prompt lengths, and the dreaded "cold start" that can tank your user experience. To actually know if your stack can handle production, you need to stop guessing and start benchmarking with realistic loads.

The Big Divide: Load Testing vs. Performance Benchmarking

Before you start spinning up GPUs, you need to understand that you're actually measuring two different things. It's easy to confuse them, but treating them as the same is how you end up with a system that is fast for one person but unusable for a thousand.

Performance benchmarking is all about the "how fast." It isolates the model's efficiency. You're looking at how quickly a single request is processed or how many tokens the engine can spit out per second. It's a controlled environment where you're testing the software's theoretical ceiling. On the other hand, load testing asks "how many." This is where you simulate a crowd. You aren't just checking if the model is fast; you're checking if the server crashes, if the autoscaling kicks in fast enough, or if the network latency becomes the actual bottleneck.

Think of it like testing a car. Performance benchmarking is the 0-60 mph sprint on a dry track. Load testing is putting that car in rush-hour traffic in downtown New York to see if it can actually get anywhere. You need both to ensure your deployment doesn't just look good in a demo but survives the real world.

Where You Measure Matters: Server-Side vs. Client-Side

One of the most common pitfalls in LLM benchmarking is ignoring the network. Depending on where your benchmark script lives, you're getting two completely different sets of data.

If you run your benchmarks on the same machine as your model server-what we call server-side benchmarking-you're essentially stripping away the internet. This is great for hardware characterization. It tells you exactly what your GPU and memory are capable of without the noise of a slow router or a congested VPC. If you want to compare an H100 against an A100, do this.

But if you want to know what your users feel, you must use client-side benchmarking. This involves running requests from a separate instance, mirroring the actual path a user's prompt takes. This captures the TCP handshake, the load balancer overhead, and the actual transit time. The golden rule here is simple: server-side tells you the maximum hardware capability; client-side tells you the actual user experience.

Key Metrics That Actually Move the Needle

Forget generic "latency" numbers. In the world of generative AI, the way tokens are delivered makes the difference between a tool that feels magical and one that feels broken. You need to track these specific indicators:

• Time-to-First-Token (TTFT): This is the most critical metric for perceived speed. It's the gap between when a user hits "Enter" and when the first word appears. If this is too high, users think the app is frozen.
• Tokens Per Second (TPS): This measures the actual throughput. While TTFT is about the start, TPS is about the flow. If the TPS is lower than the average human reading speed, the experience feels sluggish.
• Queries Per Second (QPS): This is your volume metric. How many distinct requests can your stack handle every second before the queue starts backing up?
• Requests Per Second (RPS): Similar to QPS, but often used in the context of raw network load and API gateway capacity.

Designing Realistic Production Patterns

Synthetic tests that send the same "Hello, how are you?" prompt 1,000 times are useless. They allow the system to cache the response, giving you fake, inflated numbers. To get a real picture, you need to implement a tiling pattern or a dynamic prompt strategy.

A realistic test should cycle through a diverse set of contexts. For example, if you're building a coding assistant, your benchmarks should mix short logic questions with massive 10k-token file uploads. This forces the server to manage KV Cache (Key-Value Cache) evictions and offloading, which is where most serving stacks fail. When the cache fills up, the system has to decide what to dump or how to swap memory, and that's when you'll see your TTFT spike.

Another production reality is the "warm-up" phase. Most LLM engines have a cold-start period where the first few requests are slow as the GPU kernels initialize. If you include these in your average, your data is skewed. Professional setups ignore the first 30-60 seconds of data and focus on the "steady state"-the performance level the system maintains after it's fully warmed up.

Choosing Your Serving Stack: vLLM, SGL, and Others

The infrastructure you choose dictates your performance ceiling. For a long time, vLLM has been the gold standard due to its PagedAttention mechanism, which optimizes how memory is handled for concurrent requests. It's reliable, open-source, and handles batching efficiently.

However, newer contenders like SGL (Structured Generation Language) are entering the fray. Early data suggests that in specific high-concurrency scenarios, SGL can outperform vLLM by optimizing the way it handles structured outputs. But don't just follow the leaderboard; the "best" stack depends on your specific model. A stack that excels with a 7B parameter model might struggle with a 70B model due to different memory pressures.

If you're dealing with massive models, you'll need to look at parallelism strategies. Tensor Parallelism (TP) splits the model's weights across multiple GPUs, while Pipeline Parallelism (PP) splits the model layers. Finding the right balance-like a TP2:PP1 configuration-is often the only way to hit a P90 latency target of 2 seconds for a large-scale deployment.

The Iteration Loop: Shortening the Feedback Cycle

You won't find the perfect configuration on your first try. You'll change the batch size, tweak the max sequence length, and adjust your GPU memory utilization, then run the test again. If this process takes an hour, you'll stop optimizing. If it takes five minutes, you'll find the peak.

To speed this up, invest in developer experience (DevEx) for your infra. Use automation scripts (like a simple startup_and_benchmark.sh) that boot the server, run the load test, and dump the results into a CSV. Also, leverage weight caching. There is nothing more wasteful than spending twenty minutes redownloading a 100GB model every time you want to test a new concurrency setting.

Combine this with real-time monitoring using Prometheus and Grafana. This allows you to see the correlation between a spike in GPU memory and a drop in TPS in real-time, rather than digging through logs after the crash.