Inference Throughput

While Inference Latency measures how long a single request takes, inference throughput measures how many requests a serving system can handle per unit time — a related but distinct concern, and the two are often in direct tension.

The Latency-Throughput Tradeoff

Batching multiple requests together before a single forward pass (as introduced in GPU Deployment) improves throughput by better utilizing hardware parallelism, but adds latency to individual requests that must wait briefly for a batch to accumulate. Maximizing throughput and minimizing latency generally pull in opposite directions — the right balance depends entirely on the application's actual latency tolerance.

Code — Dynamic Batching for Throughput

import asyncio
import torch

class DynamicBatcher:
    def __init__(self, model, max_batch_size=32, max_wait_ms=10):
        self.model = model
        self.max_batch_size = max_batch_size
        self.max_wait_ms = max_wait_ms
        self.queue = asyncio.Queue()

    async def predict(self, x):
        future = asyncio.Future()
        await self.queue.put((x, future))
        return await future

    async def batch_loop(self):
        while True:
            batch, futures = [], []
            deadline = asyncio.get_event_loop().time() + self.max_wait_ms / 1000

            while len(batch) < self.max_batch_size and asyncio.get_event_loop().time() < deadline:
                try:
                    x, fut = await asyncio.wait_for(self.queue.get(), timeout=0.001)
                    batch.append(x); futures.append(fut)
                except asyncio.TimeoutError:
                    continue

            if batch:
                inputs = torch.stack(batch)
                with torch.no_grad():
                    outputs = self.model(inputs)
                for fut, out in zip(futures, outputs):
                    fut.set_result(out)

Incoming requests accumulate briefly (bounded by max_wait_ms) into a batch, then run together as one GPU forward pass — significantly better throughput than one-at-a-time processing, at the cost of a small, bounded added latency per request.

Measuring Throughput

import time

def measure_throughput(model, batch_generator, duration_seconds=30):
    start = time.time()
    total_processed = 0

    while time.time() - start < duration_seconds:
        batch = next(batch_generator)
        with torch.no_grad():
            _ = model(batch)
        total_processed += batch.shape[0]

    elapsed = time.time() - start
    print(f"Throughput: {total_processed / elapsed:.1f} requests/second")

Choosing the Right Balance for the Application

Common Mistakes

• Maximizing batch size for throughput without regard for the resulting added latency, in a use case where users are actually latency-sensitive — optimizing the wrong metric for the actual application.
• Measuring throughput using unrealistically small or synthetic batches that don't reflect actual production traffic patterns, producing a misleading capacity estimate.

Interview Relevance

Q: "Explain the tradeoff between inference latency and inference throughput, and how batching affects each." Batching multiple requests together before a single forward pass improves throughput by better utilizing hardware parallelism and amortizing fixed per-call overhead across more work — but it requires individual requests to wait briefly while a batch accumulates, adding latency to each one. This makes latency and throughput a direct tradeoff, tuned via parameters like maximum batch size and maximum wait time: favoring larger batches and longer wait windows increases throughput at the cost of higher per-request latency, and the right balance depends entirely on the specific application's actual latency tolerance.

Practice Question

Why would a batch scoring system that processes millions of records overnight likely prioritize throughput over minimizing individual-request latency?

Related DL Notes

• Inference Latency
• GPU Utilization
• Batch Inference