Batch Inference

Batch inference runs a model on a large accumulated set of inputs at once, on a schedule, rather than responding to individual requests as they arrive — the right choice whenever real-time responses aren't actually required.

When Batch Inference Is the Right Choice

Code — A Simple Batch Inference Job

import torch
from torch.utils.data import DataLoader

model.eval()
dataset = load_full_dataset_to_score()   # e.g. all users, all documents
loader = DataLoader(dataset, batch_size=256, shuffle=False)

all_predictions = []
with torch.no_grad():
    for batch in loader:
        predictions = model(batch)
        all_predictions.append(predictions)

all_predictions = torch.cat(all_predictions)
save_predictions_to_storage(all_predictions)   # e.g. write to a database or data warehouse

Why Batch Inference Is Often Simpler and More Efficient

Without a real-time latency constraint, batch inference can use much larger batch sizes than real-time serving typically allows, maximizing hardware (especially GPU) utilization and overall throughput. It also avoids the operational complexity of maintaining an always-available, low-latency serving endpoint — the job simply runs, completes, and the results are stored for later use.

Batch Inference Infrastructure Patterns

Batch jobs are commonly scheduled via a workflow orchestrator (e.g. Airflow, or a simple cron job) and run on compute that only needs to exist for the duration of the job — a good fit for the cost-efficient, transient compute patterns available on most cloud platforms, since there's no need to keep an endpoint running and staffed with capacity around the clock.

Common Mistakes

• Building real-time serving infrastructure for a use case that would be perfectly well served, more simply and cheaply, by a scheduled batch job — real-time serving carries meaningfully more operational complexity and cost that should be justified by an actual need for low latency.
• Using a small batch size for a batch inference job out of habit (carried over from real-time serving code) when there's no latency constraint requiring it — this leaves throughput on the table unnecessarily.

Interview Relevance

Q: "Why would you choose batch inference over real-time inference for a use case like nightly product recommendations, even though real-time serving is technically possible?" Real-time serving carries meaningfully more operational complexity and cost — maintaining an always-available, low-latency endpoint — that isn't justified when the actual use case (recommendations refreshed once a day) doesn't require immediate responses. Batch inference can also use much larger batch sizes than real-time serving typically allows, since there's no per-request latency constraint, maximizing hardware utilization and overall throughput for the same total workload.

Practice Question

Why can batch inference typically use larger batch sizes than real-time inference serving, and why does that matter for throughput?

Related DL Notes

• Real-Time Inference
• Inference Throughput
• Cloud Deployment