Inference Latency (Research)

This note revisits inference latency specifically from a research reporting perspective — how latency should be measured and presented rigorously in papers, complementing the production-focused coverage in Inference Latency.

Why Rigorous Latency Reporting Matters in Research

When a paper claims a new method is "faster" than prior work, that claim needs the same rigor as any other research result — a fair, controlled comparison, on comparable hardware, measured correctly. Sloppy or selectively favorable latency reporting can just as easily mislead readers as sloppy accuracy reporting.

What Rigorous Latency Reporting Requires

Code — A Rigorous Research-Grade Latency Benchmark

import time
import numpy as np
import torch

def benchmark_latency(model, input_tensor, num_warmup=10, num_trials=100):
    model.eval()
    with torch.no_grad():
        for _ in range(num_warmup):
            _ = model(input_tensor)
        torch.cuda.synchronize()

        latencies = []
        for _ in range(num_trials):
            start = time.perf_counter()
            _ = model(input_tensor)
            torch.cuda.synchronize()
            latencies.append((time.perf_counter() - start) * 1000)

    return {
        "mean_ms": np.mean(latencies),
        "std_ms": np.std(latencies),
        "p50_ms": np.percentile(latencies, 50),
        "p99_ms": np.percentile(latencies, 99),
        "hardware": torch.cuda.get_device_name(0),
        "batch_size": input_tensor.shape[0],
    }

results = benchmark_latency(model, dummy_input)
print(results)   # a complete, reproducible latency report

Reporting Latency Fairly Relative to Accuracy

A method that's faster but meaningfully less accurate isn't automatically "better" — rigorous research typically reports latency alongside accuracy at multiple operating points, or explicitly acknowledges the specific accuracy-latency tradeoff being made, rather than presenting a speed improvement in isolation as unambiguously positive without this essential context.

Common Mistakes

• Comparing latency measured on different hardware configurations for different methods — this produces a fundamentally unfair, non-comparable result, regardless of how the numbers are presented.
• Reporting a single latency number without specifying batch size, sequence length, or hardware — this makes the result difficult to interpret correctly or reproduce independently.

Interview Relevance

Q: "What would you check before trusting a paper's claim that its new method is '3x faster' than a prior approach?" Whether both methods were measured on identical hardware (a fundamental requirement for a fair comparison), whether the comparison used the same batch size and input configuration for both, whether warm-up runs were excluded from timing, and whether the reported number reflects an average across multiple trials with reported variance rather than a single, potentially unrepresentative measurement. A "3x faster" claim without this context and rigor should be treated with appropriate skepticism until these details can be verified.

Practice Question

Why is reporting latency alongside accuracy, rather than in isolation, important when comparing two methods with different speed-accuracy tradeoffs?

Related DL Notes

• Inference Latency
• FLOPs
• Memory Requirements