Continuous batching

A serving technique where in-flight requests are grouped into batches dynamically: when one response finishes, its slot in the batch is immediately taken by the next request in the queue, without waiting for all responses in the batch to end. The GPU stays busy almost continuously.