Continuous Batching
Iteration-level request scheduling for LLM servers
Performance
In plain English
Letting new passengers hop onto the train at every stop instead of waiting for the entire train to empty before loading new people.
Technically
An iteration-level batching technique (Yu et al., Orca 2022) that dynamically inserts new incoming requests and evicts finished requests at each decode token step, maximizing GPU Tensor Core utilization.