Orthrus is a serving system that addresses low throughput and poor GPU utilization by executing embedding and generative models in a unified inference loop. It overcomes conflicting computational patterns through chunked embedding and optimized batch composition. This approach eliminates the inefficiencies of coarse-grained partitioning and dynamic workload bubbles.
- Unifies embedding and generation models in one inference loop
- Uses heterogeneous batching to boost GPU utilization
- Eliminates computational bubbles from isolated model execution
- Adapts to dynamic workloads without coarse-grained partitioning
