AI / ML

AWS has released a Kubernetes-native addition for SageMaker HyperPod that replaces basic round-robin load balancing with real-time inference signal routing. Deployed as an EKS managed add-on, it requires zero application changes while terminating HTTPS traffic through a single Envoy endpoint. This architecture enables one gateway to serve multiple models from a single URL by reading the model name directly from request bodies.

• Routes traffic based on real-time inference signals, not just round-robin.
• Reduces first-token latency by up to 82% in mixed-hardware scenarios.
• Cuts p99 TTFT by 97–98% during burst traffic events.
• Deploys as an EKS managed add-on with no client-side code changes.
• Uses Envoy to expose a single private endpoint per cluster for all models.

NVIDIA has released Model-Optimizer, a unified library designed to compress and accelerate deep learning models using techniques like quantization, distillation, pruning, and neural architecture search. The tool accepts Hugging Face, PyTorch, or ONNX models as input and targets downstream deployment frameworks such as TensorRT-LLM, TensorRT, and vLLM. Its primary goal is to optimize inference speed for state-of-the-art models through speculative decoding and sparsity.

• Supports Hugging Face, PyTorch, and ONNX model inputs for broad compatibility.
• Integrates with TensorRT-LLM, TensorRT, and vLLM for optimized deployment.
• Combines quantization, pruning, distillation, NAS, and speculative decoding in one library.
• Aims to accelerate inference speed for deep learning models at scale.