LLMs

Amazon SageMaker AI now enables serverless fine-tuning for the NVIDIA Nemotron 3.5 Lightning model using SFT, DPO, and RFT. This hybrid Mixture-of-Experts model features 3B active parameters within a 30B total parameter count, allowing for efficient domain adaptation. Engineers can tailor the model with proprietary data to reduce latency and costs while maintaining high accuracy on specific tasks.

• Serverless SFT, DPO, and RFT are now supported for Nemotron 3.5 Lightning.
• Model uses hybrid MoE architecture with 3B active out of 30B total parameters.
• Enables cost and latency reduction by tailoring models to specific domains.
• Labeled data improves task accuracy; preference data optimizes output alignment.

A new paper demonstrates a method to compress large language models beyond the traditional 1.58-bit limit associated with ternary quantization. This advancement allows for more extreme model compression while maintaining performance metrics that were previously unattainable at such low bit-widths. The work targets the physical limits of ternary weight representations to enable denser model storage and inference.

• Ternary LLMs can now exceed the historical 1.58-bit compression ceiling.
• Extreme quantization enables significantly denser model storage and reduced memory footprint.
• Performance degradation at ultra-low bit-widths is mitigated by the new method.
• Pushes the boundary of hardware-efficient inference for resource-constrained environments.

This paper addresses the fragmentation in LLM optimization reporting by constructing a Pareto atlas for cost, quality, and latency. The authors benchmarked 54 configurations of Qwen2.5-7B-Instruct on vLLM 0.12 across L4, A100, and H100 GPUs to create anchor data. They then calibrated a simulator that reproduces these measurements with under 1.5% drift. Quality evaluations covered FP16, AWQ 4bit, and FP8 variants on GSM8K, while sparse attention was assessed via simulation.

• Standardizes comparison of LLM optimizations across different hardware and quantization schemes.
• Validates a simulator against real vLLM 0.12 benchmarks with <1.5% cross-campaign drift.
• Provides concrete tradeoff data for FP8 weights/cache and AWQ 4bit on GSM8K tasks.
• Enables rapid identification of optimal configurations without exhaustive real-world testing.

The CapMem benchmark addresses the token and context limits of vision-language models in wearable assistants by testing if textual captions serve as efficient episodic memory. It introduces a dataset of 75 hours of egocentric video with 1,000 multiple-choice questions to evaluate Caption-Based Episodic Memory. Results show that for videos longer than 20 minutes, generating captions at 30-60 second intervals often yields higher accuracy than processing raw video frames directly.

• Caption-based retrieval beats raw video processing on >20min egocentric clips for most tested models.
• CapMem provides 75 videos and 1,000 QA pairs to standardize episodic memory evaluation.
• Textual summaries mitigate visual-token costs and long-context failures in VLMs.
• Wearable AI architectures benefit from pre-processing video into dense caption streams.
• Performance gains vary by model, but caption windows of 30-60s show consistent promise.

This paper addresses the cost of refreshing KV caches in RAG systems when underlying documents are edited. It demonstrates that while full reprefill is safe, in-place repair can save compute by selectively recomputing only affected segments. The authors compare training-free position-selection policies, finding that maintaining contiguity in the edited span significantly outperforms importance-based selection for repairing derived state dependencies.

• In-place KV cache repair avoids full reprefill costs but risks stale downstream states.
• Contiguous selection policies outperform importance-based ones for derived edits.
• All tested policies successfully repair direct edits under the specified budget.
• Derived edits clearly separate policy effectiveness, highlighting dependency complexity.
• The approach is training-free, making it broadly applicable across model families.

A new study reveals that complexity-based LLM routing systems systematically misclassify non-standard English variants, such as African American English and L2 writing, as lower-complexity queries. This bias is primarily driven by input length metrics, as these registers often omit function words, resulting in shorter strings that routing algorithms interpret as simpler tasks. Consequently, users receive responses from smaller, less capable models compared to meaning-equivalent standard English queries.

• Routing algorithms relying on input length disproportionately penalize non-standard English registers.
• Test your query router for demographic bias by comparing standard vs. AAVE/L2 equivalents.
• Shorter input length does not always correlate with lower cognitive or task complexity.
• Consider adding semantic complexity signals beyond raw token count to mitigate this gap.