AI / ML

Existing quantization-aware training often fails because it optimizes on fixed data while the deployed model generates its own prefixes, causing quantization errors to push the model into unseen states. OnlineQAT addresses this by first initializing a low-bit model via block-wise QAT and then performing on-policy distillation on the student's own generated responses. A frozen full-precision teacher provides a reverse-KL training signal at each visited prefix to correct these distributional shifts.

• Fixes accuracy loss when deploying LLMs compressed below four bits.
• Uses on-policy distillation to match the model's actual inference behavior.
• Frozen teacher applies reverse-KL signals to correct student prefix drift.
• Two-stage process: block-wise initialization followed by self-generated distillation.

AdaGuard introduces an adaptive LLM-as-a-Judge framework that dynamically enforces safety and compliance policies without requiring frequent model updates. Built via supervised fine-tuning and reinforcement learning, it allocates reasoning budgets based on runtime needs, addressing the rigidity of fixed guardrail solutions. This approach allows enterprise applications to accommodate diverse risk postures and evolving policies with greater transparency and flexibility.

• Eliminates need for frequent model updates by generalizing to user-defined policies at runtime.
• Uses SFT and GRPO reinforcement learning to build adaptive reasoning capabilities.
• Dynamically allocates reasoning budgets to balance safety checks against latency constraints.
• Addresses rigidity in current guardrails by supporting evolving enterprise risk postures.

LRCC introduces a method for low-rank model compression that replaces fixed-rank inference with conditional computation. It trains lightweight routers per Transformer block to select among nested low-rank paths based on input tokens, while keeping the underlying low-rank factors frozen. Evaluations on Llama and Qwen models show performance gains within the same active-parameter budget compared to conventional static compression.

• Replaces static low-rank compression with dynamic, token-dependent routing for better efficiency.
• Only lightweight routers are trained; low-rank factors remain frozen to reduce training cost.
• Improves language modeling and zero-shot performance without increasing average active parameters.
• Demonstrated effectiveness on major open models like Llama and Qwen.

This research investigates whether serving factors like batch size, hardware, and inference engines cause output variations in text classifiers, extending prior findings from text generation models. The study systematically measures serving-context non-invariance to determine if floating-point non-associativity and kernel selection affect classification predictions. It aims to clarify the extent to which implementation-level differences alter results even when model parameters and sampling randomness remain fixed.

• Text classification outputs may vary based on serving infrastructure, not just model weights.
• Batch size and composition can introduce non-determinism in classifier predictions.
• Hardware differences and inference engines contribute to floating-point non-associativity.
• Deterministic inference is critical for reliable and trustworthy ML systems.
• Practitioners should audit serving contexts when reproducibility is required.

This research paper demonstrates that early-stage Mixture-of-Experts models develop correlated token routing patterns, allowing for optimized expert placement. By leveraging these correlations through token shuffling and correlated placement, the study significantly reduces the all-to-all communication overhead that currently consumes up to 60% of training steps on AMD MI300X clusters. The approach improves training efficiency without requiring changes to the core model architecture.

• All-to-all collectives dominate MoE training time, reaching 60% at top-6 routing.
• Early pretraining routers exhibit strong correlation in expert selection patterns.
• Correlated expert placement exploits these patterns to minimize communication volume.
• Token shuffling further optimizes dispatch efficiency during the forward pass.
• Significant speedup potential for large-scale MoE clusters without architectural changes.

Hugging Face and Liquid AI have released open-source multimodal decision models designed specifically for edge deployment. These models aim to bring advanced reasoning capabilities to resource-constrained environments without relying on cloud inference. The release provides developers with accessible tools to integrate complex multimodal logic directly into edge devices.

• Open models enable local multimodal reasoning without cloud dependency.
• Designed for edge hardware constraints and low-latency requirements.
• Collaboration between Hugging Face and Liquid AI expands open AI tooling.
• Supports decision-making workflows on devices with limited compute.

Recent open-looped LMs fail to deliver promised savings because each iteration requires a separate KV cache level, forcing full computation. The authors introduce a 'best-available' KV caching strategy that breaks this static processing pattern. This method allows models to dynamically adjust depth per token, achieving significant efficiency gains without sacrificing performance.

• Prior looped LMs wasted resources by caching KV states for every iteration, negating dynamic benefits.
• New KV strategy enables true dynamic computation, varying depth based on token difficulty.
• Results show up to 30% reduction in FLOPs and KV memory usage compared to static approaches.
• Performance remains high, retaining full-depth accuracy while optimizing for easier tokens.