LLMs

This paper challenges the assumption that all tokens require equal computation by measuring the actual compute needed for generation. Using a Mixture-of-Agents setup with fifteen models, the authors determine the smallest model capable of correctly predicting each token as the metric for sufficient compute. This approach provides an upper bound on necessary inference cost per token, validating the potential efficiency gains of adaptive decoding methods.

• Per-token compute varies significantly based on difficulty, contradicting uniform allocation models.
• Mixture-of-Agents framework effectively identifies the minimum model capacity required per token.
• Findings support speculative decoding and routing strategies that skip unnecessary calculations.
• Sufficient compute metrics offer a concrete baseline for optimizing inference resource allocation.

Researchers deployed CourseChat, an on-premises RAG tutoring system for business education, using twin-edge AI hosts with FastAPI, a local vector database, and Ollama-served LLMs. The system isolates six courses by reference number while sharing infrastructure behind a campus gateway. Evaluation included model bake-offs and fidelity audits, noting that larger models failed to meet specific constraints.

• On-prem RAG requires dedicated edge hardware for isolation and control.
• Course isolation is handled via CRN keys in the local vector store.
• Larger LLMs may fail fidelity checks in constrained educational contexts.
• Integration targets Moodle via a FastAPI service layer.
• Ollama serves as the inference engine for local model management.

WakeKV challenges the static classification of attention heads by observing that most heads change their reading behavior during generation. The system implements a reactive policy that moves cooling heads to a recoverable CPU reservoir instead of freezing or permanently evicting them. Evaluations across multiple model sizes and regimes show improved miss rates compared to frozen or destructive eviction methods.

• Static head classification fails because attention heads often change behavior during generation.
• Reactive residency moves cooling heads to a recoverable CPU reservoir.
• Avoids permanent eviction or freezing, allowing state restoration when needed.
• Consistently improves miss rates across five model-regime combinations.
• Validated on 1.5B-8B models in retrieval, chain-of-thought, and recall tasks.

Antirez has released DwarfStar, a specialized native inference engine designed to run DeepSeek V4 and GLM models locally on consumer hardware via Metal, CUDA, and ROCm. The project focuses narrowly on specific models like DeepSeek V4 Flash, PRO, and GLM 5.2/5.3, using proprietary GGUF files rather than acting as a general GGUF runner. It prioritizes integration testing of core features such as prompt rendering, tool calls, and KV state management.

• Optimized for consumer hardware using Metal, CUDA, and ROCm backends.
• Supports DeepSeek V4 Flash/PRO, GLM 5.2/5.3, and Qwen3.8 Flash Next.
• Requires project-specific GGUF files; not a general-purpose GGUF runner.
• Focuses on native performance rather than broad compatibility.
• Integrates testing for prompt rendering, tool calls, and KV state.

This paper introduces Tropical Reinforcement Learning, which shifts from maximizing expected return (sum of probabilities) to a max-algebra approach. The authors argue that classical summation obscures which specific solution steps succeeded and causes catastrophic forgetting of valid but untried paths. By using a max-based formulation, the method better supports compositional reasoning where correct answers are assembled from disparate, often separately failed, reasoning attempts.

• Solves the 'sum obscures success' problem in LLM RL where total probability hides specific winning steps.
• Prevents forgetting of valid solution components that were never combined in a single successful trajectory.
• Max-algebra formulation aligns better with compositional reasoning tasks requiring step assembly.
• Offers a theoretical alternative to standard expected return maximization for complex reasoning models.