LLMs

This paper addresses the inefficiency of processing visual tokens in multimodal large language models by improving token pruning strategies. While prior work uses attention from a fixed middle layer to identify important tokens, the optimal layer varies significantly per sample. The authors propose learning to predict the appropriate middle-layer attention dynamically, rather than relying on a static layer selection.

• Fixed middle-layer attention is suboptimal because relevance varies by input sample
• Dynamic prediction of attention layers improves visual token pruning accuracy
• Reduces inference cost without sacrificing multimodal reasoning performance
• Bypasses the need for computationally expensive full attention maps

EntropyMoE introduces a Mixture-of-Experts architecture for byte-level LLMs that replaces uniform dense feed-forward layers with Top-K expert routing. This design allows the model to adapt its computational capacity based on the semantic complexity and byte coverage of dynamic patches. By treating each dynamic patch as the unit for expert selection, the system optimizes workload distribution across varying levels of granularity.

• Replaces dense FFN layers in byte-patch Transformers with sparse Top-K expert routing.
• Adapts model capacity dynamically to patch semantics rather than treating all patches equally.
• Uses byte coverage of dynamic patches to determine individual contribution to workload accounting.
• Enables more efficient processing for tokenizer-free LLMs by matching compute to complexity.

The LUCID method applies LLM reasoning to detect cohesive groups in graphs without requiring labels or training phases. It draws inspiration from phase-transition kinetics, allowing complex structures to emerge through initialization and merging steps. This approach aims to balance performance with interpretability, addressing limitations of both classic objective-driven methods and deep learning alternatives.

• Eliminates the need for labeled training data, enabling fully unsupervised graph analysis.
• Uses LLM world knowledge to provide interpretable community structures without black-box models.
• Employs a phase-transition kinetic model to drive initialization and merging of graph entities.
• Offers a training-free alternative to deep learning methods that often lack interpretability.

A new study compares behavioral steerability across six frontier language models from different developers. Researchers evaluated 300 paired base and steered items across three categories: values-conflict, reasoning-elicitation, and reasoning-suppression. Using 24,480 blind peer judgments scored by leave-one-out consensus, the analysis reveals significant variation in how models shift behavior when explicitly steered.

• Models differ not just in the degree of steering but in the mode of their response.
• Variations stem from distinct training data, objectives, and safety pipelines.
• Blind peer judging with consensus scoring provides robust behavioral measurement.
• Steering pressure exposes fundamental architectural or training differences across vendors.

TEXAS improves downstream adaptation of Mixture-of-Experts LLMs by identifying task-relevant experts through correctness-conditioned discovery rather than aggregate routing stats. It compares expert activations on correctly solved versus failed instances to pinpoint which experts drive success. The method then allocates token-level supervision based on these findings to refine model performance.

• Moves beyond aggregate routing statistics to identify experts linked to actual task success.
• Uses activation differences between correct and failed predictions for supervision allocation.
• Enables more precise fine-tuning of MoE models by focusing on token-level signals.
• Reduces noise in adaptation by filtering out experts not associated with correct outcomes.