LLMs

Safety guardrails trained on short text suffer a 50% drop in unsafe recall when context length increases to 32k tokens. LongGuard identifies this as a 'Safety Needle-in-a-Haystack' problem where the unsafe signal is proportionally diluted rather than lost to absolute length. The framework uses a three-layer attention-logit-behavior analysis to pinpoint the mechanistic failure in attention mechanisms across 15 mainstream guardrails.

• Guardrails degrade significantly on long contexts, losing over 50% unsafe recall.
• Failure is caused by proportional dilution of the unsafe signal, not just length.
• LongGuard provides a training-free mitigation framework for these edge cases.
• Attention logit analysis reveals specific mechanistic failure points in models.
• Evaluation should include SafetyNIAH tests, not just short-text benchmarks.

This paper examines the trade-off between inference overhead and performance in language models, specifically testing if a token budget threshold exists where planning pays off. Using GPT-5.4 mini on financial reasoning tasks, the study compares a monolithic approach against a verified search architecture across 14 budget tiers. Results indicate that low-budget inference suffers from planning costs, while higher budgets allow structural benefits to outweigh the token consumption.

• Structured inference (search/verify) consumes tokens needed for reasoning.
• A token threshold exists below which overhead hurts performance.
• Above the threshold, planning and verification improve accuracy.
• Monolithic calls are cheaper but lack revision capabilities.
• Financial reasoning tasks benefit from structure at higher budgets.

Large output embedding matrices cause memory bandwidth bottlenecks during autoregressive decoding. This work replaces the dense vocabulary projection with an HNSW-based vector index for maximum inner product search. The method retrieves a small candidate set of high-scoring tokens and scatters logits into a sparse tensor for inference.

• Replaces dense output projection with HNSW vector index to reduce memory bandwidth pressure.
• Significantly accelerates output projection and end-to-end decoding on CPU for Gemma, Llama, and Qwen models.
• Integrates via scattering retrieved logits into sparse full-vocabulary tensors for compatibility.

The Nemotron 3.5 Content Safety Moderator is a compact 4B parameter vision-language model designed to classify safety risks across user prompts, images, and assistant responses. It supports 12 languages and handles diverse inputs including documents and screenshots, addressing gaps in existing text-only guardrails. The model provides both latency-sensitive safety labels and concise reasoning traces to facilitate custom policy control.

• 4B parameter size offers a compact footprint for vision-language safety moderation.
• Supports 12 languages and multimodal inputs like images and documents.
• Generates safety labels and concise reasoning traces for auditability.
• Addresses limitations of existing text-only guardrails in complex deployments.