LLMs

LongGuard analyzes why safety guardrails trained on short text degrade with long inputs. It introduces SafetyNIAH, showing unsafe recall drops over 50% as context length increases from 0.25k to 32k. Mechanistic analysis attributes this to proportional dilution of the unsafe signal in attention heads rather than absolute length constraints.

• Safety guardrails suffer >50% recall drop on long contexts due to signal dilution
• Failure is proportional, not absolute, linked to attention mechanism behavior
• LongGuard provides training-free mitigation and mechanistic analysis tools
• Current guardrails are brittle and require re-evaluation for long-context use cases

A rigorous audit compares ConfLayers and SWIFT, two methods that skip transformer layers during LLM inference to save compute. Testing on Qwen2.5 models for reasoning and summarization shows SWIFT achieves superior accuracy in three of four scenarios. ConfLayers performs poorly, especially on complex reasoning tasks.

• SWIFT's self-speculative decoding offers better accuracy than confidence-gated early exits.
• Periodic layer skipping can maintain performance while reducing inference cost.
• ConfLayers shows significant accuracy deficits on GSM8K reasoning tasks.
• Results hold across Qwen2.5-0.5B and 1.5B model scales.

A new arXiv paper introduces CLAIMPROBE, a claim-level audit tool that decomposes deep-research reports to detect hallucinations, misattribution, and citation hygiene issues. The study reveals that standard rubric-based evaluations often mask these fine-grained factual failures, allowing pipelines to maintain stable scores while omitting key evidence. To address this, the authors propose CLAIMWRITER, a hierarchical writer that maps source facts to a query-derived outline for more faithful report generation.

• Rubric scores can hide factual errors; use claim-level audits like CLAIMPROBE for precision.
• CLAIMWRITER replaces the writer component to improve faithfulness across existing DR frameworks.
• Decomposing reports into claims exposes misattribution and missing evidence rubrics miss.

Simon Willison has released a detailed reference guide for ChatGPT's built-in tools and capabilities. The resource aggregates documentation on how the model interacts with various functions, offering a structured way to understand its operational limits and features. It serves as a quick lookup for developers integrating or testing these specific agent behaviors.

• Centralized reference for ChatGPT's native tooling capabilities
• Useful for debugging agent interactions and tool usage
• Clarifies functional boundaries within the ChatGPT environment
• Helps engineers design more reliable tool-calling workflows

Researchers demonstrate that a single 9 GB open-weight model, specifically Qwen2.5-14B quantized to 4-bit, can process the entire open Jeopardy! dataset comprising 529,939 clues from 1984 to 2025. This approach contrasts sharply with IBM Watson's 2011 era, where knowledge was locked in a curated billion-document corpus on specialized POWER7 hardware. The work highlights the shift from static, hardware-bound knowledge bases to portable, free, and queryable local LLM artifacts.

• A 9 GB 4-bit Qwen2.5-14B model handles 530k historical Jeopardy clues locally.
• Demonstrates portability advantage over IBM Watson's frozen, hardware-specific corpus.
• First known evaluation of a model against the complete 41-year Jeopardy dataset.
• Validates open-weight models for dense, long-tail factual retrieval tasks.