LLMs

AWS has general availability for a built-in server-side Web Search tool in Amazon Bedrock, enabling OpenAI GPT-5.4 through 5.6 models to ground responses in current web data. This feature allows data residency within the AWS environment with zero data egress, eliminating the need for third-party search providers. Engineers can now enable this capability via a single API parameter, removing the overhead of vendor onboarding, separate billing, and additional security reviews.

• Native web grounding for OpenAI GPT-5.4/5.5/5.6 models via a single API parameter
• Zero data egress ensures responses stay within secured AWS environment
• Eliminates third-party vendor onboarding, API key management, and orchestration
• Removes need for additional compliance reviews for external search providers

SIRIN is a new toolkit and interactive web UI designed to detect contextual hallucinations in retrieval-augmented, agentic, and memory-grounded LLM systems. It unifies three detection paradigms—representation probing, uncertainty estimation, and judge-style verification—along with pre-generation query answerability assessment. The tool supports both white-box and black-box settings, enabling response- and span-level inspection through a centralized configuration and evaluation pipeline.

• Unifies representation probing, uncertainty estimation, and judge-style verification in one interface.
• Supports live analysis of context-query-answer triples with hallucination scores and span highlighting.
• Enables span-level inspection for precise identification of unsupported information in LLM outputs.
• Operates in both white-box and black-box settings for flexible integration into existing pipelines.

JudgeArena addresses the fragmentation in LLM-as-a-judge evaluation by unifying major benchmarks like AlpacaEval and Arena-Hard under a single open-source interface. The framework allows for swappable judge models and comprehensive metadata logging to improve transparency and reproducibility. This enables systematic studies on how design choices such as prompts, benchmarks, and inference backends impact model quality conclusions.

• Unifies AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard into one interface
• Supports swappable judge models to isolate variable impacts on evaluation results
• Logs comprehensive metadata to ensure transparency and reproducibility
• Enables systematic analysis of how benchmark and prompt choices affect rankings

CoT-Core addresses the high computational cost of LLM evaluation by introducing a training-free core question selection framework. Unlike prior methods that rely on heavy historical logs or suffer from surface-level lexical bias, this approach prompts LLMs to generate zero-shot Chain-of-Thought reasoning paths. These trajectories are projected into a latent space to cluster questions based on intrinsic logical equivalence rather than superficial wording.

• Eliminates cold-start bottlenecks associated with methods like Item Response Theory.
• Reduces evaluation overhead by clustering tasks on underlying logic, not just vocabulary.
• Operates without additional training, leveraging zero-shot CoT for clustering.
• Mitigates surface lexical bias to better capture the reasoning manifold of tasks.

An arXiv study analyzing 26,804 pairwise judgments from over 736 clinicians reveals that pairwise preference is a weak indicator of clinical safety in large language models. The research, sourced from the MOOVE platform, shows that models favored in head-to-head comparisons can still generate substantial rates of unsafe or misleading content. This disconnect suggests that relying on ranking signals alone may mask critical safety failures in clinical AI applications.

• Pairwise rankings fail to capture safety-critical errors, allowing unsafe models to rank highly.
• MOOVE data uses discrete [-2, +2] rubric scores to flag unsafe content, offering better granularity.
• Clinical AI evaluation must move beyond preference rankings to explicit safety rubric scoring.
• High preference scores do not guarantee absence of misleading or dangerous medical advice.