Automation / DevOps / IaC

RagTester is an automated framework designed to test the reliability of Retrieval-Augmented Generation systems by evaluating the interaction between generative models, embeddings, and retrieval mechanisms. It generates specific test inputs, documents, and expected outputs, then uses an LLM as a judge to evaluate the system's answers. The approach specifically targets complex passages, unsupported queries, and document coverage to ensure comprehensive validation.

• Automates the creation of test inputs, documents, and ground truth for RAG pipelines.
• Uses an LLM-as-a-judge to evaluate generated answers against expected outputs.
• Targets edge cases like unsupported queries and complex document passages.
• Validated across 24 configurations of eight LLMs and six embedding models.

AWS Transform continuous modernization is now generally available, allowing engineering teams to analyze and fix technical debt across large-scale codebases. Users can integrate with GitHub, GitLab, and Bitbucket to run scheduled or on-demand analyses covering security, agentic readiness, and custom criteria. The service automatically generates branches and pull requests for findings that have associated remediation strategies.

• Integrate GitHub, GitLab, or Bitbucket to scan for technical debt and security issues.
• Automate remediation by creating branches and pull requests for fixable findings.
• Prioritize work across technical debt, agentic readiness, and custom criteria.
• Run analyses on demand or via recurring schedules directly from the web app.

A new study characterizes AI coding agent workloads using 761M LLM calls from 3.2M GitHub Copilot users. The analysis shows sessions consist of sparse user turns that trigger autonomous agent loops with frequent tool execution. This structure results in high KV cache hit rates within turns but significant drops across turn boundaries.

• Agentic coding differs fundamentally from chatbots due to interleaved LLM inference and tool execution loops.
• KV cache hit rates average 90% within a single turn but drop to 55% across turn boundaries.
• Sparse user-initiated turns unfold into complex autonomous agent loops requiring distinct system optimizations.
• Production-scale data reveals unique workload properties with direct implications for inference infrastructure.

Nova is an automated end-to-end Just-In-Time compiler built on MLIR to bridge the gap between flexible tensor frameworks and physical hardware constraints. It captures eager executions to provide whole-graph visibility, enabling operation fusion, memory hierarchy optimization, and register-level tuning. This approach aims to maximize hardware utilization by offering granular control that eager execution models inherently lack.

• Captures eager executions to enable whole-graph optimization unavailable in standard frameworks.
• Fuses operations across boundaries to reduce overhead and improve throughput.
• Tunes execution down to the register level for maximum hardware utilization.
• Uses MLIR to provide granular control over hardware mapping and memory hierarchies.