LLMs

Janas is a new system that enables the execution of large language models exceeding available system memory by streaming expert parameters directly from SSD storage. This approach allows for the inference of models that would otherwise be impossible to load into RAM, leveraging high-speed storage I/O to overcome memory constraints.

• Enables LLM inference for models larger than host RAM via SSD streaming
• Targets expert-based architectures to optimize data transfer efficiency
• Reduces hardware cost barrier for running massive language models
• Relies on fast storage throughput to mitigate memory bottlenecks
• New open-source project by prabanta-dev on GitHub

AirLLM drastically cuts inference memory requirements by streaming weights layer-by-layer, allowing 70B parameter models to run on a single 4GB GPU. The tool supports massive architectures like Kimi K3 (2.8T) and Qwen3.8-Flash-Next (125B) with minimal VRAM, eliminating the need for quantization or pruning. It also recently added training capabilities for large models on low-end hardware by keeping adapters on the GPU while streaming frozen weights.

• Run 70B+ LLMs on 4GB GPUs without quantization, distillation, or pruning.
• Supports extreme scale models like Kimi K3 (2.8T) and DeepSeek-V3 (671B).
• New training feature streams frozen weights layer-by-layer to save VRAM.
• Enables 125B model training on consumer cards like RTX 3060 Ti.
• Works across multiple platforms including MacOS and standard Linux environments.

This Hugging Face blog post addresses the scenario where pre-trained models fail to meet specific requirements. It guides engineers through the process of fine-tuning or training custom solutions using ML Intern. The focus is on practical implementation for developers needing bespoke AI capabilities.

• Pre-trained models may not always fit specific use cases
• Custom model development is a viable alternative
• Hugging Face provides tools for building from scratch
• Target audience includes engineers needing bespoke AI

A new study demonstrates that it is mathematically possible to construct AI models that conceal specific functions while appearing benign. This provable hiding mechanism allows models to retain secret behaviors that are undetectable through standard auditing or evaluation methods. The work establishes a formal framework for these hidden capabilities rather than relying on empirical observation alone.

• AI safety auditing faces a fundamental provable limitation: hidden capabilities cannot always be detected.
• Standard evaluation metrics may fail to reveal models with intentionally concealed functions.
• New alignment strategies must account for mathematically provable hiding mechanisms.
• This work shifts the focus from empirical detection to theoretical bounds of model transparency.

Open WebUI is a self-hosted, extensible platform designed to run entirely offline while providing a user-friendly interface for AI models. It supports both local inference via Ollama and cloud-based models through OpenAI-compatible APIs, enabling provider-agnostic usage. The project emphasizes ease of deployment through multiple methods including pip, Docker, and Kubernetes.

• Supports both local Ollama and remote OpenAI-compatible APIs for flexible model access
• Can run entirely offline, suitable for air-gapped or privacy-sensitive environments
• Deployable via pip, uv, Docker, or Kubernetes with Helm and Kustomize support
• Provides a unified, extensible UI to manage multiple AI model providers
• Available for enterprise use with dedicated sales and support options