FairyFuse is a new inference system designed for CPU-only platforms, enabling multiplication-free execution of large language models. It uses ternary weights ({-1, 0, +1}) to replace floating-point multiplications with conditional additions and subtractions, significantly reducing memory bandwidth bottlenecks and offering up to 16x weight compression.
This research investigates harness engineering as an inference-time technique for large language model (LLM) agents, focusing on improving long-term performance via task decomposition and guided execution. It quantifies how design elements like workflow granularity and guidance impact performance, revealing common failure modes such as over-decomposition and hallucinated execution.
A new research paper introduces a "sleep phase" for language models, consolidating context into fixed-size memory layers. This method significantly reduces quadratic inference costs and enhances performance on long-horizon tasks.
This article guides users on deploying Microsoft's Phi-3.5 Mini LLM with vLLM on a $5/month DigitalOcean Droplet. The setup offers lightweight production inference for under $60 annually, aiming to drastically cut costs compared to expensive commercial LLM APIs.
This session teaches how to deploy any Hugging Face model using Goose and Together's Dedicated Container Inference. It aims to simplify setup complexity, enabling models to run quickly in a production-grade GPU environment.
This article explores how continuous batching improves LLM inference efficiency, addressing the issues of static batching. It details dynamic scheduling and ragged batching to process multiple requests simultaneously.
Mamba-3 is introduced as a new open-source State Space Model (SSM) built for inference. It boasts superior performance, being faster than Transformers at decode and stronger than Mamba-2.
AI Hub Phase 8 announces the addition of DeepInfra and Liquid AI, expanding its provider count to 33. DeepInfra is highlighted for its cost-effectiveness and OpenAI-compatible endpoint, while Liquid AI introduces a novel, non-transformer architecture for long-context tasks.