Inkling 2 Mini on vLLM: Day-0 Support for a Looped ModelThinking Machines announced Inkling 2 Mini today. vLLM supports it and its variable-loop research checkpoint, TML-d24-varloop-research , on day 0 on NVIDIA Blackwell, Hopper and DGX Spark. Inkling 2 Mini is a looped model: its middle 12 layvLLM11/20 08:00
Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLMIntroduction Interactive LLM applications depend on low decode latency. A coding agent must finish generating its current action before it can call the next tool, so decode sits on the task’s critical path. Serving these workloads at low covLLM11/18 08:00
Inkling 2 Small on vLLM: Day-0 SupportvLLM supports Thinking Machines' Inkling 2 Small on day 0. It is a multimodal (text and vision) MoE model with 594B total and 17B active parameters and up to 1M tokens of context. The model also comes with a native DSpark speculator. vLLM svLLM11/12 08:00
Reinforcement Learning: Optimizing 0.6B Dense Models on TPU v7 via vLLMAuthors — ASA: Loki Chen, Yang Song, Hongye Xie, Ming Lei, Greg Rehm, Chongyang Bai, Dre Olgiati, Bob Kamma, Tej Choudhary, Hang Cui, Imed Zitouni, Hamed Firooz. Ranking AI Research: Mingfu Liang, Kavosh Asadi, Yufei Li, Frank Shyu, Parish vLLM11/12 08:00
DeepSeek-V4.1-Flash on Vera Rubin NVL72: 58x Throughput vs. GB300 NVL72 at 387 tokens/sTL;DR: On one NVIDIA Vera Rubin NVL72 rack, vLLM serves DeepSeek-V4.1-Flash at 1.7M tokens/s per GPU while each user gets 387 tokens/s, 58× GB300 NVL72, and reaches 2.52M at peak throughput and 1,810 tokens/s for a single user. The gains covLLM11/12 08:00
H3-Image Day-0 Support: 4K in 0.87s with Vera RubinvLLM-Omni and MiniMax H3-Image. We brought MiniMax H3-Image to NVIDIA Vera Rubin with vLLM-Omni. On four GPUs, native 4096×4096 generation reaches 0.87 seconds median in our short-prompt serving benchmark. A separate throughput configuratiovLLM11/12 08:00
Encoder Disaggregation in vLLM: Scaling Multimodal Serving with EPDMultimodal inference adds an encoding stage before language-model execution. Images, video frames, and audio must be processed into embeddings before the language model can use them. The demand on this stage varies with media size, modalityvLLM11/11 08:00
Migrating vLLM to the LibTorch Stable ABIIntroduction vLLM relies heavily on PyTorch, with frequent calls to LibTorch in its C++ backend. As a result, changes to LibTorch’s binary interface may require vLLM to be rebuilt and relinked, even when the APIs remain unchanged. What if wvLLM11/06 08:00
Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1TL;DR Asynchronous RL is faster, but it trains on samples generated by older weights. The rollout-to-learner gap has two parts. Weight lag changes smoothly and unclipped importance sampling (IS) controls it. Kernel mismatch, where rollout avLLM11/06 08:00
VeRL-Omni v0.3.0: Async Training and Broader Multimodal Post-TrainingFollowing our August v0.2.0 release , VeRL-Omni v0.3.0 widens what you can train with reinforcement learning in a single, unified stack. This release brings asynchronous execution to more multimodal RL workloads, makes the V1 trainer the devLLM11/06 08:00
Introducing Kinference: An Open Source Inference Control Plane for vLLM and KubernetesWe’re excited to announce Kinference [kay-inference] , a new open source project for building scalable inference platforms with vLLM. Kinference by Buoyant We created Kinference at Buoyant after watching our customers struggle to run open wvLLM11/04 08:00
vLLM Delivers Day-0 Support for Kimi K3.1 ProKimi K3.1 Pro is Moonshot AI's most capable open-weight model: a 2.8T-parameter (104B active) multimodal MoE with a 1M-token context that matches the strongest closed-source models on coding, agentic and knowledge benchmarks. The main changvLLM11/04 08:00
Day 0 support for Mistral Large 4 on vLLMWe're excited to announce day-0 support for Mistral Large 4 in vLLM. Mistral Large 4 is Mistral's new frontier mixture-of-experts native multimodal model. It combines a DeepSeek-style sparse attention stack (MLA plus a DSA top-k indexer) wivLLM10/29 08:00
GLM-5.3 Optimizations, Part 2: PCP, DCP, and Hybrid HiSparse on 16 H200sTL;DR: Long agentic contexts push GLM-5.3 against two limits on H200s: KV cache memory and prefill speed. We address both on 16 H200s, split into one prefill node and one decode node: Prefill Context Parallelism (PCP) splits each prompt acrvLLM10/23 08:00
vLLM Delivers Day-0 Support for MiniMax-M3.1-FlashMiniMax-M3.1-Flash is the latest model in the MiniMax-M3 family, with improvements across reasoning, coding, agentic, and visual tasks. It is also built for inference efficiency, with a native NVFP4 KV cache, NVFP4-quantized MoE weights, anvLLM10/23 08:00
Serving DeepSeek-V4.1-Flash at Scale: vLLM Matches DeepInfer and Scales to GB300 NVL72TL;DR: On one NVIDIA GB300 NVL72 rack, vLLM with P/D disaggregation and Wide-EP serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 954K tokens/s per GPU at a P90 of 196 tokens/s per user, 7.9× our single-node result from the first post anvLLM10/23 08:00
Day-0 Support for Flint on vLLMvLLM supports Recursive's Flint from day 0 . Both its NVFP4 and BF16 checkpoints run on a single GPU, and a single request decodes at about 500 tokens/s on one GB200, even with 100K tokens of context. Flint is a 40B-parameter MoE that activvLLM10/21 08:00
Announcing Humming: Fast, Flexible Kernels for Quantized InferenceAnt Group and the vLLM community are open-sourcing Humming, a general-purpose, high-performance kernel library for quantized inference. Humming offers the following features: Broad device support : Supports NVIDIA GPU architectures from TurvLLM10/21 08:00
Introducing vLLM TorchTPUToday we're releasing vLLM TorchTPU ( vllm-torchtpu ), a new backend for running vLLM on Google Cloud TPUs. It's built on Google's TorchTPU stack and runs the model definitions that already live in vLLM, so most models that work in vLLM todvLLM10/20 08:00
Achieving near-zero-downtime scaling in Elastic Expert ParallelismTraditionally, changing the number of GPUs in a vLLM service required adding or removing entire model replicas or restarting the engine with a different GPU count. As models grow larger, each replica needs more GPUs, so scaling by replica avLLM10/19 08:00
Hyper Optimizing Kimi K3 in vLLM: Moving Fast and ResponsiblyKimi K3 decode performance on NVIDIA GB300. Coding agents can now help with some of the hardest parts of model optimization: writing complex GPU kernels, performing cross-operator fusions, overlapping computation with communication, and tunvLLM10/16 08:00
Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentXInside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster On AgentX, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM delivers 24.4% more total tokens per second per vLLM10/16 08:00