"Projects\nBlog\nAbout\nDonations\nContact\nBlog\n\nLatest updates and releases by LMSYS Org are announced through our blogpost series.\n\nAll\nTech Blog\nNews\nSGLang\nChatbot Arena\nBlog\nServing GLM5.2 NVFP4 Agentic Workload with SGLang: Reaching 500 TPS in 2 Weeks\n- More than 500 TPS on 8xB300 (bs=1) - Sync free speculative decoding for GLM 5.2 MTP - Built-in IndexShare MTP with Spec V2 - 2.33x faster TopK-V2 for ISL 80k - Indexer prologue fusion - Gemm kernels...\nSGLang Team\nJuly 14, 2026\nNews\nBringing DeepSeek-V4 Flash RL Training to AMD Instinct MI355X GPUs with Miles\nDeepSeek-V4 RL is now supported in Miles on AMD Instinct™ MI355X GPUs with ROCm™! RL requires SGLang rollout and Megatron training to implement the same policy closely enough that token probabilities ...\nAMD & Miles Team\nJuly 10, 2026\nBlog\nAccelerating SGLang HiCache with Netpreme X-Mem™ MPU\nNetpreme X-Mem™ Memory Processing Unit (MPU) makes SGLang HiCache faster and more scalable by augmenting the slower Host DRAM offload tier with a purpose-built high-bandwidth KV memory tier. - Pr...\nNetpreme Team\nJuly 8, 2026\nBlog\nDSpark in SGLang: Speculative Decoding with Confidence-Driven, Variable-Length Verification\nSpeculative decoding trades extra compute for fewer decode steps, and the trade sours as load grows: at batch size B with K speculative tokens the target verifies B K tokens every step, and past a po...\nSGLang Team\nJuly 6, 2026\nBlog\nAgent-Assisted SGLang Development: An Initial Exploration\nSGLang development increasingly goes beyond isolated code changes. The same repository now spans LLM serving, distributed runtime, GPU kernels, diffusion pipelines, model-specific execution paths, and...\nSGLang Team\nJuly 2, 2026\nBlog\nImproving DeepEP MoE Load Balance in SGLang with Waterfill and LPLB\nMixture-of-Experts (MoE) models rely on Expert Parallelism (EP) to scale inference across multiple GPUs. In SGLang, DeepEP and EPLB provide high-performance serving under EP, but the workload seen by ...\nNVIDIA Team\nJune 26, 2026\nBlog\nOptimizing Ling-2.6-1T on TPU with SGLang-JAX: Hiding MoE Data Movement Behind Compute with One Pallas Kernel\nSGLang-JAX now supports efficient serving of inclusionAI's Ling-2.6-1T on TPU v7x. With a working baseline in place, profiling pointed to the Mixture-of-Experts (MoE) path as the main bottleneck: each...\nPrayer, JamesBrianD, Haolin Fu, Haoguang Cai, Qinghan Chen\nJune 17, 2026\nBlog\nMOSS-TTS Local Transformer v1.5 on SGLang-Omni: Serving Native-Streaming 48 kHz Speech\nToday we are announcing end-to-end serving for MOSS-TTS-Local-Transformer-v1.5 on SGLang-Omni, together with MOSI and the OpenMOSS Team. MOSS-TTS-Local-Transformer-v1.5 is an open TTS model for 48 kH...\nMOSI, OpenMOSS Team & SGLang-Omni Team\nJune 17, 2026\nBlog\nThe next generation of speculative decoding: DFlash and Spec V2\nUsing Modal and Z Lab's DFlash speculative decoding models with SGLang’s newly default Spec V2 engine, you can achieve state-of-the-art latencies for LLM inference serving. Our new, jointly-released D...\nZ Lab, Modal, and SGLang Teams\nJune 15, 2026\nNews\nAnnouncing the Recipient of the 2026 LMSYS PhD Fellowship\nWe are delighted to announce the first recipient of the LMSYS Fellowship Program: Will Lin. Following the launch of our Fellowship Program and careful review of applications, we selected Will for his...\nLMSYS Board\nJune 08, 2026\nBlog\nNo Token Left Behind: Demystifying Token-In-Token-Out in Miles\nIn agentic RL, a rollout is not a single generation. It is a chain of model calls, tool outputs, harness messages, and resumed generations. Token-In-Token-Out (TITO) is a design principle that address...\nMiles Team: Jiajun Li, Yuzhen Zhou, Shi Dong, Yanbin Jiang, Mao Cheng, Yusheng Su, Yueming Yuan, Zhichen Zeng, Banghua Zhu\nJune 5, 2026\nBlog\nHiggs Audio v3 TTS on SGLang-Omni: Real-Time, Controllable Speech for Voice Agents\nToday we are announcing end-to-end serving for Higgs Audio v3 TTS on SGLang-Omni. Higgs Audio v3 TTS is Boson AI's text-to-speech model for conversational voice agents: it generates natural and expres...\nBoson AI & SGLang-Omni Team\nJune 4, 2026\nBlog\nSGLang and Miles Add Day-0 Support for NVIDIA Nemotron 3 Ultra for Long-Running Autonomous Agents\nWe are excited to announce that SGLang and Miles support NVIDIA Nemotron 3 Ultra on Day 0\\. Agentic AI systems are moving from short prompt-response interactions to persistent workflows that plan, us...\nNVIDIA Nemotron Team and SGLang & Miles Team\nJune 4, 2026\nBlog\nHeterogeneous CPU + GPU EPD Disaggregation to Boost VLM Serving\nTL;DR We enabled heterogeneous Encode-Prefill-Decode (EPD) disaggregation via Dynamo and SGLang for Vision-Language Models (VLMs). By offloading vision encoding tasks to CPUs (the easiest-getting CPU...\nIntel & SGLang Team\nMay 29, 2026\nBlog\nWin on TCO: How AMD Instinct™ MI355X Achieves Cost-Competitive Distributed Inference Through SGLang with MoRI\nThe SGLang and AMD team has worked closely to unlock competitive Total Cost of Ownership (TCO) for large-scale DeepSeek-R1 disaggregated inference on AMD Instinct™ MI355X GPUs. Building on SGLang's se...\nAMD & SGLang Team\nMay 28, 2026\nBlog\nUpdating 1T parameters in seconds — P2P weight transfer in Large Scale Distributed RL\nWe introduced a RDMA-based, Peer to Peer weight update mechanism for RL workloads in SGLang as a supplement to traditional NCCL broadcast methods, compatible with all major open source models. By util...\nJiadong Guo, Xin Ji, Letian Ruan, Teng Ma, Chenyang Zhao, Yueming Yuan, Zhichen Zeng\nApril 29, 2026\nBlog\nDeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles\nWe are thrilled to announce Day-0 support for DeepSeek-V4 across both inference and RL training. SGLang and Miles form the first open-source stack to serve and train DeepSeek-V4 on launch day — with s...\nThe SGLang and Miles Team\nApril 25, 2026\nBlog\nHiSparse: Turbocharging Sparse Attention with Hierarchical Memory\nSelf-attention has become a major bottleneck in scaling LLMs to long contexts because of i"