Come and listening to all the exciting updates from @DeepSpeedAI and talk to our awesome @PKUWZP

Reminder of DeepSpeed Office Hours

DeepSpeed Office Hours: Tue Aug 25, 12pm ET v0.19.3–0.19.5: • Grad accum for custom & serverless (Tinker) • AutoEP+AutoTP folding & ZeRO-3 TP inference • Hybrid Engine for on-policy distillation • Pinned-memory CPU/NVMe offload speedups Join: zoom-lfx.platform.linuxfound…

The meeting link for tomorrow's DeepSpeed developer meetings is now available: Aug. 7, 9:00 a.m. PT — DeepCompile efficiency & robustness Aug. 7, 10:00 a.m. PT — Expert Parallelism enhancements Everyone interested is welcome to join! zoom-lfx.platform.linuxfound…

Apologies for the short notice! We're holding two DeepSpeed developer meetings on Aug. 7 to discuss our Q3 roadmap goals: 9:00 a.m. PT — DeepCompile efficiency & robustness 10:00 a.m. PT — Expert Parallelism enhancements Roadmap: github.com/deepspeedai/DeepS… Join Slack: deepspeedworkspace.slack.com…

The PyTorch Foundation continues to grow as a multi-project home dedicated to supporting collaboration across every stage of the AI lifecycle. We are excited to launch a new quarterly blog series where each hosted project - PyTorch, @vllm_project @DeepSpeedAI @raydistributed, Helion and Safetensors - shares its latest updates, technical progress, and roadmap. From core framework optimizations and hardware enablement to overall ecosystem health, our hosted projects achieved a lot over the past quarter. Read the complete update to learn more 👉 bit.ly/4fkiTJy..*

We will host our monthly Office Hour for DeepSpeed users on July 28th, 9AM PT. Feel free to join and discuss any topics ranging from issues to roadmaps. Zoom link: zoom-lfx.platform.linuxfound…

PSA for @DeepSpeedAI users - a long outstanding precision-related critical bug has been identified and fixed in github.com/deepspeedai/DeepS… and a new release has been made. The issue was about mixed precision mode downcasting buffers that had to be in fp32 - massively impacting correctness due to large static buffers - e.g. RoPE in Qwen3 models when using long sequence lengths 32K+. Hopefully this fix brings Deepspeed to a close parity with FSDP2 which has been an issue since a long time. You can still have the old behavior but you'd now need to manually configure it - by default the model's buffers will now remain in the original precision. Please install deepspeed==0.19.2 which will do the right thing. Thanks to Tunji Ruwase and Claude Opus 4.8 via Cursor for identifying and fixing the problem.

We now have native support for all ZeRO stages 1/2/3 for Muon Optimizers, providing superior performance on LLM pre-training and post-training. Feel free to try it out, kudos to @PKUWZP Guokai Ma, Peng Du and Chi for the contribution!

Reminder to join our DeepSpeed Office Hours on Tuesday, May 26 at 12:00 PM America/New_York. We'll cover general questions, Q2 roadmap progress, and requests for Q3. Everyone is welcome! Zoom: zoom-lfx.platform.linuxfound…

Don't miss @DeepSpeedAI virtual office hours on May 26 at 12:00 PM America/New_York to ask questions of @toh_tana member of DeepSpeed TSC & get the latest recent key updates, including AutoSP (sequence parallel), AutoEP (expert parallel), and AutoTP (tensor parallel).

Great News! Thanks to DeepSpeed AutoSP, efficient long context LLM training is now easily accessible.

Excited to share that our work SuperOffload received an Honorable Mention for the ASPLOS 2026 Best Paper Award 🎉 Proud of the team for pushing forward system design for large-scale AI. Xinyu gave a great talk presenting the work. In addition, it was also wonderful to spend time with collaborators and the broader community.

💡Excited to be organizing a tutorial at ASPLOS 2026 (lnkd.in/g5auexxg): "Building Efficient Large-Scale Model Systems with DeepSpeed: From Open-Source Foundations to Emerging Research" 🌀 Link: lnkd.in/gixRnAm6 📍 Room: Allegheny 🕘 Time: Monday (Mar 23), 8:30am-12pm 🎤 Speaker: Tunji Ruwase, Masahiro Tanaka, Minjia Zhang, Zhipeng Wang, PhD We will cover how @DeepSpeedAI enables new forms of parallel, distributed, and heterogeneous execution, and how modern systems tackle key challenges in parallelism, offloading, and memory efficiency. If you are working on ML systems, LLM training, or emerging hardware, would love to connect at ASPLOS!

🗓️ Plan your week: Check out the full "Meet the PyTorch Experts" schedule here: pytorch.org/event/nvidia-gtc… We'll be posting the daily lineups here in this thread all week. See you at the booth! 🤝 @NVIDIADev

I am thrilled to release our newly re-architected extremely-scale Linear Programming Solver (DuaLip-GPU), which is developed via PyTorch enabling multi-GPU computations and parallelism (github.com/linkedin/DuaLip). We also released the technical report (arxiv.org/abs/2603.04621) covering all technical details. Linear Programming Solver is a fundamental building block for solving extreme-scale matching problems, which underline many important technical domains related to social network platforms such as ranking, personalization, item-matching and recommendation systems, as well as in LLMs. To realize the available parallelism, we develop GPU execution techniques tailored to sparse matching constraints, including constraint-aligned sparse layouts, batched projection kernels, and a distributed design that communicates only dual variables. Further, we improve the underlying ridge-regularized dual ascent method with Jacobi-style row normalization, primal scaling, and a continuation scheme for the regularization parameter. On extreme-scale matching workloads, the GPU implementation achieves at least a 10x wall-clock speedup over the prior distributed CPU DuaLip solver under matched stopping criteria, while maintaining convergence guarantees. This is the superb technical work combining ML Systems, Mathematical Optimization and Machine Learning. #Optimization #AI

PSA: if you use torch>=2.10 w/ deepspeed ZeRO-3 please update to deepspeed@master - a new release should happen shortly. If you use torch<2.10 or ZeRO-1/2 nothing needs to be done. See this fix from Michael Royzen github.com/deepspeedai/DeepS… Cause: PyTorch made some grad reduction stream-related changes which could lead to borked grad reduction in Deepspeed ZeRO-3.