🚀 Exploring Llama-2’s Quality: Can we replace generalist GPT-4 endpoints with specialized OSS models? Dive deep with our technical blogpost to understand the nuances and insights of fine-tuning OSS models. 🔗 https://t.co/zVStDCoG4y 🧵 Thread 1/N👇

It has always been insightful to talk with ray developers and how they are solving their infrastructure problems with ray. Last meetup of the year happening today:

Join us for the final Ray Meetup of the year, where we will deep dive with technical talks on core advancements in Ray, as well as discuss what's coming in 2026. 🎉 Ray Meetup: A Year of Distributed Systems Innovation (End-of-Year Celebration) 🗓️ December 18 ⏱️ 5:30 - 7:30 PM 📍

vLLM delivers even more inference performance with the same GPU platform. In just 1 month, we've worked with NVIDIA to increase @nvidia Blackwell maximum throughput per GPU by up to 33% -- significantly reducing cost per token -- while also enabling even higher peak speed for

Watch @richliaw (@anyscalecompute) explain why Ray joined PyTorch Foundation, citing the ecosystem forming around PyTorch, DeepSpeed, and vLLM, and what this move signals about Ray’s role in the AI infrastructure stack. 🔗 https://t.co/1cKtUtDmvm #PyTorch #Ray #AIInfrastructure

vLLM + Ray Serve LLM APIs = 💘! It was an honor to collaborate with the vLLM team to put this together.

Scaling MoE inference is often communication + KV-cache bound: once you push expert parallelism, decode can become dominated by collectives and imbalance, and prefill stragglers can stall an entire EP group. New community benchmark results for vLLM wide-EP on multi-node H200

We are happy to announce SkyRL tx 0.2, see our blog post https://t.co/bwn5kBtCf8. It comes with lots of performance improvements, all parts of the execution now use jax jit, so there is very little overhead. Now is probably the best time to try it out if you haven't already 🧸

We recently released SkyRL-Train v0.3.0! Highlights include: - Experimental support for Pipeline-RL style Async-RL - Updated E2E Recipes page with Math, Search, SQL runs - Migration from mbridge -> Megatron-Bridge - 14 new OSS contributors! (1/n) 🧵

The team cooks 🔥 Iteration velocity on RL is key to achieving good results. SkyRL is built to modularize RL on LLMs so that researchers can focus on improving the model quality.

Announcing OpenThoughts-Agent with an incredible team — a data-centric effort on TerminalBench-style tasks, built with SkyRL+Harbor 💻🤖 Co-leading the RL team over the past month has been a blast, and we’re just getting started! (1/n) 🧵

We are starting a recurrent office hour session for our LLM APIs on Ray + vLLM. This week we’ll have prefix agenda on wide-EP demo for online inference and distributing batched embedding computation using ray data. Stop by if you are curious about these topics.

🚀 Join Anyscale at #NeurIPS2025 in San Diego. We'll be gathering a group of researchers, founders, & engineers over food and drinks. We'll be discussing Ray and the frontier of large-scale RL, multimodal model training, and multi-node LLM inference. 📅 Thursday, December 4 ·

Wise words. Just adding to this, I also think the training infra cost will still be severely dominated by inference cost (rather than pure training) for 1) data curation and synthesis and 2) RL rollouts. So it’s still inference infrastructure that is dominating the foundation.

Google TPU v6e vs AMD MI300X vs NVIDIA H100/B200: Artificial Analysis’ Hardware Benchmarking shows NVIDIA achieving a ~5x tokens-per-dollar advantage over TPU v6e (Trillium), and a ~2x advantage over MI300X, in our key inference cost metric In our metric for inference cost

1/n 🚀 Introducing SkyRL-Agent, a framework for efficient RL agent training. ⚡ 1.55× faster async rollout dispatch 🛠 Lightweight tool + task integration 🔄 Backend-agnostic (SkyRL-train / VeRL / Tinker) 🏆 Used to train SA-SWE-32B, improving Qwen3-32B from 24.4% → 39.4%

We’ve been constantly asked on how to do deepseek-style deployments with ray serve. Ideas like prefill decode dissaggregation, wide-EP, custom request routing for prefill / decode, etc require fair amount of work in the orchestration layer that might be non-trivial. In Ray 2.52

Wide-EP and prefill/decode disaggregation APIs for vLLM are now available in Ray 2.52 🚀🚀 Validated at 2.4k tokens/H200 on Anyscale Runtime, these patterns maximize sparse MoE model inference efficiency, but often require non-trivial orchestration logic. Here’s how they

We’re open-sourcing a set of high quality speculator models for Llamas, Qwens, and gpt-oss on Hugging Face. In real workloads, you can expect 1.5 to 2.5x speedups and sometimes more than 4x. Here’s how this fits into the bigger story for speculative decoding. A thread 🧵:

Need to customize vLLM? Don't fork it. 🔌 vLLM's plugin system lets you inject surgical modifications without maintaining a fork or monkey-patching entire modules. Blog by Dhruvil Bhatt from AWS SageMaker 👇 Why plugins > forks: • vLLM releases every 2 weeks with 100s of PRs

New Anyscale releases announced at Ray Summit, from Developer Central to Anyscale Runtime to Cluster Controller. Read the roll up blog: https://t.co/iMdQWtp5w5