LLM inference is too slow, too expensive, and too hard to scale. 🚨 Introducing llm-d, a Kubernetes-native distributed inference framework, to change that—using vLLM (@vllm_project), smart scheduling, and disaggregated compute. Here’s how it works—and how you can use it today:.

RT @Virginia__MM: Red Hat + @NVIDIA = a new wave of agentic AI innovation 💡. See how we're supporting NVIDIA Blackwell AI factories across….

RT @_llm_d_: Are you serving LLMs in production? We need your input for the llm-d project!. Take our 5-min anonymous survey to help guide o….

RT @Tandemn_labs: LLM inference still crawling? 🚨 Meet llm-d—a K8s-native, @vllm_project-powered framework from Red Hat @RedHat_AI that sl….

Random Samples: LLM Meets Cache: From Application to Architecture [June 27, 2025]

RT @charles_irl: spotted in the latest @RedHat_AI office hours for @vllm_project -- the LLM Engine Advisor we built on their benchmarking f….

RT @osanseviero: We've taken community feedback very seriously, and that's why for Gemma 3n launch we're so proud to partner with so many i….

[vLLM Office Hours #28] GuideLLM: Evaluate your LLM Deployments for Real-World Inference

llm-compressor v0.6.0 is out. Big improvements for anyone optimizing models for inference with @vllm_project. 1⃣ AWQ now works better for MoEs, with major runtime gains. 2⃣Calibration is faster and smoother with sequential on-loading. This cuts runtime and reduces hardware.

RT @NVIDIAAIDev: The llm-d project is a major step forward for the #opensource AI ecosystem, and we are proud to be one of the founding con….

Curious how llm-compressor is evolving?. This new deep dive covers:.-v0.6.0 updates.-API improvements.-Recipe & observer changes.-Architecture updates.-AWQ walkthrough.-FP4 walkthrough. GitHub: Watch the full breakdown:

RT @_EldarKurtic: Our flagship paper on how far careful quantization can really go in practice got accepted as an oral at ACL 2025 (top 8%)….

Will also be live streamed on X and YouTube here:

vLLM Office Hours continue this Thursday. Special topic: GuideLLM: Evaluate your LLM Deployments for Real-World Inference (with Jenny Yi and @markurtz_). our bi-weekly vLLM update (with @mgoin_). Register to get a calendar invite with a GMeet link:

RT @_EldarKurtic: Want to learn more about GuideLLM, the tool used by @charles_irl and @modal_labs' LLM Engine Advisor to easily benchmark….

RT @charles_irl: GuideLLM is a great tool -- we run it massively in parallel on @modal_labs to benchmark inference engines for the LLM Engi….

Thank you, and sorry about that. Here it is:

TL;DR: If you're building production LLM services, make GuideLLM your standard toolkit for benchmarking and deployment validation. Dive into even more details here:

What’s next? Watch for:. -Multi‑modal benchmarks.-Built‑in visual reporting.-Accuracy evaluation modules. …plus open community contributions and roadmap-driven features.

Check out GuideLLM GitHub here: BTW, we are holding vLLM office hours on GuideLLM this Thursday, June 26. Register here: What's next and detailed blog below 👇

Why it matters: no more guesswork! GuideLLM converts inference performance into actionable capacity planning. Great for cost‑efficiency, reliability, and user experience.