🙌 We’re growing the BentoML team! Come join the journey! 🔧 Forward Deployed Engineer: Work directly with customers to design, build, and deploy real-world LLM apps. You’ll own projects end-to-end. Perfect for engineers who thrive at the intersection of product, engineering,

🚀 Introducing llm-optimizer: an open-source tool for smart LLM benchmarking and inference performance tuning. Tuning LLM performance is notoriously tricky. For most AI teams, this process means endless trial and error. And even then, it’s hard to know which setup is actually

#NVIDIA may lead the #GPU market today, but AMD’s MI-series (MI325X, MI300X, MI350X) has become an important part of the conversation. https://t.co/DSpUMv8y9B With high-bandwidth memory and growing ecosystem support, #AMD GPUs are becoming a serious option for modern #AI

We benchmarked speculative decoding on H100 GPUs with two tensor parallelism setups (TP=1 vs TP=2). Key takeaways: - Throughput: Speculative decoding consistently outperformed baseline when concurrency <35. At higher loads, TP=2 scaled much better. - TTFT: TP=1 saw a sharp

What are NVIDIA Data Center GPUs? https://t.co/Qt3wHtBCIx Most people know NVIDIA from gaming, like the popular GeForce series, but those aren’t built for enterprise-scale AI. For #AIInference, what you really need are NVIDIA Data Center GPUs. In our latest blog, we cover: ✅

🚀 Check out this great post by Titus Lim Hsien Yong https://t.co/0NPO03N1N9 👋 Skip the ops-heavy stack — Kubernetes, Terraform, Ansible, etc. With #BentoCloud, you can deploy any open and custom #LLM in minutes, with fast autoscaling, comprehensive observability, and tailored

🚀 DeepSeek just dropped V3.1, a major update that fuses the strengths of V3 and R1 into a single hybrid model. Takeaways: ⚙️ 671B parameters (37B activated), 128K context length 🧠 Hybrid thinking mode: One model, two modes (thinking + non-thinking) 🛠️ Smarter tool use:

LLM leaderboards rank the top LLMs, but a high score doesn’t guarantee the best model for your use case. To find the optimal setup for inference, you need custom benchmarks tailored to your hardware, framework, and workload. This often means balancing trade-offs in throughput,

Everyone’s talking about speculative decoding for faster #LLMinference. But do you know: the actual speedup 𝗱𝗲𝗽𝗲𝗻𝗱𝘀 𝗵𝗲𝗮𝘃𝗶𝗹𝘆 𝗼𝗻 𝘁𝗵𝗲 𝗗𝗥𝗔𝗙𝗧 𝗺𝗼𝗱𝗲𝗹 𝘆𝗼𝘂 𝘂𝘀𝗲. Three metrics can be used to evaluate the performance of the draft model: - Acceptance rate

🙏 Thank you, #Ai4 2025! It was amazing to meet so many of you and connect in person! Special thanks to everyone who joined our CEO @chaoyu_'s session on #InferenceOps! If you missed it or want to dive deeper, check out our blog post 👉 https://t.co/0UebOv64If 👀 Highlights

🎤 Catch our CEO @chaoyu_ at #Ai4 2025 today! 🗓 Aug 13, 12:20 PM 📍 Room 317 (Level 3) – CAIO, CIO & IT Leaders track He’ll share why #InferenceOps is the strategic foundation for scaling enterprise #AI and how organizations can run mission-critical AI workloads with speed,

📸 Live from #Ai4 2025 in Las Vegas! 🍱 The Bento team is here at Booth #346, showing how we help enterprises run, scale, and optimize inference for mission-critical AI workloads. Come say hi 👋 If you’re here, tag us in your photos! @Ai4Conferences #AIInference #BentoML

You can get up to 3× faster LLM inference with speculative decoding, but only if you’re using the RIGHT draft model. https://t.co/pmVziNL550 Why speculative decoding? LLM inference is slow by design. Because of their autoregressive nature, every new token requires a full forward

Try LMCache

🤔 What is KV cache offloading and why does it matter for LLM inference? #LLMs use the KV cache to accelerate inference speed by avoiding recompute for KV during the prefill phase, however cache grows linearly with requests and has to be evicted to due to GPU memory limit. KV

@OpenAI 20b:

🚀 @OpenAI just released two powerful open-weight reasoning models: gpt-oss-120b and gpt-oss-20b. #BentoML supports them from day zero! Key takeaways: ✅ 120b matches OpenAI o4-mini on core benchmarks ✅ 20b rivals o3-mini, ideal for local & on-device inference ✅

🚀 We’re heading to @Ai4Conferences 2025 in Las Vegas and proud to be a sponsor! Join us August 11–13 at North America’s leading #AI industry event. 🍱 Don’t forget to visit Booth #346 to meet the Bento team! See how we help enterprises around the world run, scale, and optimize

Many enterprises are moving LLM workloads on-prem for security, performance, and cost control. But most overlook one critical piece of the stack. They need more than just GPUs and Kubernetes. They also need a purpose-built inference platform layer. Without it, they’ll likely run