We also provided an easy way for people to use some open-source models deployed on our servers at https://t.co/hDfjtPv7yo by providing free 20$ credits to all users for the hackathon! Watch the entire stream here -

We recently demoed our heterogeneous distributed inference stack at @HackMIT in a one-hour workshop, where we showcased our fully open-source stack ( https://t.co/Pl6zZmAehK), capable of running @vllm_project and @lmcache on heterogeneous GPUs that communicate via @iroh_n0.

Everyone repping up Tandemn merch!

Come check us out at @HackMIT #inferwithtandemn #MIT

Look at our work on -

Batched inference ongoing!! ⛰️🧗

Making heterogeneous GPUs feel like one server. Fewer bottlenecks, more tokens/sec and more requests/sec. Grateful for the OSS we stand on.

Tandemn just raised $1.7M! https://t.co/AvdfepH9PV Building the missing layer between heterogeneous GPUs & huge models to end underutilization and make AI workloads effortless. Hiring cracked MLsys and networking engineers. (😉Collab w/ @lmcache & @n0computer soon, stay tuned.)

📊 Two 8×H100 boxes, long-prompt/short-output bench: mean TTFT 3× faster & QPS +50 % vs replica-only baseline, SLOs intact. Tandemn’s next drop: hetero scheduler + QUIC mesh to harness every stray 3090/v100/a100/IntelArcs beside your H100s. If llm-d got you hyped, stay tuned. ⚡️

🔹 KV-cache-aware routing—send each chat to the replica that already holds its prefixes, so you skip redundant prefill and kill p95. 🔹 Prefill ≠ Decode—prefill is FLOP-bound, decode is DRAM-bound. llm-d runs them on different nodes so neither GPU half-idles.

LLM inference still crawling? 🚨 Meet llm-d—a K8s-native, @vllm_project-powered framework from Red Hat @RedHat_AI that slashes cost & tail-latency with cache-aware routing + disaggregated compute. Why it matters👇 https://t.co/uiUrvamPyp

LLM inference is too slow, too expensive, and too hard to scale. 🚨 Introducing llm-d, a Kubernetes-native distributed inference framework, to change that—using vLLM (@vllm_project), smart scheduling, and disaggregated compute. Here’s how it works—and how you can use it today: