A few of us at Georgia Tech are building Vajra, an open‑source inference engine built for the next wave of serving problems:. * Multimodal streaming first – Low‑latency realtime video, audio pipelines.* Highly distributed computing - Managing 1000s of GPUs.* Hierarchical Graph.

RT @tenderizzation:

AI coding agents are changing the tech debt equation fundamentally. It has never been easier to do iterative abstraction design.

Easiest way to burn your claude code context window -- feeding it gcc error logs.

RT @ramaramjee: Evaluation of LLM serving systems is tricky because several factors influence performance (prefill length, decode length, p….

After hitting evaluation puzzles like this in our own work, we analyzed patterns across LLM inference papers and identified 8 systematic evaluation issues that can make performance comparisons misleading. We have compiled a practical evaluation checklist to help avoid these.

When sequence lengths vary wildly (10 tokens to 100K+), normalizing by length seems like the obvious approach. But normalizing data results in loss of critical information. Let's calculate how those normalized latency numbers translate to user experience:. 📊 At 0.1 s/token ×.

The bitter lesson of AI infra: The hardest part about building faster LLM inference systems is not designing the systems, but rather it is evaluating if the system is actually faster! 🤔. This graph from a recent top systems venue paper about long-context serving shows average

Interesting work on long context inference from @nvidia, where they scale KV parallelism on gb200-nvl72 systems! To learn more about accelerating long context inference and trade-offs between different parallelism dimensions checkout out our paper, Medha:

Built a gpu cluster monitor cli dashboard, because code is apparently cheap now. pip install gpu-cluster-monitor. Github:

Super cool paper, deepseek style communication-computation overlap on steroids! Deepseek creates separate microbatches to overlap communication and communication --- thus hiding the EP communication overhead. TokenWeave splits existing batch into smaller microbatches to overlap.

RT @agrawalamey12: Super excited to share another incredible systems that we have built over the past two years! Training giant foundation….

Maya offers a transparent, accurate, and efficient way to model and optimize large-scale DL training without needing expensive hardware clusters for exploration. A crucial step towards sustainable AI!. Read the paper: Work done with @Y_Srihas , @1ntEgr8 ,.

Maya's lightweight emulation enables rapid exploration of the vast configuration space. We built Maya-Search on top, using techniques like Bayesian Optimization to automatically find near-optimal training recipes in under an hour on a single CPU node! ⚡.

The Impact: Accurate prediction means better optimization choices! Maya consistently identifies training configurations within 2% of the optimal cost, while baseline modeling errors led to choices up to 56% more expensive. That's potentially millions saved! 💸.

Benefits of Emulation:.✅ Transparency: Works with YOUR code, YOUR framework. No manual translation needed. Low adoption barrier. ✅ Generality: Not tied to specific models or optimization techniques. ✅ Accuracy: Captures real behavior, eliminates the semantic gap. Our eval.

The Maya Pipeline:.1️⃣ Emulate: Run unmodified code, capture API traces on CPU. 2️⃣ Predict: Use pre-trained models to estimate runtimes for the captured kernels/collectives on the target hardware. 3️⃣ Simulate: Replay the annotated trace in an event-driven simulator, modeling.