Sergiy Matusevych

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

On the Duolingo English Test, we're constantly researching better methods for automatically assessing writing and speaking. Check out our latest paper on automatic pronunciation scoring!

9

Building a Truly Robust RAG System: From Query Construction and Intelligent Routing to Advanced Indexing, Retrieval, and Evaluation Retrieval-Augmented Generation (RAG) is no longer just about plugging a vector database into an LLM. In real-world, production-grade systems, RAG is an end-to-end architecture problem — not a single component. A truly robust RAG system requires carefully designed layers working together: 🔹 Query Construction Moving beyond simple queries to multi-query generation, decomposition, step-back reasoning, and fusion techniques to improve recall and relevance. 🔹 Intelligent Routing Choosing the right path for every question — graph databases, relational stores, or vector databases — using logical and semantic routing strategies. 🔹 Hybrid Retrieval & Reranking Combining structured data, knowledge graphs, and unstructured documents with refinement and reranking to ensure high-quality context. 🔹 Advanced Indexing Strategies Semantic chunking, multi-representation indexing, hierarchical indexing (RAPTOR), and specialized embeddings to balance performance, accuracy, and cost. 🔹 Generation with Feedback Loops Active retrieval, Self-RAG, and reflection-based approaches that continuously improve answer quality. 🔹 Evaluation & Observability Using evaluation frameworks like RAGAS, DeepEval, and others to measure faithfulness, relevance, and overall system reliability — because what you don’t measure, you can’t improve. 👉 The key takeaway: Production RAG is a system, not a feature. Success depends on architecture decisions, data strategy, routing logic, and continuous evaluation — not just the choice of LLM. As GenAI systems move from POCs to enterprise deployments, robust RAG design will be a core differentiator.

13

PyTorch 2.9 is now available, introducing key updates to performance, portability, and the developer experience. This release includes a stable libtorch ABI for C++/CUDA extensions, symmetric memory for multi-GPU kernels, expanded wheel support to include ROCm, XPU, and CUDA 13, and enhancements for Intel, Arm, and x86 platforms. With 3,216 commits from 452 contributors, PyTorch 2.9 continues to advance open source AI for developers worldwide. 🔗 Read the full release blog: https://hubs.la/Q03NNKqV0 #PyTorch #OpenSourceAI #AI #Performance

1,875

27 Comments

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

967

21 Comments

Today’s guide by Reza Bahmanzadeh, AI Engineer on our Professional Services Team, teaches the essentials of serving large models with vLLM: https://lnkd.in/eHK9nwZm It builds from foundational neural network concepts, like transformers and attention, to introduce practical inference workflows, explore useful vLLM features and offer hands-on, operational guidance for efficient deployments. #LLMs #Serving #Inference #TechGuides

35

1 Comment

Check out Mohammad Mozaffari’s blog on the PyTorch website about why 2:4 sparsity matters and why quantization alone isn’t enough for efficient LLM deployment. With Mohammad, we’ve been exploring compression techniques for LLMs—combining quantization, sparsity, and low-rank adapters to push model efficiency and accuracy. You can see an extended version, along with our work/papers on sparsity in AI workloads, here: PyTorch: https://lnkd.in/gcr8-GMm Extended: 👉 Compression Trinity for LLMs: https://lnkd.in/genWbkuh

67

3 Comments

Comparing models.. Finding that a small 1B #llm hallucinates less than a 4B for semantic to structural matching. Small llms for targeted tasks for the win. The new Meta llama3.2 1B is better than expected and cheap on token cost.

3

Sharing here Meta's work on torchcomms as a new PyTorch comms APIs and the open-sourcing of our NCCLX/CTran implementation that got us to scale to 100K+ GPUs last year and is taking us to gigawatt-scale. If you're at the PyTorch Conference today, you can attend a talk from the team this morning: https://lnkd.in/g25BauUV So proud of our Meta teams that worked on this! Coupled with our announcement from last week at OCP summit on our fabrics, switches, and scale-up (https://lnkd.in/g-33_3CZ), this shows Meta's continued drive for open networking for AI up and down the stack to support infra builders and model researchers. From the low-level networking hardware up to the comms software stack in PyTorch, there's an amazing amount of exciting work in networking for AI! #AIInfrastructure #Meta

41