Keith Ellman

This could be big for AI engineering. MIT updated the weights of an LLM during test-time, significantly improving model performance. Normally, a model is trained on large datasets, fine-tuned on domain specific datasets, and its weights are frozen during test-time. Meaning the weights of the model do not change in production. Even a 1000 examples in real-time does not improve its performance. MIT researchers took the examples in a system prompt and turned them into real-time training samples! The result is an adaptive model that answers queries accurately. There is a long way to go (the training examples are too domain specific, and there is no general framework to generate examples). But the initial results are promising. Cheers!

190

6 Comments

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

269

11 Comments

Let me explain why everyone needs MLOps engineers, while classic ML roles are slowly losing their edge. This isn’t happening in spite of LLMs. It’s happening because of them. One data scientist with GPT-4 and Claude can now run more experiments in a day than a whole team used to manage in a sprint. Fine-tuning has turned into a collection of boilerplate scripts and pre-chewed recipes. Product teams are plugging models straight into API-first setups, skipping everything in between. And then things break. Or suddenly cost way more than expected. And who ends up fixing it in production? LLMs can write decent code. What they can’t do is build pipelines that survive data drift. They won’t notice when model performance drops overnight without a single deployment. They can’t manage training jobs when GPU spot instances vanish mid-run. They don’t resolve broken schemas in the feature store or explain where the data contracts failed. They won’t sync deployment logic with monitoring, rollback strategy, actual product behavior, and traffic from real users. They don’t know how to balance explainability, latency, infra cost, and business pressure in the same week. And they definitely don’t sit in meetings explaining what went wrong in plain English, without hiding behind “it works on my machine”. This is why MLOps and DevOps people matter more than ever. Writing code is easier now. Pretrained models are everywhere. But turning that into something stable, observable, and shippable is still hard. And someone still has to own that. You know exactly who.

459

40 Comments

Anthropic wrote about a way to decrease RAG retrieval error rate by 67%. It’s called Contextual Retrieval and it improves RAG by adding meaningful context to each chunk before embedding and indexing with BM25. First, for each chunk of your document, a concise explanation of key details is generated. For example -- In a medical study database, a chunk "Patient A showed symptoms of fatigue" becomes -- "In a 2022 clinical trial, Patient A (Group 1, receiving Drug X) showed symptoms of fatigue." These details situate every single chunk within the whole document. While creating the vectordb, embeddings are calculated on these contextualised chunks. Then, BM25 is also applied to capture exact keyword matches. BM25 is a ranking function that improves text retrieval by measuring word relevance using TF-IDF while also accounting for doct length and saturation effects. At query time, both methods are used to retrieve relevant chunks, which are ranked, deduplicated, reranked and passed to the model. Contextual Retrieval reduces retrieval failures by 49%, and combining it with reranking boosts the reduction to 67%. ♻️ Share it with anyone who might benefit :) I regularly share AI Agents and RAG projects on my newsletter: https://lnkd.in/gaJTcZBR Link to the full article: https://lnkd.in/gubfPfvN #AI #LLMs #RAG

1,351

84 Comments

12 visual explainers on MCP, AI Agent & LLMs! (for AI Engineers) 1) Function calling & MCP for LLMs Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it. Read here: https://lnkd.in/gZHYZV97 2) 4 stages of training LLMs from scratch Learn 4 stages of building LLMs from scratch to make them practically applicable. Read here: https://lnkd.in/g4s8H6Ni 3) Prompting Reasoning in LLMs Learn 3 prompting techniques that help LLMs think more clearly. Read here: https://lnkd.in/gUHn5wWE 4) Train LLMs using other LLMs LLMs don't just learn from raw text; they also learn from each other. Learn how knowledge distillation in LLMs work. Read here: https://lnkd.in/g94kW5Wm 5) SFT vs Reinforcement fine-tuning in LLMs Learn the difference between supervised fine-tuning and reinforcement fine-tuning. RFT is the technique used to build reasoning LLMs. Read here: https://lnkd.in/gh6tzmxC 6) Transformer vs. Mixture of Experts in LLMs: MoE is a popular architecture that uses different "experts" to improve Transformer models. Read here: https://lnkd.in/gPkJcXYj 7) RAG vs Agentic RAG: Naive RAG retrieves once and generates once, it cannot dynamically search for more info, and it cannot reason through complex queries. Agentic RAG solves this. Read here: https://lnkd.in/gt5mV3Zr 8) 5 popular Agentic AI design patterns Agentic behaviors allow LLMs to refine their output by incorporating self-evaluation, planning, and collaboration! Read 5 patterns here: https://lnkd.in/gTTWvqEk 9) 5 levels of Agentic AI systems: Agentic systems don't just generate text; they make decisions, call functions, and even run autonomous workflows. Read 5 levels of AI Agency here: https://lnkd.in/gd_qrUb9 10) Traditional RAG vs HyDE Questions are not semantically similar to answers, so RAG system may retrieve irrelevant context. HyDE solves this by generating a hypothetical response first. Read here: https://lnkd.in/gcvSEUhy 11) RAG vs Graph RAG: Answering questions that need global context is difficult with traditional RAG since it only retrieves the top-k relevant chunks. Graph RAG solves this. Read here: https://lnkd.in/gNupdxjh 12) KV caching: KV caching is a technique used to speed up LLM inference. Read here: https://lnkd.in/gsPPfiYk _____ Share this with your network if you found this insightful ♻️ Follow me (Akshay Pachaar) for more insights and tutorials on AI and Machine Learning!

237

3 Comments

Python library for building "AI agents that actually follow instructions" 👀. Parlant is an open-source AI agent framework designed to ensure LLMs reliably follow instructions. Instead of relying on long system prompts, developers define guidelines in natural language with "guaranteed compliance" (according to their docs). It supports tool integration, conversational journeys, guardrails against hallucination, and conversation analytics. Agents can be deployed quickly with built-in React widgets and enterprise-ready features like explainability and compliance. Use cases can be financial services, healthcare, e-commerce, and legal tech where consistent, rule-based AI behaviour is critical. IMO, while instruction following is not something anyone can guarantee (yet) for LLM-based agents, the library is quite useful for building more constrained AI workflows. Link to Github Repo: https://lnkd.in/gzAaHTuP I regularly share AI Agents and RAG projects on my newsletter: https://lnkd.in/dqJDN2NE #AI #GenAI #AIAgents

124

8 Comments

Much of deep learning still feels more like alchemy than science, but optimizing model performance doesn't have to - even at huge scale! DeepMind researchers shared an excellent 12-part series on how to scale LLMs. This series (link in comments) aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered "how expensive should this LLM be to train?" or "how much memory do I need to serve this model myself?" or "what's an AllGather?", this series will be useful. This will help researchers designing new architectures and engineers working to make current LLMs run faster. Promising model architectures routinely fail because they can't run efficiently at scale or because no one puts in the work to make them do so. The series also provides useful perspective on the evolving distinction between scientists and engineers in AI. While I believe there's still a meaningful distinction, this work illustrates that scaling considerations are engineering topics that ML researchers now need to master - something that wasn't as critical 4+ years ago. Here is the outline: Part 1: Preliminaries Chapter 1: A Brief Intro to Roofline Analysis Chapter 2: How to Think About TPUs Chapter 3: Sharded Matrices and How to Multiply Them Part 2: Transformers Chapter 4: All the Transformer Math You Need to Know Chapter 5: How to Parallelize a Transformer for Training Chapter 6: Training LLaMA 3 on TPUs Chapter 7: All About Transformer Inference Chapter 8: Serving LLaMA 3 on TPUs Part 3: Practical Tutorials Chapter 9: How to Profile TPU Code Chapter 10: Programming TPUs in JAX. Part 4: Conclusions and Bonus Content Chapter 11: Conclusions and Further Reading Chapter 12: How to Think About GPUs Have fun! Excited to hear what you think of the series. #ai #genai #llm

173

13 Comments

You're in a ML Inference engineer interview at Google, and the interviewer asks: "What's the real bottleneck in LLM serving throughput? How can PagedAttention help?" Here's how you can answer: A. Traditional LLM serving hits a memory wall fast. The problem isn't compute - it's how we manage the KV cache. 65% model weights 30% KV cache 5% activations. When KV cache is managed poorly, you're wasting 60-80% of your GPU memory. B. The PagedAttention Breakthrough: vLLM's PagedAttention solves this by borrowing from operating systems. Just like OS uses virtual memory with paging, PagedAttention splits KV cache into blocks that don't need to be contiguous. Memory fragmentation drops to near zero C. How PagedAttention Works? Instead of pre-allocating huge contiguous chunks, you divide KV cache into fixed-size blocks (like pages in virtual memory). Attention computation becomes block-wise: Query × Key blocks → Attention scores → Weighted Value blocks. D. The magic is in the block table mapping. Logical KV blocks map to physical GPU blocks dynamically. Need more tokens? Allocate one more block. Request finished? Free all blocks instantly. No more internal fragmentation from over-provisioning. E. Today's serving systems waste memory in three ways 1) Reserved slots for future tokens 2) Internal fragmentation (allocate 2048, use 200) 3) External fragmentation from buddy allocators Only 20-38% of KV memory stores actual token states! F. The performance gains are substantial 2-4× throughput improvement over state-of-the-art systems like FasterTransformer and Orca. The improvement is more pronounced with longer sequences and complex decoding algorithms like beam search. G. Memory sharing becomes trivial with PagedAttention. Parallel sampling? Share prompt blocks via copy-on-write. Beam search? Share common prefixes naturally. Shared system prompts? Cache them once, reference everywhere. H. The implementation details matter: Custom CUDA kernels for block-wise attention, fused reshape and block write operations, GPU warp-level block reading. The 20-26% kernel overhead is worth it for the massive memory savings enabling larger batches. I. Block size is critical. Too small = poor GPU utilization. Too large = internal fragmentation returns. The sweet spot is typically 16 tokens per block, balancing parallelism with memory efficiency across different workload patterns. J. The preemption story is elegant. When GPU memory is full, vLLM can swap entire sequences to CPU memory or recompute them later. All-or-nothing eviction policy exploits the fact that all KV blocks of a sequence are needed together. That's it for today folks! This thread is inspired by the brilliant PagedAttention paper. #machinelearning #datascience #inference vLLM

2,251

43 Comments

Short guide to understand NVIDIA Dynamo for large-scale LLM Deployments! If you worked with deep learning before transformers (pre-2017), you probably worked with ONNX, TorchScript, OpenVino, Tensorflow, or TensorRT. These are engines for common DL models. To serve these engines at scale, NVIDIA released Triton Inference Server as a production-ready framework. In the gen-AI era, engines like vLLM, SGLang, TGI, or TensorRT-LLM have emerged for transformer models. Similarly, NVIDIA has released the Dynamo Inference Framework to support these engines and handle inference in large-scale production deployments. 🗒️ Triton and Dynamo both act as higher-level, efficient distributed inference systems with support for multiple inference engines. 🔹 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗡𝗩𝗜𝗗𝗜𝗔 𝗗𝘆𝗻𝗮𝗺𝗼? ↳ 𝗗𝘆𝗻𝗮𝗺𝗼 is an AI inference-serving framework designed to maximize token revenue generation. ↳ It orchestrates and accelerates inference communication across thousands of GPUs. ↳ Features disaggregated serving, a concept introduced in the 2024 DistServe paper. TL;DR DistServe = the prefill and decode phases of the LLM Inference request are separated and parallelized across GPUs. ↳ Dynamo is built in 𝗥𝘂𝘀𝘁 (𝟱𝟱%), 𝗚𝗼 (𝟯𝟬%), and 𝗣𝘆𝘁𝗵𝗼𝗻 for flexibility and customizability. ↳ Dynamo builds on components, which are robust, modular, and independently scalable. 🔹 𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝘁𝗵𝗲 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀? ↻ 𝗧𝗵𝗲 𝗚𝗣𝗨 𝗣𝗹𝗮𝗻𝗻𝗲r monitors the global workload of the entire distributed cluster and can dynamically add and remove GPUs to adapt workloads. ↻ 𝗧𝗵𝗲 𝗦𝗺𝗮𝗿𝘁 𝗥𝗼𝘂𝘁𝗲𝗿 takes care of routing specific requests to the appropriate set of workers that have the best KV Cache hit match. ↻ 𝗧𝗵𝗲 𝗗𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗦𝗲𝗿𝘃𝗶𝗻𝗴 module is where prefill and decode workers are distributed, handling the workload and communicating through 𝗡𝗜𝗫𝗟 (NVIDIA Inference Transfer Library). ↻ 𝗡𝗜𝗫𝗟 is a low-latency, multi-node communication library for inference payloads. It can exchange tensors, objects, and bytes across a large distributed cluster using RDMA. ↻ 𝗧𝗵𝗲 𝗘𝘃𝗲𝗻𝘁 𝗣𝗹𝗮𝗻𝗲 monitors the cluster and identifies workers' load, then it notifies the GPU planner if more GPU resources are required. ↻ 𝗠𝗲𝗺𝗼𝗿𝘆 𝗠𝗮𝗻𝗮𝗴𝗲𝗿 takes care of KV Cache offloading using a multi-tier memory system. This is similar to RadixAttention from SGLang. ↻ 𝗔𝗣𝗜 𝗦𝗲𝗿𝘃𝗲𝗿 is the frontend interface compatible with multiple APIs (OpenAI, Llama Stack, etc). 📌 I published an in-depth article, going through the Dynamo components, find it in the comments! ---- ♻️ Share this with your network. Follow me to learn more about AI Engineering and production-ready AI Systems! ----

120

12 Comments

🚀 Fast-dLLM v2: Parallel Block-Diffusion Decoding for LLMs ⚡️ Highlights 🌟   - Blockwise bidirectional context via complementary masks   - Hierarchical caches (block + sub-block)   - Parallel sub-block decoding + token-shift training Results 📊 - ~2.5× faster vs. standard AR decoding on A100   - 102.5 tok/s (bs=1), 201.0 tok/s (bs=4) — >2× Qwen2.5   - SOTA efficiency–quality trade-off among diffusion LLMs Paper/Code 🌐: https://lnkd.in/g3AhGp96

261

What if you could time-travel through legacy POI datasets with one line of Python? That’s exactly the pain point pyplaces is solving. Big thanks to Ted Banken for sharing this with me. 👉 What is pyplaces? It’s a new open-source Python package (inspired by osmnx) designed to make it simple to work with open data sources like Overture Maps and Foursquare Open Places. Instead of wrangling queries or always pulling the latest-only dataset, you can: 🔎 Search by address, place name, or bounding box 📜 Fetch legacy versions of datasets 🗂️ Get results directly as a GeoDataFrame (EPSG:4326) 🛠️ Use SQL for local filtering at scale 🧭 Find categories faster with a semantic category matcher (no more scrolling endless lists) Why it matters:. Policy analysts and academics can quickly access reproducible datasets. Data scientists get a lightweight open-source alternative to cloud-only tools. Anyone working with places data can save hours of friction. Ted has more planned such as support for additional datasets like Layercake OSM and Source Cooperative files is already on the roadmap. Check it out and let me know what you think! 🌎 I'm Matt and I talk about modern GIS, earth observation, AI, and how geospatial is changing. 📬 Want more like this? Join 8k+ others learning from my newsletter → forrest.nyc

66

5 Comments

PyTorch is (almost) all you need!!! if you’re serious about AI, PyTorch is non-negotiable. And this is the best resource you’ll find out to learn about it! Yann LeCun summed it up brilliantly recently: - No Meta, no PyTorch - No PyTorch, no modern AI (Llama, ChatGPT, DeepSeek: all built on it) - No Llama, no open-source LLM ecosystem - No Llama2, no AI sovereignty beyond the US If you’re building anything with LLMs, chances are PyTorch is the backbone. It’s the foundation for almost everything. Lot of folks asked for beginner-friendly material over years and it’s not easy…. Most of the content is tough, pretty technical and complex. Until now…. This is the most friendly and easy to follow PyTorch material I’ve seen so far and only requires an hour from your time so it could be a good side project for the weekend! 👉 “PyTorch in One Hour: From Tensors to Multi-GPU Training” Links in the comments!!! So, if you want to stay relevant in this space, mastering PyTorch is a must. #ai #genai #pytorch #meta

132

20 Comments