Why and How I Use Distributed Inference to Run a Large Language Model (LLM)

Sometimes bigger is better

Sergio Rua

Jul 7, 2025

You can now find all my latest blog content exclusively on the Digitalis.io website. Check it out!

Get Sergio Rua’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

https://digitalis.io/post/why-and-how-i-use-distributed-inference-to-run-a-large-language-model-llm

DevOps

LLM

Hosting

Artificial Intelligence

Written by Sergio Rua

202 followers

6 following

I’m Principal DevOps at Digitalis working with many customers managing and advising on Kubernetes, Kafka, Cassandra, Elasticsearch and other cool technologies.

No responses yet

Write a response

What are your thoughts?

Recommended from Medium

A Qwen 3.5 122B LLM on a 16 GB Mac mini: MoE Expert Streaming with TurboQuant-MLX

Data Science Collective

Manjunath Janardhan

4d ago

A Qwen 3.5 122B LLM on a 16 GB Mac mini: MoE Expert Streaming with TurboQuant-MLX

Per-expert disk streaming runs a 122-billion-parameter Mixture-of-Experts model — 3× bigger than RAM — on the cheapest Mac Apple sells…

422

Stop Memorizing Design Patterns: Use This Decision Tree Instead

Women in Technology

Alina Kovtun✨

Jan 29

Stop Memorizing Design Patterns: Use This Decision Tree Instead

Choose design patterns based on pain points: apply the right pattern with minimal over-engineering in any OO language.

UX Planet

Nick Babich

Apr 6

MCP is Dead

Why you should avoid using MCP in Claude Code and what to use instead

3.3K

190

A high-contrast digital graphic with a dark, ethereal blue and purple background. Large, glowing cyan text in the center reads “USING LLMs IS A SKILL.” Below the text are three minimalist neon icons: a stack of books with a quill, a castle tower, and a human brain merged with mechanical gears. Small text at the bottom reads “Based on ‘Learning the Art’ concept.”

Leo Godin

Mar 2

Claude Code is Great

You Just Need to Learn How to Use It

4.7K

141

M5 MacBook Pro or NVIDIA DGX Spark or RTX PRO 6000?

Agent Native

May 22

M5 MacBook Pro or NVIDIA DGX Spark or RTX PRO 6000?

Before answering this in depth, please think about the following question:

180

Run Claude Code with Local & Cloud Models in 5 Minutes (Ollama, LM Studio, llama.cpp, OpenRouter)

Luong NGUYEN

Jan 31

Run Claude Code with Local & Cloud Models in 5 Minutes (Ollama, LM Studio, llama.cpp, OpenRouter)

This post exists because the old guides were written back when using non-Anthropic models in Claude Code required hacks and weird adapters…

632

See more recommendations

Help

Status

About

Careers

Press

Blog

Privacy

Rules

Terms

Text to speech

Be the first to hear about new stories from Sergio Rua

Join Medium for free to get updates from Sergio Rua sent right to your inbox.

Your email

Remember me for faster sign in

Already have an account?

By clicking "Create Account", you accept Medium's Terms of Service and Privacy Policy.

This site uses reCaptcha and the Google Privacy Policy and Terms of Service apply.

Why and How I Use Distributed Inference to Run a Large Language Model (LLM)

Sometimes bigger is better

Get Sergio Rua’s stories in your inbox

Written by Sergio Rua

No responses yet

More from Sergio Rua

Using SSH tunnels for dummies

Secure your connections

Dependabot for BitBucket

Or something like that

Rundeck with Keycloak authentication

Bring back SSO

Cerrando las brechas: logrando la soberanía de la nube a través de la arquitectura modular de…

Administracion de secretos con OpenBao