vLLM/Recipes
Moonshot AI

moonshotai/Kimi-K3

Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window

Pre-release TP8, TEP16, and disaggregated P/D profiles for the 2.8T MXFP4 checkpoint

View on HuggingFace
moe2.8T / 16 experts/token + shared (of 896 routed)1,048,576 ctxvLLM 0.27.0+multimodaltext
Verified on NVIDIA H200
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  -e VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \
  vllm/vllm-openai:kimi-k3 moonshotai/Kimi-K3 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.95 \
  -cc.pass_config.fuse_allreduce_rms=False \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --no-enable-flashinfer-autotune \
  --moe-backend marlin \
  --disable-custom-all-reduce \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3
# Set $HEAD_IP to the rank-0 node's IP before launch. Scale to N nodes by replicating # this worker command with --node-rank = i (TP/TEP) or --data-parallel-start-rank = i × local_gpus (DEP).
Hardware
NVIDIA
AMD
Variant
Strategy

Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.

Latency oriented
KV Offload
Nodes
Features
Advanced— performance tuning
Guide

Overview

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision.

Prerequisites

  • vLLM: Use the vllm/vllm-openai:kimi-k3 docker
  • Hardware: At least 8x GB300. Multi-node for real production traffic.
  • ROCm: Use vllm/vllm-openai_rocm:kimi-k3 docker and at least 8x MI355X/MI350X hardware.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K3",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Notes

  • Cross node communitcation: Use --all2all-backend deepep_v2 for RDMA and --all2all-backend flashinfer_nvlink_one_sided for NVLink.
  • MoE backend: Recommend to use deep_gemm_mega_moe for any DEP environment and flashinfer_trtllm for TP>1.
  • Model Runner v2 and Rust Frontend: VLLM_USE_V2_MODEL_RUNNER=1 and VLLM_USE_RUST_FRONTEND=1: Model Runner v2 and Rust Frontend fully supports this model and can be enabled if needed.
  • Tool calling: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.
  • AMD (MI355X / MI350X, CDNA4 gfx950): set AITER_SITUV2_A8W4 to 0 along with AITER master flag to use aiter a16w4 MoE path. Set it to 1 to use aiter a8w4 MoE path.
  • max-model-len: Adjust max-model-len for different benchmark scenarios for best performance.
  • RDMA: If RDMA is enabled, set UCX_TLS="rc,cuda_copy" to make sure KV Cache transfer goes through RDMA.
  • FP8 KV: If FP8 KV cache is needed, please also add --attention-config '{"use_prefill_query_quantization":true}' when serving vLLM.