mlx-local-server: OpenAI-Compatible Inference in 8 MB of Rust

The Problem: Local Inference Servers Are Too Heavy

Every local LLM server I tried had the same problem: they were built for Python-first workflows. Ollama, LM Studio, llama.cpp — all great tools, but they carry hundreds of megabytes of idle overhead. If you want to run a local agent that spins up an inference server on demand, that startup tax is brutal.

I wanted something different: a server that could start in under 20 milliseconds, idle at under 10 MB, and still speak the OpenAI API dialect that every agent framework expects. And since I was targeting Apple Silicon exclusively, MLX was the obvious inference backend — it's Apple's own ML framework, purpose-built for the unified memory architecture of M-series chips.

Why Rust + PyO3?

The constraint was real-time responsiveness at the Rust layer with Python/MLX doing the heavy lifting. PyO3 lets you call Python from Rust (or expose Rust to Python) with almost no overhead. The server binary embeds a Python interpreter, loads MLX at startup, and streams tokens back through Rust's async runtime.

The result: Rust handles all HTTP routing, request parsing, streaming SSE, and concurrency. Python handles exactly one thing — running the model forward pass. This split gives you the performance characteristics of Rust for everything except the GPU kernel calls, which are going to dominate latency anyway.

// Simplified: calling MLX generate from Rust via PyO3
Python::with_gil(|py| {
    let mlx = PyModule::import(py, "mlx_bridge")?;
    let stream = mlx.call_method1("stream_tokens", (prompt, max_tokens))?;
    // ... stream to SSE response
})

Three Servers, One Binary

The binary routes to three distinct backends depending on the endpoint:

• mlx-lm-server — Chat completions, embeddings, vision (LLaVA-style), LoRA adapters, and fine-tuning endpoints on :8080
• mlx-image-server — FLUX.2-klein text-to-image at :8002. On an M2 Pro, 9 seconds per image at full quality
• mlx-audio-server — TTS, STT, audio translation, and source separation on :8001

Each server starts independently. You can run just the LLM server if you don't need image generation, and the binary footprint stays at 8 MB idle — the MLX Python env is lazy-loaded on first inference.

16 ms Cold Start

The cold start number surprised me too. The key is that "cold start" here means "HTTP server accepting connections" — not "model loaded." The Rust HTTP layer starts in under 16 ms. Model loading (the first time you call /v1/models/load) takes longer, but for agent use cases, you typically pre-load the model once at session start, then the server stays warm for the session lifetime.

For truly cold starts where the model also needs to load, quantized 4-bit models on M-series chips load in 2–4 seconds from local storage — still dramatically faster than cloud API round-trips for long sessions.

WWDC 2025

The project was featured in WWDC 2025 — Build local AI agents on Mac with MLX. It was a weird feeling watching Apple engineers demo a workflow that's almost identical to what I built independently — confirms the design is solid.

Source code is on GitHub. The README has a quick-start that gets you from zero to streaming LLM output in under 5 minutes on any M-series Mac.