The Rise of Inference Engineering and the Shift to Self-Hosted AI Models

Organizations running high-volume AI workloads can cut costs by approximately 80% by moving from hosted APIs to self-hosted models, a shift that is transforming traditional platform engineering into a specialized field of inference engineering. This transition is driven by the need for data privacy, lower latency, and the increasing performance of open models, such as DeepSeek's R1, which has matched frontier models on math and coding benchmarks.

This diagram illustrates the four-layer architecture of an inference stack mentioned in the list of engineering principles.

The Shift to Self-Hosting

Platform teams are increasingly becoming inference teams as organizations seek to run models internally to protect source code and sensitive data. While hosted APIs are often sufficient for early product development, self-hosting becomes more practical as model bills rise and open models—of which there are now over two million on Hugging Face—become competitive with closed systems.

Chart drawn by HN Digest from figures reported by docs.modelplane.ai; values are shown as written in the source.

The primary goal of inference engineering is to manage the lifecycle of these models: selecting the right hardware, optimizing GPU memory, and ensuring high availability. This work is distinct from standard infrastructure management because the price of AI capability falls largely due to improvements in how models are served rather than just improvements in GPU hardware.

This visual represents the control plane layer of the inference stack, which manages model lifecycle and hardware optimization.

“The price of a fixed level of AI capability falls roughly 10x a year, and most of that comes from how models are served, not from better GPUs”

— modelplane/docs/content/guides/inference-guide.md at release-0.7 · modelplaneai/modelplane

Mechanics of the Inference Engine

An inference engine, such as vLLM or SGLang, functions like a stateful database server with a GPU attached. It loads model weights into GPU memory at startup—a process that can take several minutes—and maintains a KV (Key-Value) cache for every conversation in flight. Unlike standard databases that use system RAM, inference engines must keep their working set in expensive, limited GPU memory.

The KV cache is a critical component that stores the attention keys and values for every token in a conversation to avoid recomputing history. For a model like Llama 3 70B, a single 128K-token conversation can require about 40 GB of cache, which often becomes the primary bottleneck for how many concurrent users a single GPU can serve.

Performance Bottlenecks and Trade-offs

Model requests run in two phases with opposite bottlenecks: the 'prefill' phase reads the entire input in parallel and is limited by compute power, while the 'decode' phase generates tokens one by one and is limited by memory bandwidth. Because these phases are constrained by different hardware limits, optimizations that improve the speed of the first token (TTFT) may have little effect on the speed of subsequent tokens (tokens per second).

Engineers must balance the trade-off between latency and throughput. Batching more requests together increases the total number of tokens served per second (throughput) but slows down the response time for individual users (latency). This mirrors the classic trade-off in database tuning between transaction processing and analytical queries.