Writing asynchronous applications that perform well on Tokio is a balance between fairness and batching, contention and isolation. The post "Principles for fast Tokio applications" by Russell Cohen identifies the most common pitfalls and strategies for optimization.
**Determine whether a problem exists** Before optimizing, verify that performance issues affect user-facing metrics. The author observes that almost every real Tokio application has poll intervals (the time between `.await` points) much longer than the 10–100 microseconds recommended by Alice Ryhl in her post on blocking. However, long polls may be benign: an application can have long polls that are completely harmless, and fixing them will not measurably improve user-facing metrics. The most useful Tokio metric is the schedule latency histogram, which measures the time between a task being ready and Tokio actually polling the future.
**Split for latency, batch for throughput** Low latency across many requests requires fairness. A naive pipelined implementation can cause one client's entire pipeline to wait behind another, increasing latency dramatically. Explicitly yielding after each request can reduce latency by roughly 10×; yielding after several consecutive immediately-ready reads improves fairness with less throughput sacrifice.
The article discusses splitting work for latency versus batching for throughput, noting that yielding after each request can cut latency dramatically while sacrificing little throughput, as illustrated by these Mini‑Redis latency measurements.
For throughput, batching useful work per runtime event is more efficient. The best example is `tokio::fs`: without `io_uring`, Tokio runs each filesystem operation on a shared blocking pool, and each `spawn_blocking` call carries overhead. When performing a series of filesystem or blocking operations, batch them into the largest sensible unit. In some cases, a dedicated OS thread is a better fit.
**Beware global resources** The Tokio runtime has one global blocking pool and a global task queue. At high enough rates, pushing work onto the blocking queue becomes a bottleneck; `spawn_blocking` can appear in flamegraphs. The global queue should generally stay close to empty. Runtime-wide operations such as `spawn_blocking` are prominent in flamegraphs when the queue is consistently deep.
**Be extremely careful with mutexes** Blocking a worker on a contended mutex can stall the entire runtime. Metrics registries behind a mutex or read-write lock are especially susceptible: if a flush holds the lock while doing expensive work, every Tokio worker may eventually schedule a task that blocks on the same lock, making stealing impossible. Keep critical sections extremely short—a single hashmap update, for example. `RWLock`s are almost never the right primitive, as they still create atomic contention on the read path. Do not hold a lock while flushing, performing I/O, or awaiting another future. A common trick is to lock, then clone the data; if stale reads are tolerable, this bounds mutex-holding time.
The article warns that blocking a worker on a contended mutex can stall the entire runtime, and shows a trace where mutex contention leads to kernel scheduling delays and reduced worker activity.
`tokio::sync::Mutex` trades one issue for another: it is more expensive to lock and susceptible to subtle issues like `FutureLock`, and is really only appropriate if the critical section lasts multiple milliseconds. Consider using channels instead, where one task owns the shared data and channel messages denote mutations.
**Constrain parallelism—usually** Tokio can happily spawn far more tasks than the rest of the system can handle. Accidentally opening thousands of concurrent connections is common. The answer is boring: limit concurrency. A `Semaphore` is often enough; fancy adaptive algorithms are sometimes appropriate but add complexity.
**Isolate Tokio workers from other threads** If the operating system is highly loaded, it may take 10–20 ms (or more) for the kernel to schedule a worker after Tokio attempts to wake it. P99 latency in single-digit milliseconds is a disaster. Pinning Tokio workers to dedicated CPU cores via `cgroups` or related APIs is the most basic solution. The same issue can arise from background threads (e.g., `tracing_appender`) that do more than 100 ms of work without yielding the CPU. Moving noncritical background work to its own core and relocating Tokio workers to other cores usually improves latency.
**Tricks for when you know better** In idealized async applications, all work happens in tiny bursts with frequent yields. The real world does not always work that way, and tiny bursts are not necessarily the fastest way to run software. Under light load, Tokio's work stealing can compensate when one worker is occupied longer than usual—but this breaks down when the runtime or OS is heavily loaded. In both cases, stealing takes longer, and core maintenance (such as driving I/O) may not happen frequently enough to maintain low latency.
Blocking the executor can be acceptable under light load, but this advice does not apply when using `tokio::join!` or `tokio::select!`, which utilize in-task concurrency. Within a single task, blocking prevents any other progress, sometimes manifesting as unexpected timeouts.
The strongest isolation comes from assigning work to separate runtimes and pinning those runtimes to dedicated cores. Many network services have both latency-sensitive work and lower-priority background work; putting them on separate runtimes creates a scheduling boundary. At TokioConf, most speakers ended up moving to a solution with at least two runtimes.
**Spin to keep control** Spinning to keep control is a very advanced tactic for chasing latency measured in microseconds. It is not generally recommended.