A task round trip in the mizu framework takes less than one microsecond, a two-order-of-magnitude improvement over the ~100 µs overhead typical of standard Python process pools. This performance leap allows developers to distribute work as finely as they would with threads, making even tiny 10-microsecond tasks viable for parallel execution.
Overcoming the Serialization Bottleneck
Standard process pools in Python and R often suffer from significant overhead because each task must be serialized, copied through a socket, and deserialized by a worker. When tasks are small, this communication cost can exceed the actual computation time, while large datasets can exhaust memory by creating multiple copies across different processes.
Mizu addresses this by using a C library called libmizu to facilitate lock-free shared memory Inter-Process Communication (IPC). Instead of copying data, a task is written once into memory that both processes can see, allowing workers to read the data directly without additional overhead.
“A task round trip drops from ~100 µs over sockets to < 1 µs, so you can divide work as finely as you would with threads”
— GitHub - shikokuchuo/pymizu: Thread-speed async and parallel Python over shared memory
Architecture and Performance Gains
The architecture utilizes lock-free single-producer single-consumer rings with batched publication and spin-then-park waiting. This design ensures that the system stays out of the kernel on the hot path, only triggering OS notifications when a peer dies.
Benchmarks on an Apple M4 Pro show that pymizu achieves a 135x speedup in pipelined throughput for a single worker compared to ProcessPoolExecutor. For a parallel map of 2,000 small tasks across four workers, the speedup reaches 42x, dropping the execution time from 123 ms to just 2.9 ms.
Cross-Language Interoperability
Because mizu and pymizu share the same C core, they provide a unified wire format that allows R and Python processes to communicate directly. This enables a workflow where an R session can pass a data frame to a Python worker for processing with Polars, and the result returns as a zero-copy ALTREP view.
Chart drawn by HN Digest from figures reported by opensource.posit.co; values are shown as written in the source.
Common data structures including NumPy arrays, Polars data frames, and Arrow tables cross the boundary without marshalling or conversion. Above a small size floor where direct sending is faster, these objects are shared as views of the same memory pages.
Current Limitations and Lifecycle
The project is currently in an experimental lifecycle, meaning the API is not yet stable and may change based on user feedback. It requires Python 3.10 or later on a 64-bit platform, and the extension compiles the vendored C core during installation.
The project requires specific Python versioning and a 64-bit platform to function correctly.
While the performance gains are substantial, the current implementation is intended for single-machine use. Users seeking to submit lambdas or closures as pool tasks must install the optional cloudpickle extra.