A running amd64 guest can now move between two hosts over TCP while continuing to run, experiencing a pause of only 138–171 milliseconds during the final transfer phase. This first working state for live migration on the bhyve hypervisor moves the technology from an abstract concept to a testable implementation on FreeBSD 15.1. The project aims to provide the foundational infrastructure needed for enterprise-grade virtualization features like host maintenance and load balancing.
This image illustrates the successful execution of the live migration prototype described in the article.
The Pre-Copy Migration Mechanism
The migration process begins with a pre-copy phase where the source host sends populated guest memory to the destination. The system repeatedly obtains a bitmap of pages changed since the last scan, sending only those 'dirty' pages to keep the destination updated.
Once the remaining dirty set is small enough to meet a configured downtime target, bhyve pauses the guest and its devices. It then sends the final dirty pages and the existing snapshot state for the vCPUs, kernel state, and emulated devices.
The destination validates the VM geometry and restores the state. Only after the destination confirms it is ready does the source send a 'GO' commit point, at which point the source guest is officially handed off.
“The goal is deliberately practical: make it possible to test, discuss, and improve a real implementation rather than discuss live migration only in the abstract.”
— FreeBSD bhyve Live Migration: Moving VMs Between Hosts
Current Capabilities and Validations
The prototype includes multi-vCPU protocol support, ensuring the migration handshake carries the correct vCPU count and that all processors are paused and resumed in sync. It also features memory consistency checks that verify guest and mapped emulation writes.
To test the device pause and restore paths, the developer implemented a guest-side workload that repeatedly writes and fsyncs files. This was verified by checking checksums after a cold reboot to ensure no data was lost during the transition.
The system handles various failure modes, including unreachable targets or a destination disappearing during the pre-copy phase. In these cases, the source is designed to resume the paused guest to prevent data loss.
Roadmap and Engineering Blockers
While the prototype shows encouraging results, several engineering requirements must be met before the code can be upstreamed into FreeBSD. These include hardening the kernel-side dirty-page algorithm to ensure no writes are lost during bit examination and defining a complete compatibility contract.
A stable production version will require a hardened transport protocol with authentication and encryption, as the current TCP transport lacks these features. Additionally, the project needs to define specific coherency and fencing assumptions for shared storage.
The current scope is limited to amd64 architecture and relies on experimental snapshot support. Future work will focus on refining the commit and device-side-effect boundaries before adding performance features like compression or zero-page elision.