Security researchers at Calif have demonstrated a local privilege escalation (LPE) vulnerability in OpenBSD that stems from a bug nearly four decades old. The flaw originated in the original BSD source code, where a shared memory attachment counter was defined as a signed 16-bit short. Because this counter is signed, repeatedly forking a process with shared memory attachment records can push the value past its maximum limit of 32,767, causing it to wrap around into a negative value. This integer overflow, combined with a separate out-of-bounds write in the kernel, allows an attacker to achieve root privileges.

A visual representation of the Big Fish research project on OpenBSD.
The 'Big Fish' research project demonstrated how a 37-year-old integer overflow could be leveraged for local privilege escalation in OpenBSD.calif.io

This image represents the primary research project described in the article regarding the OpenBSD kernel bug.

The exploit mechanism relies on a complex sequence of operations to bypass modern security mitigations like KARL, SMAP, and SMEP. Researchers first wrap the shared memory attach count to a negative value (-32,643) by forking 259 processes. They then perform heap grooming, using metrics from `sysctl(KERN_MALLOCSTATS)` to track memory allocations. By winning a race condition during an `IPC_RMID` operation, the researchers were able to attach a size-confused shared memory segment. This triggered a conditional out-of-bounds write that overwrote kernel membership fields. Finally, by closing a socket to traverse group members, they triggered atomic decrements that modified the Page Table Entry (PTE) self-referential mapping, switching it from user to supervisor accessible mode and allowing for kernel code execution.

While the vulnerabilities were addressed in OpenBSD errata 026 and 018, the researchers highlighted several remaining architectural risks. They noted that while OpenBSD's KARL system randomizes function and data addresses, the kernel's memory allocators remain deterministic. This means early boot allocations consistently land at predictable locations, making them ideal targets for overwriting function pointers, such as the `lo0` and `enc0` ioctl handlers. To mitigate these risks, the researchers suggested several improvements: implementing 'safe-unlinking' to check doubly linked list invariants, randomizing the self-referential PTE map allocation address, or making the kernel allocator non-deterministic by randomizing both the base address and the runtime allocation order. They also pointed toward upcoming compiler features, such as 'Alloc Tokens,' which could isolate pointer-containing allocations from non-pointer allocations to neutralize similar classes of vulnerabilities.