There are so many things we approximate to make life simple. Wires, for example, have no resistance or other strange effects. Crystal oscillators output their exact frequency. But surely our model of how a computer stores and loads memory is accurate, right? You put data in a particular location and, later, you take it out. The [FEX-Emu] developers have a different perspective. Once you have caches and, perhaps, multiple CPUs, it isn’t that easy.
The basic problem is this: if one CPU (or, more accurately, bus master) writes to a location, will another CPU have access to the new value? X86’s Total Store Ordering model gives programmers strong guarantees about when loads and stores become visible, while ARM deliberately uses a weaker memory model that permits considerably more reordering for performance and efficiency.
An emulator can, in theory, compensate by translating ordinary x86 memory operations into ARM acquire/release operations, but doing that for nearly every memory reference can be expensive. Newer ARM extensions such as LRCPC help considerably, while Apple took a more direct approach by adding an x86-compatible TSO mode to Apple Silicon. That lets ordinary loads and stores behave the way translated x86 code expects with comparatively little overhead.
Things get much uglier with unaligned accesses and atomic operations. X86 software routinely performs accesses that ARM would consider badly aligned, and x86 provides surprisingly strong atomicity guarantees within a cache line. FEX sometimes has to catch alignment faults and dynamically patch translated code with barriers. Split-lock operations are worse still: some require excursions through the kernel and signal handlers and can be hundreds or thousands of times slower than the normal case. Qualcomm’s newer Oryon cores improve matters by supporting coherent cache-line atomics, while Valve has shipped a Linux kernel optimization that handles some troublesome unaligned atomics directly.
There’s another particularly nasty corner involving write-combined GPU memory. PC games frequently expect x86 ordering semantics while writing uncached buffers destined for a discrete GPU. ARM currently lacks a clean equivalent for some of these stores, and FEX measured worst-case bandwidth more than 800× slower, enough to reduce some games to below 1 FPS. UMA systems fare much better because drivers can often substitute ordinary cache-coherent memory.
It’s a long article, but a good illustration of why modern emulators are less about translating instructions and more about reproducing decades of architectural assumptions that software quietly depends upon. Of course, not all emulators or processor recreations are this accurate, and often that’s good enough. But sometimes you need a recreation that is truly cycle-accurate and behaves exactly like the original.
Instagram Engineering
Shopify Engineering
