The video introduces a 60-year-old design flaw in RAM (DRAM) where it periodically "goes blind" for approximately 400 nanoseconds to refresh its cells, leading to significant latency spikes (tRFC lockout).
A benchmarking program called Tailslayer was developed to measure and demonstrate these refresh stalls, showing consistent 7.82 microsecond intervals between them.
The initial attempt to predict and avoid these stalls by scheduling reads around them proved ineffective due to complex factors like opportunistic refresh scheduling by the memory controller and OS interference.
The speaker, LaurieWired, pivoted to a "hedged read" strategy, inspired by Google's "The Tail at Scale" paper, which involves duplicating memory requests across different memory channels to increase the probability of a fast read.
This strategy, while successful in theory, faces challenges with modern CPUs' out-of-order execution and reorder buffers, which can negate the benefits of parallel reads by forcing in-order commitment.
Multi-core processors circumvent the reorder buffer issue by having separate reorder buffers for each core, enabling truly independent hedged reads.
A significant hurdle is the operating system's virtual memory abstraction and undocumented channel scrambling performed by memory controllers (Intel, AMD, ARM), which obfuscates physical memory addresses and data distribution.
To overcome this, LaurieWired reverse-engineered the channel scrambling using hardware performance counters (on AMD/Intel) and later a "latency as signal" method (on AWS Graviton, a black box), mapping how different address bits affect channel placement.
The results demonstrate substantial reductions in tail latency (p99.99 and beyond) across various platforms, with up to 15x improvement on Intel Xeon, validating the effectiveness of the Tailslayer methodology.
DRAM Refresh Stalls on AMD EPYC 7713 (Milan) DDR4
[
00:04:30
]
Every 3.9 microseconds, RAM physically shuts down to recharge its capacitors to prevent data loss. [00:00]
This recharge causes a "lockout" defined as tRFC or refresh cycle time, lasting approximately 400 nanoseconds on DDR5. [01:58]
A regular DDR5 read takes about 80 nanoseconds, but hitting a tRFC lockout can increase latency to around 400 nanoseconds, potentially burning 2,000 CPU cycles. [02:04]
While unnoticeable for most users, this latency is critical for low-latency applications like high-frequency trading (HFT). [02:25]
Demonstrating Refresh Stalls:
A benchmark program was created to make RAM stalls on DDR4 obvious, using a flush read timestamp on a tight loop targeting a single address. [03:20]
The benchmark, run on an AMD cloud processor (2.65 GHz), showed latency spikes every 7.82 microseconds, precisely indicating refresh stalls. [04:24]
Initial attempts to predict and avoid these stalls were unsuccessful due to factors like cache behavior, OS scheduling, and opportunistic refresh scheduling allowed by the JEDEC spec, which permits postponing up to eight refreshes. [06:04]
DRAM Portion: Contains the "drawbridge" representing tRFC stall. [11:07]
Activate Command (tRC/tRAS): If the correct row isn't open, an activate command is issued (~15 ns). [12:04]
CAS Latency (CL): Column Access Strobe latency, a key parameter on RAM sticks (e.g., CL30 equals 10 ns). This is one of the few parts of the track that can be reduced by purchasing more expensive RAM. [12:24]
Hedged Read Implementation: Introduce a second "track" (memory channel) with an identical copy of the data. Send two requests simultaneously; the first one to return wins, and the slower one is discarded. [14:10]
Modern CPUs use out-of-order execution, allowing independent instructions to dispatch in parallel. [15:43]
However, instructions must retire (commit) strictly in order for speculative execution and rollback mechanisms. [18:12]
This means even if one hedged read finishes quickly, the CPU cannot commit its results until the slower, earlier read also completes, effectively stalling the fast read. [18:50]
Separate CPU cores have separate reorder buffers. [19:37]
By firing off hedged reads on two different threads, each pinned to a separate core, the reorder buffer of one core does not block the other, enabling true parallel execution and allowing the faster read to complete independently. [19:41]
Addressing Physical Memory and Channel Scrambling [20:19]
Operating System Abstraction (Virtual Memory):
The OS uses virtual addresses, hiding physical memory locations for security and resource management. [20:43]
Programs allocate memory in 4 KB chunks, which are physically scattered. This complicates ensuring data copies are on separate memory controllers. [22:05]
Huge Pages Cheat Code:
Huge pages provide large, physically contiguous blocks of memory, bypassing the OS's scattering and allowing precise control over physical memory layout. [22:39]
This enables calculating physical offsets directly from virtual offsets within the huge page. [22:56]
Memory Controller Undocumented Scrambling:
CPU manufacturers (Intel, AMD, ARM) use undocumented XOR hash functions to scramble physical addresses across different memory banks and channels. [24:50]
Reasons for scrambling:
Performance (Load Balancing): Prevents common linear access patterns from "hammering" a single bank, distributing load evenly across channels and banks. [26:07]
Security (Rowhammer Mitigation): Makes it harder to predict physical memory layout, raising the bar for Rowhammer attacks where aggressive access to one row can flip bits in adjacent rows. [27:09]
Flexibility: Allows manufacturers to change addressing schemes across CPU generations, steppings, or BIOS updates without breaking documented APIs. [31:51]
The goal is not to predict the exact physical location for Rowhammer, but to ensure duplicated data resides on different, uncorrelated memory channels for hedged reads. [30:11]
Method for AMD (Zen 4, EPYC): Utilizes exposed hardware performance counters (UMC counters via amd_uncore Linux module) which increment for each read through a specific channel. By changing address bits (e.g., 256-byte offset to flip bit 8), the channel mapping can be observed. [33:23]
Method for AWS Graviton (Black Box): Without performance counters or documentation, the hedged read itself is used as a signal. Iteratively flipping address bits and measuring tail latency improvement reveals which bits map to different channels (latency as signal). [45:20]
Mathematical Basis: XOR is linear over GF[2], meaning flipping an input bit will always flip the corresponding output bit, regardless of the complexity of the chained XOR operations. This allows mapping input-output behavior without full black-box reverse engineering. [36:58]
Performance Results and Real-World Application [38:46]
Tail Latency Reduction:
The hedged read strategy (named "Tailslayer") significantly reduces tail latency.
On AMD 7950X3D (DDR5, dual-channel): P99.99 latency reduced from 631 ns to 281 ns (2.25x lower). [41:31]
On AMD EPYC 8484X (Turin, DDR5): P99.99 latency shows a near-vertical blue line, meaning tail latency is almost identical to average read speed (N times better). Even on a quiet system, a 5x speedup is observed. [43:21]
On Intel Xeon 8480C (Sapphire Rapids) with 8-way hedging: P99.99 latency is 113 ns, representing a 15x reduction compared to a traditional read. [44:27]
On Intel Xeon 8592 (Granite Rapids): 7x improvement despite limited channels (4 out of 12) due to SNC3 mode. [44:56]
On AWS Graviton 3 (ARM): 9x improvement at P99.99. [47:31]
This technique works across Intel, AMD, Graviton, DDR4, DDR5, x86, and ARM. [47:50]
Tail Slayer can be used in scenarios where rapid access to large, uncacheable data (e.g., a massive pricing table) is critical.
By duplicating the large data table and using specialized NICs (e.g., SolarFlare X2522) with programmable receive queues, incoming market data packets can trigger hedged reads across separate cores and memory channels.
If one core hits a DRAM refresh stall, another core on a different channel can complete its read and send the order first. Programmable NICs can de-duplicate and drop "loser" packets from the slower core.
The project, named Tailslayer, involves setting up threads, pinning them to cores, strategically placing duplicated data in memory using channel offsets, and then performing the read, with the first successful read determining the outcome.
It avoids atomic operations to prevent synchronization overhead from negating the latency benefits.
All research and code are available on GitHub. [47:07]
