RAID Rebuild Risk: Why RAID 5 Fails While Healing

A RAID 5 rebuild can fail while healing because reconstructing the lost drive forces the controller to read every bit on all surviving drives, and a single unrecoverable read error (URE) can abort the whole job. With consumer drives rated at one URE per 10^14 bits (about 12.5 TB), a multi-terabyte rebuild may statistically expect an error before it finishes.

The mechanism: parity means reading everything

RAID 5 stripes data across N drives and stores one block of XOR parity per stripe, so any single drive can be reconstructed from the others. That elegance has a hidden cost. When a drive dies, the array is degraded, and to rebuild the replacement the controller must read every surviving sector to recompute the missing blocks. There is no shortcut. A four-drive array of 16 TB disks reads 3 × 16 = 48 TB during a single rebuild.

The danger is the unrecoverable read error. Every spinning disk occasionally hits a sector it physically cannot read after exhausting its internal error-correction and retries. Manufacturers publish this as a rate: roughly 1 URE per 10^14 bits for consumer SATA drives and 1 per 10^15 bits for enterprise drives. Convert the consumer figure and 10^14 bits is about 12.5 TB. In normal operation a URE is a minor annoyance, because parity quietly repairs the bad sector. During a degraded rebuild there is no spare parity left to fix it, so the read fails and many controllers abort.

This is the mechanism behind so many homelab stories that end with a rebuild dying at 70 or 90 percent. The reconstruction sailed through most of the capacity, then snagged on one bad sector deep in a surviving drive and stopped. Controller behavior varies: enterprise controllers often "puncture" the bad stripe, mark it as lost, and finish the rebuild with one corrupt region, while many consumer controllers abort outright. You can model your own array's exposure with the RAID rebuild calculator by entering drive size, count, and rebuild speed.

The 10^14 math that scared the industry

In 2007 storage analyst Robin Harris published "Why RAID 5 stops working in 2009" on ZDNet. His argument was uncomfortably simple. Disk capacities double every 18 to 24 months, but the URE rate stays flat at 10^14, so the total data read during a rebuild was racing toward the 12.5 TB threshold where a URE becomes likely. Harris predicted that by 2009 a seven-drive array of 2 TB disks, reading about 12 TB during rebuild, would be highly likely to hit a URE and fail.

The probability behind both Harris and later researchers is the chance of reading every bit cleanly:

P(clean rebuild) = (1 - 1/b)^a
  b = bits per URE (10^14 consumer, 10^15 enterprise)
  a = total bits read during the rebuild
P(URE)  ≈ 1 - (1 - 1/b)^a

Run real numbers. An eight-drive RAID 5 of 8 TB disks rebuilds by reading the seven survivors: 56 TB, about 4.48 × 10^14 bits. With enterprise drives at 10^15, the URE probability is roughly 35 percent per rebuild. Swap in consumer drives at 10^14 and the same rebuild carries about a 99 percent worst-case chance of hitting a URE. On paper, a clean RAID 5 rebuild of a large consumer array looks like the exception rather than the rule.

That worst-case figure deserves an honest caveat. The 10^14 rating is a conservative upper bound, not an expected error count, and real-world rebuilds succeed far more often than the formula implies. Critics also note ambiguity over whether the rate applies per bit or per read operation, and field tests have measured far fewer errors than the spec predicts. Microsoft moved roughly 2 PB through low-cost hardware in a 2005 study and saw about 20 times fewer read errors than 10^14 would forecast. The threat is real and grows with capacity, but treat the eye-popping percentages as a planning ceiling, not a guarantee of failure.

Rebuild time, second failures, and the real risk

Rebuild duration is the simplest piece of the puzzle:

rebuild_hours = drive_capacity / effective_rebuild_speed

Effective speed is far below a drive's burst rating because production arrays still serve live I/O during the rebuild. A modest 30 MB/s sustained rate means a single 6 TB drive takes about 55 hours to rebuild, and every added terabyte adds roughly four more hours. Random workloads can cut effective speed by 30 to 60 percent. The longer that degraded window stays open, the more exposure you accumulate, which is why fast rebuilds and hot spares matter so much.

There are two independent threats during that window. The first is a second whole-drive failure, modeled with an exponential survival function:

P(survive) = e^(-N × RT / MTTF)
  N = drives read during rebuild
  RT = rebuild time
  MTTF = mean time to failure

The second is the URE. Here is the counterintuitive part that storage engineers learned the hard way: for modern high-capacity disks, a URE during a degraded rebuild is typically several times more likely than an outright second-drive failure. The thing most people fear, another disk dying, is often dwarfed by the thing they ignore, one tiny unreadable sector. The URE risk scales purely with bytes read, so a 4+1 array of 600 GB drives carries the same URE exposure as an 8+1 array of 300 GB drives; both read 2.4 TB.

Note that hardware RAID controllers are usually filesystem-blind, so they read every sector on every surviving drive, including empty space. A 16 TB member that is only 30 percent full still forces the controller through the full 16 TB, which is why filesystem-aware systems like ZFS, which only rebuild live data, finish faster and read less.

How RAID 6 rescues the rebuild

RAID 6 adds a second, independent parity block to every stripe. Rather than two XOR copies, it computes a second syndrome (the "Q" block) using Reed-Solomon coding alongside the simple XOR "P" block. The payoff is that RAID 6 survives two simultaneous drive losses. During a single-drive rebuild that second parity is the safety net: if the controller hits a URE on a surviving drive, it reconstructs that lost stripe from the second parity and keeps going instead of aborting. A URE that would kill a RAID 5 rebuild is a non-event in RAID 6.

This is why guidance shifted to "use RAID 6 for any HDD array with drives larger than about 4 TB." The same arithmetic that pressured RAID 5 keeps marching, though. In 2009 ZFS engineer Adam Leventhal published "Triple-Parity RAID and Beyond" in ACM Queue, arguing that as drive capacity outpaces throughput, rebuild times grow so long that reconstruction itself effectively strips away a layer of redundancy, and that double parity alone would eventually be insufficient. His answer was triple parity, shipped in ZFS as RAIDZ3.

Practical defenses if you run spinning disks: prefer enterprise-rated 10^15 drives, choose RAID 6 (or RAIDZ2/Z3) over RAID 5 at scale, keep a hot spare so rebuilds start in seconds, and run periodic scrubs that catch latent bad sectors while parity is still intact to repair them. One more lever changes the game: SSDs are commonly rated from 10^16 (consumer) to 10^18 (hardened enterprise), roughly a hundred to ten thousand times safer per bit than consumer HDDs, which makes URE-aborted rebuilds far rarer on flash arrays. Whatever your hardware, plug your drive size, count, and rebuild speed into the RAID rebuild risk calculator to see your own rebuild hours and second-failure probability before you trust the array, not after it has already lost a disk.

Frequently Asked Questions