The RAID-5 Write Hole

This is an interesting issue that I think far too few of us know about or acknowledge. When I first learned about this I was converting my fileserver from Linux to Windows (I have good reasons). Although I found several references to "the possibility for data corruption using RAID-5", I found no explanation of the issue.

Finally, I have come across the explanation I sought.

From Wikipedia... (search for "write hole")

"In the event of a system failure while there are active writes, the parity of a stripe may become inconsistent with the data; if this is not detected and repaired before a disk or block fails, data loss may ensue as incorrect parity will be used to reconstruct the missing block in that stripe; this potential vulnerability is sometimes known as the "write hole". Battery-backed cache and other techniques are commonly used to reduce the window of vulnerability of this occurring."

From Jeff Bonwick's Weblog...

"RAID-5 (and other data/parity schemes such as RAID-4, RAID-6, even-odd, and Row Diagonal Parity) never quite delivered on the RAID promise -- and can't -- due to a fatal flaw known as the RAID-5 write hole. Whenever you update the data in a RAID stripe you must also update the parity, so that all disks XOR to zero -- it's that equation that allows you to reconstruct data when a disk fails. The problem is that there's no way to update two or more disks atomically, so RAID stripes can become damaged during a crash or power outage.

"To see this, suppose you lose power after writing a data block but before writing the corresponding parity block. Now the data and parity for that stripe are inconsistent, and they'll remain inconsistent forever (unless you happen to overwrite the old data with a full-stripe write at some point). Therefore, if a disk fails, the RAID reconstruction process will generate garbage the next time you read any block on that stripe. What's worse, it will do so silently -- it has no idea that it's giving you corrupt data."

In the following entry, Eric Lowe is talking about data corruption not from the RAID-5 Write Hole, but from disk failures. The symptom is similar. From Eric Lowe's Blog...

"it was a fatal checksum error. One of those "you might want to know that your data just went away" sort of errors. Of course, I had been running UFS on this disk for about a year, and apparently never noticed the silent data corruption. But then I reached into the far recesses of my brain, and I recalled a few strange moments -- like the one time when I did a bringover into a workspace on the disk, and I got a conflict on a file I hadn't changed. Or the other time after a reboot I got a strange panic in UFS while it was unrolling the log."

It seems that the best solution to this problem is to either use RAID-1 for redundancy, or to use RAID-Z and ZFS. I'm sure EMC has a solution, but as Jeff Bonwick writes in his blog above, the whole concept of RAID was centered around *CHEAP*. On the software RAID solution side of things, we seem to have little recourse. Buy a UPS, install a UPS agent, and set the system up to shut itself down cleanly if there is ever a power failure. For now that may have to do.

 Create your own free wiki!HelpPublic, not yet premium Privacy Policy RSS