7 ms·

this might sound like a troll comment but its coming from someone with almost zero experience with raid. What is the purpose of ZFS in 2021 if we have hardware

this might sound like a troll comment but its coming from someone with almost zero experience with raid. What is the purpose of ZFS in 2021 if we have hardware RAID and linux software RAID? BTRFS does RAID too. Why would people choose ZFS in 2021 if both Oracle and Open Source users have 2 competing ZFS? are they interoperable?

sz4kerto 5y ago
I don't want to be trolling either, but a simple Google search gives you really detailed answers. Or just look at Wikipedia: https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS Some highlights: hierarchical checksumming, CoW snapshots, deduplication, more efficient rebuilds, extremely configurable, tiered storage, various caching strategies, etc.
wyager 5y ago
ZFS RAID is the best RAID implementation in many respects. Hardware RAID is bad at actually fixing errors on disk (as opposed to just transparently correcting) and surfacing errors to the user. BTRFS is frequently not considered stable enough for production usage. ZFS has dozens of useful features besides RAID. Transparent compression, instant atomic snapshots, incremental snapshot sync, instant cloning of file systems, etc etc. Yes, different ZFS implementations are mostly compatible in my experience, and they should become totally compatible as everyone moves to OpenZFS. FreeBSD 13 and Linux currently have ZFS feature parity I believe.
- kenniskrag 5y ago
  > BTRFS is frequently not considered stable enough for production usage. synology NAS use btrfs by default. What is not stable?
  - ThatPlayer 5y ago
    Btrfs in single drive (and mirror/RAID1) modes are fine. The instability is in the RAID5/6 implementation[1]. Synology runs Btrfs (in single drive mode) on top of their own mdadm based RAID setup. [1]: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
- SkyMarshal 5y ago
  >BTRFS is frequently not considered stable enough for production usage. I'm not sure this is still true, especially after Facebook deployed it on millions of servers. https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users
  - bashinator 5y ago
    So what you’re saying is, btrfs works if you can afford an ops/sre team on par with Facebook’s?
  - chousuke 5y ago
    Do they really care if any one of those servers loses its data though? At that scale, you need higher level redundancy so it makes sense to prioritize filesystems for other features than how well they actually retain your data. For example, detecting corruption would be more important than never having any, so that you can get IO errors and immediately kill the server before it serves bad data.
  - maccam94 5y ago
    Specifically BTRFS's RAID5/RAID6 implementation is unstable and susceptible to data loss. Facebook doesn't use those RAID levels in production. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Warning-RAID5-RAID6 https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Wa... https://btrfs.wiki.kernel.org/index.php/Gotchas#Parity_RAID https://btrfs.wiki.kernel.org/index.php/Gotchas#Parity_RAID
tehbeard 5y ago
I can't speak with much experience, but what I have gleamed is. - You generally want to avoid hardware raid, if the card dies you'll likely need to source a compatible replacement vs. grabbing another SATA/was expander and reconstructing the array. - zfs handles the stack all the way from drives to filesystem, allowing them to work together (i.e filesystem usage info can better dictate what gets moved around tiered storage, or better raid recovery.
- LambdaComplex 5y ago
  My understanding is that hardware RAID is mainly a thing in the Windows world, because apparently its software RAID implementation is garbage
  - Tijdreiziger 5y ago
    I'm not a big Windows Server guy, but as I understand it, in Windows you would use Storage Spaces instead of traditional RAID.
    - tehbeard 5y ago
      Storage Spaces is their branded RAID solution. Apparently in the enterprise server versions it's fairly decent, but the desktop versions are pretty trash re: performance.
nickik 5y ago
> What is the purpose of ZFS in 2021 if we have hardware RAID Hardware RAID is actually older then ZFS style software RAID. ZFS was specifically designed fix the issues with hardware RAID. The problem with Hardware RAID is that is has no ideas what going on on top of it, and even worse, its a mostly a bunch of closed-source fireware from a vendor. And they cost money. You can find lots of terrible story about those. ZFS is open-source and battle tested. > linux software RAID Not sure what you are referring too. > BTRFS does RAID too. BTRFS is basically copied many of the features done in ZFS. BTRFS has a history of being far less stable. ZFS is far more battle tested. They say its stable now, but they had said that many times. It eat my data twice so I have not followed the project anymore. A file system in my opinion gets exactly 1 chance with me. They each have some features the other doesn't but broadly speaking they are similar technology. The new bcacheFS is also coming up and adding some interesting features. > Why would people choose ZFS in 2021 if both Oracle and Open Source users have 2 competing ZFS? Not sure what that has do with anything. Oracle is an evil company, they tried take all these great open source technologies away from people and the community thought against it. Most of the ZFS team left after the merger. The Open-Source version is arguable better, and has far more of the original designers working on it. The two code bases have diverged a lot since then. At the end of the day ZFS is incredibly battle tested, works incredibly well at what it does. And had a incredible reputation of stability basically since it came out. They question in my opinion is why not ZFS, then why ZFS.
- garmaine 5y ago
  He was referring to mdadm RAID.
- _tom_ 5y ago
  > ZFS is far more battle tested. They say its stable now, but they had said that many times. It eat my data twice Did you mean "it ate my data" to apply to ZFS? Or did you mean BTRFS?
  - znpy 5y ago
    It was probably BTRFS. I never fell for the BTRFS meme but many friends of mine did, and many of them ended up with a corrupted filesystem (and lost data).
boomboomsubban 5y ago
They are not interoperable but they're barely competing as Solaris is dead. Does Oracle Linux even offer Oracle ZFS? I assume they stick to btrfs considering they are the original developers. RAID does not feature the data protection offered by a copy on write filesystem, and OpenZFS is the most stable and portable option.
nix23 5y ago
It's pretty easy, having lots of experience with HW-Raid and SW-Raid, software it the way to go because: 1. Do you trust Firmware...i don't, i can tell you storys about freaking out san's...never had that with solaris or freebsd and zfs. 2. Why having a additional abstraction layer, HW Raid caching vs FS-Caching, no transparency for error correction, not smart raid rebuild etc. the list can go on and on, but HW-Raid is a thing of the past (exceptions are specialized san's etc)
usefulcat 5y ago
The last time I had to use HW raid it was horrible. The software for managing the RAID array was a poorly documented, difficult to use proprietary blob. I used it for years and the experience never improved. And this is a thing where if you make a mistake you can destroy the very data that you've gone to such lengths to protect. Having switched to ZFS several years ago, I lack to the words to express how much I don't miss having to deal with that.
rektide 5y ago
No matter what happens, people will seemingly forever declare BTRFS is not as stable and not as safe. There's a status page that details what BTRFS thinks of itself[1], and I doubt any of the many people docking BTRFS have read or know or care what that page says. There is one issue still being worked out to completion, a "write hole" problem, involving two separate failures, an unplanned/power-loss shut-down, followed by a second disk failure, which can result in some data being lost[2] in RAID5/6 scenarios. Other than that one extreme double-failure scenario being worked out, BTRFS has proven remarkably stable for a while now. A decade ago that wasn't quite as absolutely bulletproof, but today the situation is much different. Personally, it feels to me like there is a persistent & vocal small group of people who seemingly either have some agenda that makes them not wish to consider BTRFS, or they are unwilling to review & reconsider how things might have changed in the last decade. Not to belabor the point but it's quite frustrating, and it feels a bit odd that BTRFS is such a persistent target of slander & assault. Few other file systems seem to face anywhere near as much criticism, never so out of hand/casually, and honestly, in the end, it just seems like there's some continent of ZFS folks with some strange need to make themselves feel better by putting others down. One big sign of trust: Fedora 35 Cloud looks likely to switch to BTRFS as default[3], following Fedora 33 desktop lat year making the move. A number of big names use BTRFS, including Facebook. I have yet to see any hyperscalers interested in ZFS. I'm excited to see ZFS start to get some competent expandability. Expanding ZFS used to be a nightmare. I'll continue running BTRFS for now, but I'm excited to see file systems flourish. Things I wouldn't do? Hardware RAID. Controllers are persnickety weird devices, each with their own invisible sets of constraints & specific firmware issues. If at all possible, I'd prefer the kernel figure out how to make effective use out of multiple disks. BTRFS, and now it seems ZFS perhaps too, do a magical job of making that easy, effective, & fast, in a safe way. Edit: the current widely-adopted write hole fix is to use RAID1 or RAID1c3 or RAID1c4 (3 copy RAID1, 4 copy RAID1) for meta-data, RAID5/6 for data. [1] https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status [2] https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56 [3] https://www.phoronix.com/scan.php?page=news_item&px=Fedora-Cloud-35-Doing-Btrfs https://www.phoronix.com/scan.php?page=news_item&px=Fedora-C...
- webmobdev 5y ago
magicalhippo 5y ago
> What is the purpose of ZFS in 2021 if we have hardware RAID and linux software RAID? Others have touched on the main points, I just wanted to stress that an important distinction between ZFS and hardware RAID and linux software RAID (by which I assume you mean MD) is that the latter two present themselves as block devices. One has to put a file system on top to make use of them. In contrast, ZFS does away with this traditional split, and provides a filesystem as well as support for a virtual block device. By unifying the full stack from the filesystem down to the actual devices, it can be smarter and more resilient. The first few minutes of this[1] presentation does a good job of explaining why ZFS was built this way and how it improves on the traditional RAID solutions. [1]: https://www.youtube.com/watch?v=MsY-BafQgj4 https://www.youtube.com/watch?v=MsY-BafQgj4
Quekid5 5y ago
> hardware RAID That's just the worst of all worlds: Usually proprietary and you get the extreme aversion to improvement (or any change really) of hardware vendors. This ZFS changes is going to come, and it may end up being complex to implement for users... but it's happening. At the risk of being hyperbolic: Something like would never be possible with a HW raid system unless it had explicitly been designed for it from the start. Also: ZFS does much more than any hardware RAID ever did.
MrStonedOne 5y ago
with hardware raid, if you have data corruption that doesn't bring down a whole drive, you have no idea which data copy is the correct one. (at least not in an automated way that doesn't require manual work) With zfs, it has checksumming so it knows which data copy is the correct one. As drives get bigger, the odds of bit decay, while being low on a per-bit level, become great enough for the drive in total, and is a concern. File systems have had checksumming, but if there is no exposure into the raid layer, the file system can't use this checksum to recover from bit decay errors since it can't control which drive processes a request. This is why zfs operates at both layers. zfs vs btrfs is harder, and i'd guess its the zfs layered read and write caching systems enticing people in.
seized 5y ago
The counter is what is the point of Btrfs and Linux software RAID when ZFS is better in many ways.... Btrfs is not stable in all configurations. Mdadm and others don't do checksumming, scrubs, health checks, etc as well as ZFS (or at all). That's not even touching on built in encryption, compression, snapshots, boot environments and more.
jjav 5y ago
> What is the purpose of ZFS in 2021 if we have hardware RAID Hardware RAID controllers predate ZFS by a long time.. ZFS is a much more modern design and because it integrates the whole storage layer it can offer all the features it does, which a RAID controller hiding behind a disk interface cannot do. When ZFS came out many people (me included) considered that the end of relevance for hardware RAID controllers. I used to use hardware RAID pre-ZFS but have never again after switching to ZFS when Solaris 10 first included it.
exikyut 5y ago
The TL;DR is that, IIUC, RAID depends on disks being 100% perfect with exactly zero read errors, ever, until they go *clonk* and die all at once. Today's many-TB hard drives use mind-boggling engineering to shrink data fluctuations down almost to the size of individual atoms (not quite there yet, but making progress (!)), while higher I/O speeds push the onboard processors' ECC mechanisms almost beyond breaking point. This basically means that the likelihood of HDDs returning uncaught bit errors has gone from a maybe-once-in-a-lifetime event (with very large magnetic flux sizes on disk in the 80s) to something that individual power users using disks with great-looking SMART output should generally expect to see every few months or years+. My rule of thumb is that any storage device over maybe 250GB-500GB in size needs to be redundant. If you can shove that much data into one place, the odds that something significant will be hiding somewhere in that data easily go beyond 100% once you go past that size range, IMO.

sz4kerto 5y ago

I don't want to be trolling either, but a simple Google search gives you really detailed answers. Or just look at Wikipedia: https://en.wikipedia.org/wiki/ZFS https://en.wikipedia.org/wiki/ZFS Some highlights: hierarchical checksumming, CoW snapshots, deduplication, more efficient rebuilds, extremely configurable, tiered storage, various caching strategies, etc.

wyager 5y ago

ZFS RAID is the best RAID implementation in many respects. Hardware RAID is bad at actually fixing errors on disk (as opposed to just transparently correcting) and surfacing errors to the user. BTRFS is frequently not considered stable enough for production usage. ZFS has dozens of useful features besides RAID. Transparent compression, instant atomic snapshots, incremental snapshot sync, instant cloning of file systems, etc etc. Yes, different ZFS implementations are mostly compatible in my experience, and they should become totally compatible as everyone moves to OpenZFS. FreeBSD 13 and Linux currently have ZFS feature parity I believe.

kenniskrag 5y ago
> BTRFS is frequently not considered stable enough for production usage. synology NAS use btrfs by default. What is not stable?
- ThatPlayer 5y ago
  Btrfs in single drive (and mirror/RAID1) modes are fine. The instability is in the RAID5/6 implementation[1]. Synology runs Btrfs (in single drive mode) on top of their own mdadm based RAID setup. [1]: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
SkyMarshal 5y ago
>BTRFS is frequently not considered stable enough for production usage. I'm not sure this is still true, especially after Facebook deployed it on millions of servers. https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users https://btrfs.wiki.kernel.org/index.php/Production_Users
- bashinator 5y ago
  So what you’re saying is, btrfs works if you can afford an ops/sre team on par with Facebook’s?
- chousuke 5y ago
  Do they really care if any one of those servers loses its data though? At that scale, you need higher level redundancy so it makes sense to prioritize filesystems for other features than how well they actually retain your data. For example, detecting corruption would be more important than never having any, so that you can get IO errors and immediately kill the server before it serves bad data.
- maccam94 5y ago
  Specifically BTRFS's RAID5/RAID6 implementation is unstable and susceptible to data loss. Facebook doesn't use those RAID levels in production. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Warning-RAID5-RAID6 https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Wa... https://btrfs.wiki.kernel.org/index.php/Gotchas#Parity_RAID https://btrfs.wiki.kernel.org/index.php/Gotchas#Parity_RAID

tehbeard 5y ago

I can't speak with much experience, but what I have gleamed is. - You generally want to avoid hardware raid, if the card dies you'll likely need to source a compatible replacement vs. grabbing another SATA/was expander and reconstructing the array. - zfs handles the stack all the way from drives to filesystem, allowing them to work together (i.e filesystem usage info can better dictate what gets moved around tiered storage, or better raid recovery.

LambdaComplex 5y ago
My understanding is that hardware RAID is mainly a thing in the Windows world, because apparently its software RAID implementation is garbage
- Tijdreiziger 5y ago
  I'm not a big Windows Server guy, but as I understand it, in Windows you would use Storage Spaces instead of traditional RAID.
  - tehbeard 5y ago
    Storage Spaces is their branded RAID solution. Apparently in the enterprise server versions it's fairly decent, but the desktop versions are pretty trash re: performance.

nickik 5y ago

> What is the purpose of ZFS in 2021 if we have hardware RAID Hardware RAID is actually older then ZFS style software RAID. ZFS was specifically designed fix the issues with hardware RAID. The problem with Hardware RAID is that is has no ideas what going on on top of it, and even worse, its a mostly a bunch of closed-source fireware from a vendor. And they cost money. You can find lots of terrible story about those. ZFS is open-source and battle tested. > linux software RAID Not sure what you are referring too. > BTRFS does RAID too. BTRFS is basically copied many of the features done in ZFS. BTRFS has a history of being far less stable. ZFS is far more battle tested. They say its stable now, but they had said that many times. It eat my data twice so I have not followed the project anymore. A file system in my opinion gets exactly 1 chance with me. They each have some features the other doesn't but broadly speaking they are similar technology. The new bcacheFS is also coming up and adding some interesting features. > Why would people choose ZFS in 2021 if both Oracle and Open Source users have 2 competing ZFS? Not sure what that has do with anything. Oracle is an evil company, they tried take all these great open source technologies away from people and the community thought against it. Most of the ZFS team left after the merger. The Open-Source version is arguable better, and has far more of the original designers working on it. The two code bases have diverged a lot since then. At the end of the day ZFS is incredibly battle tested, works incredibly well at what it does. And had a incredible reputation of stability basically since it came out. They question in my opinion is why not ZFS, then why ZFS.

garmaine 5y ago
He was referring to mdadm RAID.
_tom_ 5y ago
> ZFS is far more battle tested. They say its stable now, but they had said that many times. It eat my data twice Did you mean "it ate my data" to apply to ZFS? Or did you mean BTRFS?
- znpy 5y ago
  It was probably BTRFS. I never fell for the BTRFS meme but many friends of mine did, and many of them ended up with a corrupted filesystem (and lost data).

boomboomsubban 5y ago

They are not interoperable but they're barely competing as Solaris is dead. Does Oracle Linux even offer Oracle ZFS? I assume they stick to btrfs considering they are the original developers. RAID does not feature the data protection offered by a copy on write filesystem, and OpenZFS is the most stable and portable option.

nix23 5y ago

It's pretty easy, having lots of experience with HW-Raid and SW-Raid, software it the way to go because: 1. Do you trust Firmware...i don't, i can tell you storys about freaking out san's...never had that with solaris or freebsd and zfs. 2. Why having a additional abstraction layer, HW Raid caching vs FS-Caching, no transparency for error correction, not smart raid rebuild etc. the list can go on and on, but HW-Raid is a thing of the past (exceptions are specialized san's etc)

usefulcat 5y ago

The last time I had to use HW raid it was horrible. The software for managing the RAID array was a poorly documented, difficult to use proprietary blob. I used it for years and the experience never improved. And this is a thing where if you make a mistake you can destroy the very data that you've gone to such lengths to protect. Having switched to ZFS several years ago, I lack to the words to express how much I don't miss having to deal with that.

rektide 5y ago

No matter what happens, people will seemingly forever declare BTRFS is not as stable and not as safe. There's a status page that details what BTRFS thinks of itself[1], and I doubt any of the many people docking BTRFS have read or know or care what that page says. There is one issue still being worked out to completion, a "write hole" problem, involving two separate failures, an unplanned/power-loss shut-down, followed by a second disk failure, which can result in some data being lost[2] in RAID5/6 scenarios. Other than that one extreme double-failure scenario being worked out, BTRFS has proven remarkably stable for a while now. A decade ago that wasn't quite as absolutely bulletproof, but today the situation is much different. Personally, it feels to me like there is a persistent & vocal small group of people who seemingly either have some agenda that makes them not wish to consider BTRFS, or they are unwilling to review & reconsider how things might have changed in the last decade. Not to belabor the point but it's quite frustrating, and it feels a bit odd that BTRFS is such a persistent target of slander & assault. Few other file systems seem to face anywhere near as much criticism, never so out of hand/casually, and honestly, in the end, it just seems like there's some continent of ZFS folks with some strange need to make themselves feel better by putting others down. One big sign of trust: Fedora 35 Cloud looks likely to switch to BTRFS as default[3], following Fedora 33 desktop lat year making the move. A number of big names use BTRFS, including Facebook. I have yet to see any hyperscalers interested in ZFS. I'm excited to see ZFS start to get some competent expandability. Expanding ZFS used to be a nightmare. I'll continue running BTRFS for now, but I'm excited to see file systems flourish. Things I wouldn't do? Hardware RAID. Controllers are persnickety weird devices, each with their own invisible sets of constraints & specific firmware issues. If at all possible, I'd prefer the kernel figure out how to make effective use out of multiple disks. BTRFS, and now it seems ZFS perhaps too, do a magical job of making that easy, effective, & fast, in a safe way. Edit: the current widely-adopted write hole fix is to use RAID1 or RAID1c3 or RAID1c4 (3 copy RAID1, 4 copy RAID1) for meta-data, RAID5/6 for data. [1] https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status [2] https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56 [3] https://www.phoronix.com/scan.php?page=news_item&px=Fedora-Cloud-35-Doing-Btrfs https://www.phoronix.com/scan.php?page=news_item&px=Fedora-C...

webmobdev 5y ago

magicalhippo 5y ago

> What is the purpose of ZFS in 2021 if we have hardware RAID and linux software RAID? Others have touched on the main points, I just wanted to stress that an important distinction between ZFS and hardware RAID and linux software RAID (by which I assume you mean MD) is that the latter two present themselves as block devices. One has to put a file system on top to make use of them. In contrast, ZFS does away with this traditional split, and provides a filesystem as well as support for a virtual block device. By unifying the full stack from the filesystem down to the actual devices, it can be smarter and more resilient. The first few minutes of this[1] presentation does a good job of explaining why ZFS was built this way and how it improves on the traditional RAID solutions. [1]: https://www.youtube.com/watch?v=MsY-BafQgj4 https://www.youtube.com/watch?v=MsY-BafQgj4

Quekid5 5y ago

> hardware RAID That's just the worst of all worlds: Usually proprietary and you get the extreme aversion to improvement (or any change really) of hardware vendors. This ZFS changes is going to come, and it may end up being complex to implement for users... but it's happening. At the risk of being hyperbolic: Something like would never be possible with a HW raid system unless it had explicitly been designed for it from the start. Also: ZFS does much more than any hardware RAID ever did.

MrStonedOne 5y ago

with hardware raid, if you have data corruption that doesn't bring down a whole drive, you have no idea which data copy is the correct one. (at least not in an automated way that doesn't require manual work) With zfs, it has checksumming so it knows which data copy is the correct one. As drives get bigger, the odds of bit decay, while being low on a per-bit level, become great enough for the drive in total, and is a concern. File systems have had checksumming, but if there is no exposure into the raid layer, the file system can't use this checksum to recover from bit decay errors since it can't control which drive processes a request. This is why zfs operates at both layers. zfs vs btrfs is harder, and i'd guess its the zfs layered read and write caching systems enticing people in.

seized 5y ago

The counter is what is the point of Btrfs and Linux software RAID when ZFS is better in many ways.... Btrfs is not stable in all configurations. Mdadm and others don't do checksumming, scrubs, health checks, etc as well as ZFS (or at all). That's not even touching on built in encryption, compression, snapshots, boot environments and more.

jjav 5y ago

> What is the purpose of ZFS in 2021 if we have hardware RAID Hardware RAID controllers predate ZFS by a long time.. ZFS is a much more modern design and because it integrates the whole storage layer it can offer all the features it does, which a RAID controller hiding behind a disk interface cannot do. When ZFS came out many people (me included) considered that the end of relevance for hardware RAID controllers. I used to use hardware RAID pre-ZFS but have never again after switching to ZFS when Solaris 10 first included it.

exikyut 5y ago

The TL;DR is that, IIUC, RAID depends on disks being 100% perfect with exactly zero read errors, ever, until they go *clonk* and die all at once. Today's many-TB hard drives use mind-boggling engineering to shrink data fluctuations down almost to the size of individual atoms (not quite there yet, but making progress (!)), while higher I/O speeds push the onboard processors' ECC mechanisms almost beyond breaking point. This basically means that the likelihood of HDDs returning uncaught bit errors has gone from a maybe-once-in-a-lifetime event (with very large magnetic flux sizes on disk in the 80s) to something that individual power users using disks with great-looking SMART output should generally expect to see every few months or years+. My rule of thumb is that any storage device over maybe 250GB-500GB in size needs to be redundant. If you can shove that much data into one place, the odds that something significant will be hiding somewhere in that data easily go beyond 100% once you go past that size range, IMO.