6 ms·

Tangentially related to ZFS, I’d like some advice. I’ve got a couple of projects I’d like to do that require maybe 100TB of storage: some scientometrics agains

Tangentially related to ZFS, I’d like some advice. I’ve got a couple of projects I’d like to do that require maybe 100TB of storage: some scientometrics against the sci-hub collection, as well as building a bajillion scala projects from github. I don’t really care about data redundancy. The cheapest way I’ve been able to figure out to do this is just buy a case with 15 HDD bays, eg the Anidees AI crystal case, and just get a ridiculously beefy processor + 256gb RAM and do all the computation on a single box. Does this sound right? All of the purchasable NAS cases all seem more expensive, but I’m out of my element. I expect I’m going to want to figure out ZFS to make a single logical drive. Does this seem right? Building eg a backblaze pod is outside of my budget, and my eyes glaze over whenever I try to read about NAS controllers.

threatripper 6y ago
This sounds about right. Not very reliable but cheap and easy to set up. If you make a software raid (e.g. with lvm) you could as well use any other file system.
- secabeen 6y ago
  Yep. If you're looking for more specific recommendations, the community at /r/datahoarder is a great source. With shucked USB drives, and just the biggest ATX case you can find, your costs should be pretty low.
  - roflc0ptic 6y ago
    Thanks for the input, both of y’all. Roger that about shucking drives. It is strange to me that ~10 TB external drives end up being the cheapest solution, but happy to capitalize on it
jcastro 6y ago
This is what I did, I just snagged some cheap Rosewill case off of newegg with 12 bays, a PCIe sata card and packed the case with drives. You can slice and dice the drives however you like. I found this guy's blog posts to be useful for running a homegrown NAS: - https://louwrentius.com/should-i-use-zfs-for-my-home-nas.html https://louwrentius.com/should-i-use-zfs-for-my-home-nas.htm... - https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you...
- roflc0ptic 6y ago
  Oh, cool. Being routed away from ZFS is interesting. Happy to do this as simply as possible
- bsder 6y ago
  The only real beef I have is this: "It seems though that the one URE in 10^14^ bits (an error every 12.5 TB of data read) is a worst-case specification. In real life, drives are way more reliable than this specification." I don't believe that statement without data backing it up. Disk drives are right at the engineering edge. There is a reason why consumer class drives have one MTBF and enterprise drives have a better MTBF. If a disk drive manufacturer could somehow cite a better MTBF, they absolutely would (see the whole GB vs GiB marketing stupidity, for example).
iooi 6y ago
Another +1 for the Rosewill 12-bay/15-bay cases. If you're going to run this server anywhere near you, stay away from server cases like supermicro, dell, etc. since you will suffer hearing loss if you're around them for long periods of time. If you plan on using ZFS, make sure to take into account for the overhead and parity drives. For example, I have 12x 16TB (192TB raw) drives in raidzfs2, including overhead and parity I only have ~125TiB usable.
generalizations 6y ago
You might want to consider at least a minimum of redundancy, like a RAID 5 (lets you lose 1 drive without killing the whole setup). It would be a real pain to set everything up, discover one of your drives is a dud, and have to start over from scratch. Especially with that many HDDs involved, the risk might be somewhat high.
Lammy 6y ago
Look into iStarUSA cases. Even my tiny 2U is big enough for Noctua 80mm fans that keep it quiet as well as cool.
vermaden 6y ago
Not sure if it will directly help you but can help in indirectly. This is 1080 TB (in 12 TB disks) in one server under ZFS: https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/ https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise... For 100 TB you would get smaller server, like 2U with 12 slots filled with 10/12/14 TB disks for your 'usable' 100 TB space. For more ZFS and/or FreeBSD storage options check this: https://vermaden.wordpress.com/2020/02/12/freebsd-enterprise-storage-at-pbug/ https://vermaden.wordpress.com/2020/02/12/freebsd-enterprise... Hope that helps. Regards.
- codethief 6y ago
  > 90 x Toshiba HDD MN07ACA12TE 12 TB (Data) I'm surprised you went with one single HDD model. Don't people normally say "Go for different HDDs from different manufacturers, so that in case of manufacturing defects not all HDDs will break at the same time"?

threatripper 6y ago

This sounds about right. Not very reliable but cheap and easy to set up. If you make a software raid (e.g. with lvm) you could as well use any other file system.

secabeen 6y ago
Yep. If you're looking for more specific recommendations, the community at /r/datahoarder is a great source. With shucked USB drives, and just the biggest ATX case you can find, your costs should be pretty low.
- roflc0ptic 6y ago
  Thanks for the input, both of y’all. Roger that about shucking drives. It is strange to me that ~10 TB external drives end up being the cheapest solution, but happy to capitalize on it

jcastro 6y ago

This is what I did, I just snagged some cheap Rosewill case off of newegg with 12 bays, a PCIe sata card and packed the case with drives. You can slice and dice the drives however you like. I found this guy's blog posts to be useful for running a homegrown NAS: - https://louwrentius.com/should-i-use-zfs-for-my-home-nas.html https://louwrentius.com/should-i-use-zfs-for-my-home-nas.htm... - https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you...

roflc0ptic 6y ago
Oh, cool. Being routed away from ZFS is interesting. Happy to do this as simply as possible
bsder 6y ago
The only real beef I have is this: "It seems though that the one URE in 10^14^ bits (an error every 12.5 TB of data read) is a worst-case specification. In real life, drives are way more reliable than this specification." I don't believe that statement without data backing it up. Disk drives are right at the engineering edge. There is a reason why consumer class drives have one MTBF and enterprise drives have a better MTBF. If a disk drive manufacturer could somehow cite a better MTBF, they absolutely would (see the whole GB vs GiB marketing stupidity, for example).

iooi 6y ago

Another +1 for the Rosewill 12-bay/15-bay cases. If you're going to run this server anywhere near you, stay away from server cases like supermicro, dell, etc. since you will suffer hearing loss if you're around them for long periods of time. If you plan on using ZFS, make sure to take into account for the overhead and parity drives. For example, I have 12x 16TB (192TB raw) drives in raidzfs2, including overhead and parity I only have ~125TiB usable.

generalizations 6y ago

You might want to consider at least a minimum of redundancy, like a RAID 5 (lets you lose 1 drive without killing the whole setup). It would be a real pain to set everything up, discover one of your drives is a dud, and have to start over from scratch. Especially with that many HDDs involved, the risk might be somewhat high.

Lammy 6y ago

Look into iStarUSA cases. Even my tiny 2U is big enough for Noctua 80mm fans that keep it quiet as well as cool.

vermaden 6y ago

Not sure if it will directly help you but can help in indirectly. This is 1080 TB (in 12 TB disks) in one server under ZFS: https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise-1-pb-storage/ https://vermaden.wordpress.com/2019/06/19/freebsd-enterprise... For 100 TB you would get smaller server, like 2U with 12 slots filled with 10/12/14 TB disks for your 'usable' 100 TB space. For more ZFS and/or FreeBSD storage options check this: https://vermaden.wordpress.com/2020/02/12/freebsd-enterprise-storage-at-pbug/ https://vermaden.wordpress.com/2020/02/12/freebsd-enterprise... Hope that helps. Regards.

codethief 6y ago
> 90 x Toshiba HDD MN07ACA12TE 12 TB (Data) I'm surprised you went with one single HDD model. Don't people normally say "Go for different HDDs from different manufacturers, so that in case of manufacturing defects not all HDDs will break at the same time"?