15 ms·

832 TB – ZFS on Linux

320 points by beagle3 9y ago

4ad 9y ago
you can run ZFS on Ubuntu [...] You could also build this on Solaris with necessary licensing if you wanted to that route but it’d be more expensive. I find it bewildering the author didn't even consider illumos or FreeBSD, where ZFS is a first class citizen.
- howeyc 9y ago
  I thought the same thing, they sort of answers in the comments. Something about freenas saying no support and that the put together systems from iXsystems have lower drive bay counts. My guess is they just prefer Linux. FreeBSD or illumos would definitely work.
- qeternity 9y ago
  For someone not very up to date with all the various ZFS flavors, can you please explain this a bit?
  - overcast 9y ago
    ZFS is a native filesystem to FreeBSD, where as in Linux it is not.
    - X86BSD 9y ago
      It's also had more baking time on FreeBSD.
    - qeternity 9y ago
      Sure but that alone doesn't mean it's any less suitable on Ubuntu. I'm just trying to understand what the differences are in implementation, reliability, etc.
      - cyphar 9y ago
        ZFS-on-Linux has not been mainstream (or existed) for as long as ZFS has been integrated inside FreeBSD. illumos is the free software fork of OpenSolaris -- so it's the repo of record for modern ZFS development.
        waz0wski 9y ago
        OpenZFS is the parent project from which ZFS is applied to Illumos, FreeBSD, and ZFSoL http://open-zfs.org/wiki/Main_Page http://open-zfs.org/wiki/Main_Page
        cyphar 9y ago
        I'm aware of that, I just didn't want to additionally confuse someone who is barely familiar with ZFS with the whole OpenZFS fork and so on.
      - gruturo 9y ago
        ZFS on FreeBSD and on Illumos has been used, abused, tested and stressed all the way to the Moon and beyond. Loads of people with loads of data over loads time. This makes bugs appear, and get addressed, and eventually gives you confidence that nothing nasty remains uncorrected, and it won't eat your data. ZFS on Linux, due to the unfortunate licensing situation, is considerably less tested and thus scary, data-eating bugs are at least a bit more likely to exist.
        zorked 9y ago
        On the other hand Linux is far more popular so I wouldn't be surprised if ZFS on Linux already clocks more hours of usage than on BSD.
        gruturo 9y ago
        > On the other hand Linux is far more popular so I wouldn't be surprised if ZFS on Linux already clocks more hours of usage than on BSD. Yes and no. Few people, even among the Linux crowd, are aware it exists or feel like trusting it with important data sets/jobs. Those few are the most likely to be equally at ease running FreeBSD or Illumos (or Solaris), where the damn thing is known to work really well.
- seany 9y ago
  I run napp-it on omni-os at home and work with good results. It's not too hard to adapt your linux knowledge if you're not used to the solaris-ish userspace.
- X86BSD 9y ago
  Yeah I was like uhhhh what? Illumos and illumos based OS'es and FreeBSD are the only games in town where ZFS is a first class citizen. Ill chalk it up to extreme bias.
  - floatboth 9y ago
    Yeah, many people don't consider FreeBSD because of silly reasons… :( but mentioning Solaris that "requires licensing" and not illumos?? illumos is the new Solaris, the old Solaris devs are working on illumos! Oracle Solaris is irrelevant.
    - al452 9y ago
      > Oracle Solaris is irrelevant. Amen to that.
- notpeter 9y ago
  It's likely a human resources problem. For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux. And those numbers are much worse outside of major cities. The commercial support from Ubuntu tips the scales. I made this same decision at my last job. I ran Solaris and Illumos on our file servers and loved it, but a year before I left I ported all the pools to Ubuntu so my successor only needed to be Linux competent and ZFS trainable. Sometimes when choosing tech it's not what's technologically superior solution nor what you personally could run well, it's what's best for your coworkers, your successor and the organization in the long term.
  - rdeckard0104 9y ago
    I disagree. FreeBSD is just as easy to administer as Linux. Solaris I find to be a bit more challenging. I can appreciate your commentary, but I challenge you to prove me wrong. Just for some color I've ran the Gambit of Unix and Linux systems in my career with FreeBSD being one of the easiest and most consistent out of the box technologies. Solaris had already been quite the opposite. I guess you could say AIX is a close second to Solaris, but I think that's a bit of a grey area considering how opinionated that operating system is. To be clear I'm not a FreeBSD zealot. I'm for what works. Right now what works is the emerging container based solutions albeit not relevant to this discussion.
    - ethbro 9y ago
      > For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux. Was parent wrong about that point?
      - chungy 9y ago
        Not exactly, but to any competent administrator, learning FreeBSD based on Linux experience is not that difficult. It's sort of like tasking someone to start working on a program written in Go even if they have only C++ experience. You can cope.
        9y ago
- panzerboy 9y ago
  He explained on Reddit that he has some support from Canonical as well.
twiss 9y ago
> I purchased these units through a vendor we like to use and they hooked us up, so I won’t be able to share my specific pricing. (...) If you build the systems out on there you’ll find that they come in around $35,000 (USD) each. That devided by 52 x 8 = 416TB is 0.084$/GB. For comparison, the Backblaze Storage Pod 6.0 [1] claims 0.06$/GB for the version with the same hard drives. Although this version has a bunch of extra features like 2 x 800GB SSD's for ZFS SLOG, 8x more RAM for a total of 256GB, etc. [1]: https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv...
rsync 9y ago
"I ended up between the Supermicro SSG-6048R-E1CR60L or the SSG-6048R-E1CR90L – the E1CR60L is a 60-bay 4U chassis while the E1CR90L is a 90-bay 4U chassis. This nice part is that no matter which platform you choose Supermicro sells this only as a pre-configured machine – this means that their engineers are going to make sure that the hardware you choose to put in this is all from a known compatibility list. Basically, you cannot buy this chassis empty and jam your own parts in" This is a major departure from the Supermicro business model and practices and basically broke all of our next generation expansion roadmaps. This was not a technical decision - it is the same old economic decision that every large VAR/integrator/supplier has succumbed to for the last 30 years. They aren't the first ones to try this trick and they won't be the last. We (rsync.net) are not playing ball, however. After 16 years of deploying solely on supermicro hardware (server chassis and JBODs) we bought our first non-supermicro JBOD last month.
- wmf 9y ago
  What specifically is the problem? Overpriced drives?
  - rsync 9y ago
    Yes. $30-$50 each. Multiply that by 60 or 90 and multiply that by (however many you put in a rack). That adds up to five figure premiums on drives we're going to burn in anyway. We know how to read an HCL - it's not rocket science.
    - flattone 9y ago
      Even if reading hcl were rocket science a more validating vendor would preserve buyer self determination
    - qaq 9y ago
      Ok 60X50 is 3K even with extra 3K Supermicro is still a lot cheeper
      - 9y ago
sandGorgon 9y ago
The most important line for me was "Today, you can run ZFS on Ubuntu 16.0.4.2 LTS with standard repositories and Canonical’s Ubuntu Advantage Advanced Support. That makes the decision easy." Its highly interesting that Canonical does this with ZFS. I'm not sure why they dont market this more.
- tylerjl 9y ago
  I'm only a casual user of ZFS on Linux for personal storage projects, but I've spoken with people who rely very heavily on ZFS on Linux for their small businesses, and it's interesting to hear their perspectives on this. Essentially, because btrfs has failed to deliver on the next-gen filesystem front, ZFS on Linux is such a critical piece of technology that unless Red Hat has an answer soon for out-of-the-box ZFS on Linux, Canonical has a pretty staggering advantage. One theory is that Canonical has essentially forced Oracle to decide whether it wants to crack down on inclusion of CDDL-licensed code being shipped with Ubuntu's stock kernel, so Red Hat may be waiting to see if Oracle sweeps in or not before following suit. I'd be very surprised if RHEL (by observing the progression of Fedora development) continues to bet on btrfs, as I have yet to encounter anyone (including myself) who would ever trust btrfs over ZFS on Linux with anything of importance based on their experiences with the two - my experience is anecdotal, but ZFS has been just as reliable on *BSD as with any Linux distribution.
  - majormjr 9y ago
    btrfs is getting dropped from future versions of RHEL so you can see what Red Hat thinks of the future of btrfs.
    - user5994461 9y ago
      RedHat is betting on xfs.
    - gribbly 9y ago
      Red Hat has never spent any real resources on btrfs, they've been all about XFS when it comes to enterprise needs.
SoMisanthrope 9y ago
Very impressive! It's amazing what people are doing with OTS technology.
notyourday 9y ago
Very good experience with 45drives.com storinator XLs.
jaytaylor 9y ago
What about cooling? Will the lifespan of the high-capacity platter-dense hard drives be drastically reduced by clumping them together like that with what looks like little airflow?
- comboy 9y ago
  AFAIR from the backblaze blog, a bigger issue that shortens the lifespan is vibration.
  - exikyut 9y ago
    Could that be mitigated by redesigning the chassis for slightly less density in order to fit massive rubber standoffs/grommets?
  - deelowe 9y ago
    That's probably because their design doesn't do much to mitigate it.
mikekij 9y ago
Almost big enough to archive SoundCloud!
cmurf 9y ago
Do either of these project spec hardware that would work for this use case? opencompute.org Backblaze storage pod, they're up to v 6.0 now (Netflix open connect specs supermicro hardware) Others?
- electrum 9y ago
  The next generation Facebook design for storage is Bryce Canyon, available through the Open Compute Project: https://code.facebook.com/posts/1869788206569924/introducing-bryce-canyon-our-next-generation-storage-platform/ https://code.facebook.com/posts/1869788206569924/introducing...
andreiw 9y ago
I wish Supermicro had a similar chassis around the Cavium ThunderX. That would make a lot of sense for network-attached storage, regardless of whether one goes with SATA or drops in a SAS adapter or two. Does anyone know if any of the Cavium accelerators (crypto or compression) can improve ZFS perf?
- ironMann 9y ago
  Since ZoL 0.7, Raid-Z parity and checksumming (Fletcher4) operations are accelerated using SIMD instruction sets (in case of ThunderX, using aarch64 NEON)
z3t4 9y ago
Anyone else addicted to acquiring servers and high bandwidth connections ? Any ideas on what to do with the over capacity ?
- sp332 9y ago
  ArchiveTeam is working on backing up the Internet Archive. http://iabak.archiveteam.org/ http://iabak.archiveteam.org/ http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g...
- viraptor 9y ago
  Software mirrors, http://yacy.net/ http://yacy.net/, i2p, https://filecoin.io/ https://filecoin.io/
- fs111 9y ago
  run a Tor middle relay
jjirsa 9y ago
Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtime would be the least painful part of the eventual failure.
- codys 9y ago
  > There's at least one unpatched and there are bound to be more. References, please.
  - jjirsa 9y ago
    https://news.ycombinator.com/item?id=14938092 https://news.ycombinator.com/item?id=14938092
- codys 9y ago
  It might be worthwhile mentioning that at some point in the past, HN ran on 1 server (and might still do so). https://news.ycombinator.com/item?id=5229522 https://news.ycombinator.com/item?id=5229522
  - jjirsa 9y ago
    Cute but irrelevant?
- patrickg_zill 9y ago
  Actually the real issue is, "when the system is 65% full and you reboot, how long will it take for ZFS to mount it"? Perhaps he has split up ZFS into a number of different pools and they can be mounted in parallel (depends on the init script and whether ZFS can do this). But I do recall that larger ZFS pools can take a bit of time to mount; maybe the updated ZFS for Linux is faster....
  - 9y ago
guroot 9y ago
Can I just ask. Why not use FreeBSD?
- Timothycquinn 9y ago
  Agreed. With Jails, dtrace and Linux binary support, it would be a no-brainer for me.
- ewwhite 9y ago
  I'm surprised how often people ask this.
kev009 9y ago
There are a couple needful tweaks to this BOM for anyone wanting to follow this.. Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality. I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or zero copy TCP. Supermicro seriously under-provisioned I/O on that chassis. I'd add LSI/Avago/now Broadcom cards so you can get native ports to every drive. Even if it's just a cold storage box, it will help with rebuild and scrub times and peace of mind. The cost of this is not bad compared to the frustration of SAS expander firmwares. 2x24 or 3x16 and 4 drives on the onboard if you can skip the backplane expander. Supermicro will usually do things like this if you insist, or an integrator like ixSystems can handle it. More subjectively, I would also recommend FreeBSD. It seems their main justification for Ubuntu was paid support, which can be had from ixSystems who sell and support an entire stack (Supermicro servers, FreeBSD or FreeNAS or TrueNAS, and grok ZFS and storage drivers to the tune that they have done quite a bit of development.
- matt_wulfeck 9y ago
  Why FreeBSD?
  - b34r 9y ago
    stability.
    - pbarnes_1 9y ago
      Compared to? I agree to some extent that ZFS on FreeBSD might have been more stable than ZoL, but in general? Meh.
  - olavgg 9y ago
    The reason is that the Linux and FreeBSD kernels are vastly different. The FreeBSD kernel is more similar to Solaris than Linux and this is what ZFS was developed for. So in the end you have way more kernel "workarounds" for ZFS on Linux than the FreeBSD implementation. Especially the virtual memory implementation on Linux behaves very differently and has caused many issues over time, but I think it is very good now.
rurban 9y ago
I'm not a HW guy but those drives seems to be far too close together. A few more millimeters space will keep the temperature down much better I assume.
yest 9y ago
>It’s hard – if not impossible – to beat the $/GB and durability that Amazon is able to provide with their object storage offering. what the actual fuck?? AWS S3 is a abominable rip off. After I rented to my own dedicated server, I am paying several times less.
- acdha 9y ago
  You're running three geographically separated servers with 24x7 monitoring & security, automatic rebuilds, and active bit-rot scrubbing? If not, you're doing a lot less than S3. It's possible to beat S3 pricing but you either need to be buying a lot of storage or cutting corners to do it. The most common mistake I've seen when people make those comparisons is excluding staff time, followed by presenting a system with no or manual bit-rot protection as equivalent.
  - yest 9y ago
    just buy/rent a dedicated box and put RAID1, that's all most stratups need. Your points would be valid if S3 would be allowing to disable `geographically separated servers`, `bit-rot scrubbing` (no idea wtf is this). But those who say S3 is a cheap solution for more than few GB are fools or shills
    - acdha 9y ago
      So … that box is run by a volunteer sysadmin who doesn't charge you? … and doesn't mind getting up at 3am to replace a drive? That server has perfectly reliable power and environmental setup so you never have prolonged downtime or a double disk failure? You're okay losing everything if someone makes a mistake running that server since backups cost too much? You have higher-level software which tells you when data on that RAID array is corrupted? Your free sysadmin periodically runs an audit to make sure that the data stored on disk is what you originally stored? That's what I was referring to with scrubbing: even with RAID corruption happens and most storage admins have stories about the time they found out it'd happened after the only good disk failed, been written to tape, etc. The best solution is to actively scan every copy and verify it against the stored hashes for what you originally stored, which also protects against cases where a bug or human error meant that e.g. your RAID array faithfully protected a truncated file because the original write failed and nobody noticed in time. S3 provides a strong guarantee that you will get back the original data you stored or an error but never a corrupt copy and that you can prevent storing a partial or corrupted upload. If you roll your own, you need to provide those same protections for the full stack or accept a higher level of risk and perhaps mitigate it in other ways (e.g. Git-style distributed full copies with integrity checks). Again, I'm not saying that it's impossible to pay less than S3 but your response is a bingo card for the corners people cut until something breaks and they learn the hard way why raw storage costs less than a supported storage service. Doing this for real adds support cost for the OS, your software, security, monitoring, backups & other DR planning, etc. If you use S3, Google, etc. you get all of that built into a price which is known in advance, which is a significant draw for anyone who wants to spend their mental capacity on other issues. Many places don't have enough storage demand for that overhead to pay off in less than years and startups in particular should be extremely careful about spending their limited staff time on commodity functions rather than something which furthers their actual business. If you're Dropbox, sure, invest in a capable storage team because that's a core function but if your business is different it's time to look long and close at whether it makes any sense to devote staff time to saving a few grand a year.
vc00000 9y ago
We bought our initial 2 TrueNAS servers from IX Systems (SuperMicro) back in 2011, have been upgrading over the years and they have been very reliable servers. Currently each server has 63 drives (4TB HGST NL-SAS) with 1 hot spare, configured as RAIDZ. Right now there is 200TB of usable storage, we initially started with 29TB and have been expanding as needed when it hits about 79%, I buy 18 drives roughly every 6-8 months, 9 drives per server and expand the pool. To say that we never had issues is lying, we did have some major issues when upgrading from versions, but this was early on, now it is a rock solid storage system. Although there is less than 300 active users connecting to the primary server, there is a lot of very important pre & post production high dev videos. Reboot with 63 drives is around 10 minutes or less. Resilvering could take 24-48 hours, depending on load, depending on how much data the failed drive contained. Performance has been great, reliability has been great, support has been great. Sadly IX Systems can no longer provide support after the end of this year, they've extended support beyond the expected lifetime of the hardware.
BigIQ 9y ago
Biggest question: why? At that scale something like Ceph would be more reasonable. Just because ZFS can handle those filesystem sizes doesn't necessarily mean that it's the best tool for the job. There's a reason why all big players like Google, Amazon and Facebook go for the horizontal scaling approach.

320 points by beagle3 9y ago

4ad 9y ago

you can run ZFS on Ubuntu [...] You could also build this on Solaris with necessary licensing if you wanted to that route but it’d be more expensive. I find it bewildering the author didn't even consider illumos or FreeBSD, where ZFS is a first class citizen.

howeyc 9y ago
I thought the same thing, they sort of answers in the comments. Something about freenas saying no support and that the put together systems from iXsystems have lower drive bay counts. My guess is they just prefer Linux. FreeBSD or illumos would definitely work.
qeternity 9y ago
For someone not very up to date with all the various ZFS flavors, can you please explain this a bit?
- overcast 9y ago
  ZFS is a native filesystem to FreeBSD, where as in Linux it is not.
  - X86BSD 9y ago
    It's also had more baking time on FreeBSD.
  - qeternity 9y ago
    Sure but that alone doesn't mean it's any less suitable on Ubuntu. I'm just trying to understand what the differences are in implementation, reliability, etc.
    - cyphar 9y ago
      ZFS-on-Linux has not been mainstream (or existed) for as long as ZFS has been integrated inside FreeBSD. illumos is the free software fork of OpenSolaris -- so it's the repo of record for modern ZFS development.
      - waz0wski 9y ago
        OpenZFS is the parent project from which ZFS is applied to Illumos, FreeBSD, and ZFSoL http://open-zfs.org/wiki/Main_Page http://open-zfs.org/wiki/Main_Page
        cyphar 9y ago
        I'm aware of that, I just didn't want to additionally confuse someone who is barely familiar with ZFS with the whole OpenZFS fork and so on.
    - gruturo 9y ago
      ZFS on FreeBSD and on Illumos has been used, abused, tested and stressed all the way to the Moon and beyond. Loads of people with loads of data over loads time. This makes bugs appear, and get addressed, and eventually gives you confidence that nothing nasty remains uncorrected, and it won't eat your data. ZFS on Linux, due to the unfortunate licensing situation, is considerably less tested and thus scary, data-eating bugs are at least a bit more likely to exist.
      - zorked 9y ago
        On the other hand Linux is far more popular so I wouldn't be surprised if ZFS on Linux already clocks more hours of usage than on BSD.
        gruturo 9y ago
        > On the other hand Linux is far more popular so I wouldn't be surprised if ZFS on Linux already clocks more hours of usage than on BSD. Yes and no. Few people, even among the Linux crowd, are aware it exists or feel like trusting it with important data sets/jobs. Those few are the most likely to be equally at ease running FreeBSD or Illumos (or Solaris), where the damn thing is known to work really well.
seany 9y ago
I run napp-it on omni-os at home and work with good results. It's not too hard to adapt your linux knowledge if you're not used to the solaris-ish userspace.
X86BSD 9y ago
Yeah I was like uhhhh what? Illumos and illumos based OS'es and FreeBSD are the only games in town where ZFS is a first class citizen. Ill chalk it up to extreme bias.
- floatboth 9y ago
  Yeah, many people don't consider FreeBSD because of silly reasons… :( but mentioning Solaris that "requires licensing" and not illumos?? illumos is the new Solaris, the old Solaris devs are working on illumos! Oracle Solaris is irrelevant.
  - al452 9y ago
    > Oracle Solaris is irrelevant. Amen to that.
notpeter 9y ago
It's likely a human resources problem. For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux. And those numbers are much worse outside of major cities. The commercial support from Ubuntu tips the scales. I made this same decision at my last job. I ran Solaris and Illumos on our file servers and loved it, but a year before I left I ported all the pools to Ubuntu so my successor only needed to be Linux competent and ZFS trainable. Sometimes when choosing tech it's not what's technologically superior solution nor what you personally could run well, it's what's best for your coworkers, your successor and the organization in the long term.
- rdeckard0104 9y ago
  I disagree. FreeBSD is just as easy to administer as Linux. Solaris I find to be a bit more challenging. I can appreciate your commentary, but I challenge you to prove me wrong. Just for some color I've ran the Gambit of Unix and Linux systems in my career with FreeBSD being one of the easiest and most consistent out of the box technologies. Solaris had already been quite the opposite. I guess you could say AIX is a close second to Solaris, but I think that's a bit of a grey area considering how opinionated that operating system is. To be clear I'm not a FreeBSD zealot. I'm for what works. Right now what works is the emerging container based solutions albeit not relevant to this discussion.
  - ethbro 9y ago
    > For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux. Was parent wrong about that point?
    - chungy 9y ago
      Not exactly, but to any competent administrator, learning FreeBSD based on Linux experience is not that difficult. It's sort of like tasking someone to start working on a program written in Go even if they have only C++ experience. You can cope.
      - 9y ago
panzerboy 9y ago
He explained on Reddit that he has some support from Canonical as well.

twiss 9y ago

> I purchased these units through a vendor we like to use and they hooked us up, so I won’t be able to share my specific pricing. (...) If you build the systems out on there you’ll find that they come in around $35,000 (USD) each. That devided by 52 x 8 = 416TB is 0.084$/GB. For comparison, the Backblaze Storage Pod 6.0 [1] claims 0.06$/GB for the version with the same hard drives. Although this version has a bunch of extra features like 2 x 800GB SSD's for ZFS SLOG, 8x more RAM for a total of 256GB, etc. [1]: https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv...

rsync 9y ago

"I ended up between the Supermicro SSG-6048R-E1CR60L or the SSG-6048R-E1CR90L – the E1CR60L is a 60-bay 4U chassis while the E1CR90L is a 90-bay 4U chassis. This nice part is that no matter which platform you choose Supermicro sells this only as a pre-configured machine – this means that their engineers are going to make sure that the hardware you choose to put in this is all from a known compatibility list. Basically, you cannot buy this chassis empty and jam your own parts in" This is a major departure from the Supermicro business model and practices and basically broke all of our next generation expansion roadmaps. This was not a technical decision - it is the same old economic decision that every large VAR/integrator/supplier has succumbed to for the last 30 years. They aren't the first ones to try this trick and they won't be the last. We (rsync.net) are not playing ball, however. After 16 years of deploying solely on supermicro hardware (server chassis and JBODs) we bought our first non-supermicro JBOD last month.

wmf 9y ago
What specifically is the problem? Overpriced drives?
- rsync 9y ago
  Yes. $30-$50 each. Multiply that by 60 or 90 and multiply that by (however many you put in a rack). That adds up to five figure premiums on drives we're going to burn in anyway. We know how to read an HCL - it's not rocket science.
  - flattone 9y ago
    Even if reading hcl were rocket science a more validating vendor would preserve buyer self determination
  - qaq 9y ago
    Ok 60X50 is 3K even with extra 3K Supermicro is still a lot cheeper
    - 9y ago

sandGorgon 9y ago

The most important line for me was "Today, you can run ZFS on Ubuntu 16.0.4.2 LTS with standard repositories and Canonical’s Ubuntu Advantage Advanced Support. That makes the decision easy." Its highly interesting that Canonical does this with ZFS. I'm not sure why they dont market this more.

tylerjl 9y ago
I'm only a casual user of ZFS on Linux for personal storage projects, but I've spoken with people who rely very heavily on ZFS on Linux for their small businesses, and it's interesting to hear their perspectives on this. Essentially, because btrfs has failed to deliver on the next-gen filesystem front, ZFS on Linux is such a critical piece of technology that unless Red Hat has an answer soon for out-of-the-box ZFS on Linux, Canonical has a pretty staggering advantage. One theory is that Canonical has essentially forced Oracle to decide whether it wants to crack down on inclusion of CDDL-licensed code being shipped with Ubuntu's stock kernel, so Red Hat may be waiting to see if Oracle sweeps in or not before following suit. I'd be very surprised if RHEL (by observing the progression of Fedora development) continues to bet on btrfs, as I have yet to encounter anyone (including myself) who would ever trust btrfs over ZFS on Linux with anything of importance based on their experiences with the two - my experience is anecdotal, but ZFS has been just as reliable on *BSD as with any Linux distribution.
- majormjr 9y ago
  btrfs is getting dropped from future versions of RHEL so you can see what Red Hat thinks of the future of btrfs.
  - user5994461 9y ago
    RedHat is betting on xfs.
  - gribbly 9y ago
    Red Hat has never spent any real resources on btrfs, they've been all about XFS when it comes to enterprise needs.

SoMisanthrope 9y ago

Very impressive! It's amazing what people are doing with OTS technology.

notyourday 9y ago

Very good experience with 45drives.com storinator XLs.

jaytaylor 9y ago

What about cooling? Will the lifespan of the high-capacity platter-dense hard drives be drastically reduced by clumping them together like that with what looks like little airflow?

comboy 9y ago
AFAIR from the backblaze blog, a bigger issue that shortens the lifespan is vibration.
- exikyut 9y ago
  Could that be mitigated by redesigning the chassis for slightly less density in order to fit massive rubber standoffs/grommets?
- deelowe 9y ago
  That's probably because their design doesn't do much to mitigate it.

mikekij 9y ago

Almost big enough to archive SoundCloud!

cmurf 9y ago

Do either of these project spec hardware that would work for this use case? opencompute.org Backblaze storage pod, they're up to v 6.0 now (Netflix open connect specs supermicro hardware) Others?

electrum 9y ago
The next generation Facebook design for storage is Bryce Canyon, available through the Open Compute Project: https://code.facebook.com/posts/1869788206569924/introducing-bryce-canyon-our-next-generation-storage-platform/ https://code.facebook.com/posts/1869788206569924/introducing...

andreiw 9y ago

I wish Supermicro had a similar chassis around the Cavium ThunderX. That would make a lot of sense for network-attached storage, regardless of whether one goes with SATA or drops in a SAS adapter or two. Does anyone know if any of the Cavium accelerators (crypto or compression) can improve ZFS perf?

ironMann 9y ago
Since ZoL 0.7, Raid-Z parity and checksumming (Fletcher4) operations are accelerated using SIMD instruction sets (in case of ThunderX, using aarch64 NEON)

z3t4 9y ago

Anyone else addicted to acquiring servers and high bandwidth connections ? Any ideas on what to do with the over capacity ?

sp332 9y ago
ArchiveTeam is working on backing up the Internet Archive. http://iabak.archiveteam.org/ http://iabak.archiveteam.org/ http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/g...
viraptor 9y ago
Software mirrors, http://yacy.net/ http://yacy.net/, i2p, https://filecoin.io/ https://filecoin.io/
fs111 9y ago
run a Tor middle relay

jjirsa 9y ago

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtime would be the least painful part of the eventual failure.

codys 9y ago
> There's at least one unpatched and there are bound to be more. References, please.
- jjirsa 9y ago
  https://news.ycombinator.com/item?id=14938092 https://news.ycombinator.com/item?id=14938092
codys 9y ago
It might be worthwhile mentioning that at some point in the past, HN ran on 1 server (and might still do so). https://news.ycombinator.com/item?id=5229522 https://news.ycombinator.com/item?id=5229522
- jjirsa 9y ago
  Cute but irrelevant?
patrickg_zill 9y ago
Actually the real issue is, "when the system is 65% full and you reboot, how long will it take for ZFS to mount it"? Perhaps he has split up ZFS into a number of different pools and they can be mounted in parallel (depends on the init script and whether ZFS can do this). But I do recall that larger ZFS pools can take a bit of time to mount; maybe the updated ZFS for Linux is faster....
- 9y ago

guroot 9y ago

Can I just ask. Why not use FreeBSD?

Timothycquinn 9y ago
Agreed. With Jails, dtrace and Linux binary support, it would be a no-brainer for me.
ewwhite 9y ago
I'm surprised how often people ask this.

kev009 9y ago

There are a couple needful tweaks to this BOM for anyone wanting to follow this.. Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality. I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or zero copy TCP. Supermicro seriously under-provisioned I/O on that chassis. I'd add LSI/Avago/now Broadcom cards so you can get native ports to every drive. Even if it's just a cold storage box, it will help with rebuild and scrub times and peace of mind. The cost of this is not bad compared to the frustration of SAS expander firmwares. 2x24 or 3x16 and 4 drives on the onboard if you can skip the backplane expander. Supermicro will usually do things like this if you insist, or an integrator like ixSystems can handle it. More subjectively, I would also recommend FreeBSD. It seems their main justification for Ubuntu was paid support, which can be had from ixSystems who sell and support an entire stack (Supermicro servers, FreeBSD or FreeNAS or TrueNAS, and grok ZFS and storage drivers to the tune that they have done quite a bit of development.

matt_wulfeck 9y ago
Why FreeBSD?
- b34r 9y ago
  stability.
  - pbarnes_1 9y ago
    Compared to? I agree to some extent that ZFS on FreeBSD might have been more stable than ZoL, but in general? Meh.
- olavgg 9y ago
  The reason is that the Linux and FreeBSD kernels are vastly different. The FreeBSD kernel is more similar to Solaris than Linux and this is what ZFS was developed for. So in the end you have way more kernel "workarounds" for ZFS on Linux than the FreeBSD implementation. Especially the virtual memory implementation on Linux behaves very differently and has caused many issues over time, but I think it is very good now.

rurban 9y ago

I'm not a HW guy but those drives seems to be far too close together. A few more millimeters space will keep the temperature down much better I assume.

yest 9y ago

>It’s hard – if not impossible – to beat the $/GB and durability that Amazon is able to provide with their object storage offering. what the actual fuck?? AWS S3 is a abominable rip off. After I rented to my own dedicated server, I am paying several times less.

acdha 9y ago
You're running three geographically separated servers with 24x7 monitoring & security, automatic rebuilds, and active bit-rot scrubbing? If not, you're doing a lot less than S3. It's possible to beat S3 pricing but you either need to be buying a lot of storage or cutting corners to do it. The most common mistake I've seen when people make those comparisons is excluding staff time, followed by presenting a system with no or manual bit-rot protection as equivalent.
- yest 9y ago
  just buy/rent a dedicated box and put RAID1, that's all most stratups need. Your points would be valid if S3 would be allowing to disable `geographically separated servers`, `bit-rot scrubbing` (no idea wtf is this). But those who say S3 is a cheap solution for more than few GB are fools or shills
  - acdha 9y ago
    So … that box is run by a volunteer sysadmin who doesn't charge you? … and doesn't mind getting up at 3am to replace a drive? That server has perfectly reliable power and environmental setup so you never have prolonged downtime or a double disk failure? You're okay losing everything if someone makes a mistake running that server since backups cost too much? You have higher-level software which tells you when data on that RAID array is corrupted? Your free sysadmin periodically runs an audit to make sure that the data stored on disk is what you originally stored? That's what I was referring to with scrubbing: even with RAID corruption happens and most storage admins have stories about the time they found out it'd happened after the only good disk failed, been written to tape, etc. The best solution is to actively scan every copy and verify it against the stored hashes for what you originally stored, which also protects against cases where a bug or human error meant that e.g. your RAID array faithfully protected a truncated file because the original write failed and nobody noticed in time. S3 provides a strong guarantee that you will get back the original data you stored or an error but never a corrupt copy and that you can prevent storing a partial or corrupted upload. If you roll your own, you need to provide those same protections for the full stack or accept a higher level of risk and perhaps mitigate it in other ways (e.g. Git-style distributed full copies with integrity checks). Again, I'm not saying that it's impossible to pay less than S3 but your response is a bingo card for the corners people cut until something breaks and they learn the hard way why raw storage costs less than a supported storage service. Doing this for real adds support cost for the OS, your software, security, monitoring, backups & other DR planning, etc. If you use S3, Google, etc. you get all of that built into a price which is known in advance, which is a significant draw for anyone who wants to spend their mental capacity on other issues. Many places don't have enough storage demand for that overhead to pay off in less than years and startups in particular should be extremely careful about spending their limited staff time on commodity functions rather than something which furthers their actual business. If you're Dropbox, sure, invest in a capable storage team because that's a core function but if your business is different it's time to look long and close at whether it makes any sense to devote staff time to saving a few grand a year.

vc00000 9y ago

We bought our initial 2 TrueNAS servers from IX Systems (SuperMicro) back in 2011, have been upgrading over the years and they have been very reliable servers. Currently each server has 63 drives (4TB HGST NL-SAS) with 1 hot spare, configured as RAIDZ. Right now there is 200TB of usable storage, we initially started with 29TB and have been expanding as needed when it hits about 79%, I buy 18 drives roughly every 6-8 months, 9 drives per server and expand the pool. To say that we never had issues is lying, we did have some major issues when upgrading from versions, but this was early on, now it is a rock solid storage system. Although there is less than 300 active users connecting to the primary server, there is a lot of very important pre & post production high dev videos. Reboot with 63 drives is around 10 minutes or less. Resilvering could take 24-48 hours, depending on load, depending on how much data the failed drive contained. Performance has been great, reliability has been great, support has been great. Sadly IX Systems can no longer provide support after the end of this year, they've extended support beyond the expected lifetime of the hardware.

BigIQ 9y ago

Biggest question: why? At that scale something like Ceph would be more reasonable. Just because ZFS can handle those filesystem sizes doesn't necessarily mean that it's the best tool for the job. There's a reason why all big players like Google, Amazon and Facebook go for the horizontal scaling approach.