3 ms·
A few things here: 1) What/where exactly are they using GlusterFS for? Has Gluster fixed their scaling problems yet? Specifically the issue where new storage s
by scurvy 11y ago
A few things here:
1) What/where exactly are they using GlusterFS for? Has Gluster fixed their scaling problems yet? Specifically the issue where new storage spaces/nodes were only available to new directories and files, but not existing directories? Granted, the last time I looked at this was 2009 or so, but it was a flaw due to their "no master node" topology.
2) FB has an entire team to manage Hadoop/HBase. This shows just how much of a beast that stack is. Anyone who has run Hadoop on "Internet time" knows what I'm talking about. It's great at running time insensitive, deferred compute jobs in an academic or scientific setting. It's really hard to keep it all 100% running in an on-demand setting. Aside, I couldn't imagine just working on 1 product in an operations setting as my full-time job. Boredom/fatigue must be a problem on that team.
3) I'd like to see more information on the networking side. What transport protocol? How large are the average updates in frame size? Etc etc.
We've built something similar to Gorilla in-house, so I'm happy to see that we've come to some of the same conclusions.
- vidarh 11y agoFor GlusterFS, new storage space is immediately available to new directories and files, and can be quickly made available for new files in existing directories by fixing the layout. You can then run a rebalance in the background if you want to evenly distribute the files, but that's of course a slow operation in a large cluster.
- nwmcsween 11y agoGlusterfs will just always be slower due to the userspace -> kernelspace -> userspace switching fuse has to do. I don't understand why facebook hasn't jumped on ceph yet.
- linuxhansl 11y agoFor #2, show me any system that holds over 2PB of data on a large set of machines that does not need a team to be managed.
- scurvy 11y agoI meant dedicated team. Most places run shared ops teams. Every place I've seen that runs a "big Hadoop" deployment also has a dedicated "Hadoop team" to go with it. It's pretty easy to build a 2PB storage system on Ceph that the average group of sysadmins can run.