7 ms·
Google Acquires Elastifile
- choppaface 7y agoIt will be interesting to see how well this scales. I have seen AWS EFS fall over under heavy read load (lots of reads for the same TB-scale data). A lot of poor souls using Horovod with no data team use EFS to “distribute” data in data-parallel SGD.
- khc 7y agoDo you have more details about this?
- spydum 7y agoEFS generally only falls down when you hit your IOPs limit for the measurement period -have seen this soooo many times. Buy more storage/IOPS, and monitor that puppy.
- hobofan 7y agoIOPs is one of the main pitfalls of cloud platforms and I'm baffled that neither GCP nor AWS make it easier to detect that you are hitting a limit there. I've had multiple gigs where I was tasked with solving performance pitfalls, and most of the time it was as simple as upping the IOPS. In one case even though the problem was resolved in the end, the client migrated away from the cloud again because of that bad experience.
- objectivefs 7y agoFor TB-scale heavy read loads ObjectiveFS[0] is an alternative to EFS that don't have EFS's IO/bandwidth limits. It is often used for distributing data for machine learning and large scale simulations on EC2 and GCE. [0]: https://objectivefs.com https://objectivefs.com
- geerlingguy 7y agoI guess this will be the Google Cloud equivalent of Amazon’s EFS?
- p_l 7y agoThere's already GCP Filestore.
- sciurus 7y agoFilestore is basically a single google-managed instance running NFS. It's much less sophisticated and resilient than EFS, but also can be much faster since it doesn't have the overhead of a distributed system. Don't try running something that touches lots of files like mercurial on EFS, trust me... The article says Elastifile (which I know nothing about) will become part of filestore.
- dogismycopilot 7y agoFilestore is also really expensive for what is basically just NFS.
- ethbro 7y agoManaged services are generally for enterprise, where you're fully costing out support + security mitigation. Eliminating team coordination tips the balance.
- geerlingguy 7y agoEFS is... surprisingly cheap, as long as you don’t need heavy sustained IO for more than a few hours at a time. Great for a simple file store and some big data purposes.
- nodesocket 7y agoA disk that can be attached to multiple VM's and support simultaneous reads/writes is a huge addition to GCP.
- zachberger 7y agoGoogle Cloud Filestore has existed for some time: https://cloud.google.com/filestore/ https://cloud.google.com/filestore/
- christi_meyer 7y agoElastic is next
- ppod 7y agohttps://www.youtube.com/watch?v=H27rfr59RiE https://www.youtube.com/watch?v=H27rfr59RiE
- strin 7y agoLooks like Elastifile allows users to "mount" cloud buckets to their servers as NFS. This allows them to run old desktop workflows without the need to migrate these workflows to the cloud. The upside is that Google Cloud is getting serious about the enterprise because most of them still use desktop applications. The question is how long this will last since the future is the cloud.
- truth_seeker 7y agoI would be interested to see how it compares with AWS EFS under a variety of load. AWS EFS implements NFS 4.1 which support has support for massive parallel IO. http://www.pnfs.com/ http://www.pnfs.com/