5 ms·
It will be interesting to see how well this scales. I have seen AWS EFS fall over under heavy read load (lots of reads for the same TB-scale data). A lot of p
by choppaface 7y ago
It will be interesting to see how well this scales. I have seen AWS EFS fall over under heavy read load (lots of reads for the same TB-scale data). A lot of poor souls using Horovod with no data team use EFS to “distribute” data in data-parallel SGD.
- khc 7y agoDo you have more details about this?
- spydum 7y agoEFS generally only falls down when you hit your IOPs limit for the measurement period -have seen this soooo many times. Buy more storage/IOPS, and monitor that puppy.
- hobofan 7y agoIOPs is one of the main pitfalls of cloud platforms and I'm baffled that neither GCP nor AWS make it easier to detect that you are hitting a limit there. I've had multiple gigs where I was tasked with solving performance pitfalls, and most of the time it was as simple as upping the IOPS. In one case even though the problem was resolved in the end, the client migrated away from the cloud again because of that bad experience.
- objectivefs 7y agoFor TB-scale heavy read loads ObjectiveFS[0] is an alternative to EFS that don't have EFS's IO/bandwidth limits. It is often used for distributing data for machine learning and large scale simulations on EC2 and GCE. [0]: https://objectivefs.com https://objectivefs.com