4 ms·
For ML training, where you repeatedly open the same files over and over again, the NFS 'close to open' coherency protocol (where every open() and close() is a r
by geertj 5y ago
For ML training, where you repeatedly open the same files over and over again, the NFS 'close to open' coherency protocol (where every open() and close() is a round trip) is not a good fit. Once we support NFS delegations, this will become a lot faster, but local training data set will very likely continue to be faster (we have ideas on this though!) Given that ML training uses very expensive instance types, a local file system is the way to go for now.
ML inference on the other hand is a use case that we've specifically targeted with this update, and it should work very well. Many of our customers do inference using AWS Lambda, directly connected to their EFS file system.
- ZeroCool2u 5y agoAppreciate the insight, that's really good to know! It's one of those things that's been bugging me every once in a while when I think back to it. Also, does today's update apply to GovCloud regions as well or just Commercial for now?
- geertj 5y ago> Also, does today's update apply to GovCloud regions as well or just Commercial for now? It also applies to GovCloud.
- ZeroCool2u 5y agoThat's awesome, thank you!