4 ms·
This is good to see. I didn't realize just how much this latency could effect me until I recently was doing some fine tuning on an NLP model on Sagemaker and no
by ZeroCool2u 5y ago
This is good to see. I didn't realize just how much this latency could effect me until I recently was doing some fine tuning on an NLP model on Sagemaker and noticed my training loss was looking pretty rough when I used a working directory that was in EFS as opposed to on my EC2 instance. Testing it out, performance was dramatically improved when I moved everything to the directly attached storage, so I've just been avoiding EFS since then during training.
- geertj 5y agoFor ML training, where you repeatedly open the same files over and over again, the NFS 'close to open' coherency protocol (where every open() and close() is a round trip) is not a good fit. Once we support NFS delegations, this will become a lot faster, but local training data set will very likely continue to be faster (we have ideas on this though!) Given that ML training uses very expensive instance types, a local file system is the way to go for now. ML inference on the other hand is a use case that we've specifically targeted with this update, and it should work very well. Many of our customers do inference using AWS Lambda, directly connected to their EFS file system.
- ZeroCool2u 5y agoAppreciate the insight, that's really good to know! It's one of those things that's been bugging me every once in a while when I think back to it. Also, does today's update apply to GovCloud regions as well or just Commercial for now?
- geertj 5y ago> Also, does today's update apply to GovCloud regions as well or just Commercial for now? It also applies to GovCloud.
- ZeroCool2u 5y agoThat's awesome, thank you!