3 ms·
Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.
by nightski 9y ago
Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.
- treehau5 9y agoextremely frustrating. Internet service is getting worse this in country, not better, sadly. The only lucky ones are those who live in competitive markets (lol) and those who had google fiber break some ground. The anti-competitiveness is getting worse too. Charter just took over TWC in my area.
- zeptomu 9y agoIt is not that bad. Either you've your training data local (unlikely) or it is already available in the "cloud" (i.e. a public-facing service). Let's say a typical training set (raw data) is 100G. Assuming 2.5MB/s Upload capacity with local data, you've uploaded it to your deep learning machine in half a day (100000 / 2.5 / 3600 ~ 11 hours) - which is not that much, as most of your time will be used for development and fine-tuning of your deep learning tool chain anyway. In most cases the data is accessible via a public-facing service, and assuming 1GBit/s bandwidth you've downloaded 100G in 13 minutes (100000 / 125 / 60 ~ 13 minutes).
- chronic940 9y agoYou forgot the AWS Internet egress charge per GB..
- zeptomu 9y agoWhat do you mean? I know egress is one of the more expensive cloud services (e.g. compared to compute and storgage) at AWS, GCP, etc., but if I upload data to my learning system that's ingress AFAIK which is mostly free or less expensive. Btw. current Egress is like 0.1$/GB, so 100G ~ 10$. Don't get me wrong, I am not saying you should always train in the cloud, but I do not think slower Upload or Ingress are the limiting factor.