5 ms·

I'm guessing you don't... you use the cloud, right?

exelius 9y ago
If you're doing a lot of experimentation, it can make more sense to train your models locally (so you're not paying server / data transfer costs every time you test).
antognini 9y ago
For a hobbyist, it probably makes sense to just use some GPU instances on AWS. But it doesn't really take that long for it to become cost-effective to buy a GPU. It only takes a couple of months of full-time use before it becomes cost-effective to purchase the GPU. Even if you're building the entire machine from scratch it takes under a year. (At least, that was my estimate when I was building a rig earlier this year.) If you're doing any serious deep learning projects it's pretty easy to have a model training nearly 100% of the time.
- nightski 9y ago
  Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.
  - treehau5 9y ago
    extremely frustrating. Internet service is getting worse this in country, not better, sadly. The only lucky ones are those who live in competitive markets (lol) and those who had google fiber break some ground. The anti-competitiveness is getting worse too. Charter just took over TWC in my area.
  - zeptomu 9y ago
    It is not that bad. Either you've your training data local (unlikely) or it is already available in the "cloud" (i.e. a public-facing service). Let's say a typical training set (raw data) is 100G. Assuming 2.5MB/s Upload capacity with local data, you've uploaded it to your deep learning machine in half a day (100000 / 2.5 / 3600 ~ 11 hours) - which is not that much, as most of your time will be used for development and fine-tuning of your deep learning tool chain anyway. In most cases the data is accessible via a public-facing service, and assuming 1GBit/s bandwidth you've downloaded 100G in 13 minutes (100000 / 125 / 60 ~ 13 minutes).
    - chronic940 9y ago
      You forgot the AWS Internet egress charge per GB..
      - zeptomu 9y ago
        What do you mean? I know egress is one of the more expensive cloud services (e.g. compared to compute and storgage) at AWS, GCP, etc., but if I upload data to my learning system that's ingress AFAIK which is mostly free or less expensive. Btw. current Egress is like 0.1$/GB, so 100G ~ 10$. Don't get me wrong, I am not saying you should always train in the cloud, but I do not think slower Upload or Ingress are the limiting factor.
ex3ndr 9y ago
I have done simple math - one 1080 ti will cover costs for a only month or two on AWS.

If you're doing a lot of experimentation, it can make more sense to train your models locally (so you're not paying server / data transfer costs every time you test).

antognini 9y ago

For a hobbyist, it probably makes sense to just use some GPU instances on AWS. But it doesn't really take that long for it to become cost-effective to buy a GPU. It only takes a couple of months of full-time use before it becomes cost-effective to purchase the GPU. Even if you're building the entire machine from scratch it takes under a year. (At least, that was my estimate when I was building a rig earlier this year.) If you're doing any serious deep learning projects it's pretty easy to have a model training nearly 100% of the time.

nightski 9y ago
Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.
- treehau5 9y ago
  extremely frustrating. Internet service is getting worse this in country, not better, sadly. The only lucky ones are those who live in competitive markets (lol) and those who had google fiber break some ground. The anti-competitiveness is getting worse too. Charter just took over TWC in my area.
- zeptomu 9y ago
  It is not that bad. Either you've your training data local (unlikely) or it is already available in the "cloud" (i.e. a public-facing service). Let's say a typical training set (raw data) is 100G. Assuming 2.5MB/s Upload capacity with local data, you've uploaded it to your deep learning machine in half a day (100000 / 2.5 / 3600 ~ 11 hours) - which is not that much, as most of your time will be used for development and fine-tuning of your deep learning tool chain anyway. In most cases the data is accessible via a public-facing service, and assuming 1GBit/s bandwidth you've downloaded 100G in 13 minutes (100000 / 125 / 60 ~ 13 minutes).
  - chronic940 9y ago
    You forgot the AWS Internet egress charge per GB..
    - zeptomu 9y ago
      What do you mean? I know egress is one of the more expensive cloud services (e.g. compared to compute and storgage) at AWS, GCP, etc., but if I upload data to my learning system that's ingress AFAIK which is mostly free or less expensive. Btw. current Egress is like 0.1$/GB, so 100G ~ 10$. Don't get me wrong, I am not saying you should always train in the cloud, but I do not think slower Upload or Ingress are the limiting factor.

ex3ndr 9y ago

I have done simple math - one 1080 ti will cover costs for a only month or two on AWS.