3 ms·

Smaller datasets will work on laptop/desktop. For DL work with large datasets you need to build a GPU workstation. Moreover there setting up environment and dep

Smaller datasets will work on laptop/desktop. For DL work with large datasets you need to build a GPU workstation. Moreover there setting up environment and dependencies on different hardware setups is not straight forward. Cloud provides the flexibility of choosing the hardware, many open source projects allow you to manage your dependencies and setup better. From 0 to something, cloud is better than custom.

PeterisP 8y ago
For quite many companies their whole "big data" dataset is small enough to be processed by a single beefy machine. In that case it's far more cost effective to simply plug in $1000 worth of extra ram in a workstation rather than spend some extra engineer-hours to do it remotely.
- raghavsb 8y ago
  Couldn't agree more on the data size. In most cases beefy machine work. Would on-demand (cloud) make it simple? Also beefy machine works for training jobs. But we need to deploy the models too.
  - cup-of-tea 8y ago
    With the many container solutions available today it's incredibly easy to move from dev to prod. You don't need to pay for a prod environment to do your development just to avoid ever having to migrate.
    - anandology 8y ago
      Yes, it is incredibly easy, except when you upgrade to tensorflow 1.6 and it fails with [a cuda error][1] and after couple of sleepless nights you realize nvidia has deleted the docker image of cuda version 70xx from dockerhub and you need to find the right commit that works from their git repo and build everything yourself. [1]: https://github.com/tensorflow/tensorflow/issues/17566 https://github.com/tensorflow/tensorflow/issues/17566
walshemj 8y ago
So if you struggle to "setting up environment and dependencies on different hardware setups" then maybe Data science and technical programming is not for you.
- cup-of-tea 8y ago
  I wish this view was held more widely. Unfortunately the vast majority of "data science" jobs out there will not ask for any such technical ability.

PeterisP 8y ago

For quite many companies their whole "big data" dataset is small enough to be processed by a single beefy machine. In that case it's far more cost effective to simply plug in $1000 worth of extra ram in a workstation rather than spend some extra engineer-hours to do it remotely.

raghavsb 8y ago
Couldn't agree more on the data size. In most cases beefy machine work. Would on-demand (cloud) make it simple? Also beefy machine works for training jobs. But we need to deploy the models too.
- cup-of-tea 8y ago
  With the many container solutions available today it's incredibly easy to move from dev to prod. You don't need to pay for a prod environment to do your development just to avoid ever having to migrate.
  - anandology 8y ago
    Yes, it is incredibly easy, except when you upgrade to tensorflow 1.6 and it fails with [a cuda error][1] and after couple of sleepless nights you realize nvidia has deleted the docker image of cuda version 70xx from dockerhub and you need to find the right commit that works from their git repo and build everything yourself. [1]: https://github.com/tensorflow/tensorflow/issues/17566 https://github.com/tensorflow/tensorflow/issues/17566

walshemj 8y ago

So if you struggle to "setting up environment and dependencies on different hardware setups" then maybe Data science and technical programming is not for you.

cup-of-tea 8y ago
I wish this view was held more widely. Unfortunately the vast majority of "data science" jobs out there will not ask for any such technical ability.