4 ms·

> GUI or not data science has to move to cloud-based tools. This is a pretty odd statement. Care to explain why?

raghavsb 9y ago
Smaller datasets will work on laptop/desktop. For DL work with large datasets you need to build a GPU workstation. Moreover there setting up environment and dependencies on different hardware setups is not straight forward. Cloud provides the flexibility of choosing the hardware, many open source projects allow you to manage your dependencies and setup better. From 0 to something, cloud is better than custom.
- PeterisP 9y ago
  For quite many companies their whole "big data" dataset is small enough to be processed by a single beefy machine. In that case it's far more cost effective to simply plug in $1000 worth of extra ram in a workstation rather than spend some extra engineer-hours to do it remotely.
  - raghavsb 9y ago
    Couldn't agree more on the data size. In most cases beefy machine work. Would on-demand (cloud) make it simple? Also beefy machine works for training jobs. But we need to deploy the models too.
    - cup-of-tea 9y ago
      With the many container solutions available today it's incredibly easy to move from dev to prod. You don't need to pay for a prod environment to do your development just to avoid ever having to migrate.
      - anandology 9y ago
        Yes, it is incredibly easy, except when you upgrade to tensorflow 1.6 and it fails with [a cuda error][1] and after couple of sleepless nights you realize nvidia has deleted the docker image of cuda version 70xx from dockerhub and you need to find the right commit that works from their git repo and build everything yourself. [1]: https://github.com/tensorflow/tensorflow/issues/17566 https://github.com/tensorflow/tensorflow/issues/17566
- walshemj 9y ago
  So if you struggle to "setting up environment and dependencies on different hardware setups" then maybe Data science and technical programming is not for you.
  - cup-of-tea 9y ago
    I wish this view was held more widely. Unfortunately the vast majority of "data science" jobs out there will not ask for any such technical ability.

Smaller datasets will work on laptop/desktop. For DL work with large datasets you need to build a GPU workstation. Moreover there setting up environment and dependencies on different hardware setups is not straight forward. Cloud provides the flexibility of choosing the hardware, many open source projects allow you to manage your dependencies and setup better. From 0 to something, cloud is better than custom.

PeterisP 9y ago
For quite many companies their whole "big data" dataset is small enough to be processed by a single beefy machine. In that case it's far more cost effective to simply plug in $1000 worth of extra ram in a workstation rather than spend some extra engineer-hours to do it remotely.
- raghavsb 9y ago
  Couldn't agree more on the data size. In most cases beefy machine work. Would on-demand (cloud) make it simple? Also beefy machine works for training jobs. But we need to deploy the models too.
  - cup-of-tea 9y ago
    With the many container solutions available today it's incredibly easy to move from dev to prod. You don't need to pay for a prod environment to do your development just to avoid ever having to migrate.
    - anandology 9y ago
      Yes, it is incredibly easy, except when you upgrade to tensorflow 1.6 and it fails with [a cuda error][1] and after couple of sleepless nights you realize nvidia has deleted the docker image of cuda version 70xx from dockerhub and you need to find the right commit that works from their git repo and build everything yourself. [1]: https://github.com/tensorflow/tensorflow/issues/17566 https://github.com/tensorflow/tensorflow/issues/17566
walshemj 9y ago
So if you struggle to "setting up environment and dependencies on different hardware setups" then maybe Data science and technical programming is not for you.
- cup-of-tea 9y ago
  I wish this view was held more widely. Unfortunately the vast majority of "data science" jobs out there will not ask for any such technical ability.