2 ms·

Unfortunately for Google, NVIDIA's offerings are very strong, and TPUs are a pain the rear to use and require TensorFlow, which in itself is a pain to use, maki

by m0zg 8y ago

Unfortunately for Google, NVIDIA's offerings are very strong, and TPUs are a pain the rear to use and require TensorFlow, which in itself is a pain to use, making it doubly painful, to the extent that using their offering requires a significant degree of desperation or not knowing any better.

akoumis 8y ago
Have you ever used Keras?
- m0zg 8y ago
  Yes I used Keras before PyTorch came out. IMO, Google needs to start over rather than try extensive wart removal on something that consists almost entirely of warts. These days Keras has subpar multi-GPU support (two GPUs are often slower than one), less flexibility, and more un-debuggable magic than PyTorch. And it doesn't look like things will be improving much (if at all) with TF 2.0. And the TPU is a story of its own when it comes to usability. Take a look at their docs. TPU actually has to run on a separate machine, which you need to provision manually using their `ctpu` tool, and training of a trivial network on CIFAR looks like this: https://github.com/tensorflow/tpu/blob/master/models/experimental/cifar_keras/cifar_keras.py https://github.com/tensorflow/tpu/blob/master/models/experim...
  - so_tired 8y ago
    Err - i don't get it. What alternative r u suggesting for development which is aimed at eventual production? I am a complete noob, so i just did Keras (to TF) and GPUs. If i switch to TPU - its still the same code. When i get the budget to cluster, i think I am better off with one of the Deepmind libs which TF based. No?
    - m0zg 8y ago
      If you're just using off-the-GitHub models and pressing them into service, sure, Keras is adequate. If your work actually requires research and experimentation (as most practical problems on which actual cash money can be made do) you'll be much better served with PyTorch. The main cost in all this is not actually implementing or training your final model. That can be done in any framework. The cost is finding a model that works really well for a particular task. You will get there a lot sooner and with a lot fewer headaches if you use PyTorch for this more "researchy" part. As to deployment, once you figure out what works, there's also libtorch, or if that doesn't work for you for whatever reason, implementing a model you know working hyperparameters for can be done in a day or two on whatever framework works for your backend, including TF.
nl 8y ago
Well you'll be glad to know that PyTorch is available (in development form) on the TPU: https://github.com/pytorch/xla https://github.com/pytorch/xla
- m0zg 8y ago
  If it's "available" but doesn't do anything useful, it's IMO not really "available". It's a good start though. I hope they don't throttle it in favor of TF because that'd only prolong TF's agony.
  - nl 8y ago
    I think this is pretty unfair. I know a bunch of PyTorch using people who have switched back to TF 2.0, and that is ignoring things like the Swift TF implementation. The deployment story around TF is a lot better too.

by m0zg 8y ago

akoumis 8y ago

Have you ever used Keras?

m0zg 8y ago
Yes I used Keras before PyTorch came out. IMO, Google needs to start over rather than try extensive wart removal on something that consists almost entirely of warts. These days Keras has subpar multi-GPU support (two GPUs are often slower than one), less flexibility, and more un-debuggable magic than PyTorch. And it doesn't look like things will be improving much (if at all) with TF 2.0. And the TPU is a story of its own when it comes to usability. Take a look at their docs. TPU actually has to run on a separate machine, which you need to provision manually using their `ctpu` tool, and training of a trivial network on CIFAR looks like this: https://github.com/tensorflow/tpu/blob/master/models/experimental/cifar_keras/cifar_keras.py https://github.com/tensorflow/tpu/blob/master/models/experim...
- so_tired 8y ago
  Err - i don't get it. What alternative r u suggesting for development which is aimed at eventual production? I am a complete noob, so i just did Keras (to TF) and GPUs. If i switch to TPU - its still the same code. When i get the budget to cluster, i think I am better off with one of the Deepmind libs which TF based. No?
  - m0zg 8y ago
    If you're just using off-the-GitHub models and pressing them into service, sure, Keras is adequate. If your work actually requires research and experimentation (as most practical problems on which actual cash money can be made do) you'll be much better served with PyTorch. The main cost in all this is not actually implementing or training your final model. That can be done in any framework. The cost is finding a model that works really well for a particular task. You will get there a lot sooner and with a lot fewer headaches if you use PyTorch for this more "researchy" part. As to deployment, once you figure out what works, there's also libtorch, or if that doesn't work for you for whatever reason, implementing a model you know working hyperparameters for can be done in a day or two on whatever framework works for your backend, including TF.

nl 8y ago

Well you'll be glad to know that PyTorch is available (in development form) on the TPU: https://github.com/pytorch/xla https://github.com/pytorch/xla

m0zg 8y ago
If it's "available" but doesn't do anything useful, it's IMO not really "available". It's a good start though. I hope they don't throttle it in favor of TF because that'd only prolong TF's agony.
- nl 8y ago
  I think this is pretty unfair. I know a bunch of PyTorch using people who have switched back to TF 2.0, and that is ignoring things like the Swift TF implementation. The deployment story around TF is a lot better too.