3 ms·

As someone who hasn't really used either, what's pytorch doing that's so much better?

jwjohnson314 2y ago
PyTorch is just much more flexible. Implementing a custom loss function, for example, is straightforward in PyTorch and a hassle in Keras (or was last time I used it, which was several years ago).
minimaxir 2y ago
A few things from personal experience: - LLM support with PyTorch is better (both at a tooling level and CUDA level). Hugging Face transformers does have support for both TensorFlow and PyTorch variants of LLMs but... - Almost all new LLMs are in PyTorch first and may or may not be ported to TensorFlow. This most notably includes embeddings models which are the most important area in my work. - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible. PyTorch has a DataLoader which can handle CPU/GPU data movement and processing. - PyTorch has better implementations for modern ML training improvments such as fp16, multi-GPU support, better native learning rate schedulers, etc. PyTorch can also override the training loop for very specific implementations (e.g. custom loss functions). Implementing them in TensorFlow/Keras is a buggy pain. - PyTorch was faster to train than TensorFlow models using the same hardware and model architecture. - Keras's serialization for model deployment is a pain in the butt (e.g. SavedModels) while PyTorch both has better implementations with torch.jit, and also native ONNX export.
- perturbation 2y ago
  I think a lot of these may have improved since your last experience with Keras. It's pretty easy to override the training loop and/or make custom loss. The below is for overriding training / test step altogether, custom loss is easier by making a new loss function/class. https://keras.io/examples/keras_recipes/trainer_pattern/ https://keras.io/examples/keras_recipes/trainer_pattern/ > - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible. The Tensorflow backend has the excellent tf.data.Dataset API, which allows for out of core data and processing in a streaming way.
  - minimaxir 2y ago
    That's a fair implementation of custom loss. Hugging Face's Trainer with transformers suggests a similar implementation, although their's has less boilerplate. https://huggingface.co/docs/transformers/main/en/trainer#customize-the-trainer https://huggingface.co/docs/transformers/main/en/trainer#cus...
adultSwim 2y ago
Being successful is also why it's better. PyTorch has a thriving ecosystem of software around it and a large userbase. Picking it comes with many network benefits.

PyTorch is just much more flexible. Implementing a custom loss function, for example, is straightforward in PyTorch and a hassle in Keras (or was last time I used it, which was several years ago).

minimaxir 2y ago

A few things from personal experience: - LLM support with PyTorch is better (both at a tooling level and CUDA level). Hugging Face transformers does have support for both TensorFlow and PyTorch variants of LLMs but... - Almost all new LLMs are in PyTorch first and may or may not be ported to TensorFlow. This most notably includes embeddings models which are the most important area in my work. - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible. PyTorch has a DataLoader which can handle CPU/GPU data movement and processing. - PyTorch has better implementations for modern ML training improvments such as fp16, multi-GPU support, better native learning rate schedulers, etc. PyTorch can also override the training loop for very specific implementations (e.g. custom loss functions). Implementing them in TensorFlow/Keras is a buggy pain. - PyTorch was faster to train than TensorFlow models using the same hardware and model architecture. - Keras's serialization for model deployment is a pain in the butt (e.g. SavedModels) while PyTorch both has better implementations with torch.jit, and also native ONNX export.

perturbation 2y ago
I think a lot of these may have improved since your last experience with Keras. It's pretty easy to override the training loop and/or make custom loss. The below is for overriding training / test step altogether, custom loss is easier by making a new loss function/class. https://keras.io/examples/keras_recipes/trainer_pattern/ https://keras.io/examples/keras_recipes/trainer_pattern/ > - Keras's training loop assumes you can fit all the data in memory and that the data is fully preprocessed, which in the world of LLMs and big data is infeasible. The Tensorflow backend has the excellent tf.data.Dataset API, which allows for out of core data and processing in a streaming way.
- minimaxir 2y ago
  That's a fair implementation of custom loss. Hugging Face's Trainer with transformers suggests a similar implementation, although their's has less boilerplate. https://huggingface.co/docs/transformers/main/en/trainer#customize-the-trainer https://huggingface.co/docs/transformers/main/en/trainer#cus...

adultSwim 2y ago

Being successful is also why it's better. PyTorch has a thriving ecosystem of software around it and a large userbase. Picking it comes with many network benefits.