4 ms·

Is it just me or is there zero explanation to what this actually is? It somehow "helps" me understand deep learning but its tutorial / doc is one python notebo

by zwaps 7y ago

Is it just me or is there zero explanation to what this actually is? It somehow "helps" me understand deep learning but its tutorial / doc is one python notebook with three cells where some nondescript unknown API is called to train a transformer. Huh?

ma2rten 7y ago
From the README: "Trax code is structured in a way that allows you to understand deep learning from scratch. We start with basic maths and go through layers, models, supervised and reinforcement learning."
- zwaps 7y ago
  But what is it? It says it is an API or module? Is it a DL framework? Is it an extension of TF/Jax with new models? Is it a set of tutorials? Are we to work through the code? Is a part of the doc for Jax? Is it a set of implementations of Jax code into concrete models? All these things come to mind when going through this Github. By now I think to have figured out that it is a module that implements some DL models using Jax. So it is like an extension to Jax. It "helps" you to understand how to build the models in Jax if you go through the code. You could also load these abstractions as a Python module and use them in code, but I would doubt that this would "help you understand deep learning". So the real intention is to go through the code and see how Jax implements, for example, a transformer. Alternatively, this could be an early stage of some PyTorch-type framework over Jax and TF. So essentially, another Keras.
  - kavalg 7y ago
    Thank you
  - ma2rten 7y ago
    It's the successor of tensor2tensor, so it's a high level library that implements different the algorithms. Unlike other libraries the code is very readable.
WnZ39p0Dgydaz1 7y ago
You're not alone. Even as an ML researcher at ex-FAANG I have no idea what this is. Is it a collection of well-documented models built on top of jax? I could probably figure out what exactly this is if I spend an hour looking through the code, but it should be made clear in the README. It's kind of funny, I think it's completely opaque what this actually is to 99% of HN users, but for some reason it's being upvoted because it has Google and Deep Learning in the name.
- WnZ39p0Dgydaz1 7y ago
  15min of looking through the code and my best guess right now is that it's a reimplementation of higher-level primitives, such as optimizers and layers, found in Tensorflow/Pytorch/etc, but based on a variable backend (you can pick jax or TF), together with a collection of models and training loops. I think the idea is that most of the code is simpler and more modular than what you would find in TF, which makes the models easier to read. However, I don't yet understand what the use case is, or how it helps you to "learn" anything.
chillee 7y ago
Trax is meant as a successor to Tensor2Tensor. From my understanding of what Tensor2Tensor was (and thus what Trax is): Basically, at it's core, it's a library of models, datasets, and utilities that are meant to be clear and work together nicely. This is meant to be good for education, as you have an easy and unified way to run SOTA trained models on a wide variety of datasets. This is also meant to be good for research, as much of research involves building on top of other people's models, and having a library of models with shared API (as opposed to the fragmented wilderness that is public research code) is meant to facilitate that. I believe that it also ended up being used in industry as well, simply as a reliable source of pre-trained models. PS: As Jax did not have an "official" neural network library until very recently, this also served as a neural network library for Jax.
- shoyer 7y ago
  JAX still doesn't have an "official" neural network library yet. Source: I work closely with the JAX team at Google.
  - chillee 7y ago
    Poor wording perhaps. I meant that there was no project that "officially" called itself a neural network library for Jax, not that Jax had "officially" chosen a neural network library. I'm talking, of course, about Flax.

by zwaps 7y ago

ma2rten 7y ago

From the README: "Trax code is structured in a way that allows you to understand deep learning from scratch. We start with basic maths and go through layers, models, supervised and reinforcement learning."

zwaps 7y ago
But what is it? It says it is an API or module? Is it a DL framework? Is it an extension of TF/Jax with new models? Is it a set of tutorials? Are we to work through the code? Is a part of the doc for Jax? Is it a set of implementations of Jax code into concrete models? All these things come to mind when going through this Github. By now I think to have figured out that it is a module that implements some DL models using Jax. So it is like an extension to Jax. It "helps" you to understand how to build the models in Jax if you go through the code. You could also load these abstractions as a Python module and use them in code, but I would doubt that this would "help you understand deep learning". So the real intention is to go through the code and see how Jax implements, for example, a transformer. Alternatively, this could be an early stage of some PyTorch-type framework over Jax and TF. So essentially, another Keras.
- kavalg 7y ago
  Thank you
- ma2rten 7y ago
  It's the successor of tensor2tensor, so it's a high level library that implements different the algorithms. Unlike other libraries the code is very readable.

WnZ39p0Dgydaz1 7y ago

You're not alone. Even as an ML researcher at ex-FAANG I have no idea what this is. Is it a collection of well-documented models built on top of jax? I could probably figure out what exactly this is if I spend an hour looking through the code, but it should be made clear in the README. It's kind of funny, I think it's completely opaque what this actually is to 99% of HN users, but for some reason it's being upvoted because it has Google and Deep Learning in the name.

WnZ39p0Dgydaz1 7y ago
15min of looking through the code and my best guess right now is that it's a reimplementation of higher-level primitives, such as optimizers and layers, found in Tensorflow/Pytorch/etc, but based on a variable backend (you can pick jax or TF), together with a collection of models and training loops. I think the idea is that most of the code is simpler and more modular than what you would find in TF, which makes the models easier to read. However, I don't yet understand what the use case is, or how it helps you to "learn" anything.

chillee 7y ago

Trax is meant as a successor to Tensor2Tensor. From my understanding of what Tensor2Tensor was (and thus what Trax is): Basically, at it's core, it's a library of models, datasets, and utilities that are meant to be clear and work together nicely. This is meant to be good for education, as you have an easy and unified way to run SOTA trained models on a wide variety of datasets. This is also meant to be good for research, as much of research involves building on top of other people's models, and having a library of models with shared API (as opposed to the fragmented wilderness that is public research code) is meant to facilitate that. I believe that it also ended up being used in industry as well, simply as a reliable source of pre-trained models. PS: As Jax did not have an "official" neural network library until very recently, this also served as a neural network library for Jax.

shoyer 7y ago
JAX still doesn't have an "official" neural network library yet. Source: I work closely with the JAX team at Google.
- chillee 7y ago
  Poor wording perhaps. I meant that there was no project that "officially" called itself a neural network library for Jax, not that Jax had "officially" chosen a neural network library. I'm talking, of course, about Flax.