7 ms·

Julia on Google TPU: Shakespeare RNN

209 points by KenoFischer 8y ago

Mizza 8y ago
That's a whole lot of work for a result that's indistinguishable from a Markov chain generator..
- KenoFischer 8y ago
  Sure, the point is 1. To use a small dataset that can be fed from the relatively small Colab VM (so people can play with it themselves) 2. To use a well known model (so people can focus on the implementation parts of it) There's a resnet example in the previous example in the series (though the dataset is too large to feed from Colab). Also, this is very early stage software, so things are a bit more verbose than they should be :).
  - dnautics 8y ago
    Am I mistaken that this is interesting (and possibly also complex) because it's using the TensorFlow (out of julia) library to leverage the TPUs, while the actual ML algo is implemented in Flux, a library which has no TPU support?
    - KenoFischer 8y ago
      There's very little TensorFlow here. We're using it as a glorified grpc client basically to ship XLA to the TPU. The actual mechanism is described in https://arxiv.org/abs/1810.09868 https://arxiv.org/abs/1810.09868. The model definition itself uses the layers from Flux, but there's a couple assumptions that Flux makes that don't hold for the TPU so we don't get to use everything from Flux (e.g. the training loop and the optimizers). Luckily everything is julia, so we can get TPU-compatiable versions in just a few lines of code. Unifying the abstractions is an active work in progress, but this shows off that Julia runs on TPUs at all (an on freely available, public infrastructure at that), which is pretty cool, because right now the only other TPU frontend that can say the same is TensorFlow (which had a bit of a head start ;).
      - marmaduke 8y ago
        Not to detract from Julia, but I’ve always found it odd that Julia users report that “Julia runs on X” when in fact it’s more that one can use X from Julia or generate code for X by rewriting Julia code. By analogy, and contrast, no one says Python runs on GPUs, just because TensorFlow allows describing models in Python that then get run on the GPU, or Numba rewrites Python loops to CUDA PTX. It looks like marketing tbh despite know that Julia is a very solid language technically and shouldn’t need these kinds of rhetorical tricks.
        KenoFischer 8y ago
        So to me the biggest semantic question are the following: - Does what you have to write to run on X follow the semantics of the language? - Can you use data structures/code defined in libraries that don't know about your thing? In the case of julia on TPU here, the answer is yes! (surprisingly perhaps, and getting this to work is pretty hard). In particular, you get a lot of julia's language features: Multiple dispatch, control flow, etc. It's a bit of a subset of the full language (e.g. no mutation at the moment), but everything that's supported is just standard julia and we're working on growing that subset more and more. That approach is very different from something like TensorFlow where you're essentially metaprogramming an expression graph. Numba probably counts for python (yeah, you have to put an annotation on things, but if the python people really wanted, they could probably import Numba into core cpython and make it work more smoothly). Of course in python, you have the additional complication that most of the core implementation is not in python itself, so even if you satisfy my two criteria above for the core language, you're still gonna have to rewrite the whole standard library.
        skybrian 8y ago
        It's a nice demo, but reusing Julia libraries not intended for TensorFlow seems like a fragile thing? Just because it works today doesn't mean the authors won't inadvertently break it by using something outside the portable subset. It seems like for non-demo usage, you would want upstream maintainers to agree that their code should be TensorFlow-compatible, and have tests keeping it working.
- minimaxir 8y ago
  It depends on the dataset. Char-RNNs do pretty well compared to Markov chains when used to generate smaller texts. (which is why I've become less of a fan of the Shakespeare example the original Karpathy post used).
- jefftk 8y ago
  > indistinguishable from a Markov chain generator A Markov chain generator wouldn't: * Capitalize the first word of each line * Make lines of approximately the right length * Mark text with who is to speak it While this is just a toy example, it's powerful enough to start showing the ways RNNs can produce text that looks superficially correct. (Generating Shakespeare is actually one of the examples given in the classic http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
  - AstralStorm 8y ago
    You're mistaking a Markov chain toy with an actual Markov chain generator. 1. Yes it would. It would see capitalized words as high probability for first word on a line or after a point. 2. Obviously it could, depending on stop condition. Especially if you include line length. 3. If trained on corpus of plays, for sure it would. The strength on the RNN is supposed to be in context and memory... Perhaps handling of grammar. There are advanced hierarchical grammars that are related to Markov random field models that are about on par with RNN based on many text and music analysis loads. (In fact probabilistic math is often used to describe results and workings of a deep NN anyway.)
- mopeloi 8y ago
  I had a similar experience trying to use an LSTM model / TensorFlow based on the Shakespeare RNN example. This doesn't generate novel, meaningful text content without a template, but I'm unaware of any machine learning model that does this well (OpenAI's included). This model does absorb the 'rules' of the text, both grammar and structure (character names and dialogue, scene introductions, etc) In my own project, I made use of this to find spelling and grammatical errors on Esperanto Wikipedia: https://medium.com/@mapmeld/esperanto-nlp-part-3-correcting-grammar-6d34f93bded1 https://medium.com/@mapmeld/esperanto-nlp-part-3-correcting-...
aperrien 8y ago
How large is the project? I'd like to clone it to my drive account, but not if the data sets are too huge.
- KenoFischer 8y ago
  The notebook is about 400kB. It also downloads a 4MB dataset and julia, tensorflow and dependencies need about 400MB, but that gets stored in the ephemeral Colab VM, not on drive if you copy it there.
KenoFischer 8y ago
Oh, I should have mentioned somewhere. Credit for putting the notebook together goes to Elliot Saba (https://github.com/staticfloat https://github.com/staticfloat) :).
- zackmorris 8y ago
  This is brilliant work. I'm not strong in NNs yet, but I am strong in prerequisites/blockers. This demos: * Working in a rapid application development (RAD) fashion by operating on vectors using a language like Julia/MATLAB/Octave/Scilab which allows focusing on abstractions instead of implementation details and other distractions. * Running code optimized automagically on GPU/TPU/etc. * Sharing work over the web in a standard fashion (Jupyter Notebook on colab.research.google.com) It's not clear to me where in this process the code is actually run on TPU (maybe someone has a tutorial?) but that doesn't really matter. The specific machine learning algorithm used is also not really that important. The important part is that this enables amateurs to tinker with machine learning, see results quickly and share their work. Which means that now we'll finally see the accelerated evolution of machine learning. Any of these blockers alone hindered the evolution of AI for decades, but seeing all three knocked down in one fell swoop is pretty astonishing, at least for me. I favorited it as a watershed moment in the history of AI! Congrats to him.
fartcannon 8y ago
"Error Could not access the resources needed to display output. This is probably because third-party cookies are not allowed by your browser. SecurityError: The operation is insecure." You know what doesn't throw errors whenever I just try to look at it? Products like Jupyter notebooks - stuff not made by google. I believe Colab is a using dark patterns to discourage blocking 3rd party cookies by subtly breaking what should render as a simple non interactive webpage to passive viewers. To me, at least, Google and their dark patterns are considered harmful.
- 3JPLW 8y ago
  Here's what I white-listed on uMatrix: * raw.githubusercontent.com — it's loading the notebook directly from GitHub, so yeah, this one is fairly essential. Just need one XHR here. * googleusercontent.com — no cookies, but it does load scripts, css, a frame and an XHR here. That's it. There are some other domains it hits (like fonts.google.com and gstatic.com), but they're not needed to view the file.
  - __random__ 8y ago
    https://github.com/googlecolab/colabtools/issues/17#issuecomment-459030596 https://github.com/googlecolab/colabtools/issues/17#issuecom...
  - fartcannon 8y ago
    Thanks! But instead, I'll just avoid Google.
FridgeSeal 8y ago
I’ve been using Julia for a project at work and it’s been a pretty fantastic experience. I’ve only being doing some analysis stuff at the moment, but I’ve got a few machine learning/NLP projects coming up that I’m super excited to use Julia and Flux for!
- xiaodai 8y ago
  How did u find the compilation process
  - FridgeSeal 8y ago
    Package compilation or code compilation? Package compilation was painless, code compilation isn't really noticeable once it starts running and the benefits you get from native speed are evident. I recently learnt that the IO functions are async as well (powered by libUV!) and the parallelism is easily an order of magnitude nicer to work with than Python's (and it's only going to get better).

209 points by KenoFischer 8y ago

Mizza 8y ago

That's a whole lot of work for a result that's indistinguishable from a Markov chain generator..

KenoFischer 8y ago
Sure, the point is 1. To use a small dataset that can be fed from the relatively small Colab VM (so people can play with it themselves) 2. To use a well known model (so people can focus on the implementation parts of it) There's a resnet example in the previous example in the series (though the dataset is too large to feed from Colab). Also, this is very early stage software, so things are a bit more verbose than they should be :).
- dnautics 8y ago
  Am I mistaken that this is interesting (and possibly also complex) because it's using the TensorFlow (out of julia) library to leverage the TPUs, while the actual ML algo is implemented in Flux, a library which has no TPU support?
  - KenoFischer 8y ago
    There's very little TensorFlow here. We're using it as a glorified grpc client basically to ship XLA to the TPU. The actual mechanism is described in https://arxiv.org/abs/1810.09868 https://arxiv.org/abs/1810.09868. The model definition itself uses the layers from Flux, but there's a couple assumptions that Flux makes that don't hold for the TPU so we don't get to use everything from Flux (e.g. the training loop and the optimizers). Luckily everything is julia, so we can get TPU-compatiable versions in just a few lines of code. Unifying the abstractions is an active work in progress, but this shows off that Julia runs on TPUs at all (an on freely available, public infrastructure at that), which is pretty cool, because right now the only other TPU frontend that can say the same is TensorFlow (which had a bit of a head start ;).
    - marmaduke 8y ago
      Not to detract from Julia, but I’ve always found it odd that Julia users report that “Julia runs on X” when in fact it’s more that one can use X from Julia or generate code for X by rewriting Julia code. By analogy, and contrast, no one says Python runs on GPUs, just because TensorFlow allows describing models in Python that then get run on the GPU, or Numba rewrites Python loops to CUDA PTX. It looks like marketing tbh despite know that Julia is a very solid language technically and shouldn’t need these kinds of rhetorical tricks.
      - KenoFischer 8y ago
        So to me the biggest semantic question are the following: - Does what you have to write to run on X follow the semantics of the language? - Can you use data structures/code defined in libraries that don't know about your thing? In the case of julia on TPU here, the answer is yes! (surprisingly perhaps, and getting this to work is pretty hard). In particular, you get a lot of julia's language features: Multiple dispatch, control flow, etc. It's a bit of a subset of the full language (e.g. no mutation at the moment), but everything that's supported is just standard julia and we're working on growing that subset more and more. That approach is very different from something like TensorFlow where you're essentially metaprogramming an expression graph. Numba probably counts for python (yeah, you have to put an annotation on things, but if the python people really wanted, they could probably import Numba into core cpython and make it work more smoothly). Of course in python, you have the additional complication that most of the core implementation is not in python itself, so even if you satisfy my two criteria above for the core language, you're still gonna have to rewrite the whole standard library.
        skybrian 8y ago
        It's a nice demo, but reusing Julia libraries not intended for TensorFlow seems like a fragile thing? Just because it works today doesn't mean the authors won't inadvertently break it by using something outside the portable subset. It seems like for non-demo usage, you would want upstream maintainers to agree that their code should be TensorFlow-compatible, and have tests keeping it working.
minimaxir 8y ago
It depends on the dataset. Char-RNNs do pretty well compared to Markov chains when used to generate smaller texts. (which is why I've become less of a fan of the Shakespeare example the original Karpathy post used).
jefftk 8y ago
> indistinguishable from a Markov chain generator A Markov chain generator wouldn't: * Capitalize the first word of each line * Make lines of approximately the right length * Mark text with who is to speak it While this is just a toy example, it's powerful enough to start showing the ways RNNs can produce text that looks superficially correct. (Generating Shakespeare is actually one of the examples given in the classic http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
- AstralStorm 8y ago
  You're mistaking a Markov chain toy with an actual Markov chain generator. 1. Yes it would. It would see capitalized words as high probability for first word on a line or after a point. 2. Obviously it could, depending on stop condition. Especially if you include line length. 3. If trained on corpus of plays, for sure it would. The strength on the RNN is supposed to be in context and memory... Perhaps handling of grammar. There are advanced hierarchical grammars that are related to Markov random field models that are about on par with RNN based on many text and music analysis loads. (In fact probabilistic math is often used to describe results and workings of a deep NN anyway.)
mopeloi 8y ago
I had a similar experience trying to use an LSTM model / TensorFlow based on the Shakespeare RNN example. This doesn't generate novel, meaningful text content without a template, but I'm unaware of any machine learning model that does this well (OpenAI's included). This model does absorb the 'rules' of the text, both grammar and structure (character names and dialogue, scene introductions, etc) In my own project, I made use of this to find spelling and grammatical errors on Esperanto Wikipedia: https://medium.com/@mapmeld/esperanto-nlp-part-3-correcting-grammar-6d34f93bded1 https://medium.com/@mapmeld/esperanto-nlp-part-3-correcting-...

aperrien 8y ago

How large is the project? I'd like to clone it to my drive account, but not if the data sets are too huge.

KenoFischer 8y ago
The notebook is about 400kB. It also downloads a 4MB dataset and julia, tensorflow and dependencies need about 400MB, but that gets stored in the ephemeral Colab VM, not on drive if you copy it there.

KenoFischer 8y ago

Oh, I should have mentioned somewhere. Credit for putting the notebook together goes to Elliot Saba (https://github.com/staticfloat https://github.com/staticfloat) :).

zackmorris 8y ago
This is brilliant work. I'm not strong in NNs yet, but I am strong in prerequisites/blockers. This demos: * Working in a rapid application development (RAD) fashion by operating on vectors using a language like Julia/MATLAB/Octave/Scilab which allows focusing on abstractions instead of implementation details and other distractions. * Running code optimized automagically on GPU/TPU/etc. * Sharing work over the web in a standard fashion (Jupyter Notebook on colab.research.google.com) It's not clear to me where in this process the code is actually run on TPU (maybe someone has a tutorial?) but that doesn't really matter. The specific machine learning algorithm used is also not really that important. The important part is that this enables amateurs to tinker with machine learning, see results quickly and share their work. Which means that now we'll finally see the accelerated evolution of machine learning. Any of these blockers alone hindered the evolution of AI for decades, but seeing all three knocked down in one fell swoop is pretty astonishing, at least for me. I favorited it as a watershed moment in the history of AI! Congrats to him.

fartcannon 8y ago

"Error Could not access the resources needed to display output. This is probably because third-party cookies are not allowed by your browser. SecurityError: The operation is insecure." You know what doesn't throw errors whenever I just try to look at it? Products like Jupyter notebooks - stuff not made by google. I believe Colab is a using dark patterns to discourage blocking 3rd party cookies by subtly breaking what should render as a simple non interactive webpage to passive viewers. To me, at least, Google and their dark patterns are considered harmful.

3JPLW 8y ago
Here's what I white-listed on uMatrix: * raw.githubusercontent.com — it's loading the notebook directly from GitHub, so yeah, this one is fairly essential. Just need one XHR here. * googleusercontent.com — no cookies, but it does load scripts, css, a frame and an XHR here. That's it. There are some other domains it hits (like fonts.google.com and gstatic.com), but they're not needed to view the file.
- __random__ 8y ago
  https://github.com/googlecolab/colabtools/issues/17#issuecomment-459030596 https://github.com/googlecolab/colabtools/issues/17#issuecom...
- fartcannon 8y ago
  Thanks! But instead, I'll just avoid Google.

FridgeSeal 8y ago

I’ve been using Julia for a project at work and it’s been a pretty fantastic experience. I’ve only being doing some analysis stuff at the moment, but I’ve got a few machine learning/NLP projects coming up that I’m super excited to use Julia and Flux for!

xiaodai 8y ago
How did u find the compilation process
- FridgeSeal 8y ago
  Package compilation or code compilation? Package compilation was painless, code compilation isn't really noticeable once it starts running and the benefits you get from native speed are evident. I recently learnt that the IO functions are async as well (powered by libUV!) and the parallelism is easily an order of magnitude nicer to work with than Python's (and it's only going to get better).