4 ms·

You're bearish on new hardware for AI training because it takes 2 months to setup? That's like saying you're bearish on the invention of transistor because ther

You're bearish on new hardware for AI training because it takes 2 months to setup? That's like saying you're bearish on the invention of transistor because there are no computers yet. This chip is for research organizations such as LLNL and ANL. Tooling and commoditization will happen. > “if we build it, they’ll buy it.” This is literally how any new thing is invented and commercialized.

sillysaurusx 6y ago
Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today. Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons). Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with. The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning. Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood. Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.
- durst 6y ago
  Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?
  - sillysaurusx 6y ago
    Sadly I have nothing to offer yet, other than to DM me on Twitter with questions. Always happy to answer basic ones; I love this stuff. But you’re right, I should write up something. That guide is pretty good, but it doesn’t walk you through anything specific; it shows you a map, but doesn’t take you on a trip, so to speak. Some tips: Use tf.name_scopes! You probably use them for variables, but there’s a different one for operations. If you make an @op_scope decorator, you can use it on all your ML functions and immediately get lots of insights as to where the XLA ops end up mapping to in your source code. As much as it pains me to say this, avoid tensorflow 2 style code like the plague. Pretend that if you use eager execution, someone will jump out of the bushes and shoot you. I technically use TF2.4 now, but it’s still session / graph-based, not the new tf.function magic. The new magic pipeline seems to be slower, harder to use, harder to debug, and very likely to explode when you do anything even slightly different than the tutorial examples. YMMV, and maybe things are better now (or in a future release). The profiler is magical; leverage it whenever you can. My workflow is to start up a TPU run, then ssh into my server and fire up a Tensorboard to that model dir. Then I manually navigate to <tensorboard_url>/#profile (because the “profile” button doesn’t seem to show up in the menu anymore) which then lets you “capture profile”. Make sure your TPU version matches your Tensorboard version. At this point we use TPU version 2.4.0, Tensorboard 2.4.1. If you get mysterious errors, this is likely the reason things are going wrong. Even TPU version 2.3.0 wasn’t enough. Happily, the new profiling tooling is totally badass. The trace viewer is great, the op profiling is ok (though I wish it would show me all the damn ops, instead of “helpfully” hiding all but the top N ops), and the memory viewer is incredible. You can see exactly where in your pipeline is causing “peak memory usage”, what the peak is (down to the kilobyte), and have at least some idea of what’s causing it. It’s not effortless though. The XLA fusion ops sort of make it harder to track down what’s doing what. (TF compiler is very powerful, but the trade off is that you almost never have manual control over memory usage, which can be frustrating). All in all (or all-to-all, ha) it’s a lot of fun if you like seeing expensive hardware go brrrr, as I do. It makes it all worth it when your loss drops from 11 to 3 overnight on a 430m parameter gpt model. :)
    - durst 6y ago
      Thank you for the tips! Whenever I hear about new accelerators, my first question is: "how do people in the real world run fast code on this"? Because (related to your above points), you don't use an accelerator for things to just run, they have to run fast. Otherwise, you'd just use a CPU. Detailed Question 1: do you have examples of fusion making things hard? Is there a way to nudge the compiler to not fuse or create a symbol table tracking fusion? Does fusion cause issues with the name_scopes? Detailed Question 2: isn't Pytorch eager execution? Do you know how it compares to Tensorflow's eager execution? General Request: I'm in a more theoretical position, writing papers on programming languages for accelerators https://aetherling.org/ https://aetherling.org/ and TAing courses on accelerators http://cs149.stanford.edu/fall20 http://cs149.stanford.edu/fall20. So, I'm excited to see people's practical experiences using these accelerators in industry. It would be very enlightening (if you have time) to write up a comparison of tuning a model for an A100's tensor cores vs a TPU. This seems like the key trade-off in comparing architectures.

sillysaurusx 6y ago

Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today. Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons). Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with. The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning. Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood. Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.

durst 6y ago
Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?
- sillysaurusx 6y ago
  Sadly I have nothing to offer yet, other than to DM me on Twitter with questions. Always happy to answer basic ones; I love this stuff. But you’re right, I should write up something. That guide is pretty good, but it doesn’t walk you through anything specific; it shows you a map, but doesn’t take you on a trip, so to speak. Some tips: Use tf.name_scopes! You probably use them for variables, but there’s a different one for operations. If you make an @op_scope decorator, you can use it on all your ML functions and immediately get lots of insights as to where the XLA ops end up mapping to in your source code. As much as it pains me to say this, avoid tensorflow 2 style code like the plague. Pretend that if you use eager execution, someone will jump out of the bushes and shoot you. I technically use TF2.4 now, but it’s still session / graph-based, not the new tf.function magic. The new magic pipeline seems to be slower, harder to use, harder to debug, and very likely to explode when you do anything even slightly different than the tutorial examples. YMMV, and maybe things are better now (or in a future release). The profiler is magical; leverage it whenever you can. My workflow is to start up a TPU run, then ssh into my server and fire up a Tensorboard to that model dir. Then I manually navigate to <tensorboard_url>/#profile (because the “profile” button doesn’t seem to show up in the menu anymore) which then lets you “capture profile”. Make sure your TPU version matches your Tensorboard version. At this point we use TPU version 2.4.0, Tensorboard 2.4.1. If you get mysterious errors, this is likely the reason things are going wrong. Even TPU version 2.3.0 wasn’t enough. Happily, the new profiling tooling is totally badass. The trace viewer is great, the op profiling is ok (though I wish it would show me all the damn ops, instead of “helpfully” hiding all but the top N ops), and the memory viewer is incredible. You can see exactly where in your pipeline is causing “peak memory usage”, what the peak is (down to the kilobyte), and have at least some idea of what’s causing it. It’s not effortless though. The XLA fusion ops sort of make it harder to track down what’s doing what. (TF compiler is very powerful, but the trade off is that you almost never have manual control over memory usage, which can be frustrating). All in all (or all-to-all, ha) it’s a lot of fun if you like seeing expensive hardware go brrrr, as I do. It makes it all worth it when your loss drops from 11 to 3 overnight on a 430m parameter gpt model. :)
  - durst 6y ago
    Thank you for the tips! Whenever I hear about new accelerators, my first question is: "how do people in the real world run fast code on this"? Because (related to your above points), you don't use an accelerator for things to just run, they have to run fast. Otherwise, you'd just use a CPU. Detailed Question 1: do you have examples of fusion making things hard? Is there a way to nudge the compiler to not fuse or create a symbol table tracking fusion? Does fusion cause issues with the name_scopes? Detailed Question 2: isn't Pytorch eager execution? Do you know how it compares to Tensorflow's eager execution? General Request: I'm in a more theoretical position, writing papers on programming languages for accelerators https://aetherling.org/ https://aetherling.org/ and TAing courses on accelerators http://cs149.stanford.edu/fall20 http://cs149.stanford.edu/fall20. So, I'm excited to see people's practical experiences using these accelerators in industry. It would be very enlightening (if you have time) to write up a comparison of tuning a model for an A100's tensor cores vs a TPU. This seems like the key trade-off in comparing architectures.