5 ms·

I recently released a codebase in beta that modernizes a tiny model that gets really good performance on CIFAR-10 in about 18.1 or so seconds on the right singl

I recently released a codebase in beta that modernizes a tiny model that gets really good performance on CIFAR-10 in about 18.1 or so seconds on the right single GPU -- a number of years ago the world record was 10 minutes, down from several days a few years previously. While most of my work was porting and cleaning up certain parts of the code for a different purpose (just-clone-and-hack experimentation workbench), I've spent years optimizing neural networks at a very fine grained level, and many of the lessons learned here in debugging reflected that. I believe that there are fundamentally a few big NP-hard layers (at least two that I can define, and likely several other smaller ones) unfortunately but they are not hard blockers to progress. The model I mentioned above is extremely simple and has little "extra fat" where it is not needed. It also importantly seems to have good gradient and such flow throughout, something that's important for a model to be able to learn quickly. There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data. That does a shocking amount of work in stabilizing and speeding up training. Ultimately, the network is simple, and there are a number of other methods to help it reach near-SOTA, but they are as simple as can be. I think as this project evolves and we get nearer to the goal (<2 seconds in a year or two), we'll keep uncovering good puzzle pieces showing exactly what it is that's allowing such a tiny network to perform so well. There's a kind of exponential value to having ultra-short training times -- you can somewhat open-endedly barrage-test your algorithm, something that's already led to a few interesting discoveries that I'd like to refine before publishing to the repo. If you're interested, the code is here. The running code is a single .py with the upsides and downsides that come with that. If you're interested or have any questions, let me know! :D :)))) https://github.com/tysam-code/hlb-CIFAR10 https://github.com/tysam-code/hlb-CIFAR10

jacquesm 4y ago
Very nice work! What I've found with optimizations is that once you have exhausted one avenue that has almost magically caused another one to be opened up which you can then squash and so on. The compounding effect of such serial optimizations can be considerable. I wonder if converting a model back into code and then optimizing that is a viable path, you could then try to drop out little bits of the code to see if they meaningfully affect the output or not.
- tbalsam 4y ago
  Thanks for the comment! As far as seeing what the core parts are, I'm really partial to the method where you have a graph network with a really high L2 communication penalty which seems to approach the intrinsic dimension of the problem for certain simple problems. How well that scales to larger problems I have no idea (probably not well, but techniques like variational dropout are pretty analogous in their own kinds of ways). Thankfully with a fast-training network you can play/dork around with the numbers a bit and see what's what. One could distill the tiny ResNet into a graph network, which with the right constraints could theoretically accomplish the same as the original neural network, and then compress that as small as possible. There's probably an interesting tradeoff in "maximal compression" and "number of iterate rounds" for said graph network. I recently got enough runs (25) and performance difference to have a p=.0014 result or something like that in half an hour for something I was experimenting with against the baseline recently and it felt so good because I wasn't diddling with 5 runs, which in some cases for certain papers take days to finish. It's just a very satisfying feeling. I guess back to the explainability side of things.... -- that alone I don't think would necessarily provide answers to the explainability problem but I think it would be like a refining-oil type step before diving into the L2-compressed feature representations....
  - jacquesm 4y ago
    Interested in how that will pan out. Unfortunately I don't know enough about the subject matter or I would definitely give it a shot. And if you hack this to the point where it is explainable then that in turn might generalize to larger networks and/or different problems.
    - tbalsam 4y ago
      That's the hope! :D :)))) <3 <3 :)
      - jacquesm 4y ago
        Best of luck with this, if you have any interesting results and write them up please do post them and ping me (email in profile).
m3at 4y ago
Nice experiment! Not to minimize your work, in fact some ideas might be complementary to yours, a couple years ago there were already some approaches reaching 30s on a single V100: https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/ https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr...
- tbalsam 4y ago
  Yes, David Page's work is lovely. This initial release is almost a bit for bit remake of the functionality of the original code, but built to be linear and hackable at basically any stage of the pipeline. Page gets a ton of respect for me for all of the novel stuff introduced, I spent like 80-90 hours plus trying to debug the minutiae it takes to get things working properly at that accuracy -- and doing that thing has been my career. It's a seriously impressive accomplishment to me and the ease of which he presents some of those changes in the blog feels like one of those baking shows where you get a good idea of how truly difficult the achievement is when you do it yourself. I wanted to start with his baseline but as that hackable workbench for my own purposes to explore some information theory concepts w.r.t. deep learning and etc. His code is beautiful but also a framework-within-a-framework and nearly purely functional so quick hacks are basically impossible beyond a certain point. There are tradeoffs of course. Continuing to drop bit depth and a few other improvements will probably carry things surprisingly far, so long as hardware compatibility with said hacks remains Gucci. There's also some Triton kernel hacks that we could dip into but it would taint some of the "pure simple python" goals for the project. But yes -- this is a port of David Page's work designed for a 1-2 hour quick-sketch experimenting researcher. I've found a few other improvements that I hope to refine and contribute to the repo at some point -- after I fix a few basic, glaring bugs like the console printing the whole progress chart again each time. But yes, we're well on our way and thank you so much for linking that -- I'm a rather large fan of his work and I truly hope that some more of his wizardry comes to public light for us to glean from. :D :)))) <3
  - m3at 4y ago
    Now I realise that I _somehow_ totally missed your link to Page's work right in the README… Thanks for the detailed comment, I definitely will dive into your script soon! One early suggestion, you might want to try torch dynamo [1], anecdotally I had good speedups (~20%) on some image models; though not sure how significant the impact might be at this (relatively) small scale. [1] https://pytorch.org/docs/master/dynamo/ https://pytorch.org/docs/master/dynamo/
elcritch 4y ago
> There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data. That does a shocking amount of work in stabilizing and speeding up training. Wow, that sounds nifty! Could you elaborate or point to more resources for that sort of technique?
- tbalsam 4y ago
  Someone else linked this in a different tree but I do truly love this blogpost series -- you can see the section "input patch whitening" for more details. https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/ https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr... Alternatively, you can read through the direct code here if you're willing to wade through some of the mathy code on it: https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#L250 https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#...
  - elcritch 4y ago
    Thanks! Also that code seems pretty straightforward. Later there's this bit: conv_layer.weight.data = (eigenvectors/torch.sqrt(eigenvalues+eps))[-shape[0]:, :, :, :] ## We don't want to train this, since this is implicitly whitening over the whole dataset Oh it looks like you then just "remove" the correlated data.. Actually, you're just pre-"padding" the basic (gaussian?) covariances. Ah so then you'd skip training basic statistical information that's already easy to calculate directly. Clever!
ayhanfuat 4y ago
> a few big NP-hard layers What does NP-hard mean in the context of neural networks? You mean the loss minimization of those layers is NP-hard?
- bmc7505 4y ago
  There are architectures with layers which can approximate MAXSAT [1] (or rather, an SDP relaxation thereof), but I doubt this is what OP was referring to. [1]: https://arxiv.org/abs/1905.12149 https://arxiv.org/abs/1905.12149
- tbalsam 4y ago
  That's a good question, and I provided scant few hints towards it in my original post. There are a few layers in which the order of certain things matter -- basically, any chaotic system arising from the choices made in training neural networks. We oftentimes just randomly choose an answer as that's a "best guess", but in pushing the territory of world records, something more principled is in order. Weight initialization is one, and what data we show to the network when is another. Each choice at any point influences every single choice that comes after it, so even if it was quantized into discrete decision bins (which it can be for both of those, I believe -- even the weight initialization if you're a lottery ticket hypothesis fan. I cringe in saying that though as that phrase can summon an interesting mix of people.) In that sense, calculating which order of operations/order of values is ideal is I believe an NP-hard problem by definition, and not too much weirdness if we're looking at the discrete case(s) I think. Maybe solving that up front from a structure perspective is untenable, but if we're able to crack some of the mystery of the solution manifold and turn that into portable priors for architectures like this, then I think that opens the door to connecting things to a more universal, pure mathematical solution. And then that ends up unpacking nicely in other problem domains even if we maybe don't know up front which priors work well for that particular subdomain. If we have some sort of mathematically-connected rule, we can be more sure about it. That's a loose form of a general workflow I follow, it's a bit more of a crapshoot though where any discrete chaotic processes are involved, unfortunately. Hope that helps answer some of the question, there's definitely other layers to be had, though. Which means job security for a lot of people for a long time to come, I personally think. :D :))))
naasking 4y ago
Switching to Hinton's new Forward-Forward algorithm might get you that faster training time, rather than focusing on tweaking existing approaches.
- tbalsam 4y ago
  This is an interesting statement to me, especially as the original paper from Hinton notes that it converges more slowly than traditional algorithms. I like the new shiny shiny too, but I've been in this field too long to chase all the new stuff that comes along (and I do love me some Hinton too). I thought about FF for this application but didn't see anything that would make it work in this context, is there anything in particular that you were seeing that would benefit us in this particular usecase?

jacquesm 4y ago

Very nice work! What I've found with optimizations is that once you have exhausted one avenue that has almost magically caused another one to be opened up which you can then squash and so on. The compounding effect of such serial optimizations can be considerable. I wonder if converting a model back into code and then optimizing that is a viable path, you could then try to drop out little bits of the code to see if they meaningfully affect the output or not.

tbalsam 4y ago
Thanks for the comment! As far as seeing what the core parts are, I'm really partial to the method where you have a graph network with a really high L2 communication penalty which seems to approach the intrinsic dimension of the problem for certain simple problems. How well that scales to larger problems I have no idea (probably not well, but techniques like variational dropout are pretty analogous in their own kinds of ways). Thankfully with a fast-training network you can play/dork around with the numbers a bit and see what's what. One could distill the tiny ResNet into a graph network, which with the right constraints could theoretically accomplish the same as the original neural network, and then compress that as small as possible. There's probably an interesting tradeoff in "maximal compression" and "number of iterate rounds" for said graph network. I recently got enough runs (25) and performance difference to have a p=.0014 result or something like that in half an hour for something I was experimenting with against the baseline recently and it felt so good because I wasn't diddling with 5 runs, which in some cases for certain papers take days to finish. It's just a very satisfying feeling. I guess back to the explainability side of things.... -- that alone I don't think would necessarily provide answers to the explainability problem but I think it would be like a refining-oil type step before diving into the L2-compressed feature representations....
- jacquesm 4y ago
  Interested in how that will pan out. Unfortunately I don't know enough about the subject matter or I would definitely give it a shot. And if you hack this to the point where it is explainable then that in turn might generalize to larger networks and/or different problems.
  - tbalsam 4y ago
    That's the hope! :D :)))) <3 <3 :)
    - jacquesm 4y ago
      Best of luck with this, if you have any interesting results and write them up please do post them and ping me (email in profile).

m3at 4y ago

Nice experiment! Not to minimize your work, in fact some ideas might be complementary to yours, a couple years ago there were already some approaches reaching 30s on a single V100: https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/ https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr...

tbalsam 4y ago
Yes, David Page's work is lovely. This initial release is almost a bit for bit remake of the functionality of the original code, but built to be linear and hackable at basically any stage of the pipeline. Page gets a ton of respect for me for all of the novel stuff introduced, I spent like 80-90 hours plus trying to debug the minutiae it takes to get things working properly at that accuracy -- and doing that thing has been my career. It's a seriously impressive accomplishment to me and the ease of which he presents some of those changes in the blog feels like one of those baking shows where you get a good idea of how truly difficult the achievement is when you do it yourself. I wanted to start with his baseline but as that hackable workbench for my own purposes to explore some information theory concepts w.r.t. deep learning and etc. His code is beautiful but also a framework-within-a-framework and nearly purely functional so quick hacks are basically impossible beyond a certain point. There are tradeoffs of course. Continuing to drop bit depth and a few other improvements will probably carry things surprisingly far, so long as hardware compatibility with said hacks remains Gucci. There's also some Triton kernel hacks that we could dip into but it would taint some of the "pure simple python" goals for the project. But yes -- this is a port of David Page's work designed for a 1-2 hour quick-sketch experimenting researcher. I've found a few other improvements that I hope to refine and contribute to the repo at some point -- after I fix a few basic, glaring bugs like the console printing the whole progress chart again each time. But yes, we're well on our way and thank you so much for linking that -- I'm a rather large fan of his work and I truly hope that some more of his wizardry comes to public light for us to glean from. :D :)))) <3
- m3at 4y ago
  Now I realise that I _somehow_ totally missed your link to Page's work right in the README… Thanks for the detailed comment, I definitely will dive into your script soon! One early suggestion, you might want to try torch dynamo [1], anecdotally I had good speedups (~20%) on some image models; though not sure how significant the impact might be at this (relatively) small scale. [1] https://pytorch.org/docs/master/dynamo/ https://pytorch.org/docs/master/dynamo/

elcritch 4y ago

> There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data. That does a shocking amount of work in stabilizing and speeding up training. Wow, that sounds nifty! Could you elaborate or point to more resources for that sort of technique?

tbalsam 4y ago
Someone else linked this in a different tree but I do truly love this blogpost series -- you can see the section "input patch whitening" for more details. https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/ https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr... Alternatively, you can read through the direct code here if you're willing to wade through some of the mathy code on it: https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#L250 https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#...
- elcritch 4y ago
  Thanks! Also that code seems pretty straightforward. Later there's this bit: conv_layer.weight.data = (eigenvectors/torch.sqrt(eigenvalues+eps))[-shape[0]:, :, :, :] ## We don't want to train this, since this is implicitly whitening over the whole dataset Oh it looks like you then just "remove" the correlated data.. Actually, you're just pre-"padding" the basic (gaussian?) covariances. Ah so then you'd skip training basic statistical information that's already easy to calculate directly. Clever!

ayhanfuat 4y ago

> a few big NP-hard layers What does NP-hard mean in the context of neural networks? You mean the loss minimization of those layers is NP-hard?

bmc7505 4y ago
There are architectures with layers which can approximate MAXSAT [1] (or rather, an SDP relaxation thereof), but I doubt this is what OP was referring to. [1]: https://arxiv.org/abs/1905.12149 https://arxiv.org/abs/1905.12149
tbalsam 4y ago
That's a good question, and I provided scant few hints towards it in my original post. There are a few layers in which the order of certain things matter -- basically, any chaotic system arising from the choices made in training neural networks. We oftentimes just randomly choose an answer as that's a "best guess", but in pushing the territory of world records, something more principled is in order. Weight initialization is one, and what data we show to the network when is another. Each choice at any point influences every single choice that comes after it, so even if it was quantized into discrete decision bins (which it can be for both of those, I believe -- even the weight initialization if you're a lottery ticket hypothesis fan. I cringe in saying that though as that phrase can summon an interesting mix of people.) In that sense, calculating which order of operations/order of values is ideal is I believe an NP-hard problem by definition, and not too much weirdness if we're looking at the discrete case(s) I think. Maybe solving that up front from a structure perspective is untenable, but if we're able to crack some of the mystery of the solution manifold and turn that into portable priors for architectures like this, then I think that opens the door to connecting things to a more universal, pure mathematical solution. And then that ends up unpacking nicely in other problem domains even if we maybe don't know up front which priors work well for that particular subdomain. If we have some sort of mathematically-connected rule, we can be more sure about it. That's a loose form of a general workflow I follow, it's a bit more of a crapshoot though where any discrete chaotic processes are involved, unfortunately. Hope that helps answer some of the question, there's definitely other layers to be had, though. Which means job security for a lot of people for a long time to come, I personally think. :D :))))

naasking 4y ago

Switching to Hinton's new Forward-Forward algorithm might get you that faster training time, rather than focusing on tweaking existing approaches.

tbalsam 4y ago
This is an interesting statement to me, especially as the original paper from Hinton notes that it converges more slowly than traditional algorithms. I like the new shiny shiny too, but I've been in this field too long to chase all the new stuff that comes along (and I do love me some Hinton too). I thought about FF for this application but didn't see anything that would make it work in this context, is there anything in particular that you were seeing that would benefit us in this particular usecase?