4 ms·
Very nice work! What I've found with optimizations is that once you have exhausted one avenue that has almost magically caused another one to be opened up which
by jacquesm 4y ago
Very nice work! What I've found with optimizations is that once you have exhausted one avenue that has almost magically caused another one to be opened up which you can then squash and so on. The compounding effect of such serial optimizations can be considerable.
I wonder if converting a model back into code and then optimizing that is a viable path, you could then try to drop out little bits of the code to see if they meaningfully affect the output or not.
- tbalsam 4y agoThanks for the comment! As far as seeing what the core parts are, I'm really partial to the method where you have a graph network with a really high L2 communication penalty which seems to approach the intrinsic dimension of the problem for certain simple problems. How well that scales to larger problems I have no idea (probably not well, but techniques like variational dropout are pretty analogous in their own kinds of ways). Thankfully with a fast-training network you can play/dork around with the numbers a bit and see what's what. One could distill the tiny ResNet into a graph network, which with the right constraints could theoretically accomplish the same as the original neural network, and then compress that as small as possible. There's probably an interesting tradeoff in "maximal compression" and "number of iterate rounds" for said graph network. I recently got enough runs (25) and performance difference to have a p=.0014 result or something like that in half an hour for something I was experimenting with against the baseline recently and it felt so good because I wasn't diddling with 5 runs, which in some cases for certain papers take days to finish. It's just a very satisfying feeling. I guess back to the explainability side of things.... -- that alone I don't think would necessarily provide answers to the explainability problem but I think it would be like a refining-oil type step before diving into the L2-compressed feature representations....
- jacquesm 4y agoInterested in how that will pan out. Unfortunately I don't know enough about the subject matter or I would definitely give it a shot. And if you hack this to the point where it is explainable then that in turn might generalize to larger networks and/or different problems.