3 ms·

Very research-y, but I am working on a pet idea of mine called "compact models" in Machine Learning. Essentially you take a model - say a decision tree of depth

by abhgh 7y ago

Very research-y, but I am working on a pet idea of mine called "compact models" in Machine Learning. Essentially you take a model - say a decision tree of depth 20 - and "compact" it to a decision tree of size 10, without losing accuracy. The motivation is "interpretability": models are interpretable by humans if they are small. Another application would be low memory footprint models - but frankly, I haven't thought of this use case much. I have had some success with producing a model agnostic algorithm [1] (i.e. works for any model with some notion of size e.g. decision trees with size being the depth, linear models with size being the number of non-zero terms) - now I am exploring options to speed up the compaction process. The quarantine has given me some time to think about this. [1] https://www.frontiersin.org/articles/10.3389/frai.2020.00003/pdf https://www.frontiersin.org/articles/10.3389/frai.2020.00003...

stfwn 7y ago
You might be interested in pruning neural networks. Some networks can be reduced in size by over 90% without performance loss. The lottery ticket hypothesis paper is good place to start, if you don’t already know about it of course. https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 Some pruning methods have recently been added in the latest feature release of PyTorch.
- abhgh 7y ago
  I am! And the lottery ticket (LT) hypothesis is something I have thought about quite a bit. I believe the LT hypothesis and our work are related in a subtle way: in the former we build out the larger structure first and then prune, while in our approach we conservatively construct the "winning lottery" so to speak. Another difference with the LT hypothesis is that the pruning there is very specific to neural networks. Thanks!
  - stfwn 7y ago
    Pruning in general is about trimming larger structures down. This is nice for using a network, but indeed doesn’t help with training it. Lottery tickets on the other hand are configurations of weights that rival the full network in performance when both are trained from scratch, with the ticket having far fewer parameters to optimize. Pruning is only the method to find the tickets for now, but the ideal would be to have a weight init strategy that can create winning tickets in one-shot. There is work being done on this front, and also to see if winning tickets for one image classification task generalize to image classification as a whole. This would do a lot to reduce the size of networks from the start, and so far results are promising. But yes, this is all very specific to neural networks. Do you have a blog or other place where you post about some of the things you’re working on? It would be nice to read about.
    - abhgh 7y ago
      I haven't had time to write a blog post about this. The paper linked to in the first comment - which is [1] - present the first cut of our ideas. We have made two additional extensions beyond that, we will be submitting them to journals soon. I am yet to release code publicly but that would take a couple of months, given the papers are my priority now - downsides of working on a PhD while having a day job :-| To be clear I am not claiming that compact models are related to the LT hypothesis, just that I think they are and this, of course, needs to be rigorously established. For a while I would be spending time on the compact model arc (my primary interest), till I can get to investigating its connections with the LT hypothesis. In fact [1] doesn't talk about the LT hypothesis at all, instead focusing solely on compacting models in a model-agnostic manner. [1] (PDF) https://www.frontiersin.org/articles/10.3389/frai.2020.00003/pdf https://www.frontiersin.org/articles/10.3389/frai.2020.00003...

by abhgh 7y ago

stfwn 7y ago

You might be interested in pruning neural networks. Some networks can be reduced in size by over 90% without performance loss. The lottery ticket hypothesis paper is good place to start, if you don’t already know about it of course. https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635 Some pruning methods have recently been added in the latest feature release of PyTorch.

abhgh 7y ago
I am! And the lottery ticket (LT) hypothesis is something I have thought about quite a bit. I believe the LT hypothesis and our work are related in a subtle way: in the former we build out the larger structure first and then prune, while in our approach we conservatively construct the "winning lottery" so to speak. Another difference with the LT hypothesis is that the pruning there is very specific to neural networks. Thanks!
- stfwn 7y ago
  Pruning in general is about trimming larger structures down. This is nice for using a network, but indeed doesn’t help with training it. Lottery tickets on the other hand are configurations of weights that rival the full network in performance when both are trained from scratch, with the ticket having far fewer parameters to optimize. Pruning is only the method to find the tickets for now, but the ideal would be to have a weight init strategy that can create winning tickets in one-shot. There is work being done on this front, and also to see if winning tickets for one image classification task generalize to image classification as a whole. This would do a lot to reduce the size of networks from the start, and so far results are promising. But yes, this is all very specific to neural networks. Do you have a blog or other place where you post about some of the things you’re working on? It would be nice to read about.
  - abhgh 7y ago
    I haven't had time to write a blog post about this. The paper linked to in the first comment - which is [1] - present the first cut of our ideas. We have made two additional extensions beyond that, we will be submitting them to journals soon. I am yet to release code publicly but that would take a couple of months, given the papers are my priority now - downsides of working on a PhD while having a day job :-| To be clear I am not claiming that compact models are related to the LT hypothesis, just that I think they are and this, of course, needs to be rigorously established. For a while I would be spending time on the compact model arc (my primary interest), till I can get to investigating its connections with the LT hypothesis. In fact [1] doesn't talk about the LT hypothesis at all, instead focusing solely on compacting models in a model-agnostic manner. [1] (PDF) https://www.frontiersin.org/articles/10.3389/frai.2020.00003/pdf https://www.frontiersin.org/articles/10.3389/frai.2020.00003...