4 ms·
Jeremy from fast.ai here. Let me know if you have any questions about the methods we used, or anything else relevant to this project.
by jph00 8y ago
Jeremy from fast.ai here. Let me know if you have any questions about the methods we used, or anything else relevant to this project.
- karandwivedi42 8y agoHow important was momentum cycle (The decrease then increase)? Unrelated but it would be great if you could answer: In general, how important is momentum tuning and what are the heuristics for the same?
- karandwivedi42 8y agoHow important was momentum cycle (The decrease then increase)? Unrelated but it would be great if you could answer: In general, how important is momentum tuning and what are the heuristics for the same?
- jph00 8y agoBased on Smith's paper and our experiments, it seems pretty important. But there's still much research to be done to understand momentum tuning and related concepts.
- throw98987 8y agoIs the code used for the competition available?
- jph00 8y agoYes it's linked from the competition web site. It'll be integrated into the fastai library in the coming weeks.
- tegansnyder 8y agoHi Jermey... Great article. I have a labeled product image dataset (thumbnails, and various sizes) that I'd like to use for the model training. Can you recommend any lightweight methods for commodity hardware?
- jph00 8y agoYou can use the approach we used to win the CIFAR 10 training cost section of the competition. If you use fastai/pytorch, then it's ~5 lines of code. Check out lesson 1 of http://course.fast.ai http://course.fast.ai for the basic approach, but when calling `fit()`, add the param `use_clr_beta=(20,20,0.95,0.85)` which will enable 1cycle, and should allow of super convergence. Then train with SGD with a really high learning rate (somewhere from 1-3, generally).
- tegansnyder 8y agoExcellent. I will check it out.
- cs702 8y agoI do have a question, but first I must say that as I was reading your post I found myself nodding in agreement again and again and again. Yes, the Silicon-Valley-centric AI research community (including UK-based DeepMind and its surrounding satellites) tends to overlook or even ignore important research from elsewhere. Yes, "algorithmic creativity" and focused tinkering often prove more important than computational resources. Yes, innovation in deep learning does not require big data nor massive hardware, but "engineers are drawn to using the biggest datasets they can get, on the biggest machines they can access, like moths flitting around a bright light" (so true!). And yes, "genuine advances [have] consistently come from doings things differently, not doing things bigger." Ah-men! With that out of the way, here's my question: Have you guys tried or managed to achieve learning super-convergence with attention or residual attention models?
- jph00 8y agoNo I haven't tried yet. One of our students has started experimenting with language models (AWD LSTM) and getting encouraging results (about 3x speed improvement), but not with attention. I really hope that these results will help encourage more people to both see where and how super-convergence can be achieved. I suspect that we're only scratching the surface of what's possible.
- cs702 8y agoThank you for the prompt answer -- and for your intellectual generosity. I too hope these results encourage more people to see where/how super convergence can be achieved. FWIW, trying to achieve it with fully attentional models is on my "R&D things to try" list at work.
- nutanc 8y agoDoes the solution use pre trained Resnet models?
- jph00 8y agoNo you can't use pre-trained models in this competition.
- nutanc 8y agoThanks for the prompt response. Do you use data augmentation? Sorry for all the questions which could maybe be answered by looking at the code, but I am just getting started with pytorch etc and I am trying to understand the exact input format.
- jph00 8y agoYes data augmentation is critical