6 ms·
Winter is coming, again.
by marvel_boy 6y ago
Winter is coming, again.
- jmortin 6y agoYeah, if I got a penny everytime I heard that one...
- WanderPanda 6y agoJust a quick belief of myself (might change tomorrow): ML winter will not come until we hit the flattening of the specialized hardware s-curve. I know people believe that ML can still scale in funding (for even bigger models than GPT-3 with current hardware) but that shouldn’t be our hope. I also can not imagine that the exponential efficiency increase in architectures with optimized structure can continue for a couple more years (happy to be proven wrong here). Also the scaling in dataset sizes will ultimately halt (or might be already) for supervised learning which might give a bump for reinforcement learning where there is usually unlimited data (due to simulation)
- mlthoughts2018 6y agoMost commercial use of ML, including neural nets, is small data application-specific business logic. Think phishing / spam / fraud detection, anomaly detection, semantic search, image search, keyword labeling, classifying or segmenting customer data. These applications have well-understood business value propositions. Much, much less often the ML application focused on truly large data. I think we might see a winter in super big applications like self-driving cars or voice assistants, but ML in general is just a boring, non-controversial business tool with hundreds of valuable applications. You’ll still need statistical specialists to train and operate models and ensure systems avoid pitfalls like overfitting, poor convergence, multicollinearity, confounders, etc. So I doubt this will have much impact on ML job market. Companies that invest in ML will continue to run circles around companies that don’t. You’ll just see the unjustified over-focus on SOTA neural networks die down and become just another boring tool in the toolbox like everything else in ML.
- WanderPanda 6y agoI agree, or at least I hope you are right. I have some doubts about investment into ML without the hype (at least for legacy big corps) but probably their hype-driven ML efforts are/were misguided by poor incentives. But I agree again that companies that invest in ML from a grassroots/first principles way should run circles around the legacy ones. I think the „boring“ part you mention might be what I refer to as ML/AI winter. But then again do you think there is much space left in the ML toolbox apart from neural networks? I think at least for e.g. computer vision we can agree that neural nets ate all the cake, right?
- superbcarrot 6y agoThat's unlikely. There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. The worst case for the field is that progress slows down, people realise that their expectations were unrealistic and the hype inevitably dies down. Which has to happen eventually. So even if ML isn't the hottest thing or a massively growing field, it will still be used for a long time.
- otabdeveloper4 6y ago> There are many machine learning applications that have been shown to be good enough for commercial use and they aren't going anywhere. If you could name three of them I'd be really grateful. Serious question; everything surrounding ML seems to be only good for (non-monetizable) art projects. As art it is amazing, not going to lie, but "commercial use" seems like a huge stretch.
- Tenoke 6y agoHere's 3 off the top of my head, but there's more especially when you get into less flashy territory. Translation (Google translate, DeepL) Automatically generated product descriptions, sometimes also edited by humans (Alibaba) Image Tagging (Facebook photos)
- otabdeveloper4 6y agoOkay, I should have worded my comment more carefully. These applications seem to firmly fall into the "I'm willing to compromise on quality if I don't have to pay a living person a wage" niche, so they're value-destroying, not value-creating. Are there examples of value-creating applications for ML? (From a business point of view; obviously the "shitty translations but at no cost" proposition creates value for the average Internet user.)
- Tenoke 6y agoI don't understand this. Do you think that e.g. the average engineer is value destroying because if the business hires a more expensive and experienced one they will do a better job? In either case it is only 'value-destroying' if the business has unlimited resources.
- 314 6y agoWinter is always coming, the important question is when it will arrive. Current ML research is still destroying new problems with ease so the current velocity is high. It will slow down first, before people start to question why the new crop of problems are too hard, and then the cycle will start again.
- ausbah 6y agoI think there's already a good amount of discussion around where current ML methods. stuff like lack of sample efficiency, adjusting for distribution shift, etc.
- indymike 6y agoNot sure that it's as much stagnation as commoditization.