4 ms·
The unreasonable effectiveness of character-level language models (2015)
- osigurdson 4y agoPhrases for the dustbin: - Considered harmful - Unreasonable effectiveness - Root of all evil
- _Microft 4y agoThe title is alluding to an old article called „The Unreasonable Effectiveness of Mathematics in the Natural Sciences“ by physicist Eugene Wigner. https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...
- balder1991 4y agoAnd "considered harmful" was popularized by the Dijkstra’s article "GOTO Statement Considered Harmful".
- ASalazarMX 4y agoMost people recognize the reference, it's a very overused reference, hence PCs complaint.
- WithinReason 4y agoI think at this point everyone knows that.
- PotatoPancakes 4y agoRelevant xkcd: https://xkcd.com/1053/ https://xkcd.com/1053/
- CaptainNegative 4y agoI think at this point everyone's seen this comic.
- Jverse 4y agohttps://xkcd.com/2501/ https://xkcd.com/2501/
- itsboring 4y agoI’d like to add the phrase “solved problem”.
- modernpink 4y ago* is all you need
- osigurdson 4y agoAnother one that is starting to get on my nerves (though not really in the same category as the above) is "first world problem". Person A states they have some kind of problem and person B says "tisk, tisk, first world problems" in order to attenuate whatever problem person A has. If person B is currently living in the 3rd world under 3rd world conditions, fine, otherwise stop cheaply leveraging the poor in order to win arguments.
- the_optimist 4y agoEach representing failures of prose to appropriately characterize multi-dimensional performance into simplistic concepts of good and evil.
- xeonmc 4y agoThe unreasonable effectiveness of considered harmful essays is all you need
- kwertyoowiyop 4y agoWhat we talk about when we talk about…
- creatonez 4y agoI remember being insanely impressed by GPT-2 in 2019. It's remarkable how something just a few years behind the state-of-the-art completely fails to impress anymore.
- petulla 4y agohttps://www.theverge.com/2019/11/7/20953040/openai-text-generation-ai-gpt-2-full-model-release-1-5b-parameters https://www.theverge.com/2019/11/7/20953040/openai-text-gene...
- deleted 4y ago[deleted]
- promiseofbeans 4y agoReading this, all the changes in modern models like gpt-3 seem like pretty incremental improvements - better structured training data (and a lot more of it), using tokens instead of characters, etc.
- loveparade 4y agoThey are. Nobody in the AI research community expected that scaling up models several orders of magnitude would make them that good. Even the original selling point of the Transformer paper was "can be much bigger and faster than RNNs, even if not as good" - Of course everyone was wrong and it's all about the data and scale. Btw, tokens are nothing new in modern models. They've been used since 2015/2016 [0] [0] https://arxiv.org/abs/1508.07909 https://arxiv.org/abs/1508.07909
- williamtrask 4y agoWhich goes to show you one of the best strategies in ML isn’t necessarily thinking about learning theory, it’s thinking about what our fastest computers are most capable of doing. But this does run out of steam (or rather…data) eventually. The largest LLM was ibm’s tangora model back in the 1980s. 8 trillion params. We’ll go through the cycle again with a new paradigm.
- IIAOPSW 4y ago"It just goes to show the best strategy in assembling wood together isn't learning to use a screw driver, its about pounding it in with the hardest and most capable hammer."
- simonh 4y agoYep. Sometimes so. https://www.thesprucecrafts.com/wood-joinery-methods-use-no-fasteners-3536634 https://www.thesprucecrafts.com/wood-joinery-methods-use-no-...
- 4y ago