6 ms·
Should we really call it "engineering" if its a case of "try random things until one of them works without really knowing why"?
by mangecoeur 3y ago
Should we really call it "engineering" if its a case of "try random things until one of them works without really knowing why"?
- mnky9800n 3y agoThat's called machine learning.
- boyka 3y agoGradient descent type optimization is far from "trying random things until one of them works without really knowing why". You can calculate all partial derivatives and understand the impact.
- deleted 3y ago[deleted]
- varispeed 3y agoSure you can, but you can also throw things in randomly and see what works then build theory around it.
- quickthrower2 3y agoI think it is a freaking miracle it even works. I understand how it works for say 100 parameter linear regression, but that it would work for billions of parameters (billion dimensional space) by nudging the parameters each a little (based on purely it's impact on the loss, assuming everything stays the same), is not obvious to me. It is a kind of magic. Regarding randomness, the initialization of the weights is random, and if they use dropouts that is random too, plus the order in which to process the text might be random.
- rmnclmnt 3y agoWhat’s even crazier is the fact that the underlying numerical math solution was exposed early 1800’s by Gauss!
- bravura 3y agoTo be honest, there’s always been a tension in machine learning between the: “we won’t do it unless the theory is complete and sound” versus the “we don’t understand this but it works much better consistently so we do it” crowd. In the 90s and even 00s, the theory first crowd was mainstream and the empirical first crowd was considered fringe. Very fringe. Personally I appreciated it when LeCun was like: “you can’t find the solution if you only search where the lamplight is shining.” Or other early deep learning practitioners note that ML theory is usually so far disconnected from practice in terms of tightness and bounds that you might as well ignore pure theory completely. Anyway, it wasn’t until deep learning methods really smashed benchmarks across the board did people give in to the black magic / alchemy driven approaches of empiricism based upon intuition and bias developed through long-held experience.
- deleted 3y ago[deleted]
- creata 3y agoTo a layperson like me, the "try random things" part of machine learning doesn't seem to lie in optimizing the parameters, but in designing the model.
- muzani 3y agoLooks more like science here. Run a bunch of experiments and see which does better. But literally the first sentence of the readme is "Prompt engineering is kind of like alchemy."
- bryanrasmussen 3y agoalchemy is kind of like science before a lot of principles were formalized.
- f1shy 3y agoThis comes to mind: http://lambda-the-ultimate.org/node/5335 http://lambda-the-ultimate.org/node/5335
- namaria 3y ago> Run a bunch of experiments and see which does better. That's empiricism. Scientific method implies formulating falsifiable theoretical claims, producing meaningful analysis of experimental conditions, publishing peer-reviewed and reproducible results.
- judge2020 3y agoThat's if you're trying to become a scientist. You can do science even if your experiment doesn't contain any of those elements besides trying multiple things and observing the outcomes.
- namaria 3y agoSure, the same way I can practice medicine by putting leeches on people. I'm not trying to become a physician so it's alright to call that medicine.
- olddustytrail 3y agoWell, not you, but other people could because leeches are still used because they work. You're trying to correct people on a subject you know nothing about.
- classified 3y agoHasn't software "engineering" been like that forever?
- js8 3y agoI think the real etymology of it was "social engineering". Which also means clever hacks, not a real engineering, it deliberately subverted the meaning.
- andrewdb 3y ago"I don't play craps. I'm a dice engineer!" - prompt engineers, satirically