5 ms·
Why don't machine learning research agents overfit?
- dominotw 1mo ago> Machine learning, at its core, is about generalization, not memorization. Well they memorize the patterns. memorization doesnt mean rote learning.
- porridgeraisin 1mo agoThat's a bit pedantic no. Memorization in ML refers to the model having the wrong level of capacity such that it's too hard to optimise it such that it doesn't memorize the _training examples_ themselves.
- tomrod 1mo agoI don't take issue with that. Attempting memorized pattern generalization through holdout / validation strategies is a big part of ML that you would not typically see with econometrics / psychometrics / possibly sabermetrics / most other -metrics. Philosophically the explain versus predict divide. https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf https://www.stat.berkeley.edu/~aldous/157/Papers/shmueli.pdf
- red75prime 1mo agoDoes '110001111000000011111111111' contain n>1∧∀d(d|n→(d=1∨d=n)) as well as infinite number of other generalizations?
- demibabs 1mo agoEven tech giants are putting out articles seemingly fully written by Claude.
- ks2048 1mo agoThe animated graphic labeled "Occam's razor, formalized" is bizarre. Is that really visualizing "Occam's razor, formalized"?
- mrbungie 1mo agoAh, over-the-top larger-than-life LLM-isms, they are really funny when you see them in a company blog, but they are vomitive when it's your coworker copy-pasting it and insisting you on reading it.
- smashah 1mo agoI was expecting Occam wearing a suit.
- e12e 1mo agoAnimated SVG of a pelican shaving with Occam's razor?
- serial_dev 1mo agoTime to first detected slop in this article is <1s. Claudisms per paragraph is also very high. Is it too much to ask from people to read their own article anymore? If anyone read this at all, they would have had the ick, and would have fired off a prompt to get rid of the most popular AI slop tells...
- cj 1mo agoWhat I really dislike is having to edit my own non-LLM assisted writing to make sure I'm not accidentally confused with AI. I caught myself writing "And that matters because..." in a HN comment but had to edit myself. Also miss uising emdashes.
- bee_rider 1mo agoThese models are trained on human language, which belongs to us, we shouldn’t surrender it to them. Keep the em-dashes. IMO don’t overuse negative parallelisms though, they were always bad and lazy.
- srean 1mo agoAn arms race on style would be interesting. Essentially a real life GAN.
- elzbardico 1mo agoFor the first time ever in my life, I consider the fact that English is not my first language and that I never had any formal education in it as an advantage. Nobody is going to consider my weird and barbarian prose with the well polished product of a SOTA model.
- cj 1mo agoEnglish as a 2nd language speakers often have better grammar than natives in my experience! I wouldn't be surprised if the average native english speaker doesn't know the different between there and their.
- percentcer 1mo agoNobody wants to work anymore!
- exit 1mo ago[dead]
- dguest 1mo agoarXiv link: https://arxiv.org/abs/2606.11045 https://arxiv.org/abs/2606.11045
- gwern 1mo agoThe methodology is partially based on https://www.offconvex.org/2021/04/07/ripvanwinkle/ https://www.offconvex.org/2021/04/07/ripvanwinkle/ , for those thinking this sounded familiar.
- novaapi 1mo ago[flagged]
- diddid 1mo agoI always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
- bnmik2 1mo agoMarcus Hutter formalized this in his AIXI work.
- gowld 1mo agoThat's not true. It's pretty clear that she meant "do something you knew they were going to say no to and now you are trying to get away with something." https://youtu.be/wHdHCoeUbU4?t=861s https://youtu.be/wHdHCoeUbU4?t=861s > So I want to tell something to all the young people here on many many occasions you'll find it is much easier to apologize than it is to get permission. You do it then when somebody comes after you and say are you supposed to do that, "oh gee I didn't know I wasn't supposed to do that" ... so just remember it's frequently much easier to apologize than it is to get permission do it She goes on further, explaining how to deceive your superiors to manipulate them to get what you want.
- neutronicus 1mo agoDamn no wonder she got a supercomputer named after her
- diddid 1mo agoBut I still don’t think that means eat all the cookies in the cookie jar and then apologize after because nobody would have given permission. That’s still about doing what you believe to be right. She even frames the fallout as “where you supposed to do that?” and not “you shouldn’t have done that”.
- fragmede 1mo ago
- nyeah 1mo agoThey tend not to overfit ... when there are way more data points than parameters.
- wmedrano 1mo agoOr when there are way more parameters than data points https://en.wikipedia.org/wiki/Double_descent https://en.wikipedia.org/wiki/Double_descent
- deleted 1mo ago[deleted]
- 32df179 1mo agoWherein Claude gives an honest assessment that it genuinely does not overfit. I also had Grok telling me that it isn't quantized. Do the submitters really not notice that this is AI slop? Do they like this? It is a complete pain to read.
- vatsachak 1mo agoNo point in reading anything AI related anymore. It's all slop. We need to retvrn to rss feeds
- exit 1mo agowhat would returning to rss feeds achieve?
- nyeah 1mo agoForce the clankers to support rss!
- fwip 1mo agoThe pattern that RSS feeds best supports is subscribing to individual creators who produce things you value. Presumably, if you don't like AI word-vomit, you wouldn't subscribe to a blog that uses it heavily. This is in contrast to the social-media state of affairs (like HN, reddit, twitter, etc), wherein much of the posts you read are recommended to you by either a person or an algorithm, regardless of whether you believe that author will write something you like. A shorter way of writing this is that RSS promotes writing stuff good enough to get into somebody's "favorite authors" list, and social media instead incentivizes "lots of people will share this."
- jimmaswell 1mo agoIs there a study demonstrating that AI-assisted articles are less substantive or is it purely a kneejerk bias towards humanslop?
- signalbright 1mo ago> Why don't machine learning research agents overfit? they do.
- sigbottle 1mo agoCompression in this modern day and age is so slop. Yes, I'm familiar with keystone results such as Solomonoff induction. It's a direct counterexample to compression - your intensional algorithm can completely outrun reality. I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them, and it's an optimal compressor. It's completely vacuous though. You can always hide the "heavy work" in your mappings and descriptions. It's ironic that a kolomogorov complexity minimizer is so loaded that it's vacuous. This is pretty much why I roll my eyes at this point at all the compression is intelligence memes. I wonder when intervention and causality will hit the mainstream. These tools were designed specifically to counteract purely predictive theories. But your average compression dude will hold tight to their paradigms and slogans, not realize their internal contradictions (that their own field has brought up), and then whenever a new paradigm suddenly becomes visible and mainstream, they'll latch onto that. It's not principled at all. And to be clear - I do think intelligence is some amount of compression, and I am well aware of formal results such as the arithmetic decoding theoretical and empricial result. Just annoyed. It's literally no different than the whole Bayesianism meme. If you're not actually practicing that type of intelligence as a basis, then you don't get to go around beating the drum about how it's the ultimate reality. You're just spouting dogma to feel like part of an in-group.
- srean 1mo ago> I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them Then why don't you ? and did you mean all or those that halt ? I presume you have a way of separating those.
- Xcelerate 1mo agoI’m guessing your downvotes were for tone? You’re correct though regarding Solomonoff induction, as the choice of reference universal partial recursive function gives drastically different results for predictions based on finite data (even with access to a halting oracle). Asymptotically, any choice eventually converges to the same predictions, but that’s no help when there are infinitely many choices for U and no obvious natural prior over universal functions. And I don’t find the argument that our natural environment “implements some choice of U” particularly convincing. There’s definitely an open mystery there.
- jsrozner 1mo agoWhy is this being published as a blog post and not as a peer-reviewed submission? If it's going to be a blog post, why isn't there a corresponding scientific version for me to look at? Someone else already found it. I don't understand why the link isn't in the blog post. https://arxiv.org/abs/2606.11045 https://arxiv.org/abs/2606.11045 Use of claude for writing it should be disclosed.
- deleted 1mo ago[deleted]
- ubutler 1mo agoIf anything, the latest generation of AI models, Astra and Fable, are prime example of overfitting—whereas benchmarks suggest they’re AGI-tier, users (including myself) report the same old gaslighting, hallucination, context rot, cheating, incomprehensibility patterns as with prior models, sometimes even more pronounced. Fable and Opus 5, I suspect, will become textbook examples of RL collapse.
- jayle 1mo ago[dead]
- paidx 1mo ago[flagged]
- llflw 1mo agoDo you know what the scaling law actually is? Overfit everything as much as you can.
- nandanadileep29 1mo agoWhat fits (into few tokens) doesn't overfit.