A lot of rambling/yapping for sure. Context Window, parameters, whatever, one current fact remains: If it wasn’t present in one form or another in pretraining,

A lot of rambling/yapping for sure. Context Window, parameters, whatever, one current fact remains: If it wasn’t present in one form or another in pretraining, it goes off the rails real quick, yet people constantly keep overestimating the capabilities. I use LLMs daily but I’d really appreciate it if people would stop cryptobro-ing it up to AGI, replacing X, and so on. The phrase “the technology is only going to improve” is the worst offender, there is so much tech used in the past that was simpler, better produced and more productive. Yeah, this is AI we’re talking about, but does model collapse exist or not? What happens today if your synthetic dataset has a ton of slop and benchmark data included? What’s the result? I wish for some genuine developments instead of blind faith.