4 ms·
> still think it’s noteworthy that a model running on my 2.5 year old laptop (a 64GB MacBook Pro M2) is able to produce code like this—especially code that wor
by bob1029 1y ago
> still think it’s noteworthy that a model running on my 2.5 year old laptop (a 64GB MacBook Pro M2) is able to produce code like this—especially code that worked first time with no further edits needed.
I believe we are vastly underestimating what our existing hardware is capable of in this space. I worry that narratives like the bitter lesson and the efficient compute frontier are pushing a lot of brilliant minds away from investigating revolutionary approaches.
It is obvious that the current models are deeply inefficient when you consider how much you can decimate the precision of the weights post-training and still have pelicans on bicycles, etc.
- jonas21 1y agoWasn't the bitter lesson about training on large amounts of data? The model that he's using was still trained on a massive corpus (22T tokens).
- yahoozoo 1y agoWhat does that have to do with quantizing?
- itsalotoffun 1y agoI think GP means that if you internalize the bitter lesson (more data more compute wins), you stop imagining how to squeeze SOTA minus 1 performance out of constrained compute environments.
- reactordev 1y agoThis. When we ran out of speed on the CPU, we moved to the GPU. Same thing here. The more we work with (22T) models, quants, and decimating precision - the more we learn and find more novel ways to do things.
- Breza 1y agoVery well put. There's a lot to be gained from using smaller models and existing hardware. So many enterprise PMs skip straight to using a cutting edge LLM via API. There are many tasks where a self-hosted LLM or even a finetuned small language model can either complete a preliminary step or even handle the full task for much less money. And if a self-hosted model can do the job today, imagine what you'll be able to do in a year or five when you have more powerful hardware and even better models.