3 ms·

I've been at this longer than most. After three major generations of models the "intuition" I've build isn't about what AI can do, but about what a specific mo

by noosphr 9mo ago

I've been at this longer than most. After three major generations of models the "intuition" I've build isn't about what AI can do, but about what a specific model family can do. No one cares what the gotchas in gpt3 are because it's a stupid model. In two years no one will care what they were for gpt5 or Claude 4 for the same reason. We currently have the option of wasting months of our lives to get good at a specific model, or burn millions to try and get those models to do things by themselves. Neither option is viable long term.

CuriouslyC 9mo ago
My philosophy is to try and model the trajectories of these systems and build rigging around where the curve is flat (e.g. models have been producing big balls of mud since the beginning and this hasn't improved meaningfully). Models also have a strong mean bias that I don't expect to go away any time soon. Trying to outsmart the models at core behaviors over time is asking to re-learn the bitter lesson though.
- noosphr 9mo ago
  The issue is that finding where the curve is flat reuiqres either intuition or a lot of money. The contrapositive of the bitter lesson is that any hand crafted system will over any market meanginfil time scale outperform a system using data and compute.
pjbk 9mo ago
It seems that common sense is very difficult to program. Perhaps because we don't really know how to properly define it or how an encoding of it would look like. All of these models keep trying to convince me they can solve the Post Correspondence Problem.
umairnadeem123 9mo ago
[dead]
- noosphr 9mo ago
  Dynamic evals of simple tasks are the only ones that matter, the closer to your target domain the better. If the model can't get simple tasks right it won't be able to get complex tasks right. Finding parse trees given a grammar and a sentence. Paths between vertices in a graph. Asking it to do a search and replace on a document. Anything that you can scale and test against an algorithms answer automatically.

by noosphr 9mo ago

CuriouslyC 9mo ago

My philosophy is to try and model the trajectories of these systems and build rigging around where the curve is flat (e.g. models have been producing big balls of mud since the beginning and this hasn't improved meaningfully). Models also have a strong mean bias that I don't expect to go away any time soon. Trying to outsmart the models at core behaviors over time is asking to re-learn the bitter lesson though.

noosphr 9mo ago
The issue is that finding where the curve is flat reuiqres either intuition or a lot of money. The contrapositive of the bitter lesson is that any hand crafted system will over any market meanginfil time scale outperform a system using data and compute.

pjbk 9mo ago

It seems that common sense is very difficult to program. Perhaps because we don't really know how to properly define it or how an encoding of it would look like. All of these models keep trying to convince me they can solve the Post Correspondence Problem.

umairnadeem123 9mo ago

[dead]

noosphr 9mo ago
Dynamic evals of simple tasks are the only ones that matter, the closer to your target domain the better. If the model can't get simple tasks right it won't be able to get complex tasks right. Finding parse trees given a grammar and a sentence. Paths between vertices in a graph. Asking it to do a search and replace on a document. Anything that you can scale and test against an algorithms answer automatically.