3 ms·

My definition is that I can be much less precise with AI the more intelligent it is. It can extract the intent from my fuzzy description of the problem. Which m

by f6v 2mo ago

My definition is that I can be much less precise with AI the more intelligent it is. It can extract the intent from my fuzzy description of the problem. Which means I can offload some of the thinking effort. It wasn't possible a couple years ago. I used to make fun of people who were trying to get ChatGPT to think about the problem when all it could do was write code from the pseudocode you provide. But now I can say: "Look at the latest log and make a plan to fix". And it takes it from there.

eru 2mo ago
Sounds like a good working definition in the context you are using it in. > But now I can say: "Look at the latest log and make a plan to fix". And it takes it from there. I usually tell the agents to first work on reliably reproducing the problem in the log, and only then even start thinking about a fix.
versteegen 2mo ago
I think this is the best and most useful way to measure model intelligence. In my experience it's what really sets apart the capable models from the best. A small model can be RL trained to be extremely good at programming or narrow problem solving for its size (eg 5.6 Luna, DS4 Flash, Qwen 3.6 27B), but even Luna is IME comparatively awful at understanding intent and making good decisions with limited guidance.
okamiueru 2mo ago
I'm not sure if you are aware as to the extent certain processes and functions are being anthropomorphized. These systems are not "intelligent" if you follow the dictionary definition. Hence the question posed to get a better understanding of how it is being used in this context. They also do not "extract intent". There is for sure some intent behind your input to the service. What follows is a predictive text that uses your input, together with a LLM trained on a corpus with similar relations, that ultimately gives you a series of words. That isn't to say a service like this cannot be useful. But I'm often wondering if the people who rely on these, and are particularly enthused by them, are actually aware that the terms they used are in fact anthropomorphized. I start by giving the benefit of the doubt, but it rarely lasts. 'Reasoning', 'agent', 'skill' 'hallucinate', 'know', 'think', 'train', 'learn', 'understand', 'harness', 'attention', 'context', 'prompt'.

by f6v 2mo ago

eru 2mo ago

Sounds like a good working definition in the context you are using it in. > But now I can say: "Look at the latest log and make a plan to fix". And it takes it from there. I usually tell the agents to first work on reliably reproducing the problem in the log, and only then even start thinking about a fix.

versteegen 2mo ago

I think this is the best and most useful way to measure model intelligence. In my experience it's what really sets apart the capable models from the best. A small model can be RL trained to be extremely good at programming or narrow problem solving for its size (eg 5.6 Luna, DS4 Flash, Qwen 3.6 27B), but even Luna is IME comparatively awful at understanding intent and making good decisions with limited guidance.

okamiueru 2mo ago

I'm not sure if you are aware as to the extent certain processes and functions are being anthropomorphized. These systems are not "intelligent" if you follow the dictionary definition. Hence the question posed to get a better understanding of how it is being used in this context. They also do not "extract intent". There is for sure some intent behind your input to the service. What follows is a predictive text that uses your input, together with a LLM trained on a corpus with similar relations, that ultimately gives you a series of words. That isn't to say a service like this cannot be useful. But I'm often wondering if the people who rely on these, and are particularly enthused by them, are actually aware that the terms they used are in fact anthropomorphized. I start by giving the benefit of the doubt, but it rarely lasts. 'Reasoning', 'agent', 'skill' 'hallucinate', 'know', 'think', 'train', 'learn', 'understand', 'harness', 'attention', 'context', 'prompt'.