5 ms·

Coming soon: PhD-level super-agents

17 points by geox 2y ago

braunjohnson 2y ago
[dead]
elashri 2y ago
Unless there is a significant increase in the effective context window in LLMs. Pursuing the goal of having agents working on complex goals is not going to work well. All the tricks and hacks trying to work around this problem is not going to fundamentally change that. LLM agents will lose track of what they are trying to do after couple of trials. That's something that would differentiate human PhD is that while not fast or always creative, they have better attention memory span.
- xnx 2y ago
  What's would be a significant increase to you? Gemini does 2 million tokens and Google just released a research paper to go beyond that.
- OutOfHere 2y ago
  https://github.com/MiniMax-AI/MiniMax-01 https://github.com/MiniMax-AI/MiniMax-01 is an open model that claims a 4 million context. Note however that longer context makes evaluation expensive as you are paying for every token. Still, it is true that OpenAI seriously needs a better solution for it.
  - AstralStorm 2y ago
    How many tokens in a Bible? A typical technical book? Thing will lose context before it has finished reading...
    - OutOfHere 2y ago
      I think it's time to partition the context into L1, L2, and L3 contexts. L1 is the current context with a quadratic memory requirement. L2 is based on fancy mechanisms such as what is used by Gemini and MiniMax-01, having a sub-quadratic to linear memory requirement. L3 is based on document and chunk embeddings having a linear to logarithmic memory requirement. LLMs don't use this approach, but I think it might make sense. As for how this partitioning would work at the neural layers, that remains to be determined.
      - OutOfHere 2y ago
        Titans [1] goes sort of a bit in this direction. [1] https://arxiv.org/abs/2501.00663 https://arxiv.org/abs/2501.00663
- mountainriver 2y ago
  Reasoning is probably enough
checker659 2y ago
Ah, so anybody will be able to create rockets to go to Mars or design GPUs or set up semiconductor foundries?
- JTyQZSnP3cQGa8B 2y ago
  No, but you’ll get PhD-level ReactJS Hello World templates while people will be replaced by those agents. My vision of the future is bleak but I have yet to be proven wrong.
prettyStandard 2y ago
I'm easily in the top 95-99th percentile of people I know and probably most people in general in most respects. Net worth, IQ, height, athletic achievements... A good 10-20 years ago I anticipated these AI moments, and even some of the specifics like our use of ANNs(when everyone thought they were a failure), and yet I failed to capitalize on it. It's hard to describe just how defeated and irrelevant these news articles make me feel. I don't say this to brag or anything, I keep this account pseudo anonymous. Some follow up points and questions - What hope do the normies have? - Does anyone expect our representatives to do anything? - How does this not end badly?
- qrsjutsu 2y ago
  You are not bragging. You are honest. We are just as disappointed about your performance as you are. Nice height, though. And the size of your hands ... brrr And you did not mention any actual achievement, IQy, which was enlightening enough.
  - prettyStandard 2y ago
    Thanks for the laugh
    - qrsjutsu 2y ago
      right back at ya. all that caring about Normies
arresin 2y ago
I’m a bit burned out atm from the hype headlines that have all turned out to be BS in some way or another.
- qrsjutsu 2y ago
  I'm amused by all of it. Especially the way the companies make LLMs appear more human. It will be the same with "PhD level" ... utterly useless in terms of evolution or cognitive performance. Efficient and fast scripts that can take any request and data and give a precise result is all super-agents are good at. And that's a lot. And we are happy. The rest is just pretty standard PR & content media for another 10 years.
leymed 2y ago
Hoping this forces universities to change their archaic PhD system, to adapt something similar to european system. Then people who really have passion for a field and want to have normal job do their research, without working for labs as a cheap labor.
deleted 2y ago
[deleted]
actsasbuffoon 2y ago
I’ve spent the day working in C++ with a popular, well known framework. Both ChatGPT and Claude have been horrible. They hallucinate APIs, they remove things that were already working for no apparent reason, they severely struggle with basic logic. Even a big leap in model quality would still be dog shit at these kinds of tasks. I am vastly better at programming than even the best LLMs out there right now. The only real advantages they have over me are that they type a lot more quickly than I do, and their hourly rate is lower. I’m not saying this is complete BS. I confess I feel a twinge of panic every time I see a headline like this. But years of working with these models and seeing how terrible they are for many tasks makes me skeptical that we’re suddenly about to get AGI.

17 points by geox 2y ago

[dead]

Unless there is a significant increase in the effective context window in LLMs. Pursuing the goal of having agents working on complex goals is not going to work well. All the tricks and hacks trying to work around this problem is not going to fundamentally change that. LLM agents will lose track of what they are trying to do after couple of trials. That's something that would differentiate human PhD is that while not fast or always creative, they have better attention memory span.

xnx 2y ago
What's would be a significant increase to you? Gemini does 2 million tokens and Google just released a research paper to go beyond that.
OutOfHere 2y ago
https://github.com/MiniMax-AI/MiniMax-01 https://github.com/MiniMax-AI/MiniMax-01 is an open model that claims a 4 million context. Note however that longer context makes evaluation expensive as you are paying for every token. Still, it is true that OpenAI seriously needs a better solution for it.
- AstralStorm 2y ago
  How many tokens in a Bible? A typical technical book? Thing will lose context before it has finished reading...
  - OutOfHere 2y ago
    I think it's time to partition the context into L1, L2, and L3 contexts. L1 is the current context with a quadratic memory requirement. L2 is based on fancy mechanisms such as what is used by Gemini and MiniMax-01, having a sub-quadratic to linear memory requirement. L3 is based on document and chunk embeddings having a linear to logarithmic memory requirement. LLMs don't use this approach, but I think it might make sense. As for how this partitioning would work at the neural layers, that remains to be determined.
    - OutOfHere 2y ago
      Titans [1] goes sort of a bit in this direction. [1] https://arxiv.org/abs/2501.00663 https://arxiv.org/abs/2501.00663
mountainriver 2y ago
Reasoning is probably enough

checker659 2y ago

Ah, so anybody will be able to create rockets to go to Mars or design GPUs or set up semiconductor foundries?

JTyQZSnP3cQGa8B 2y ago
No, but you’ll get PhD-level ReactJS Hello World templates while people will be replaced by those agents. My vision of the future is bleak but I have yet to be proven wrong.

prettyStandard 2y ago

I'm easily in the top 95-99th percentile of people I know and probably most people in general in most respects. Net worth, IQ, height, athletic achievements... A good 10-20 years ago I anticipated these AI moments, and even some of the specifics like our use of ANNs(when everyone thought they were a failure), and yet I failed to capitalize on it. It's hard to describe just how defeated and irrelevant these news articles make me feel. I don't say this to brag or anything, I keep this account pseudo anonymous. Some follow up points and questions - What hope do the normies have? - Does anyone expect our representatives to do anything? - How does this not end badly?

qrsjutsu 2y ago
You are not bragging. You are honest. We are just as disappointed about your performance as you are. Nice height, though. And the size of your hands ... brrr And you did not mention any actual achievement, IQy, which was enlightening enough.
- prettyStandard 2y ago
  Thanks for the laugh
  - qrsjutsu 2y ago
    right back at ya. all that caring about Normies

arresin 2y ago

I’m a bit burned out atm from the hype headlines that have all turned out to be BS in some way or another.

qrsjutsu 2y ago
I'm amused by all of it. Especially the way the companies make LLMs appear more human. It will be the same with "PhD level" ... utterly useless in terms of evolution or cognitive performance. Efficient and fast scripts that can take any request and data and give a precise result is all super-agents are good at. And that's a lot. And we are happy. The rest is just pretty standard PR & content media for another 10 years.

leymed 2y ago

Hoping this forces universities to change their archaic PhD system, to adapt something similar to european system. Then people who really have passion for a field and want to have normal job do their research, without working for labs as a cheap labor.

deleted 2y ago

[deleted]

actsasbuffoon 2y ago

I’ve spent the day working in C++ with a popular, well known framework. Both ChatGPT and Claude have been horrible. They hallucinate APIs, they remove things that were already working for no apparent reason, they severely struggle with basic logic. Even a big leap in model quality would still be dog shit at these kinds of tasks. I am vastly better at programming than even the best LLMs out there right now. The only real advantages they have over me are that they type a lot more quickly than I do, and their hourly rate is lower. I’m not saying this is complete BS. I confess I feel a twinge of panic every time I see a headline like this. But years of working with these models and seeing how terrible they are for many tasks makes me skeptical that we’re suddenly about to get AGI.