13 ms·

> Its main purpose is to produce synthetic data for Orion, their next big LLM https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502 "This is gener

by thesz 2y ago

> Its main purpose is to produce synthetic data for Orion, their next big LLM https://arxiv.org/abs/2404.03502 https://arxiv.org/abs/2404.03502 "This is generally useful, but widespread reliance on recursive AI systems could lead to a process we define as "knowledge collapse", and argue this could harm innovation and the richness of human understanding and culture... In our default model, a 20% discount on AI-generated content generates public beliefs 2.3 times further from the truth than when there is no discount."

audessuscest 2y ago
If it's proven to be a real issue, we might expect to see models trained on a lot of synthetic data with less knowledge but highly capable to reason, and other models less capable to reason but with large knowledge.
JumpCrisscross 2y ago
Having one inscrutable AI train another should just thrill the safetyists.
- xg15 2y ago
  Wasn't aware this has already become an -ism...
  - trashtester 2y ago
    Aka "doomers".
aucisson_masque 2y ago
yeah, i'm by no mean an llm engineer but even with the basic knowledge on how it works, i can understand how it's a bad idea to feed a LLM with the data from another LLM. Yeah, you are probably going to sanitize it and it will have less hallucinations but in the same time it's scope will be much more limited. for instance, they speak about product marketing strategies. it requires creativity, which ai is not capable but currently it can still borrow human creativity. With llm data fed to another llm, it's going to be diluted even more and what will be left is extremely standard and common knowledge. Tho it could be interesting for corporation running chatbot, but even here there is always the tiny risk it hallucinates and screw the company big time, which is a deal breaker. No i don't see the benefit.
- trashtester 2y ago
  It becomes less bad if the LLM is learning from something that is not a (pure) LLM, though. Imagine if you let an LLM-like model learn to predict the next move from 1 billion AlphaZero self-play chess games. The next-move prediction it ended up with might represent a much better chess player than games just trained on human online games. And it might ALSO be faster than AlphaZero, meaning it could possibly even beat AlphaZero if time controls were restricted to 1 minute each, or something (for the whole game).
  - skywhopper 2y ago
    Chess has such a tiny scope and tightly defined rules that it’s hilarious to map this idea of generated chess games as training input onto anything resembling real-world concepts. Also, do you have any evidence whatsoever for your bald assertion that an LLM might get faster than and/or defeat AlphaZero? When has such a thing occurred in a simpler problem space?
    - trashtester 2y ago
      A chess move is a single token. An network that would predict the single token without traversing a search space would most likely be faster than a model that needed to do some kind of recursive search. This kind of benefit is what sets AlphaZero apart from earlier engines in the first place. It was better able to evaluate a position just from the pattern on the board, and needed to search a much smaller part of the search space than the older chess engines did. An engine that at a glance would be quite good at predicting what move AlphaZero would do, given enough time, would take this principle to the next level. You see the same in humans like Magnus Carlsen. Even if he doesn't "calculate" a single move (meaning traversing part of the search space in chess lingo), he can beat most good amateurs by just going for the move that looks "obvious" to him. Anyway, the point isn't chess. The point is that the search part of the Alpha family of models (which tend to be specialized) seems to be making its way to multi-modal models. And that this makes them slower and more expensive to run. That's fine for some applications, but since their output is not really just regurgitating the input, their output MAY be more useful as training data for other models than other available training data. Now there is another difference between AlphaZero and traditional LLM's, and that is in the RL-through-self-play. I don't really know if something like Strawberry would also require RL-training to actually outperform the original training data.
trashtester 2y ago
There is a caveat to this. Strawberry/Q* relies on elements similar to an element of AlphaZero to find "strategies" suitable for a problem. That takes it furter from "next-word-prediction" than current models, and improves quality of the output. The downside is that this requires more compute during inference. That makes it too expensive to deploy directly. Still, at least to some extent, this could allow a larger model to achieve similar performance to a Strawberry enhanced GPT-4o by adding more parameters, without the impact on speed and compute cost. Humans often do the same. While we first learn some topic, we often first use conscious reasoning (which has elements of a tree-search) to find a way to solve it. But if we practice enough times, it becomes "muscle memory".
- JumpCrisscross 2y ago
  > Strawberry/Q relies on elements similar to an element of AlphaZero to find "strategies" suitable for a problem. That takes it furter from "next-word-prediction" than current models, and improves quality of the output* Genuine question: is this independently substantiated? Or Altmanspeak?
  - mjburgess 2y ago
    The whole comment is "altmanspeak", which we might more directly call, "ideological descriptions of AI consistent with increasing stock prices"
  - trashtester 2y ago
    The specifics are kept secret, but all the labs appear to work on some variant. It seems to be related to the DeepMind reference in [1] and most of [2]. [1] https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Q-learning [2] https://arxiv.org/pdf/2403.09629 https://arxiv.org/pdf/2403.09629
    - fire_lake 2y ago
      This sounds like chess AI

by thesz 2y ago

audessuscest 2y ago

If it's proven to be a real issue, we might expect to see models trained on a lot of synthetic data with less knowledge but highly capable to reason, and other models less capable to reason but with large knowledge.

JumpCrisscross 2y ago

Having one inscrutable AI train another should just thrill the safetyists.

xg15 2y ago
Wasn't aware this has already become an -ism...
- trashtester 2y ago
  Aka "doomers".

aucisson_masque 2y ago

yeah, i'm by no mean an llm engineer but even with the basic knowledge on how it works, i can understand how it's a bad idea to feed a LLM with the data from another LLM. Yeah, you are probably going to sanitize it and it will have less hallucinations but in the same time it's scope will be much more limited. for instance, they speak about product marketing strategies. it requires creativity, which ai is not capable but currently it can still borrow human creativity. With llm data fed to another llm, it's going to be diluted even more and what will be left is extremely standard and common knowledge. Tho it could be interesting for corporation running chatbot, but even here there is always the tiny risk it hallucinates and screw the company big time, which is a deal breaker. No i don't see the benefit.

trashtester 2y ago
It becomes less bad if the LLM is learning from something that is not a (pure) LLM, though. Imagine if you let an LLM-like model learn to predict the next move from 1 billion AlphaZero self-play chess games. The next-move prediction it ended up with might represent a much better chess player than games just trained on human online games. And it might ALSO be faster than AlphaZero, meaning it could possibly even beat AlphaZero if time controls were restricted to 1 minute each, or something (for the whole game).
- skywhopper 2y ago
  Chess has such a tiny scope and tightly defined rules that it’s hilarious to map this idea of generated chess games as training input onto anything resembling real-world concepts. Also, do you have any evidence whatsoever for your bald assertion that an LLM might get faster than and/or defeat AlphaZero? When has such a thing occurred in a simpler problem space?
  - trashtester 2y ago
    A chess move is a single token. An network that would predict the single token without traversing a search space would most likely be faster than a model that needed to do some kind of recursive search. This kind of benefit is what sets AlphaZero apart from earlier engines in the first place. It was better able to evaluate a position just from the pattern on the board, and needed to search a much smaller part of the search space than the older chess engines did. An engine that at a glance would be quite good at predicting what move AlphaZero would do, given enough time, would take this principle to the next level. You see the same in humans like Magnus Carlsen. Even if he doesn't "calculate" a single move (meaning traversing part of the search space in chess lingo), he can beat most good amateurs by just going for the move that looks "obvious" to him. Anyway, the point isn't chess. The point is that the search part of the Alpha family of models (which tend to be specialized) seems to be making its way to multi-modal models. And that this makes them slower and more expensive to run. That's fine for some applications, but since their output is not really just regurgitating the input, their output MAY be more useful as training data for other models than other available training data. Now there is another difference between AlphaZero and traditional LLM's, and that is in the RL-through-self-play. I don't really know if something like Strawberry would also require RL-training to actually outperform the original training data.

trashtester 2y ago

There is a caveat to this. Strawberry/Q* relies on elements similar to an element of AlphaZero to find "strategies" suitable for a problem. That takes it furter from "next-word-prediction" than current models, and improves quality of the output. The downside is that this requires more compute during inference. That makes it too expensive to deploy directly. Still, at least to some extent, this could allow a larger model to achieve similar performance to a Strawberry enhanced GPT-4o by adding more parameters, without the impact on speed and compute cost. Humans often do the same. While we first learn some topic, we often first use conscious reasoning (which has elements of a tree-search) to find a way to solve it. But if we practice enough times, it becomes "muscle memory".

JumpCrisscross 2y ago
> Strawberry/Q relies on elements similar to an element of AlphaZero to find "strategies" suitable for a problem. That takes it furter from "next-word-prediction" than current models, and improves quality of the output* Genuine question: is this independently substantiated? Or Altmanspeak?
- mjburgess 2y ago
  The whole comment is "altmanspeak", which we might more directly call, "ideological descriptions of AI consistent with increasing stock prices"
- trashtester 2y ago
  The specifics are kept secret, but all the labs appear to work on some variant. It seems to be related to the DeepMind reference in [1] and most of [2]. [1] https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Q-learning [2] https://arxiv.org/pdf/2403.09629 https://arxiv.org/pdf/2403.09629
  - fire_lake 2y ago
    This sounds like chess AI