3 ms·

It seems like that is undesirable. Surely you get compounding errors and a set of weights that skew towards weird. I mean, just feed English to Japanese in a t

It seems like that is undesirable. Surely you get compounding errors and a set of weights that skew towards weird. I mean, just feed English to Japanese in a translator and back again, or encode a jpeg 100 times. What stops them suffering the same entropy issues? Is there a reinforcement step with human feedback?

Xmd5a 2y ago
>What stops them suffering the same entropy issues? Is there a reinforcement step with human feedback? It seems the situation changed with inference scaling/reasoning models (o1 and beyond). I found the tweet I was looking for, but it wasn't from an OpenAI employee as I seemed to recall: https://x.com/kimmonismus/status/1879961110507581839 https://x.com/kimmonismus/status/1879961110507581839 >>Gwern: we have reached the threshold of “recursively self-improving”, “where o4 or o5 will be able to automate AI R&D and finish off the rest.” >Now that we have received various cryptic messages from renowned OpenAI scientists, “Gwern” assumes the following: >- we have reached the threshold of “recursively self-improving”, “where o4 or o5 will be able to automate AI R&D and finish off the rest.” >- the purpose of o1 is primarily to generate synthetic data for models like o3, which is why he is surprised that o1-pro was released at all >- he thinks that Anthropic Opus 3.5 is not released for the same reason, the compute is needed to generate synthetic data The rest of the tweet is interesting too, it starts with the same facts as this article, but makes the point for "reverse distillation" instead (i.e. using synthetic data to train the next model). In particular: >If you look at the relevant scaling curves - may I yet again recommend reading Jones 2021?* - the reason for this becomes obvious. Inference-time search is a stimulant drug that juices your score immediately, but asymptotes hard. Quickly, you have to use a smarter model to improve the search itself, instead of doing more. gwern.net https://news.ycombinator.com/from?site=gwern.net https://news.ycombinator.com/from?site=gwern.net
- sirsinsalot 2y ago
  Yeah that doesn't address the point. The output of an LLM is a compression. It has errors. Recursive training would seem to create iterations that become more noisy. There's no new information, just a lossy distillation of the previous iteration. I'm not sure what stops that.
  - Xmd5a 2y ago
    I'm not into ML so I can't answer your question specifically, but it seems your circular google translate or jpeg comparisons are missing the essential element: self-evaluation https://x.com/search?q=synthetic%20data%20self-improvement https://x.com/search?q=synthetic%20data%20self-improvement
    - sirsinsalot 2y ago
      You're not into ML, got it.

Xmd5a 2y ago

>What stops them suffering the same entropy issues? Is there a reinforcement step with human feedback? It seems the situation changed with inference scaling/reasoning models (o1 and beyond). I found the tweet I was looking for, but it wasn't from an OpenAI employee as I seemed to recall: https://x.com/kimmonismus/status/1879961110507581839 https://x.com/kimmonismus/status/1879961110507581839 >>Gwern: we have reached the threshold of “recursively self-improving”, “where o4 or o5 will be able to automate AI R&D and finish off the rest.” >Now that we have received various cryptic messages from renowned OpenAI scientists, “Gwern” assumes the following: >- we have reached the threshold of “recursively self-improving”, “where o4 or o5 will be able to automate AI R&D and finish off the rest.” >- the purpose of o1 is primarily to generate synthetic data for models like o3, which is why he is surprised that o1-pro was released at all >- he thinks that Anthropic Opus 3.5 is not released for the same reason, the compute is needed to generate synthetic data The rest of the tweet is interesting too, it starts with the same facts as this article, but makes the point for "reverse distillation" instead (i.e. using synthetic data to train the next model). In particular: >If you look at the relevant scaling curves - may I yet again recommend reading Jones 2021?* - the reason for this becomes obvious. Inference-time search is a stimulant drug that juices your score immediately, but asymptotes hard. Quickly, you have to use a smarter model to improve the search itself, instead of doing more. gwern.net https://news.ycombinator.com/from?site=gwern.net https://news.ycombinator.com/from?site=gwern.net

sirsinsalot 2y ago
Yeah that doesn't address the point. The output of an LLM is a compression. It has errors. Recursive training would seem to create iterations that become more noisy. There's no new information, just a lossy distillation of the previous iteration. I'm not sure what stops that.
- Xmd5a 2y ago
  I'm not into ML so I can't answer your question specifically, but it seems your circular google translate or jpeg comparisons are missing the essential element: self-evaluation https://x.com/search?q=synthetic%20data%20self-improvement https://x.com/search?q=synthetic%20data%20self-improvement
  - sirsinsalot 2y ago
    You're not into ML, got it.