6 ms·

Catch up to what? LLMs have clearly stalled in terms of advancement, they just kind of sort of get a little bit different at this point. Not even better, just d

by gishh 11mo ago

Catch up to what? LLMs have clearly stalled in terms of advancement, they just kind of sort of get a little bit different at this point. Not even better, just different.

bubblelicious 11mo ago
Where does this view come from? I’m not aware of any real evidence for this. Also consider our data center buildouts in 26 and 27 will be absolutely extraordinary, and scaling is only at the beginning. You have a growing flywheel and plenty of synthetic data to break the data wall
- candiddevmike 11mo ago
  We need a fundamental paradigm shift beyond transformers. Throwing more compute or data at it isn't pushing the needle.
  - bubblelicious 11mo ago
    And you don’t think that’s already happening? Also where is your evidence for this?
    - bigyabai 11mo ago
      > Also where is your evidence for this? The fact that "scaling laws" didn't scale? Go open your favorite LLM in a hex editor, oftentimes half the larger tensors are just null bytes.
      - bubblelicious 11mo ago
        Show me a paper, this makes no sense of course scaling laws are scaling
  - marcosdumay 11mo ago
    Just to point, but there's no more data. LLMs would always bottleneck on one of those two, as computing demand grows crazy quickly with the data amount, and data is necessarily limited. Turns out people threw crazy amounts of compute into it, so the we got the other limit.
    - bigyabai 11mo ago
      Synthetic data works.
      - marcyb5st 11mo ago
        There's a limit to that according to: https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y . Basically, if you use an LLM to augment a training dataset it will become "dumber" every subsequent generation and I am not sure how you can generate synthetic data for a language model without using a language model
        yorwba 11mo ago
        Synthetic data doesn't have to come from an LLM. And that paper only showed that if you train on a random sample from an LLM, the resulting second LLM is a worse model of the distribution that the first LLM was trained on. When people construct synthetic data with LLMs, they typically do not just sample at random, but carefully shape the generation process to match the target task better than the original training distribution.
    - Alex2037 11mo ago
      [dead]
    - Mistletoe 11mo ago
      Yeah I’m constantly reminded of a quote about this- you can’t make another internet. LLMs already digested the one we have.
    - bubblelicious 11mo ago
- ModernMech 11mo ago
  Let me put it this way: when ChatGPT tells me I've hit the "Free plan limit for GPT-5", I don't even notice a difference when it goes away or when it comes back. There's no incentive for me to pay them for access to 5 if the downgraded models are just as good. That's a huge problem for them.
  - _aavaa_ 11mo ago
    It is a problem easily solved with advertising.
    - ModernMech 11mo ago
      No, because as the history of hardware scaling shows us, things that run on supercomputers today will run on smartphones tomorrow. Current models already run fairly well on beefy desktop systems. Eventually models the quality of ChatGPT 4 will be open sourced and running on commodity systems. Then what? There's no moat.
      - treis 11mo ago
        10-20 years of your data in the form of chat history Billions of users allowing them to continually refund their models Hell by then your phone might be the OpenAI 1. The world's first AI powered phone (tm)
        overfeed 11mo ago
        > The world's first AI powered phone Do you remember the Facebook phone? Not many people do, because it was a failed project, and that was back when Android was way more open. Every couple of years, a tech company with billions has the brilliant idea: "Why don't we have a mobile platform that we control?", followed by failure. Amazon is the only qualified success in this area.
        treis 11mo ago
        I agree that a slight twist on android doesn't make sense. A phone with a in integrated LLM with apps that are essentially prompts to the LLM might be different enough to gain market share.
- skywhopper 11mo ago
  There is zero evidence that synthetic data will provide any real benefit. All common sense says it can only reinforce and amplify the existing problems with LLMs and other generative “AI”.
  - bubblelicious 11mo ago
    Sounds like someone has no knowledge of the literature, synthetic data isn’t like asking ChatGPT to give you a bunch of fake internet data.
epolanski 11mo ago
I don't see that stall. All of the tools I use get increasingly better every quarter at the very least (coding tools, research, image generation, etc).
- afavour 11mo ago
  As always it’s hype vs reality. Today we’re told AGI is just around the corner. It isn’t. You’re right that a lot of tools are improving iteratively and that’s great but the hyped up valuations we’re seeing are valuing coding assistants, they’re valuing a fictional reality where AGI solves everything and the first one there gets all the rewards.
  - epolanski 11mo ago
    I haven't said a word about hype nor AGI, I merely said that LLM evolution in the tools available right now, not in the future has neither plateaued nor stalled. I'm not expressing any judgement on the economics of it.
- Bender 11mo ago
  It probably depends on the topics people are interacting with. I've spent the last few days teaching Grok how to manage iptables and nftables when all I wanted was to translate my u32 module rules into nftables and I was being lazy, something the provided translate scripts can not do. Grok would confidently give me an answer, I would say that is wrong and it would admit straight away it was wrong and would confidently give me another wrong answer and I would teach it the right answer after a bit of bumbling on my part. It feels worse than being an editor on serverfault but that's just the silly topics I play with. It could be that most topics it does fine but that has not been my experience thus far. At the end of the day I would have been better off just sticking with the man pages and tcpdump.
  - Bender 11mo ago
    Adding today I tested some u32 nftable equiv rules and ran into similar issues on chatgpt. Provided I set chatgpt what OS+version and nftables version I am using it will get some of the IP header locations correct but it still gets the syntax wrong for the older version of nftables that Alpine ships with. Sometimes it does get the older syntax right, sometimes not. If I correct it chatgpt it will agree that is a safer method so I should probably start by saying to use the older methods. Both Grok and ChatGPT appear to have learned from the same sub-optimal locations.

by gishh 11mo ago

Catch up to what? LLMs have clearly stalled in terms of advancement, they just kind of sort of get a little bit different at this point. Not even better, just different.

bubblelicious 11mo ago

Where does this view come from? I’m not aware of any real evidence for this. Also consider our data center buildouts in 26 and 27 will be absolutely extraordinary, and scaling is only at the beginning. You have a growing flywheel and plenty of synthetic data to break the data wall

candiddevmike 11mo ago
We need a fundamental paradigm shift beyond transformers. Throwing more compute or data at it isn't pushing the needle.
- bubblelicious 11mo ago
  And you don’t think that’s already happening? Also where is your evidence for this?
  - bigyabai 11mo ago
    > Also where is your evidence for this? The fact that "scaling laws" didn't scale? Go open your favorite LLM in a hex editor, oftentimes half the larger tensors are just null bytes.
    - bubblelicious 11mo ago
      Show me a paper, this makes no sense of course scaling laws are scaling
- marcosdumay 11mo ago
  Just to point, but there's no more data. LLMs would always bottleneck on one of those two, as computing demand grows crazy quickly with the data amount, and data is necessarily limited. Turns out people threw crazy amounts of compute into it, so the we got the other limit.
  - bigyabai 11mo ago
    Synthetic data works.
    - marcyb5st 11mo ago
      There's a limit to that according to: https://www.nature.com/articles/s41586-024-07566-y https://www.nature.com/articles/s41586-024-07566-y . Basically, if you use an LLM to augment a training dataset it will become "dumber" every subsequent generation and I am not sure how you can generate synthetic data for a language model without using a language model
      - yorwba 11mo ago
        Synthetic data doesn't have to come from an LLM. And that paper only showed that if you train on a random sample from an LLM, the resulting second LLM is a worse model of the distribution that the first LLM was trained on. When people construct synthetic data with LLMs, they typically do not just sample at random, but carefully shape the generation process to match the target task better than the original training distribution.
  - Alex2037 11mo ago
    [dead]
  - Mistletoe 11mo ago
    Yeah I’m constantly reminded of a quote about this- you can’t make another internet. LLMs already digested the one we have.
  - bubblelicious 11mo ago
ModernMech 11mo ago
Let me put it this way: when ChatGPT tells me I've hit the "Free plan limit for GPT-5", I don't even notice a difference when it goes away or when it comes back. There's no incentive for me to pay them for access to 5 if the downgraded models are just as good. That's a huge problem for them.
- _aavaa_ 11mo ago
  It is a problem easily solved with advertising.
  - ModernMech 11mo ago
    No, because as the history of hardware scaling shows us, things that run on supercomputers today will run on smartphones tomorrow. Current models already run fairly well on beefy desktop systems. Eventually models the quality of ChatGPT 4 will be open sourced and running on commodity systems. Then what? There's no moat.
    - treis 11mo ago
      10-20 years of your data in the form of chat history Billions of users allowing them to continually refund their models Hell by then your phone might be the OpenAI 1. The world's first AI powered phone (tm)
      - overfeed 11mo ago
        > The world's first AI powered phone Do you remember the Facebook phone? Not many people do, because it was a failed project, and that was back when Android was way more open. Every couple of years, a tech company with billions has the brilliant idea: "Why don't we have a mobile platform that we control?", followed by failure. Amazon is the only qualified success in this area.
        treis 11mo ago
        I agree that a slight twist on android doesn't make sense. A phone with a in integrated LLM with apps that are essentially prompts to the LLM might be different enough to gain market share.
skywhopper 11mo ago
There is zero evidence that synthetic data will provide any real benefit. All common sense says it can only reinforce and amplify the existing problems with LLMs and other generative “AI”.
- bubblelicious 11mo ago
  Sounds like someone has no knowledge of the literature, synthetic data isn’t like asking ChatGPT to give you a bunch of fake internet data.

epolanski 11mo ago

I don't see that stall. All of the tools I use get increasingly better every quarter at the very least (coding tools, research, image generation, etc).

afavour 11mo ago
As always it’s hype vs reality. Today we’re told AGI is just around the corner. It isn’t. You’re right that a lot of tools are improving iteratively and that’s great but the hyped up valuations we’re seeing are valuing coding assistants, they’re valuing a fictional reality where AGI solves everything and the first one there gets all the rewards.
- epolanski 11mo ago
  I haven't said a word about hype nor AGI, I merely said that LLM evolution in the tools available right now, not in the future has neither plateaued nor stalled. I'm not expressing any judgement on the economics of it.
Bender 11mo ago
It probably depends on the topics people are interacting with. I've spent the last few days teaching Grok how to manage iptables and nftables when all I wanted was to translate my u32 module rules into nftables and I was being lazy, something the provided translate scripts can not do. Grok would confidently give me an answer, I would say that is wrong and it would admit straight away it was wrong and would confidently give me another wrong answer and I would teach it the right answer after a bit of bumbling on my part. It feels worse than being an editor on serverfault but that's just the silly topics I play with. It could be that most topics it does fine but that has not been my experience thus far. At the end of the day I would have been better off just sticking with the man pages and tcpdump.
- Bender 11mo ago
  Adding today I tested some u32 nftable equiv rules and ran into similar issues on chatgpt. Provided I set chatgpt what OS+version and nftables version I am using it will get some of the IP header locations correct but it still gets the syntax wrong for the older version of nftables that Alpine ships with. Sometimes it does get the older syntax right, sometimes not. If I correct it chatgpt it will agree that is a safer method so I should probably start by saying to use the older methods. Both Grok and ChatGPT appear to have learned from the same sub-optimal locations.