3 ms·

>People notice that while AI can now write programs, design websites, etc, it still often makes mistakes or goes in a wrong direction, and then they somehow jum

>People notice that while AI can now write programs, design websites, etc, it still often makes mistakes or goes in a wrong direction, and then they somehow jump to the conclusion that AI will never be able to do these tasks at human levels, or will only have a minor impact. When just a few years ago, having AI do these things was complete science fiction! Both things can be true, since they're orthogonal. Having AI do these things was complete fiction 10 years ago. And after 5 years of LLM AI, people do start to see serious limits and stunted growth with the current LLM approaches, while also seeing that nobody has proposed another serious contended to that approach. Similarly, going to the moon was science finction 100 years ago. And yet, we're now not only not in Mars, but 50+ years without a new moon manned landing. Same for airplanes. Science fiction in 1900. Mostly stale innovation wise for the last 30 years. A lot of curves can fit an exponential line plot, without the progress going forward being exponential. We would have 1 trillion transistor cpus following Moore's "exponential curve"

Tenemo 1y ago
I agree with all your points, just wanted to say that transistor count is probably a counter example. We have been keeping with the Moore's Law more or less[1] and M3 Max, a 2023 consumer-grade CPU, has ~100B of transistors, "just" one order of magnitude away from yout 1T. I think that shows we haven't stagnated much in transistor density and the progress is just staggering! [1] https://en.m.wikipedia.org/wiki/Transistor_count https://en.m.wikipedia.org/wiki/Transistor_count
- szatkus 1y ago
  That one order of magnitude is about 7 years behind the Moore's Law. We're still progressing but it's slower, more expensive and we hit way more walls than before.
senordevnyc 1y ago
Except it’s not been five years, it’s been at most three, since approximately no one was using LLMs prior to ChatGPT’s release, which was just under three years ago. We did have Copilot a year before that, but it was quite rudimentary. And really, we’ve had even less than that. The first large scale reasoning model was o1, which was released 12 months ago. More useful coding agents are even newer than that. This narrative that we’ve been using these tools for many years and are now hitting a wall doesn’t match my experience at all. AI-assisted coding is way better than it was a year ago, let alone five.
- coldtea 1y ago
  >Except it’s not been five years, it’s been at most three, Why would it be "at most" 3? We had Chat GPT commercially available as private beta API on 2020. It's only the mass public that got 3.5 3 years ago. But those who'd do the noticing as per my argument is not just Joe Public (which could be oblivious), but people already starting in 2020, and includes people working in the space, who worked with LLM and LLM-like architectures 2-3 years before 2020.
  - senordevnyc 1y ago
    No, we didn’t. We had the GPT-3 API available in 2020, and approximately no one was using it.
    - coldtea 1y ago
      Seems like you missed most of my comment
      - senordevnyc 1y ago
        I stopped once it became clear that the first half was both wrong and irrelevant.
dist-epoch 1y ago
> We would have 1 trillion transistor cpus following Moore's "exponential curve" Cerebras wafer scale chip has 4 trillion transistors. https://www.cerebras.ai/chip https://www.cerebras.ai/chip
- solid_fuel 1y ago
  > Cerebras wafer scale chip has 4 trillion transistors. It is also, notably, _wafer-scale_. The metric is not just "number of transistors", but in fact "number of transistors per cm2"

Tenemo 1y ago

I agree with all your points, just wanted to say that transistor count is probably a counter example. We have been keeping with the Moore's Law more or less[1] and M3 Max, a 2023 consumer-grade CPU, has ~100B of transistors, "just" one order of magnitude away from yout 1T. I think that shows we haven't stagnated much in transistor density and the progress is just staggering! [1] https://en.m.wikipedia.org/wiki/Transistor_count https://en.m.wikipedia.org/wiki/Transistor_count

szatkus 1y ago
That one order of magnitude is about 7 years behind the Moore's Law. We're still progressing but it's slower, more expensive and we hit way more walls than before.

senordevnyc 1y ago

Except it’s not been five years, it’s been at most three, since approximately no one was using LLMs prior to ChatGPT’s release, which was just under three years ago. We did have Copilot a year before that, but it was quite rudimentary. And really, we’ve had even less than that. The first large scale reasoning model was o1, which was released 12 months ago. More useful coding agents are even newer than that. This narrative that we’ve been using these tools for many years and are now hitting a wall doesn’t match my experience at all. AI-assisted coding is way better than it was a year ago, let alone five.

coldtea 1y ago
>Except it’s not been five years, it’s been at most three, Why would it be "at most" 3? We had Chat GPT commercially available as private beta API on 2020. It's only the mass public that got 3.5 3 years ago. But those who'd do the noticing as per my argument is not just Joe Public (which could be oblivious), but people already starting in 2020, and includes people working in the space, who worked with LLM and LLM-like architectures 2-3 years before 2020.
- senordevnyc 1y ago
  No, we didn’t. We had the GPT-3 API available in 2020, and approximately no one was using it.
  - coldtea 1y ago
    Seems like you missed most of my comment
    - senordevnyc 1y ago
      I stopped once it became clear that the first half was both wrong and irrelevant.

dist-epoch 1y ago

> We would have 1 trillion transistor cpus following Moore's "exponential curve" Cerebras wafer scale chip has 4 trillion transistors. https://www.cerebras.ai/chip https://www.cerebras.ai/chip

solid_fuel 1y ago
> Cerebras wafer scale chip has 4 trillion transistors. It is also, notably, _wafer-scale_. The metric is not just "number of transistors", but in fact "number of transistors per cm2"