5 ms·

> NASA spending has created a huge pile of technologies that we use day to day We're a little too early to know if that's the case here too. I do foresee a cha

> NASA spending has created a huge pile of technologies that we use day to day We're a little too early to know if that's the case here too. I do foresee a chance at a reality where AI is a dead end, but after it we have a ton of cheap GPU compute lying about, which we all rush to somehow convert into useful compute (by emulating CPU's or translating traditional algorithms into GPU oriented ones or whatever).

PunchyHamster 6mo ago
I think there is little chance it is a "dead end", it's here to stay but at least LLMs seem to have hit the diminishing returns curve already, despise what investors might think, and so far none of the big providers actually makes money for all that investment
- etempleton 6mo ago
  I think for many, if LLMs and AI only improves marginally in the next 5-10 years it is effectively a dead end. The capital expenditure necessitates AI does something exponentially more valuable than what it does now. I think we are saying the same thing.i just think the pull back on AI will be dramatic unless something amazing happens very soon.
  - brookst 6mo ago
    I just don’t see it. Both professionally and personally I’m producing so much more now. Back burner projects that weren’t worth months of my time are easily worth a few hours and $20 or whatever. Why would I pull back?
    - grandchild 6mo ago
      You're forgetting that the 20$ are not a sustainable price point. Would your backburner personal app thingy be worth 200$?
      - BobbyTables2 6mo ago
        I’ve wondered about this too. Perhaps if we used something exotic like solid gold cookware, there might be some amazing benefits that people would love. But it would be far from practical without being wildly subsidized… With AI, it feels too much like the “grownups” are acting worse than the kids…
      - brookst 6mo ago
        Yes, I have often paid more than that to have someone else develop a personal side project.
      - nradov 6mo ago
        Compute capacity for the same workload always gets cheaper over time.
        jclulow 6mo ago
        No piece of compute capacity (in the form of equipment) I have bought this year has been in any way cheaper than last year.
        nradov 6mo ago
        So what. Fluctuations over a year or two are meaningless. Do you really believe that the constant-dollar price of an LLM token will be higher in 20 years?
        edmundsauto 6mo ago
        I can see a world where energy costs rise at a rate faster than overall inflation, or are a leading indicator. In that scenario then yes I could see LLM token costs going up.
      - theptip 6mo ago
        This is thoroughly debunked at this point. The frontier labs are profitable on the tokens they serve. They are negative when you bake in the training costs for the next generation.
    - wr2 6mo ago
      Lol are people like you going to be enough to support the large revenues? Nope. A firm that see's rising operating expenses but no not enough increase in revenue will start to cut back on spending on LLMs and become very frugal (e.g. rationing).
      - brookst 6mo ago
        Before they cut back on human programmers?
    - close04 6mo ago
      You’re probably already experienced at your job and using AI to enhance that, or at least using that experience to keep the AI results clean. That’s something you or a company would want to pay for but it has to be a lot more than today’s prices to make it profitable. Companies want to get more out of you, or get a better price/performance ratio (an AI that delivers cheaper than the equivalent human). But current gen AIs are like eternal juniors, never quite ready to operate independently, never learning to become the expert that you are, they are practically frozen in time to the capabilities gained during training. Yet these LLMs replaced the first few rungs of the ladder so human juniors have a canyon to jump if they want the same progression you had. I’m seeing inexperienced people just using AI like a magic 8 ball. “The AI said whatever”. [0] LLMs are smart and cheap enough to undercut human juniors, especially in the hands of a senior. But they’re too dumb to ever become a senior. Where’s the big money in that? What company wants to pay for the “eternal juniors” workforce and whatever they save on payroll goes to procuring external seniors which they’re no longer producing internally? So I’m not too sure a generation of people who have to compete against the LLMs from day 1 will really be producing “so much more” of value later on. Maybe a select few will. Without a big jump in model quality we might see “always junior” LLMs without seniors to enhance. This is not sustainable. And you enhancing your carpentry skills for your free time isn’t what pays for the datacenters and some CEO’s fat paycheck. [0] I hire trainees/interns every year, and pore through hundreds of CVs and interviews for this. The quality of a significant portion of them has gone way down in the past years, coinciding with LLMs gaining popularity.
m_mueller 6mo ago
e.g. the climate models that could be run on some of these systems would dwarve anything we’ve been able to do so far.
TeMPOraL 6mo ago
AI progress may fizzle out, but everything it produced so far would still be there. Models are just big bags of floats - once trained, they're around forever (well, at least until someone deletes them), same is true about harnesses they run in (it's just programs). But AI proliferation is not stopping soon, because we've not picked up even the low hanging fruits just yet. Again, even if no new SOTA models were to be trained after today, there's years if not decades of R&D work into how to best use the ones we have - how to harness the big ones, where to embed the small ones, and of course, more fundamental exploration of the latent spaces and how they formed, to inform information sciences, cognitive sciences, and perhaps even philosophy. And if that runs out or there is an Anti AI Revolution, we can still run those weather models and route planners on the chips once occupied by LLMs - just don't tell the proles that those too are AI, or it's guillotine o'clock again.
- delusional 6mo ago
  > there's years if not decades of R&D work into how to best use the ones we have - how to harness the big ones, where to embed the small ones, and of course, more fundamental exploration of the latent spaces and how they formed, to inform information sciences, cognitive sciences, and perhaps even philosophy. I think my sense of "dead end" would entail none of those directions panning out into anything interesting. You would "explore the latent spaces" only to find nothing of value. Embedding the LLM models wouldn't end up doing anything useful for whatever reason, and philosophy would continue on without any change.
lukewarm707 6mo ago
when ai is dead we can use all those gpus for zucc's metaverse xD s
theptip 6mo ago
If all AI progress somehow immediately halted, the models that have currently been built will still have more economic impact than the Internet. Not least because the slower the frontier advances, the cheaper ASICs get on a relative basis, and therefore the cheaper tokens at the frontier get. We have a massive scaffolding capability overhang, give it ten years to diffuse and most industries will be radically different. Again, all of this is obvious if you spend 1k hours with the current crop, this isn’t making any capability gain forecasts. Just for a dumb example, there is a great ChatGPT agent for Instacart, you can share a photo of your handwritten shopping list and it will add everything to your cart. Just following through the obvious product conclusions of this capability for every grocery vendor’s app, integrating with your fridge, learning your personal preferences for brands, recipe recommendation systems, logistics integrations with your forecasted/scheduled demand, etc is I contend going to be equivalent engineering effort and impact to the move from brick and mortar to online stores.
- delusional 6mo ago
  You have to agree that it's totally possible that none of those things you are envisioning getting built out actually end up working as products, right? AI (LLM) progress would stop, and then everything people try to do with those last and most capable models would end up uninteresting or at least temporary. That's the world I'm calling a "dead end". No matter how unlikely you think that is, you have to agree that it's at least possible, right?
  - theptip 6mo ago
    > then everything people try to do with those last and most capable models would end up uninteresting I believe that some of my made up examples won’t end up getting built, but my point is that there is _so much_ low hanging fruit like this. Of course, anything is _possible_, but let’s talk likelihood. In my forecast the possible worlds where progress stops and then the existing models don’t end up making anything interesting are almost exclusively scenarios like “Taiwan was invaded, TSMC fabs were destroyed, and somehow we deleted existing datacenters’ installed capacity too” or “neo-Luddites take over globally and ban GPUs”, all of this gives sub-1% likelihood. You can imagine 5-10% likelihood worlds where the growth rate of new chips dramatically decreases for a decade due to a single black-swan event like Taiwan getting glassed, but that’s a temporary setback not a permanent blocker. Again, I’m just looking at all the things that can obviously be built now, and just haven’t made it to the top of the list yet. I’m extremely confident that this todo list is already long enough that “this all fizzles to nothing” is basically excluded. I think if model progress stops then everyone investing in ASI takes a big haircut, but the long-term stock market progression will look a lot like the internet after the dot com boom, ie the bloodbath ends up looking like a small blip in the rear view mirror. I guess, a question for you - how do you think about coding agents? Don’t they already show AI is going to do more than “end up uninteresting”?
petra 6mo ago
Even if chatbot LLM's stop at their current capability, There's a whole ecosystem of scientific language models(in drug discovery, chemistry, materials design, etc), and engineering language models(software, chip design, etc) that are very valuable in their fields. And even if chatbot LLM's seem to be a dead end, them and other machine learning algo's will be happy to use the data centers to create/discover a lot of stuff.
vrighter 6mo ago
What will happen is that new buzzwords will be invented, and a new fad will take its place. And we will be stuck with the short end of the stick again. You can hope, but shit doesn't really get cheaper for us common folk, ever. :/

PunchyHamster 6mo ago

I think there is little chance it is a "dead end", it's here to stay but at least LLMs seem to have hit the diminishing returns curve already, despise what investors might think, and so far none of the big providers actually makes money for all that investment

etempleton 6mo ago
I think for many, if LLMs and AI only improves marginally in the next 5-10 years it is effectively a dead end. The capital expenditure necessitates AI does something exponentially more valuable than what it does now. I think we are saying the same thing.i just think the pull back on AI will be dramatic unless something amazing happens very soon.
- brookst 6mo ago
  I just don’t see it. Both professionally and personally I’m producing so much more now. Back burner projects that weren’t worth months of my time are easily worth a few hours and $20 or whatever. Why would I pull back?
  - grandchild 6mo ago
    You're forgetting that the 20$ are not a sustainable price point. Would your backburner personal app thingy be worth 200$?
    - BobbyTables2 6mo ago
      I’ve wondered about this too. Perhaps if we used something exotic like solid gold cookware, there might be some amazing benefits that people would love. But it would be far from practical without being wildly subsidized… With AI, it feels too much like the “grownups” are acting worse than the kids…
    - brookst 6mo ago
      Yes, I have often paid more than that to have someone else develop a personal side project.
    - nradov 6mo ago
      Compute capacity for the same workload always gets cheaper over time.
      - jclulow 6mo ago
        No piece of compute capacity (in the form of equipment) I have bought this year has been in any way cheaper than last year.
        nradov 6mo ago
        So what. Fluctuations over a year or two are meaningless. Do you really believe that the constant-dollar price of an LLM token will be higher in 20 years?
        edmundsauto 6mo ago
        I can see a world where energy costs rise at a rate faster than overall inflation, or are a leading indicator. In that scenario then yes I could see LLM token costs going up.
    - theptip 6mo ago
      This is thoroughly debunked at this point. The frontier labs are profitable on the tokens they serve. They are negative when you bake in the training costs for the next generation.
  - wr2 6mo ago
    Lol are people like you going to be enough to support the large revenues? Nope. A firm that see's rising operating expenses but no not enough increase in revenue will start to cut back on spending on LLMs and become very frugal (e.g. rationing).
    - brookst 6mo ago
      Before they cut back on human programmers?
  - close04 6mo ago
    You’re probably already experienced at your job and using AI to enhance that, or at least using that experience to keep the AI results clean. That’s something you or a company would want to pay for but it has to be a lot more than today’s prices to make it profitable. Companies want to get more out of you, or get a better price/performance ratio (an AI that delivers cheaper than the equivalent human). But current gen AIs are like eternal juniors, never quite ready to operate independently, never learning to become the expert that you are, they are practically frozen in time to the capabilities gained during training. Yet these LLMs replaced the first few rungs of the ladder so human juniors have a canyon to jump if they want the same progression you had. I’m seeing inexperienced people just using AI like a magic 8 ball. “The AI said whatever”. [0] LLMs are smart and cheap enough to undercut human juniors, especially in the hands of a senior. But they’re too dumb to ever become a senior. Where’s the big money in that? What company wants to pay for the “eternal juniors” workforce and whatever they save on payroll goes to procuring external seniors which they’re no longer producing internally? So I’m not too sure a generation of people who have to compete against the LLMs from day 1 will really be producing “so much more” of value later on. Maybe a select few will. Without a big jump in model quality we might see “always junior” LLMs without seniors to enhance. This is not sustainable. And you enhancing your carpentry skills for your free time isn’t what pays for the datacenters and some CEO’s fat paycheck. [0] I hire trainees/interns every year, and pore through hundreds of CVs and interviews for this. The quality of a significant portion of them has gone way down in the past years, coinciding with LLMs gaining popularity.

m_mueller 6mo ago

e.g. the climate models that could be run on some of these systems would dwarve anything we’ve been able to do so far.

TeMPOraL 6mo ago

AI progress may fizzle out, but everything it produced so far would still be there. Models are just big bags of floats - once trained, they're around forever (well, at least until someone deletes them), same is true about harnesses they run in (it's just programs). But AI proliferation is not stopping soon, because we've not picked up even the low hanging fruits just yet. Again, even if no new SOTA models were to be trained after today, there's years if not decades of R&D work into how to best use the ones we have - how to harness the big ones, where to embed the small ones, and of course, more fundamental exploration of the latent spaces and how they formed, to inform information sciences, cognitive sciences, and perhaps even philosophy. And if that runs out or there is an Anti AI Revolution, we can still run those weather models and route planners on the chips once occupied by LLMs - just don't tell the proles that those too are AI, or it's guillotine o'clock again.

delusional 6mo ago
> there's years if not decades of R&D work into how to best use the ones we have - how to harness the big ones, where to embed the small ones, and of course, more fundamental exploration of the latent spaces and how they formed, to inform information sciences, cognitive sciences, and perhaps even philosophy. I think my sense of "dead end" would entail none of those directions panning out into anything interesting. You would "explore the latent spaces" only to find nothing of value. Embedding the LLM models wouldn't end up doing anything useful for whatever reason, and philosophy would continue on without any change.

lukewarm707 6mo ago

when ai is dead we can use all those gpus for zucc's metaverse xD s

theptip 6mo ago

If all AI progress somehow immediately halted, the models that have currently been built will still have more economic impact than the Internet. Not least because the slower the frontier advances, the cheaper ASICs get on a relative basis, and therefore the cheaper tokens at the frontier get. We have a massive scaffolding capability overhang, give it ten years to diffuse and most industries will be radically different. Again, all of this is obvious if you spend 1k hours with the current crop, this isn’t making any capability gain forecasts. Just for a dumb example, there is a great ChatGPT agent for Instacart, you can share a photo of your handwritten shopping list and it will add everything to your cart. Just following through the obvious product conclusions of this capability for every grocery vendor’s app, integrating with your fridge, learning your personal preferences for brands, recipe recommendation systems, logistics integrations with your forecasted/scheduled demand, etc is I contend going to be equivalent engineering effort and impact to the move from brick and mortar to online stores.

delusional 6mo ago
You have to agree that it's totally possible that none of those things you are envisioning getting built out actually end up working as products, right? AI (LLM) progress would stop, and then everything people try to do with those last and most capable models would end up uninteresting or at least temporary. That's the world I'm calling a "dead end". No matter how unlikely you think that is, you have to agree that it's at least possible, right?
- theptip 6mo ago
  > then everything people try to do with those last and most capable models would end up uninteresting I believe that some of my made up examples won’t end up getting built, but my point is that there is _so much_ low hanging fruit like this. Of course, anything is _possible_, but let’s talk likelihood. In my forecast the possible worlds where progress stops and then the existing models don’t end up making anything interesting are almost exclusively scenarios like “Taiwan was invaded, TSMC fabs were destroyed, and somehow we deleted existing datacenters’ installed capacity too” or “neo-Luddites take over globally and ban GPUs”, all of this gives sub-1% likelihood. You can imagine 5-10% likelihood worlds where the growth rate of new chips dramatically decreases for a decade due to a single black-swan event like Taiwan getting glassed, but that’s a temporary setback not a permanent blocker. Again, I’m just looking at all the things that can obviously be built now, and just haven’t made it to the top of the list yet. I’m extremely confident that this todo list is already long enough that “this all fizzles to nothing” is basically excluded. I think if model progress stops then everyone investing in ASI takes a big haircut, but the long-term stock market progression will look a lot like the internet after the dot com boom, ie the bloodbath ends up looking like a small blip in the rear view mirror. I guess, a question for you - how do you think about coding agents? Don’t they already show AI is going to do more than “end up uninteresting”?

petra 6mo ago

Even if chatbot LLM's stop at their current capability, There's a whole ecosystem of scientific language models(in drug discovery, chemistry, materials design, etc), and engineering language models(software, chip design, etc) that are very valuable in their fields. And even if chatbot LLM's seem to be a dead end, them and other machine learning algo's will be happy to use the data centers to create/discover a lot of stuff.

vrighter 6mo ago

What will happen is that new buzzwords will be invented, and a new fad will take its place. And we will be stuck with the short end of the stick again. You can hope, but shit doesn't really get cheaper for us common folk, ever. :/