4 ms·

> That's the default assumption but in the new GPU+Memory constrained age isn't true. Is it an age or a temporary situation?

frognumber 4mo ago
It's very unclear to me. The key question is on direction of LLMs. Right now, LLMs are taking over human jobs. If the cost of silicon+power < cost of human being doing the same work, what rational reason is there to employ a human being? If this applies to SWEs, lawyers, business analysts, many research scientists, .... this situation could persist for a long, long time. While capital costs less than the inputs of labor (nominal food, housing, etc.), there is no need for labor. The key question is about continued progress in models, and of the tooling around them: - Plateau: Old silicon obsoletes in due course - Rise quickly: Old silicon maintains value for a long time
- paulhebert 4mo ago
  > what rational reason is there to employ a human being? To maintain a functioning society and social contract? Is wanting low unemployment in our society not rational?
  - hackyhacky 4mo ago
    Correct. Unfortunately, that's not how capitalism makes decisions.
    - tirant 4mo ago
      Capitalism does not decide anything. Capitalism allows individuals to take decisions in a free market. If you want to complain about selfishness then do it on selfish individuals, which by the way, are present in all types of economic systems.
      - goatlover 4mo ago
        Which includes SCOTUS recognizing corporations as persons.
      - hackyhacky 4mo ago
        > Capitalism allows individuals to take decisions in a free market. Capitalism provides a set of incentives that shape how people make decisions. Anyone can be selfish, but selfishness in capitalist society has a particular shape. To ignore the external incentives when looking at human behavior is horribly naive and shortsighted, but is frequently done by capitalism-apologists who seek to disregard any criticism of their favorite incentive system.
  - Earw0rm 4mo ago
    It's ethnically rational, and morally right. However. It's not rational relative to the short-term incentives of a typical corporation or investment vehicle. PE, VC, fund managers aren't paid to give a fuck about the social contract. Literally not in their job description.
  - donaldjbiden 4mo ago
    [dead]
  - ben_w 4mo ago
    > Is wanting low unemployment in our society not rational? Only conditionally on there being bad consequences for high unemployment. I don't particularly trust politicians, but there's a whole host of hypothetical scenarios about futures where work is essentially optional. Unfortunately, they're all either in the sci-fi or religion sections of the book store: Despite people occasionally investigating UBI, the efforts to research UBI seriously have the same problems that Marx had with literal Communism, in that there's an obvious difference between any partial transition as compared to a global transition, and we don't have a completely disconnected parallel world to be a petri dish for us to test the economic outcomes on.
- fluidcruft 4mo ago
  What I don't understand is if nobody has jobs, who's paying the machines to do anything? So okay cool you don't need people to design and build cars. Who's going to buy the cars and where exactly are they finding money? But see also the "radiologists driving to work" meme for why I think tech in general is currently getting high off their own farts.
  - saulpw 4mo ago
    Rich people become the only consumers.
    - andy_ppp 4mo ago
      Yes, the plan seems to be anti human in the extreme. Why do you need the plebs if they can be entirely replaced by AI? But the question then becomes why does the AI (and before that their security detail in a post money world) need billionaires?
      - iugtmkbdfil834 4mo ago
        This likely is the tertiary reason as to why llms are so heavily kneecapped. Granted, at this point, projects do exist to remove those arbitrary restrictions, but the effort that goes into it suggests it is a real concern.
  - somenameforme 4mo ago
    The overwhelming majority of the labor force remains service, manual labor, and other such stuff that LLMs will have no real effect on. So the economy will be fine, but I do agree with you from a different angle. The entire goal of LLMs seems self destructive. If they're successful then the endgame is completely removing the barriers to entry to producing software and other digital tech. But if we do reach that endgame then the value of tech is going to plummet because there will be absolutely no barriers to entry to compete, or even just individuals homebrewing up what they need on demand. Like imagine there was something you could buy where you insert some lumber, give it some passable description of furniture, and it outputs it. And you paid $20/month for access to this. And this was all being bankrolled by the furniture industry? I mean, sure guys - it's much appreciated, but I don't think I've ever seen anybody so enthusiastic about digging their own grave. I think it's already obvious that the gazillion dollars of API calls isn't going to materialize - it seems the handful of companies that trialed that are already reversing course hard. And in the future where LLMs are successful, that'd be even more true.
mikepurvis 4mo ago
Are current datacenter deployments structured in such a way that the memory can later be moved to newer GPU dies? Or is it all packaged together as on consumer graphics cards? I assumed the latter and therefore that the memory is depreciating along with the GPU cores it's soldered onto PCBs with. ... or is it a different argument being made, perhaps that depreciation for GPUs has slowed because rising demand will keep them in service longer?
- nl 4mo ago
  The argument is that all GPUs are currently appreciating (!!) Google is still running 10 year old Tesla T4s at full capacity. This is way beyond the expected lifetime.
- fragmede 4mo ago
  Removing RAM chips off old cards is uneconomical, until it isn't. With things going the way they are, if you've got a card with soldered on RAM that could be transplanted to a newer card, I think you'll start seeing that happening.
  - fer 4mo ago
    It has already become economic. While not exactly the same, the NVIDIA 2080 11GB cards are notorious for being upcycled with extra RAM: https://www.reddit.com/r/nvidia/comments/146us12/nvidia_geforce_rtx_2080_ti_gets_a_44gb_memory_mod/ https://www.reddit.com/r/nvidia/comments/146us12/nvidia_gefo...
    - deleted 4mo ago
      [deleted]
  - donaldjbiden 4mo ago
    Chinese recyclers already do this with laptops
baq 4mo ago
Everything is a temporary situation on long enough timeframes, especially if it’s exponentially growing. Moore’s law which dictates that compute depreciates quickly has been slowing down a lot in the last few years, coupled with the explosion in demand we’ve found ourselves in a prolonged shortage situation. The bubble will pop, but if you predict when correctly, you will be a rich man.
nl 4mo ago
Memory is unlikely to drop in price before mid-2027 when new capacity starts to come online. The GPU shortage looks to be even longer lived.
- williamdclt 4mo ago
  So, temporary situation then. That's a pretty short period with no paradigm shift, just a delay in capacity.
  - __blockcipher__ 4mo ago
    It’s gonna take a lot longer than mid 2027. 2029 earliest IMO. Hyperscaler spend is basically already spoken for the next 2 years.
  - Seanambers 4mo ago
    Temporary until its not. It's the new normal, get used to it. The MAG7 isn't pumping all their FCF + new debt issuance into DC's just for fun. The world is seemingly moving into a era where compute is becoming expensive and scarce. Only thing that can possibly change this is LLMs hitting a vertical unscalable wall. More AI compute = more CPU, memory, storage needs.
    - _1100 4mo ago
      Do you think we will recognize any walls? Or is there a point where the output might look different with respect to different paradigms / modalities we throw at it, but we won't be able to understand the quantitative differences as good/bad/scalable?

frognumber 4mo ago

It's very unclear to me. The key question is on direction of LLMs. Right now, LLMs are taking over human jobs. If the cost of silicon+power < cost of human being doing the same work, what rational reason is there to employ a human being? If this applies to SWEs, lawyers, business analysts, many research scientists, .... this situation could persist for a long, long time. While capital costs less than the inputs of labor (nominal food, housing, etc.), there is no need for labor. The key question is about continued progress in models, and of the tooling around them: - Plateau: Old silicon obsoletes in due course - Rise quickly: Old silicon maintains value for a long time

paulhebert 4mo ago
> what rational reason is there to employ a human being? To maintain a functioning society and social contract? Is wanting low unemployment in our society not rational?
- hackyhacky 4mo ago
  Correct. Unfortunately, that's not how capitalism makes decisions.
  - tirant 4mo ago
    Capitalism does not decide anything. Capitalism allows individuals to take decisions in a free market. If you want to complain about selfishness then do it on selfish individuals, which by the way, are present in all types of economic systems.
    - goatlover 4mo ago
      Which includes SCOTUS recognizing corporations as persons.
    - hackyhacky 4mo ago
      > Capitalism allows individuals to take decisions in a free market. Capitalism provides a set of incentives that shape how people make decisions. Anyone can be selfish, but selfishness in capitalist society has a particular shape. To ignore the external incentives when looking at human behavior is horribly naive and shortsighted, but is frequently done by capitalism-apologists who seek to disregard any criticism of their favorite incentive system.
- Earw0rm 4mo ago
  It's ethnically rational, and morally right. However. It's not rational relative to the short-term incentives of a typical corporation or investment vehicle. PE, VC, fund managers aren't paid to give a fuck about the social contract. Literally not in their job description.
- donaldjbiden 4mo ago
  [dead]
- ben_w 4mo ago
  > Is wanting low unemployment in our society not rational? Only conditionally on there being bad consequences for high unemployment. I don't particularly trust politicians, but there's a whole host of hypothetical scenarios about futures where work is essentially optional. Unfortunately, they're all either in the sci-fi or religion sections of the book store: Despite people occasionally investigating UBI, the efforts to research UBI seriously have the same problems that Marx had with literal Communism, in that there's an obvious difference between any partial transition as compared to a global transition, and we don't have a completely disconnected parallel world to be a petri dish for us to test the economic outcomes on.
fluidcruft 4mo ago
What I don't understand is if nobody has jobs, who's paying the machines to do anything? So okay cool you don't need people to design and build cars. Who's going to buy the cars and where exactly are they finding money? But see also the "radiologists driving to work" meme for why I think tech in general is currently getting high off their own farts.
- saulpw 4mo ago
  Rich people become the only consumers.
  - andy_ppp 4mo ago
    Yes, the plan seems to be anti human in the extreme. Why do you need the plebs if they can be entirely replaced by AI? But the question then becomes why does the AI (and before that their security detail in a post money world) need billionaires?
    - iugtmkbdfil834 4mo ago
      This likely is the tertiary reason as to why llms are so heavily kneecapped. Granted, at this point, projects do exist to remove those arbitrary restrictions, but the effort that goes into it suggests it is a real concern.
- somenameforme 4mo ago
  The overwhelming majority of the labor force remains service, manual labor, and other such stuff that LLMs will have no real effect on. So the economy will be fine, but I do agree with you from a different angle. The entire goal of LLMs seems self destructive. If they're successful then the endgame is completely removing the barriers to entry to producing software and other digital tech. But if we do reach that endgame then the value of tech is going to plummet because there will be absolutely no barriers to entry to compete, or even just individuals homebrewing up what they need on demand. Like imagine there was something you could buy where you insert some lumber, give it some passable description of furniture, and it outputs it. And you paid $20/month for access to this. And this was all being bankrolled by the furniture industry? I mean, sure guys - it's much appreciated, but I don't think I've ever seen anybody so enthusiastic about digging their own grave. I think it's already obvious that the gazillion dollars of API calls isn't going to materialize - it seems the handful of companies that trialed that are already reversing course hard. And in the future where LLMs are successful, that'd be even more true.

mikepurvis 4mo ago

Are current datacenter deployments structured in such a way that the memory can later be moved to newer GPU dies? Or is it all packaged together as on consumer graphics cards? I assumed the latter and therefore that the memory is depreciating along with the GPU cores it's soldered onto PCBs with. ... or is it a different argument being made, perhaps that depreciation for GPUs has slowed because rising demand will keep them in service longer?

nl 4mo ago
The argument is that all GPUs are currently appreciating (!!) Google is still running 10 year old Tesla T4s at full capacity. This is way beyond the expected lifetime.
fragmede 4mo ago
Removing RAM chips off old cards is uneconomical, until it isn't. With things going the way they are, if you've got a card with soldered on RAM that could be transplanted to a newer card, I think you'll start seeing that happening.
- fer 4mo ago
  It has already become economic. While not exactly the same, the NVIDIA 2080 11GB cards are notorious for being upcycled with extra RAM: https://www.reddit.com/r/nvidia/comments/146us12/nvidia_geforce_rtx_2080_ti_gets_a_44gb_memory_mod/ https://www.reddit.com/r/nvidia/comments/146us12/nvidia_gefo...
  - deleted 4mo ago
    [deleted]
- donaldjbiden 4mo ago
  Chinese recyclers already do this with laptops

baq 4mo ago

Everything is a temporary situation on long enough timeframes, especially if it’s exponentially growing. Moore’s law which dictates that compute depreciates quickly has been slowing down a lot in the last few years, coupled with the explosion in demand we’ve found ourselves in a prolonged shortage situation. The bubble will pop, but if you predict when correctly, you will be a rich man.

nl 4mo ago

Memory is unlikely to drop in price before mid-2027 when new capacity starts to come online. The GPU shortage looks to be even longer lived.

williamdclt 4mo ago
So, temporary situation then. That's a pretty short period with no paradigm shift, just a delay in capacity.
- __blockcipher__ 4mo ago
  It’s gonna take a lot longer than mid 2027. 2029 earliest IMO. Hyperscaler spend is basically already spoken for the next 2 years.
- Seanambers 4mo ago
  Temporary until its not. It's the new normal, get used to it. The MAG7 isn't pumping all their FCF + new debt issuance into DC's just for fun. The world is seemingly moving into a era where compute is becoming expensive and scarce. Only thing that can possibly change this is LLMs hitting a vertical unscalable wall. More AI compute = more CPU, memory, storage needs.
  - _1100 4mo ago
    Do you think we will recognize any walls? Or is there a point where the output might look different with respect to different paradigms / modalities we throw at it, but we won't be able to understand the quantitative differences as good/bad/scalable?