4 ms·

compute gets cheaper and cheaper every year. This model will be in your phone by 2030 if we continue at the pace we've been at the last few years.

agentultra 2y ago
There’s probably enough VC money to subsidize the costs for a few more years. But the data centres running the training for models like this are bringing up new methane power plants at a fast rate at a time when we need to be reducing reliance on O&G. But let’s assume that the efficiency gains out pace the resource consumption with the help of all the subsidies being thrown in and we achieve AGI. What’s the benefit? Do we get more fresh water?
- hamburga 2y ago
  Yeah, good question. I think it depends on our politics. If we’re in a techno-capital-oligarchy, people are going to have a hard time making fresh water a priority when the robots would prefer to build nuclear power everywhere and use it to desalinate sea water. OTOH if these data centers are sufficiently decentralized and run for public benefit, maybe there’s a chance we use them to solve collective action problems.
- fastball 2y ago
  Politically anything can happen. Maybe the billionaire class controls everything with an army of robots and it's a horrible prison-like dystopia, or maybe we end up in a post-scarcity utopia a la The Culture. Regardless, once we have AGI (and it can scale), I don't think O&G reliance (/ climate change) is going to be something that we need concern ourselves with.
  - agentultra 2y ago
    I think climate is something we need to concern ourselves with now and at the rate we’re putting up new methane power plants to power the DC’s for these AI apps today, we might not make it to this hypothetical future where magic algorithms will reset everything. Like it or not we already know what we need to do to avert the worst of the climate disasters to come.
hajile 2y ago
These models are nearing 2+ trillion parameters. At 4 bits each, we're talking about somewhere around 1tb of RAM. The problem is that RAM stopped scaling a long time ago now. We're down to the size where a single capacitor's charge is held by a mere 40,000 or so electrons and all we've been doing is making skinnier, longer cells of that size because we can't find reliable ways to boost even weaker signals, but this is a dead end because as the math shows, if the volume is consistent and you are reducing X and Y dimensions, that Z dimension starts to get crazy big really fast. The chemistry issues of burning a hole a little at a time while keeping wall thickness somewhat similar all the way down is a very hard problem. Another problem is that Moore's law hit a wall when Dennard Scaling failed. When you look at SRAM (it's generally the smallest and most reliable stuff we can make), you see that most recent shrinks can hardly be called shrinks. Unless we do something very different like compute in storage or have some radical breakthrough in a new technology, I don't know that we will ever get a 2T parameter model inside a phone (I'd love for someone in 10 years to show up and say how wrong I was).

agentultra 2y ago

There’s probably enough VC money to subsidize the costs for a few more years. But the data centres running the training for models like this are bringing up new methane power plants at a fast rate at a time when we need to be reducing reliance on O&G. But let’s assume that the efficiency gains out pace the resource consumption with the help of all the subsidies being thrown in and we achieve AGI. What’s the benefit? Do we get more fresh water?

hamburga 2y ago
Yeah, good question. I think it depends on our politics. If we’re in a techno-capital-oligarchy, people are going to have a hard time making fresh water a priority when the robots would prefer to build nuclear power everywhere and use it to desalinate sea water. OTOH if these data centers are sufficiently decentralized and run for public benefit, maybe there’s a chance we use them to solve collective action problems.
fastball 2y ago
Politically anything can happen. Maybe the billionaire class controls everything with an army of robots and it's a horrible prison-like dystopia, or maybe we end up in a post-scarcity utopia a la The Culture. Regardless, once we have AGI (and it can scale), I don't think O&G reliance (/ climate change) is going to be something that we need concern ourselves with.
- agentultra 2y ago
  I think climate is something we need to concern ourselves with now and at the rate we’re putting up new methane power plants to power the DC’s for these AI apps today, we might not make it to this hypothetical future where magic algorithms will reset everything. Like it or not we already know what we need to do to avert the worst of the climate disasters to come.

hajile 2y ago

These models are nearing 2+ trillion parameters. At 4 bits each, we're talking about somewhere around 1tb of RAM. The problem is that RAM stopped scaling a long time ago now. We're down to the size where a single capacitor's charge is held by a mere 40,000 or so electrons and all we've been doing is making skinnier, longer cells of that size because we can't find reliable ways to boost even weaker signals, but this is a dead end because as the math shows, if the volume is consistent and you are reducing X and Y dimensions, that Z dimension starts to get crazy big really fast. The chemistry issues of burning a hole a little at a time while keeping wall thickness somewhat similar all the way down is a very hard problem. Another problem is that Moore's law hit a wall when Dennard Scaling failed. When you look at SRAM (it's generally the smallest and most reliable stuff we can make), you see that most recent shrinks can hardly be called shrinks. Unless we do something very different like compute in storage or have some radical breakthrough in a new technology, I don't know that we will ever get a 2T parameter model inside a phone (I'd love for someone in 10 years to show up and say how wrong I was).