4 ms·
With only a 100x increase in cost, we improved performance by 0.1x and continued plotting this concave-down diminishing-returns type graph! Hurray for logarith
by __MatrixMan__ 2y ago
With only a 100x increase in cost, we improved performance by 0.1x and continued plotting this concave-down diminishing-returns type graph! Hurray for logarithmic x-axes!
Joking aside, better than ever before at any cost is an achievement, it just doesn't exactly scream "breakthrough" to me.
- kvetching 2y agoIt may eventually be able to solve any problem
- iterance 2y agoAh. Me, too.
- HDThoreaun 2y agocompute gets cheaper and cheaper every year. This model will be in your phone by 2030 if we continue at the pace we've been at the last few years.
- agentultra 2y agoThere’s probably enough VC money to subsidize the costs for a few more years. But the data centres running the training for models like this are bringing up new methane power plants at a fast rate at a time when we need to be reducing reliance on O&G. But let’s assume that the efficiency gains out pace the resource consumption with the help of all the subsidies being thrown in and we achieve AGI. What’s the benefit? Do we get more fresh water?
- hamburga 2y agoYeah, good question. I think it depends on our politics. If we’re in a techno-capital-oligarchy, people are going to have a hard time making fresh water a priority when the robots would prefer to build nuclear power everywhere and use it to desalinate sea water. OTOH if these data centers are sufficiently decentralized and run for public benefit, maybe there’s a chance we use them to solve collective action problems.
- fastball 2y agoPolitically anything can happen. Maybe the billionaire class controls everything with an army of robots and it's a horrible prison-like dystopia, or maybe we end up in a post-scarcity utopia a la The Culture. Regardless, once we have AGI (and it can scale), I don't think O&G reliance (/ climate change) is going to be something that we need concern ourselves with.
- agentultra 2y agoI think climate is something we need to concern ourselves with now and at the rate we’re putting up new methane power plants to power the DC’s for these AI apps today, we might not make it to this hypothetical future where magic algorithms will reset everything. Like it or not we already know what we need to do to avert the worst of the climate disasters to come.
- hajile 2y agoThese models are nearing 2+ trillion parameters. At 4 bits each, we're talking about somewhere around 1tb of RAM. The problem is that RAM stopped scaling a long time ago now. We're down to the size where a single capacitor's charge is held by a mere 40,000 or so electrons and all we've been doing is making skinnier, longer cells of that size because we can't find reliable ways to boost even weaker signals, but this is a dead end because as the math shows, if the volume is consistent and you are reducing X and Y dimensions, that Z dimension starts to get crazy big really fast. The chemistry issues of burning a hole a little at a time while keeping wall thickness somewhat similar all the way down is a very hard problem. Another problem is that Moore's law hit a wall when Dennard Scaling failed. When you look at SRAM (it's generally the smallest and most reliable stuff we can make), you see that most recent shrinks can hardly be called shrinks. Unless we do something very different like compute in storage or have some radical breakthrough in a new technology, I don't know that we will ever get a 2T parameter model inside a phone (I'd love for someone in 10 years to show up and say how wrong I was).
- swalsh 2y ago[flagged]
- zeofig 2y agoAgi level my ass
- haidev 2y ago+1
- adamtaylor_13 2y agoI know AGI is a bit of a moving goalpost these days, but by my personally anecdotal and irrelevant opinion this ain’t AGI. I’ll let those smarter than me debate the merits of AGI, but if it can’t learn and self-improve it isn’t “general” intelligence. This is a very smart computer, accomplishing a very niche set of problems. Cool? Yes. AGI? No.
- BobbyJo 2y agoI think what current technology is missing is in situ learning. I don't think that's a massive leap from where we are though.
- Jensson 2y agoHow is in situ learning not a massive leap? We have no clue how to make an intelligent general task judger, and without that we don't know how to do in situ learning that actually improves performance on general tasks.
- danenania 2y agoNot to say it wouldn’t be a leap, but is there anything particularly special about learning that makes it unachievable? If you can reason and you can do trial and error, given enough time and compute I think most things should be learnable?
- whalee 2y agoimo it's a mistake to interpret the marginal increases in the upper echelons of benchmarks as materially marginal gains. Chess is an example. ELO narrows heavily at the top, but each ELO point carries more relative weight. This is a bit apples and oranges since chess is adversarial, but I think the point stands.
- wavemode 2y ago> ELO narrows heavily at the top What do you mean by this? I'm assuming you're not speaking about simple absolute differences in value - there have been top players rated over 100 points higher than the average of the rest of the top ten.
- dyauspitr 2y agoI mean going from 10% to 85% doesn’t seem like a 0.1% improvement
- __MatrixMan__ 2y agoOh crap I made a mistake. I was comparing o3 low to o3 high. I'm a little disappointed by all the upvotes I got for being flat wrong. I guess as long as you're trashing AI you can get away with anything. Really I was just trying to nitpick the chart parameters.
- energy123 2y agoo3-mini (high) uses 1/3rd of the compute of o1, and performs about 200 Elo higher than o1 on Codeforces. o1 is the best code generation model according to Livebench. So how is this not a breakthrough? It's a genuine movement of the frontier.
- handzhiev 2y agoHow much time does a top sprinter take a 100 m run for compared to a mediocre sprinter?
- deleted 2y ago[deleted]