7 ms·
Energy, not compute, will be the #1 bottleneck to AI progress – Mark Zuckerberg [video]
- aixpert 3y agoPhotonics will make energy much less relevant again
- croes 3y agoI hope AI doesn't become a version of Universal Paperclips.
- resource_waste 3y agoYou can always slam more proccessors together, makes sense compute isnt the bottleneck. You have people who have proven you can run on CPU, even if its not realistic in the real-world. Memory currently is a concern. Algorithms are my only concern. Its not like these algorithms can snag some protons and electrons and analyze them, they are at the whim of human progress/data collection. Energy, above my paygrade.
- anonzzzies 3y agoWe need to bring models closer to animal brains; more research, better models. Not just keep adding gpus as fast as we can. Everyone knows this obviously, however seems we are going to push the limits even though we know we need something vastly better in the basics.
- hiAndrewQuinn 3y agoIf true, it could be big for countries which already are or are trying to become strong on nuclear. Q There's no reason to suspect most training runs could only be run e.g. during the workday. So a 24/7, more or less constant supply of electricity, like what nuclear characteristically provides, is something training centers could really slurp up.
- anticensor 3y agoAnother quote from the full interview: "I guess a big part of my theory on this is that there's not just going to be like one singular AI that you interact with because I think every business is going to want an AI that represents their interests. They're not going to want to primarily interact with you through an AI that is going to sell their competitors' customers." A huge Freudian slip there.
- rchaud 3y agoWhat's the slip? That AI in practice will not be neutral but instead tailored to the preferences of the vendor?
- gnz11 3y agoThe slip is that they aren’t developing AI for the betterment of human kind, just to better sell to you —- so you keep buying more crap. At least that was my hot take.
- anticensor 3y agoNo, the slip is that Mark subconsciously equates customers and products, hence he said "sell their competitors' customers" instead of "sell their competitors' products".
- max_ 3y agoI am really surprised by how tunnel visioned top scientists, VCs and CEOs are in the AI space. This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end. This is like saying if human beings just flap their wings hard enough, they will be able to fly like birds. We need to finance original research, alternative research and think of working on an actual theory of intelligence. The stage AI is at right now is more like the Ptolemaic model of the universe. We need a Galileo and Kepler of AI. Maybe they already exist but are being marginalized by the current establishment?
- juliangamble 3y agoPerhaps the more apt analogy is that we need "a John Von Neuman" to come and organise the pieces to give us that leap.
- coffeebeqn 3y agoEither way - if/when that breakthrough happens would you rather have n compute or n^4 compute ready to go?
- Retric 3y agoBuying compute before such breakthroughs is a terrible investment compared to just waiting.
- imtringued 3y agoJust rent it out. It's not like we have an abundance of GPUs as of now.
- coffeebeqn 3y agoIt’s not like it’s sitting idle - the companies feel like they’re getting a ROI in whatever Gen AI nonsense they’re pushing to their apps
- DueDilligence 3y ago[dead]
- alangibson 3y agoIf we assume this is true, then it's a massive problem for AI. Building energy infra is slow and hellishly expensive. Near and medium term use cases just aren't compelling enough to start upgrading the power grid.
- coffeebeqn 3y agoI wonder how feasible it would be to place a data center in somewhere like Arizona or Nevada and just run it during daytime hours on a massive solar array and then pause the work for the night time. That would probably take a few decades less than trying to permit and build a nuclear plant next to it
- ArchD 3y agoYou are right that nuclear power tends to have a very long lead time. China has already conveniently installed vast amounts of solar power.
- awongh 3y agoIf you look at the arguments for/against resource consumption for bitcoin, a lot of the same arguments hold true- that actually we don’t necessarily have a deficit of power production. We have an energy capacity problem and distribution problem. I wonder how much of this over supply can be taken up with AI training and how much that would change the calculations.
- macrolime 3y agoIt's not really all that useful for AI training. The reason why energy is a problem for AI training is that you need a lot of compute at one place, distributing backpropagation doesn't work that well. When the energy problems hit, it's when the datacenter need exceeds the capacity of nearby power plants. If we want an alternative I think we'd need to try alternative training algorithms that are easier to distribute. Predictive coding for example can approximate backprop, but requires more compute. Something in that direction might work https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2006.04182
- baerrie 3y agoActually, he says energy is the current bottleneck that if overcome, could lead to another bottleneck in the rate of scaling AIs capabilities
- nicklecompte 3y agoThere is a very intelligent system, far more intelligent than GPT-4, which uses a tiny fraction of the energy. It's called a "rat," and despite requiring only 2 watts of power, it can perform the following tasks which are far beyond any multimodal LLM: - creating and executing complex plans in arbitrary scenarios - counting to 6 with 99% reliability in arbitrary scenarios - understanding object permanence with 99% reliability - understanding simple, intuitive causality with 99% reliability But rats can't answer Python questions, so according to a truly depressing number of tech folks, of course GPT-4 is smarter than a rat! AI researchers/capitalists seriously need to spend more time reading animal cognition. If Silicon Valley was in charge of animal cognition research, spiders would be considered smarter than crows.
- macrolime 3y agoHow much energy was spent on the training run though? It's all the energy spent through the lifetime of trillions of organisms.
- BiteCode_dev 3y agoEnergy is a bottleneck to life in general, and the whole humanity. It's like saying your employees need oxygen to breath or they can't work at scale.
- Havoc 3y agoAI compute is not super latency sensitive. Surely you can just stick the DC somewhere that has power? A bit like the Chinese crypto miners moved to hydro dams
- latchkey 3y agoCrypto (Proof of Work) has the advantage of being extremely low bandwidth, very static compute. You just stick boxes somewhere, plug them in and they mostly just work. AI requires high bandwidth and dynamic compute. You need a whole supply chain to a data center. Repairs become expensive when you have to ship in highly trained people to fix the machines. Swapping out an OAM/UBB AIA isn't like pulling a gpu card from a PCIe slot.
- Havoc 3y agoThose all sound like surmountable problems to me. At least compared to the starting premise that we just don’t have enough energy
- latchkey 3y agoWhat if AI took over bitcoin's power usage?
- waldrews 3y agoWhat's static/dynamic compute in this context?
- latchkey 3y agoBitcoin ASIC's are basically nothing more than kitchen appliances at this point. Plug and play. Everything can be replaced with a few screws. They are extremely static. Put it on a shelf, plug in power/network, and run it until it dies. AIA servers are complex 350lbs beasts that have a lot of moving components and require dedicated trained technicians to replace parts. They have multiple ways they connect to multiple networks. They require a lot of software. They are very dynamic with continuous updates.
- pilgrim0 3y agoCan anyone here argue with confidence (read: evidence) that this AI boom is, in fact, yielding real-life, non-superfluous evolutionary advantages to mankind in general? Judging from a primitive metric, I see no one around me benefiting from AI, nor the people closest to them. Considering the current scale and volume of investments in the field, I would expect that finding AI success stories from a random sample within metropolitan communities should be common by now. That seems not to be the case from my point of view. How can anyone argue that investing in AI indeed yields superior results over merely distributing all the resources diversely, when we have much stronger evidence of advantageous outcomes for other domains? I understand this looks like a false duality, but it may not be considering the established, narrow, and concentrated investment structures. Overall, software has a really flimsy connection to actual human necessities, being more of a vector than a need in itself. But AI seems to be all about itself for now. Recall, at least, that AI winters are about a steep decay in public confidence over the promised outcomes, rather than necessarily a decline in research progress and investments. So the more the investment rates outpace the actual outcomes, the higher the probability of it happening. Ironically, I revised this text with GPT, but this is as far as it’s been useful to me. It’s just marginally more useful than what we actually have, and substantially more expensive.
- tim333 3y agoDeepmind has done some interesting stuff with proteins.