8 ms·
The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing po
by duluca 2y ago
The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle.
AGI won’t happen until you can fit enough compute that’d take several data center’s worth of compute into a brain sized vessel. So the thing can move around process the world in real time. This is all going to take some time to say the least. Progress is progress.
- lxgr 2y ago> take several data center’s worth of compute into a brain sized vessel. So the thing can move around process the world in real time How so? I'd imagine a robot connected to the data center embodying its mind, connected via low-latency links, would have to walk pretty far to get into trouble when it comes to interacting with the environment. The speed of light is about three orders of magnitude faster than the speed of signal propagation in biological neurons, after all.
- waldrews 2y ago6 orders of magnitude if we use 120 m/s vs 300 km/s
- byw 2y agoThe robot brain could be layered so that more basic functions are embedded locally while higher-level reasonings and offloaded to the cloud.
- arthurcolle 2y agoblue strip from iRobot?
- lumost 2y agoThe concern here is mainly on practicality. The original mainframes did not command startup valuations counted in fractions of the US economy, they did qualify for billions in investment. This is a great milestone, but OpenAI will not be successful charging 10x the cost of a human to perform a task.
- BriggyDwiggs42 2y agoI wouldn’t expect it to cost 10x in five years, if only because parallel computing still seems to be roughly obeying moore’s.
- raincole 2y agoThe cost of inference has be dropping by ~100x in the past 2 years. https://a16z.com/llmflation-llm-inference-cost/ https://a16z.com/llmflation-llm-inference-cost/
- nico 2y ago*inference
- gritzko 2y ago*infernonce
- christianqchung 2y agoHmm the link is saying the price of an LLM that scores 42 or above on MMLU has dropped 100x in 2 years, equating gpt 3.5 and llama 3.2 3B. In my opinion gpt 3.5 was significantly better than llama 3B, and certainly much better than the also-equated llama 2 7B. MMLU isn't a great marker of overall model capabilities. Obviously the drop in cost for capability in the last 2 years is big, but I'd wager it's closer to 10x than 100x.
- owenpalmer 2y ago> OpenAI will not be successful charging 10x the cost of a human to perform a task. True, but they might be successful charging 20x for 2x the skill of a human.
- otabdeveloper4 2y agoIntelligence has nothing at all whatever to do with compute.
- oefnak 2y agoUnless you're a dualist who believes in a magic spirit, I cannot understand how you think that's the case. Can you please explain?
- freehorse 2y agoIntelligence is about learning from few examples and generalising to novel solutions. Increasing compute so that exploring the whole problem space is possible is not intelligence. There is a reason the actual ARC-AGI price has efficiency as one of the success requirements. It is not so that the solutions scale to production and whatnot, these are toy tasks. It is to help ensure that it is actually an intelligent system solving these. So yeah, the o3 result is impressive but if the difference between o3 and the previous state of art is more compute to do a much longer CoT/evaluation loop, I am not so impressed. Reminder that these problems are solved by humans in seconds, ARC-AGI is supposed to be easy.
- lambdaphagy 2y agoPhilosophy of mind is the branch of philosophy that attempts to account for a very difficult problem: why there are apparently two different realms of phenomena, physical and mental, that are at once tightly connected and yet as different from one another as two things can possibly be. Broadly speaking you can think that the mental reduces to the physical (physicalism), that the physical reduces to the mental (idealism), both reduce to some other third thing (neutral monism) or that neither reduces to the other (dualism). There are many arguments for dualism but I’ve never heard a philosopher appeal to “magic spirits” in order to do so. Here’s an overview: https://plato.stanford.edu/entries/dualism/ https://plato.stanford.edu/entries/dualism/
- otabdeveloper4 2y agoDualism has nothing to do with it. There are more things on heaven and earth then just computable functions in the mathematical sense. (In fact, the very idea of "computable functions" was invented to narrow down the space of "all things" to something much smaller, tighter and manageable. And now we've come full circle and apparently everything in the universe is a computable function? Well, if all you have is a hammer, I guess everything must necessarily look like a nail.)
- TechDebtDevin 2y agoBatteries..
- deleted 2y ago[deleted]
- pera 2y agoMaybe AGI as a goal is overvalued: If you have a machine that can, on average, perform symbolic reasoning better than humans, and at a lower cost, that's basically the end game, isn't it? You won capitalism.
- harrall 2y agoRight now I can ask an (experienced) human to do something for me and they will either just get it done or tell me that they can’t do it. Right now when I ask an LLM… I have to sit there and verify everything. It may have done some helpful reasoning for me but the whole point of me asking someone else (or something else) was to do nothing at all… I’m not sure you can reliably fulfill the first scenario without achieving AGI. Maybe you can, but we are not at that point yet so we don’t know yet.
- raincole 2y agoYou do need to verify humans work though. The difference, to me, is that humans seem to be good at canceling each other's mistakes when put in a proper environment.
- harrall 2y agoNot with the same depth. I might ask a friend to drop off a letter and I might verify that they did it, but I don’t have to verify that they didn’t mistake a Taco Bell or a dumpster as the post office. It’s very scary to ask a friend to drop off a letter if the last scenario is even 1% within the realm of possibility.
- pera 2y agoIt's not clear to me whether AGI is necessary for solving most of the issues in the current generation of LLMs. It is possible you can get there by hacking together CoTs with automated theorem provers and bruteforcing your way to the solution or something like that. But if it's not enough then maybe it might come as a second-order effect (e.g. reasoning machines having to bootstrap an AGI so then you can have a Waymo taxi driver who is also a Fields medalist)
- 8n4vidtmkvmk 2y agoI thought you were going to say that now we're back to bigger-than-room sized computers that cost many millions just to perform the same tasks we could 40 years ago. I of course mean we're using these LLMs for a lot of tasks that they're inappropriate for, and a clever manually coded algorithm could do better and much more efficiently.
- arthurcolle 2y agojust ask the LLM to solve enough problems (even new problems), cache the best, do inference time compute for the rest, figure out the best/ fastest implementations, and boom, you have new training data for future AIs
- owenpalmer 2y ago> cache the best How do you quantify that?
- martinkallstrom 2y ago"Assume the role of an expert in cache invalidation..."
- DyslexicAtheist 2y ago"one does not just assume", "because the hardest problems in Tech are Johnny Cash invalidations" --Lao Tzi
- Terr_ 2y ago> "Those who invalidate caches know nothing; Those who know retain data." These words, as I am told, were spoken by Lao Tzi. If we are to believe that Lao Tzi was himself one who knew, why did he erase /var/tmp to make space for his project? -- Poem by Cybernetic Bai Juyi, "The Philosopher [of Caching]"
- pavlov 2y ago
- nopinsight 2y agoMany of humans' capabilities are pretrained with massive computing through evolution. Inference results of o3 and its successors might be used to train the next generation of small models to be highly capable. Recent advances in the capabilities of small models such as Gemini-2.0 Flash suggest the same. Recent research from NVIDIA suggests such an efficiency gain is quite possible in the physical realm as well. They trained a tiny model to control the full body of a robot via simulations. --- "We trained a 1.5M-parameter neural network to control the body of a humanoid robot. It takes a lot of subconscious processing for us humans to walk, maintain balance, and maneuver our arms and legs into desired positions. We capture this “subconsciousness” in HOVER, a single model that learns how to coordinate the motors of a humanoid robot to support locomotion and manipulation." ... "HOVER supports any humanoid that can be simulated in Isaac. Bring your own robot, and watch it come to life!" More here: https://x.com/DrJimFan/status/1851643431803830551 https://x.com/DrJimFan/status/1851643431803830551 --- This demonstrates that with proper training, small models can perform at a high level in both cognitive and physical domains.
- bigprof 2y ago> Similarly, many of humans' capabilities are pretrained with massive computing through evolution. Hmm .. my intuition is that humans' capabilities are gained during early childhood (walking, running, speaking .. etc) ... what are examples of capabilities pretrained by evolution, and how does this work?
- nopinsight 2y agoThe brain is predisposed to learn those skills. Early childhood experiences are necessary to complete the training. Perhaps that could be likened to post-training. It's not a one-to-one comparison but a rather loose analogy which I didn't make it precise because it is not the key point of the argument. Maybe evolution could be better thought of as neural architecture search combined with some pretraining. Evidence suggests we are prebuilt with "core knowledge" by the time we're born [1]. See: Summary of cool research gained from clever & benign experiments with babies here: [1] Core knowledge. Elizabeth S. Spelke and Katherine D. Kinzler. https://www.harvardlds.org/wp-content/uploads/2017/01/SpelkeKinzler07-1.pdf https://www.harvardlds.org/wp-content/uploads/2017/01/Spelke...
- Existenceblinks 2y agoHonestly, it doesn't need to be local, API is some 200ms away is ok-ish, make it 50ms it will be practically usable for every majority of interaction.
- deleted 2y ago[deleted]