9 ms·

>Live systems in nature seem to solve similar problems with way less compute available Do they really? They're certainly more energy-efficient in business-as-u

• pdimitar 3y ago
  You are ignoring the fact that a toddler, once their musculature develops, is able to learn to walk after several tries. Show me a humanoid robot that can do that.
  • Drakim 3y ago
    To be fair a toddler's brain isn't "offline" while it's musculature develops, they kick and move their limbs constantly, and then later crawl and and even stand, before they finally take the first walking steps. It's just objectively wrong to say a toddler goes from zero to walking in "several tries".
    • pdimitar 3y ago
      Sure, not several. Not several trillions either though. As mentioned in a sibling comment, HN anthropomorphizes AI too much. And is too optimistic about it. I just don't see the results and the value, people trip up all over themselves to congratulate themselves and the researches, yet 99.99% of the problems in the world persist. I am one of these a-holes that wants to see results when money are invested. It still comes as a shock to some apparently.
      • flangola7 3y ago
        Blank checks are flying like it is the dot com era again. Money is most definitely being invested in it.
      • pixl97 3y ago
        I mean, if we're taking the random walk that nature took to get human level intelligence from the first multicellular organisms you'll have to wait 600 million years or so. AI takes a shortcut around having a timeline of a trillion different versions history optimizing the learning algorithm and putting the successful ones in a timeline, but at a cost of trillions of operations to teach it. TL;DR, the problem space is huge and evidently must be iterated.
        • pdimitar 3y ago
          You are extremely generously assuming that AI evolution has a direction and a purpose/goal like the organic one has. It doesn't. We are steering it. And I doubt that we know better than the natural evolution.
          • pixl97 3y ago
            Natural evolution has no direction itself, huge amounts of it are negative. It just happens those with positive traits get to hang around and breed some more.
            • pdimitar 3y ago
              Well, you are in fact correct, yep.
  • M4v3R 3y ago
    > You are ignoring the fact that a toddler, once their musculature develops, is able to learn to walk after several tries. It's definitely not several tries. I've actually witnessed that very recently with my daughter. It takes a child literally several thousand attempts and falls (around a hundred falls per day) and 1,000 hours of dedicated practice, before they can make their first step without holding anything [1]. And a lot more until they can walk reliably. [1] Source: Becoming You, S1E2, from 21:30
  • garblegarble 3y ago
    At that point, the toddler's still run years of self-reinforced learning (as well as learning supported by parents, and observing other humans solving the problem) on controlling their muscles... and again, think of the computational power expended over that time (by a computational approach that's been highly-optimised over millions of years of trial-and-error recursive descent style). Our intellect and capabilities are the result of a truly vast amount of computation in the grand scheme of things. Don't get me wrong, I'm not claiming that Machine Learning is as generally capable as animals/humans, and I don't entirely disagree with the OP (i.e. I don't know if our current approach has a chance of scaling to human-level capabilities). I just don't think computation-wise it compares that badly to animals, considering that it's the result of a few decades of work, largely on repurposed silicon
    • cgio 3y ago
      Human is not the most efficient learner for walking. See other animals, e.g. deer or elephant, that walk straight after being born. Relevant since the discussion is not about human learning but animal learning in nature.
      • jimbokun 3y ago
        Isn't that just biped vs quadruped difference in skill required to walk?
        • notahacker 3y ago
          Nope, puppies are walking within the first month of their life, a stage where human babies have barely any voluntary limb control and no apparent desire to move. The puppies are benefitting from a different type of "learning" process of course, in having genetic instincts tuned by many generations of ancestors who had good enough genetic wiring of motor capabilities and instincts sufficiently well attuned to their environments to survive. Humans have weaker priors and reach physical and mental maturity much more slowly but ultimately achieve greater mastery of their environment by learning it from first principles or communication with other humans (there may be a tradeoff, and of course humans are so good at protecting their young over years of infancy the early instincts aren't that important to retain)
  • lynx23 3y ago
    A toddler capable of walking is still miles away from the topic at hand, a LLM capable of processing relatively complex text. It is well know that humans have a particularily long training period compared to other animals. So I dont understand why you are bringing up the walking-training. It seems quite unrelated.
    • pdimitar 3y ago
      Because humans learn from much less samples sizes than "AI". The overly optimistic point of view on "AI" is over-represented in HN and that makes the discussion unbalanced and puts nearly everyone in it in a filter bubble. That's not OK.
      • pixl97 3y ago
        Again, as per my previous response, you're missing the optimization timeline and how long it took nature to do it. "Walking" at least on dry land has been around 420 million years or so. The locomotion part of the learning algorithms has been around a very, very long time, and it's going to be very hard to duplicate the efficiency that nature brute forced. Nature has not hype optimized human level intelligence. Really human intelligence is at the limits of birth. Our heads can't be any larger at birth or we'll kill the mother too much of the time. Then the human body has a power budget that's been the defining factor of human/animal survival for just about forever. Humans were already 'smart' before we invented agriculture and excess fuel storage. AI takes that a step further. We're not using reversible computing so the heat generated is pretty high. We've not been doing it very long and our algorithms are not optimal in any sense. But what we do have that nature does not is nearly unlimited energy, nearly unlimited cooling capacity, and a lot of computer systems working on the problem at once. In an evolutionary sense we're doing closer to the bacteria method, we have a lot of different experiments going on at once and at least a few of them are likely to be fruitful. Personally I hope the AI development is no insanely rapid. Rapid and powerful changes in society can cause as many problems as they find fixes for, and the pace of society change is not that fast.
        • pdimitar 3y ago
  • sebzim4500 3y ago
    Not sure about humanoids but there are RL algorithms that can learn to walk from scratch in a quadruped in minutes.
    • pdimitar 3y ago
      Well that now is a proper technical revolution. The rest seems to mostly justify swallowing 9-digit numbers from investors from where I am standing, without the beneficiaries being able to show much for it.
  • burnished 3y ago
    You're missing all of the subskills that are developed along the way. They don't just grow, braindead, and twitch a couple times until they get the hang of walking. Its a joy to watch a child grow up, but also its super interesting watching them figure out the most basic shit. Would highly recommend if you get the opportunity.
    • pdimitar 3y ago
      I would love it if AI is raised like that but so far it seems to be not the case. :|
      • YeGoblynQueenne 3y ago
        Turing wrote about "the child machine" in his "Intelligent machinery" paper. See: Bringing Up Turing’s ‘Child-Machine’ by Susan G. Sterrett https://link.springer.com/chapter/10.1007/978-3-642-30870-3_71 https://link.springer.com/chapter/10.1007/978-3-642-30870-3_...
• chongli 3y ago
  86 billion (or 600+ trillion) times 20 years is a lot of computation What is that in kilowatt hours? Human brains are remarkably energy-efficient with their compute. We should get credit for that. Comparing one of us to an AI being trained at a data centre with the energy budget of a small city isn’t really fair, is it?
  • deleted 3y ago
    [deleted]
  • garblegarble 3y ago
    Seems that ~100W is regularly quoted as the average human energy output, so that'd be something like 18 MWh for 20 years (assuming 100% efficiency of food input). I suppose there's also the energy cost of all the training infrastructure (daycare, school teachers, homes, transportation, etc.), and also all the energy consumption of all humans that have come before and built those cities and knowledge infrastructure (although LLMs sit atop that anyway), and then the millions (billions?) of generations of creatures going back to the earliest organisms, who have been optimising how cells work to lower their energy cost. For that setup cost, you get a single-threaded human, starting to specialise in a single field. They will work in that field for around 7.5 hours every weekday, with ~20 days of holiday a year, for around 45 years, and then retire. Llama v2 70B cost 1,720,320 GPU-hours[1] at 400W, so 688MWh. Once trained, it can be run 24/7, and you can spin up as many instances as you want on much lower spec hardware. That model produces output faster than a human while consuming around ~30W on my Macbook Pro. Now, I know we're in pretty shaky spherical cow territory here, comparing a human (albeit a highly educated one) to Llama v2 in logical reasoning... but consider that this is the state of the art in generally-available LLMs after a few decades of research into machine learning & we're using repurposed silicon to compute vs the amazing complexity and physics-leveraging approach of human neurons... and the training cost is only 1 order of magnitude off humans. Again, I'm not disagreeing with the general point of the OP, I'm not saying that the models we're using right now are the best/right ones (or that the hardware they're running on are the most efficient way of executing them), but I don't think the energy efficiency gap is actually all that high considering Edit: If you look at the total footprint (considering not just the efficiency of the neurons, but the whole animal) the figures are very close - the llama model card indicates that v2 70B caused the emission of ~300T of CO2, and an average human in the US emits 16T of CO2 a year, so a human would emit ~320T of CO2 in 20 years. I assume children don't have as high a CO2 output, but even so it seems like it's the same order of magnitude. 1: https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md#hardware-and-software https://github.com/facebookresearch/llama/blob/main/MODEL_CA...
• nine_k 3y ago
  Indeed, the human brain has much more logical parts. They all run at sub-MHz speeds though, and don't seem to be using matrix multiplication (which is still worse than quadratic with the best algorithms we have). Also, human brains do a lot more than language processing. LLMs are definitely touching something very important about intelligence, much like counting sticks touches something very important about numbers. But the real power of numbers is unleashed with the invention of digits and positional notation, a different representation that makes things literally exponentially easier. I hope there is a transition step from what we do now using billion-parameter models, to some better representations, more compact and thus more powerful. Also I hope that ready-made logic and efficient numerical computation can be connected to the learning systems more directly, not taught painstakingly from first principles, much like vision is relatively directly wired into the human brain.