11 ms·

> but LLMs do solve problems that people thought were extremely difficult to solve ten years ago Agreed. I would have laughed you out of the room 5 years ago i

• gizmo686 2y ago
  > It turns out that the larger these models get, the more unexpected emergent capabilities they have, so I'm mostly in the camp of thinking AGI is just a matter of time and resources. AI research has a long history of people saying this. Whenever there is a new fundamental improvement, it looks like you can just keep getting better results by throwing more resources at it. However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance. LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them. However, unlike most other AI systems we have developed, LLMs are actively polluting their data pool, so this intitial generation of LLMs are probably going to have the best data set of any that we ever develop. Of course, today's data will continue to be available, but will loose value as it ages.
  • scotty79 2y ago
    Currently we are throwing everything at LLMs and hope good things stick. At one point we might use AI to select best training data from what's available to best train the next AI.
    • oco101 2y ago
      "That is just one very narrow task that basically anyone can do regardless of intelligence or talent. It takes less than a year to train a distracted 16 year old to do it, but three decades to train an AI and even then you probably need to hand tune it for specific locations because it won’t know what do in unusual road layouts." It takes three decades to train an AI, but of course, like everything humanity does is not linear, it is exponential. Before the Wright brothers, it was believed that powered, controlled, heavier-than-air flight was impossible. Then, in 1903, they achieved the first successful airplane flight, which lasted 12 seconds and covered 120 feet. By 1914, the first commercial flight covered approximately 21 miles and took about 23 minutes. This is just one example and I don't see why AI should be any different
      • thebigspacefuck 2y ago
        On the other hand, airplanes haven’t really improved that much since the 737 was introduced in 1966 and have even gotten worse in some ways (737 Max). We actually don’t know what the AI equivalent of Wright brothers plane is and what the 737 is. Maybe Eliza, 60 years ago, was the Kitty Hawk flight and ChatGPT 4 is the 737.
        • fhehdbehevfjhf 2y ago
          I’m not sure I would say that the 737 Max has gotten worse in “some ways”, by which I assume you are referring to safety. New aircraft models up to about the 1990s frequently had serious design defects that resulted in fatalities, including the 737 (most notably related to the rudder).
          • falcor84 2y ago
            Indeed, we've seen a slight regression in safety, but definitely not to 1980s levels.
      • perilunar 2y ago
        > Before the Wright brothers, it was believed that powered, controlled, heavier-than-air flight was impossible. Not by the many people around the world working on the problem, and their supporters. Also, “powered, controlled, heavier-than-air flight” is exactly was birds, bats, and many insects do — it clearly wasn’t impossible. You need to add a couple more adjectives to qualify what the Wright’s achieved.
  • etempleton 2y ago
    It is hard for me to not think that given the current approach it will always be almost there, but not quite good enough to be more than a nice assistant or tool that has to be meticulously built and maintained by a company. Take, for example, self driving vehicles, we have been 5 years out for 25 years. That is not to say there has not been significant progress, but to go from a really powerful driver assist to full self driving has still not really come to full fruition and that is kind of a best case scenario for AI/ machine learning. That is just one very narrow task that basically anyone can do regardless of intelligence or talent. It takes less than a year to train a distracted 16 year old to do it, but three decades to train an AI and even then you probably need to hand tune it for specific locations because it won’t know what do in unusual road layouts. I think we will get there with driverless vehicles, but only because it is being tailored by humans to handle all of the edge cases. Self driving cars, if truly self driving with no user intervention, will of course be worth it in the end. The trillions of dollars spent will eventually add countless dollars in added efficiency and unlimited revenue for those who get there first, but how many applications can we really say that is true for? That the decades of development and training it takes to remove humans will be worth the initial investment? I guess my contention is that the current development path seems unlikely to ever achieve AGI and so instead you will have to do heavy customization to get anything that is much more useful than what we have now.
    • andrewchambers 2y ago
      I feel like your point would be better if there weren't already self driving cars around like waymo.
      • ileonichwiesz 2y ago
        Except they’re not fully self-driving, they’re all at least partially driven by an underpaid contractor in a remote warehouse.
      • etempleton 2y ago
        That is kind of my point. They exist only in a few select locations. If they were truly self driving Google would be rapidly rolling them out around the World, but they are not.
  • jebarker 2y ago
    > LLMs have an additional problem related to training data. We are already throwing all the data we can get our hands on at them. I think it's plausible we'll see a breakthrough in data efficiency that helps here. Humans are an existence proof that language is learnable through much less data and, in theory, facts should only need to be seen once. LLMs in their current form seem very data inefficient.
    • deleted 2y ago
      [deleted]
    • deleted 2y ago
      [deleted]
    • dartos 2y ago
      You can’t predict breakthroughs. We probably will see a breakthrough, but there’s no more reason to believe it’ll happen tomorrow than 100 years from now.
      • jebarker 2y ago
        I didn't make a prediction, let alone a timeline - I just said it's plausible. I suppose I did say "we'll see" so I'm implicitly predicting within a few decades.
  • falcor84 2y ago
    > However, eventually we end up reaching a point where throwing more resources at it stops meaningfully improving performance. Honestly asking - why? It's been my understanding that based on the Universal Approximation Theorem, given sufficient resources, a deep learning neural net can approximate any function to an arbitrary degree of accuracy. Is there any other theorem or even conjecture that would lead us to believe that the progress that we're seeing will slow down? Or is it just that you're claiming that we/it would run out of physical resources before reaching AGI? As for training data, as I see it, with the wide deployment of AIs, and gradually of AI-driven robots, they'll soon be able to "takeoff" and rely primarily on the live data that they are collecting directly.
    • deleted 2y ago
      [deleted]
    • gizmo686 2y ago
      The universal approximation therorem is not as powerful as it sounds. Polynomials essentially satisfy it as well [0], the only hiccup being that the Universal Approximation Theroum is explicitly about neural networks. The UAT is an existence proof, it says nothing about any particular method being capable of constructing such a network. In contrast, with polynomials we have several methods of constructing polynomials that are proven to converge to the desired function. Indeed, polynomials have been widely used as universal approximators for centuries now, and are often amazingly successful. However, polynomials in this context are only good in low degrees, where they are inherently limited in how well they can approximate [1]. Beyond a certain point, increasing your degrees of freedom with polynomial approximators simply does not help and is generally counter productive, even though a higher degree polynomial is strictly more powerful than a lower degree one. Looking at the current generative AI breakthrough, the UAT would say that today's transformer based architecture is no more powerful than a standard neurul net. However, it produces vastly superior results that could simply not be achieved by throwing more compute at the problem. Sure, if you have an infinite dataset and infinite compute, you might have AGI. But at that point , you have basically just replicated the Chinese room thought experiment. [0] See the Stone–Weierstrass theorem [1] They are also used as arbitrary precision approximators, but that is when you compute them analytically instead of interporlating them from data.
• kylecazar 2y ago
  Are capabilities truly 'emerging'? Or are we just observing new competencies/applications we hadn't previously considered.
  • bamboozled 2y ago
    We've built a machine to complicated, we don't really know what to make of it, fun times.
  • dfilppi 2y ago
    [dead]
• jiggawatts 2y ago
  One thing that keeps me up at night is that the human genome is only 3 giga base pairs, of which only a fraction encodes the design of our brains — and quite inefficiently at that, through layers of indirections. That’s sufficient information to produce a system that can learn to think like us! Not just learn but efficiently, with far less input data needed than any current LLM. Literally just a couple of decades of video and audio, only a small fraction of that text! From what I can tell, the human brain achieves this through scale alone. There’s nothing else that can explain the observed learning capability. The genome is too small to encode learned weights, and we don’t clone our parents’s brains in development. It’s possible, and a meat computer can do it. Replicating this in silicon is just a matter of time, and it might require only scale and nothing else.
  • chipsrafferty 2y ago
    Pretty sure it's not scale but sensors. We have sensors for vision, sound, taste, temperature, texture, etc that are constantly observing the world and affecting not only our current behavior but changing us in real time.
    • jiggawatts 2y ago
      You can feed an LLM of typical size (~1 TB) all the video and audio you can find, but it won't turn into a human. I suspect but can't prove that even if you wired up a bunch of other sensors, gave it a robot body, and let it "explore" the world on its own, that would still be woefully insufficient. A gazelle just days(!) old can control its body and four legs sufficiently well to outrun a cheetah. This is a complex motor-control loop involving all of its senses. Compare that to Tesla's autopilot training system, which uses many millions of hours of training data and still struggles to move a car... slowly. The equivalent would be a training routine that can take just a handful of days of footage and produce an AI that can win a car race. There's something magical about neural networks when scaled up to brain sizes. From what I gather, there's little else encoded in the genome except for the high-level pattern of wiring, the equivalent to the PyTorch model configuration.
      • 2y ago
• cmsj 2y ago
  General Intelligence is constantly learning. Transformer based models can't do that. All bets are still firmly on.