5 ms·

>It feels like computers should be able to get much more out of what is already available I mean why ? It took millions of years of optimization for humanity t

• kelseyfrog 3y ago
  Because humans with less language data outperform LLMs with more language data. This either says we need better models, not more data. Or, the human ability to be multi-modal augments our ability to perform language tasks in which case, we need to pump LLMs with much more image and video input than we currently do.
  • famouswaffles 3y ago
    The point I'm making is that humans do not in fact have "less language data". We're heavily predisposed to learning languages. We don't start with random weights. GPT has no such predisposition.
    • kelseyfrog 3y ago
      Better init and architecture aren't what people think of when they think of "giving models more data" - they mean a larger training set.
      • famouswaffles 3y ago
        You're not making much sense here. The better init comes from training and little else. GPT needing lots of training data doesn't mean we need a better architecture. You would expect it to have a lot of training because humans have a lot of training too, spanning millions of years..
        • kelseyfrog 3y ago
          Init means the distribution of weights prior to training. Human training begins at birth. Evolution might result in better architecture and init(inductive biases), but that's a separate thing than training.
          • HDThoreaun 3y ago
            > Human training begins at birth. Is this true? My understanding is that people are born with many pre trained weights. Was the evolutionary convergence of those weights not itself training?
            • kelseyfrog 3y ago
              > Was the evolutionary convergence of those weights not itself training? No, inductive biases are not training. I'm saying that better models(ie: better inductive biases) or non-language data is needed to advance LLMs and somehow we've arrived at "evolution is training." I'm not sure how that's relevant to the point.
              • famouswaffles 3y ago
                Evolutionary training is not just inductive bias. They're not comparable at all lol. And the more inductive bias we've shoved into models, the worse they've performed. Transformers have a lot less bias than either RNNs or CNNs and are better for it. Same story with what preceded both.
                • kelseyfrog 3y ago
                  This is a good time to stop and ask, what point do you think I'm making?
                  • stevenhuang 3y ago
                    Pretty much all of your assertions in this thread and https://news.ycombinator.com/item?id=37797108 https://news.ycombinator.com/item?id=37797108 in particular are what we're disagreeing with.
  • blovescoffee 3y ago
    You're comparing humans - a multimodal model with billions of years of training epochs - to a unimodal language model that's been around for a few months.
    • kelseyfrog 3y ago
      Correct. That's a crucial part if the point I'm making.
    • bamboozled 3y ago
      Yes but heavily inspired by the billion year old model ?
    • quickthrower2 3y ago
      Let alone no hardware squeeze and boundless energy supply through the food chain. Remember for most of the billions of years we were not human and were low powered
• deleted 3y ago
  [deleted]
• ToValueFunfetti 3y ago
  Millions of years of evolutionary computation is a fairly small amount of computational time. LLMs also benefit from decades of neurological computation in that their structure was invented and optimized by humans, which is already orders of magnitude faster than evolution.
• marcosdumay 3y ago
  > If you think you're "starting from scratch", you really aren't. Our genomes have ~600MB, so where are you claiming that humans transmit millions of years of language optimization between generations?
  • zo1 3y ago
    You could store a pretty gigantic graph using 600MB. They're not saying we store the weights, they're saying that we store the "architecture" that you overlay the weights on with training.
  • lukeschlather 3y ago
    That 600MB is the result of millions of years of optimization. For every human genome that exists today there are many other genomes which were tried and discarded over the years. It also contains a remarkable amount of compression. Even if you assume that a genome contains a complete template for a human (it likely doesn't) the fact that the compressed version is 600MB doesn't really relate to the unpacked amount of information. Especially since the compression has seen millions of years of optimization.