2 ms·

If you look at the original transformer paper, it was not a revolution. It was yet another few % improvement over state of the art. The reason it is still aroun

If you look at the original transformer paper, it was not a revolution. It was yet another few % improvement over state of the art. The reason it is still around is because it was actually a step back from recurrent networks and re-allowed scaling once more. Recurrent models used to be the big hope towards AGI for a long time (and some people still cling to them), but they simply turned out to be too limited when you want to scale up. And in the end, it's all about scaling compute. No matter which way you look at it. This is the true constant over the past 60 years. >GPT-4 remains incapable of the things I claimed GPT models were incapable of back in the GPT-2 days. Which are...? Anything we have seen so far from scaling LLMs is that capability is purely an issue of training/fine-tuning. It's not a question of the architecture.

wizzwizz4 3y ago
My money was on recurrent networks being the next big thing, then I looked at Attention is All You Need, and I went "wow, that's a good idea". I don't remember ever looking at the benchmarks. Quantitative improvement over state-of-the-art is nigh-irrelevant. You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it (with a few architecture / dataset decisions to push it in a particular direction). That's not the kind of thing I consider an improvement: it doesn't make anything possible that wasn't already possible. The measure of an AI system is fundamentally about doing stuff, but primarily about doing stuff efficiently. Computers have always been able to play chess by exhaustive brute-force search, given a galaxy-sized GPU and a few million years.
- sigmoid10 3y ago
  >You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it Thank you for confirming my point. Because this is what it's all about. It might not seem like much if you only think on timeframes of startups, but in the long run this will beat any architecture improvement. Yeah, maybe you can come up with a fancy revolutionary design and achieve instant 50% improvement over SOTA. You will get all the attention of the media and prizes and stuff. But how often does that happen? On the other hand, you could also just silently wait one or two generations of Nvidia cards and get the same thing. And even better, this also works when there is absolutely no improvement in architectures by anyone. This is only highlighted by transformers, because they came out in 2017, but only now have they been able to solve e.g. the bar exam. Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with.
  - Jensson 3y ago
    In order to use scale you need a good architecture. A quadratic sort works just fine for 10 elements, the architecture efficiency of faster algorithms isn't very relevant until higher scale. > Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with. Architecture -> scale. Current LLM architecture doesn't scale very well, you need massive increase in compute, memory and data for tiny improvements. Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics.
    - coldtea 3y ago
      >Architecture -> scale Yeah, "scale > architecture" assumes that those are indepedent, whereas architecture affects scaling reach, needs, and options. Also would a Perceptron with the original architecture do what an LLM does just given scale?
      - sigmoid10 3y ago
        >Also would a Perceptron with the original architecture do what an LLM does just given scale? Yes. The answer is yes. This is exactly what the universal approximation theorem [1] says, which was proven decades ago. It is also just another line of reasoning that compute matters more than architecture, but fewer people outside of ML research know about it. And it's also less practical, since the formal theorem has few empirics to go by for really large models. But mathematically, a scaled perceptron is 100% able to replicate any LLM. [1] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
        wizzwizz4 3y ago
        Given an LLM, you can construct a perceptron-based model that approximates it, but that doesn't mean there's any obvious way of training a perceptron model that would produce something like an LLM. What the universal approximation theorem tells you is that there's some algorithm you could use to train a (sufficiently-large) perceptron model to behave arbitrarily-closely to the equivalently-trained LLM – which, yeah, of course there is. There's a way of training an array of floating point numbers to do that, too. It's called the transformer architecture. (Your misunderstandings are similar to the sorts of misunderstandings I had before I got in the habit of trying to (abstractly) implement every CS concept I came across. Kudos for trying to reason about AI from first principles, but you have to do a lot of reasoning before you stop being much wronger than the empiricists.)

wizzwizz4 3y ago

My money was on recurrent networks being the next big thing, then I looked at Attention is All You Need, and I went "wow, that's a good idea". I don't remember ever looking at the benchmarks. Quantitative improvement over state-of-the-art is nigh-irrelevant. You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it (with a few architecture / dataset decisions to push it in a particular direction). That's not the kind of thing I consider an improvement: it doesn't make anything possible that wasn't already possible. The measure of an AI system is fundamentally about doing stuff, but primarily about doing stuff efficiently. Computers have always been able to play chess by exhaustive brute-force search, given a galaxy-sized GPU and a few million years.

sigmoid10 3y ago
>You can drive that up arbitrarily high, for almost any given metric, just by throwing more compute at it Thank you for confirming my point. Because this is what it's all about. It might not seem like much if you only think on timeframes of startups, but in the long run this will beat any architecture improvement. Yeah, maybe you can come up with a fancy revolutionary design and achieve instant 50% improvement over SOTA. You will get all the attention of the media and prizes and stuff. But how often does that happen? On the other hand, you could also just silently wait one or two generations of Nvidia cards and get the same thing. And even better, this also works when there is absolutely no improvement in architectures by anyone. This is only highlighted by transformers, because they came out in 2017, but only now have they been able to solve e.g. the bar exam. Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with.
- Jensson 3y ago
  In order to use scale you need a good architecture. A quadratic sort works just fine for 10 elements, the architecture efficiency of faster algorithms isn't very relevant until higher scale. > Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with. Architecture -> scale. Current LLM architecture doesn't scale very well, you need massive increase in compute, memory and data for tiny improvements. Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics.
  - coldtea 3y ago
    >Architecture -> scale Yeah, "scale > architecture" assumes that those are indepedent, whereas architecture affects scaling reach, needs, and options. Also would a Perceptron with the original architecture do what an LLM does just given scale?
    - sigmoid10 3y ago
      >Also would a Perceptron with the original architecture do what an LLM does just given scale? Yes. The answer is yes. This is exactly what the universal approximation theorem [1] says, which was proven decades ago. It is also just another line of reasoning that compute matters more than architecture, but fewer people outside of ML research know about it. And it's also less practical, since the formal theorem has few empirics to go by for really large models. But mathematically, a scaled perceptron is 100% able to replicate any LLM. [1] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
      - wizzwizz4 3y ago
        Given an LLM, you can construct a perceptron-based model that approximates it, but that doesn't mean there's any obvious way of training a perceptron model that would produce something like an LLM. What the universal approximation theorem tells you is that there's some algorithm you could use to train a (sufficiently-large) perceptron model to behave arbitrarily-closely to the equivalently-trained LLM – which, yeah, of course there is. There's a way of training an array of floating point numbers to do that, too. It's called the transformer architecture. (Your misunderstandings are similar to the sorts of misunderstandings I had before I got in the habit of trying to (abstractly) implement every CS concept I came across. Kudos for trying to reason about AI from first principles, but you have to do a lot of reasoning before you stop being much wronger than the empiricists.)