3 ms·

In order to use scale you need a good architecture. A quadratic sort works just fine for 10 elements, the architecture efficiency of faster algorithms isn't ver

In order to use scale you need a good architecture. A quadratic sort works just fine for 10 elements, the architecture efficiency of faster algorithms isn't very relevant until higher scale. > Scale > architecture is the uncomfortable reality that all ML researchers eventually come to terms with. Architecture -> scale. Current LLM architecture doesn't scale very well, you need massive increase in compute, memory and data for tiny improvements. Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics.

coldtea 3y ago
>Architecture -> scale Yeah, "scale > architecture" assumes that those are indepedent, whereas architecture affects scaling reach, needs, and options. Also would a Perceptron with the original architecture do what an LLM does just given scale?
- sigmoid10 3y ago
  >Also would a Perceptron with the original architecture do what an LLM does just given scale? Yes. The answer is yes. This is exactly what the universal approximation theorem [1] says, which was proven decades ago. It is also just another line of reasoning that compute matters more than architecture, but fewer people outside of ML research know about it. And it's also less practical, since the formal theorem has few empirics to go by for really large models. But mathematically, a scaled perceptron is 100% able to replicate any LLM. [1] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
  - wizzwizz4 3y ago
    Given an LLM, you can construct a perceptron-based model that approximates it, but that doesn't mean there's any obvious way of training a perceptron model that would produce something like an LLM. What the universal approximation theorem tells you is that there's some algorithm you could use to train a (sufficiently-large) perceptron model to behave arbitrarily-closely to the equivalently-trained LLM – which, yeah, of course there is. There's a way of training an array of floating point numbers to do that, too. It's called the transformer architecture. (Your misunderstandings are similar to the sorts of misunderstandings I had before I got in the habit of trying to (abstractly) implement every CS concept I came across. Kudos for trying to reason about AI from first principles, but you have to do a lot of reasoning before you stop being much wronger than the empiricists.)
sigmoid10 3y ago
>Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics. Oh really? And what would that look like? People have literally been looking at this for decades. And yet we're still using gradient descent with a few tweaks. So I would not bet on anyone coming up with something revolutionary. Especially not while hardware regularly revolutionizes the capabilities of models trained in this ancient way. Chipmakers have always allowed us to overcome the hurdle created by our lack of advances on the training architecture front. No matter whether you look at convnets or transformers.

coldtea 3y ago

>Architecture -> scale Yeah, "scale > architecture" assumes that those are indepedent, whereas architecture affects scaling reach, needs, and options. Also would a Perceptron with the original architecture do what an LLM does just given scale?

sigmoid10 3y ago
>Also would a Perceptron with the original architecture do what an LLM does just given scale? Yes. The answer is yes. This is exactly what the universal approximation theorem [1] says, which was proven decades ago. It is also just another line of reasoning that compute matters more than architecture, but fewer people outside of ML research know about it. And it's also less practical, since the formal theorem has few empirics to go by for really large models. But mathematically, a scaled perceptron is 100% able to replicate any LLM. [1] https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
- wizzwizz4 3y ago
  Given an LLM, you can construct a perceptron-based model that approximates it, but that doesn't mean there's any obvious way of training a perceptron model that would produce something like an LLM. What the universal approximation theorem tells you is that there's some algorithm you could use to train a (sufficiently-large) perceptron model to behave arbitrarily-closely to the equivalently-trained LLM – which, yeah, of course there is. There's a way of training an array of floating point numbers to do that, too. It's called the transformer architecture. (Your misunderstandings are similar to the sorts of misunderstandings I had before I got in the habit of trying to (abstractly) implement every CS concept I came across. Kudos for trying to reason about AI from first principles, but you have to do a lot of reasoning before you stop being much wronger than the empiricists.)

sigmoid10 3y ago

>Likely there are much more efficient ways to train these models, it shouldn't be this hard for them to learn arithmetics. Oh really? And what would that look like? People have literally been looking at this for decades. And yet we're still using gradient descent with a few tweaks. So I would not bet on anyone coming up with something revolutionary. Especially not while hardware regularly revolutionizes the capabilities of models trained in this ancient way. Chipmakers have always allowed us to overcome the hurdle created by our lack of advances on the training architecture front. No matter whether you look at convnets or transformers.