4 ms·

I'd argue that yours is not a counter argument, but the same thing said by grandparent rephrased. It's obvious that with the same amount of computing power, spe

I'd argue that yours is not a counter argument, but the same thing said by grandparent rephrased. It's obvious that with the same amount of computing power, specialized > generalized. But in 2-5 years, the future generalized models can beat the current specialized ones.

goldenkey 3y ago
The future of this is the same as what happened with PCs. The specificity will initially be thrown away, added back later as an accelerator, and eventually, brought back into the fold as an automatically used accelerator. It all comes full circle as optimizations, hinting, tiering, and heuristics. AI will be used to select the net to automatically load. Nets will be cached, branch predicted etc. The future of AI software and hardware doesn't yet support the scale we need for this type of generalized AI processor (think CPU but call it an AIPU.) And no, GPUs aren't an AIPU, we can't even fit whole some of the largest models on these things without running them in pieces. They don't have a higher level language yet, like C, which would compile down to more specific actions after optimizations are borne (not PTX/LLVM/Cuda/OpenCL.)
marcosdumay 3y ago
No, the GP is stating that the short term advantage will always exist, for fundamental reasons. It's just short term because it has to be reacquired often, not because there stops being an advantage. In 5 years, the specialized model would still beat the generalized ones. They would just be different from the ones today.
- PaulHoule 3y ago
  Couldn't you apply the same prompt engineering tricks to the specialized model that their system beats?
  - marcosdumay 3y ago
    Well, I'm not sure how valid is the entire thing. It's not clear how much cross-polination there is between those datasets and the ChatGPT training set (IMO, if I was training something like it, I'd take any specialized dataset I can find), or how useful answering those questions is to any real task. Also, it's not only the prompt engineering. Training ChatCPT was a decades long process, with billions of dollars invested, and it takes a small cluster to run the thing. How would the other models compare if given similar resources? Besides that, on the context of this thread, the bitter lesson has absolutely no relation to the specialized vs. general purpose model dichotomy. It's about the underlining algorithms, that for this article are all very similar. (And it's also not a known truth, and looking less and less as an absolute truth as deep learning advances, so take it with a huge grain of salt.)

goldenkey 3y ago

The future of this is the same as what happened with PCs. The specificity will initially be thrown away, added back later as an accelerator, and eventually, brought back into the fold as an automatically used accelerator. It all comes full circle as optimizations, hinting, tiering, and heuristics. AI will be used to select the net to automatically load. Nets will be cached, branch predicted etc. The future of AI software and hardware doesn't yet support the scale we need for this type of generalized AI processor (think CPU but call it an AIPU.) And no, GPUs aren't an AIPU, we can't even fit whole some of the largest models on these things without running them in pieces. They don't have a higher level language yet, like C, which would compile down to more specific actions after optimizations are borne (not PTX/LLVM/Cuda/OpenCL.)

marcosdumay 3y ago

No, the GP is stating that the short term advantage will always exist, for fundamental reasons. It's just short term because it has to be reacquired often, not because there stops being an advantage. In 5 years, the specialized model would still beat the generalized ones. They would just be different from the ones today.

PaulHoule 3y ago
Couldn't you apply the same prompt engineering tricks to the specialized model that their system beats?
- marcosdumay 3y ago
  Well, I'm not sure how valid is the entire thing. It's not clear how much cross-polination there is between those datasets and the ChatGPT training set (IMO, if I was training something like it, I'd take any specialized dataset I can find), or how useful answering those questions is to any real task. Also, it's not only the prompt engineering. Training ChatCPT was a decades long process, with billions of dollars invested, and it takes a small cluster to run the thing. How would the other models compare if given similar resources? Besides that, on the context of this thread, the bitter lesson has absolutely no relation to the specialized vs. general purpose model dichotomy. It's about the underlining algorithms, that for this article are all very similar. (And it's also not a known truth, and looking less and less as an absolute truth as deep learning advances, so take it with a huge grain of salt.)