2 ms·
Well, I'm not sure how valid is the entire thing. It's not clear how much cross-polination there is between those datasets and the ChatGPT training set (IMO, if
by marcosdumay 3y ago
Well, I'm not sure how valid is the entire thing. It's not clear how much cross-polination there is between those datasets and the ChatGPT training set (IMO, if I was training something like it, I'd take any specialized dataset I can find), or how useful answering those questions is to any real task.
Also, it's not only the prompt engineering. Training ChatCPT was a decades long process, with billions of dollars invested, and it takes a small cluster to run the thing. How would the other models compare if given similar resources?
Besides that, on the context of this thread, the bitter lesson has absolutely no relation to the specialized vs. general purpose model dichotomy. It's about the underlining algorithms, that for this article are all very similar. (And it's also not a known truth, and looking less and less as an absolute truth as deep learning advances, so take it with a huge grain of salt.)