3 ms·

ANNs aren't top down either though. The architecture isn't as important as people think. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ http

ANNs aren't top down either though. The architecture isn't as important as people think. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat... Even then, the architectures we use are essentially stumbled on. This is alchemy, not modern chemistry.

p1esk 3y ago
A quote from the link you posted: "Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset." Yet all four of these examples use the same model architecture (transformers).
- famouswaffles 3y ago
  "Everything else is a means to an end in efficiently delivery compute..." Without tweaking anything (so not RWKV), you could train a GPT level RNN...if you had the compute to burn.
  - p1esk 3y ago
    We don't know that. No one has demonstrated it. It's very likely that at larger scales, for a given amount of compute you cannot train a traditional RNN to be as a good as a transformer.
    - famouswaffles 3y ago
      We are saying the same thing. Transformers are more compute efficient than RNNs. Nobody is denying that but the switch from RNNs didn't precede some performance wall(i.e it's not like we were training bigger RNNs that weren't getting better). We use Transformers today in large part because they got rid of recursion and in effect could massively parallelize compute.
HarHarVeryFunny 3y ago
Sure - one only needs a sufficiently powerful learning architecture to extract the signal from the data, and assuming that is the case then what is learnt depends on the data not the architecture. That said, I don't think that LLMs are yet close to fully utilizing the training data. e.g. they are prone to hallucinating due to not thinking ahead and backing themselves into a corner where they have to say something. One obvious improvement for that one is more forward looking prediction (i.e. "thinking ahead" - engage brain before opening mouth), which for LLMs can be addressed by tree of thought rollouts and RL learning. So, while architecture is not important if all architectures are equally powerful (in which case you'll learn what's available to learn), it certainly does matter if not all are fully up to the job, as it would appear none currently are.
edgyquant 3y ago
ANNs are a predesigned topology so maybe you can make a case they aren’t “top down” but they definitely aren’t an emergent architecture.

p1esk 3y ago

A quote from the link you posted: "Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset." Yet all four of these examples use the same model architecture (transformers).

famouswaffles 3y ago
"Everything else is a means to an end in efficiently delivery compute..." Without tweaking anything (so not RWKV), you could train a GPT level RNN...if you had the compute to burn.
- p1esk 3y ago
  We don't know that. No one has demonstrated it. It's very likely that at larger scales, for a given amount of compute you cannot train a traditional RNN to be as a good as a transformer.
  - famouswaffles 3y ago
    We are saying the same thing. Transformers are more compute efficient than RNNs. Nobody is denying that but the switch from RNNs didn't precede some performance wall(i.e it's not like we were training bigger RNNs that weren't getting better). We use Transformers today in large part because they got rid of recursion and in effect could massively parallelize compute.

HarHarVeryFunny 3y ago

Sure - one only needs a sufficiently powerful learning architecture to extract the signal from the data, and assuming that is the case then what is learnt depends on the data not the architecture. That said, I don't think that LLMs are yet close to fully utilizing the training data. e.g. they are prone to hallucinating due to not thinking ahead and backing themselves into a corner where they have to say something. One obvious improvement for that one is more forward looking prediction (i.e. "thinking ahead" - engage brain before opening mouth), which for LLMs can be addressed by tree of thought rollouts and RL learning. So, while architecture is not important if all architectures are equally powerful (in which case you'll learn what's available to learn), it certainly does matter if not all are fully up to the job, as it would appear none currently are.

edgyquant 3y ago

ANNs are a predesigned topology so maybe you can make a case they aren’t “top down” but they definitely aren’t an emergent architecture.