4 ms·
you say obviously doesn't. These language models do indeed work by computing a distribution over all possible next words given the previous words using transfor
by fleischhauf 4y ago
you say obviously doesn't. These language models do indeed work by computing a distribution over all possible next words given the previous words using transformers, and it seems using enough training data and compute gives you the results we see. Everyone I know is completely surprised that it works so well by just adding more data and compute (and probably lots of training tricks)..
- mpweiher 4y ago> using enough training data and compute gives you the results we see. I think this is key. We don't have a good intuition for the truly staggering amount of data and compute that goes into this. An example that we have come to terms with is weather forecasting: weather models have distinctly super-human capabilities when it comes to forecasting the weather. This is due to the amount of compute and data they have available, neither of which a human mind can come close to matching. We have gotten used to this.