The architecture is understood, but the specifics about how they calculate which words are high-probability are mostly a mystery. Here’s a good blog post though

The architecture is understood, but the specifics about how they calculate which words are high-probability are mostly a mystery. Here’s a good blog post though: We Found An Neuron in GPT-2 https://clementneo.com/posts/2023/02/11/we-found-an-neuron https://clementneo.com/posts/2023/02/11/we-found-an-neuron If anyone knows of any other research like this, I’d love to read it.