A mental model of GPTs that's pretty successful at predicting their behaviour at the limit of large scale is an N-Gram model trained on a much larger internet t

A mental model of GPTs that's pretty successful at predicting their behaviour at the limit of large scale is an N-Gram model trained on a much larger internet than the real one. If you search the real world's internet for the text ``` You are a helpful and harmless AI assistant. What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again and drank a tea with her in the kitchen. ``` It would be exponentially unlikely to show up even once. But if you could keep re-rolling the internet from whatever probability distribution generated it, eventually (and I mean eventually- every additional token in the prompt will require ~10,000 times more re-rolls of the whole internet) the whole prompt will show up, and you can just grab the next word. An N-Gram model trained on that massive multi-internet will answer the question in a human fashion. (likely, copied from a human writing science fiction).