> Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40

> Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte. Thanks, I was looking for that estimate but could not quickly find it. (Although are you sure about that 'petabyte' figure? I see 1e13 * 5 = 50e13 = 50 TB?) I love these sorts of bulk comparisons, since they let us start to reason about how much humans learn from "each bit of experience" versus LLMs. On one hand humans process a lot of sensory information (most of it by discarding, mind), but on the other humans do a lot of 'active learning' whereby experience is directly related to actions. LLMs don't have the latter, since they're trained on passive feeds of tokens.