My understanding is, if you have 175 billion parameters of 16-bit values that all effectively transact (eg, multiply) together, the realm of possibility is 175b

My understanding is, if you have 175 billion parameters of 16-bit values that all effectively transact (eg, multiply) together, the realm of possibility is 175b^65536; really rather a large number of encodable potentials. The length and number of probability chains that can be discovered in such a space is therefore sufficient for the level of complexity being analysed and effectively "encoded" from the source text data. Which is why it works. Obviously, as the weights become fixed on particular values by the end of training, not all of those possibilities are required. But they are all in some sense "available" during training, and required and so utilised in that sense. Think of it as expanding the corpus as water molecules into a large cloud of possible complexity, analysing to find the channels of condensation that will form drops, then compress it by encoding only the final droplet locations.