3 ms·
So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me
by neverokay 2y ago
So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me.
Will compute allow that number to go up? Or is that an optimal number?
- WhitneyLand 2y agoDefinitely has trended upward, there’s no special number. It’s just a matter of how much compute, storage, time to allocate to that part of the architecture.
- causal 2y agoHas it? It's my understanding that GPT-3 was 12,288 and GPT-4 went down to 3072.
- WhitneyLand 2y agoI’m not aware that internal embedding dimension size has been made public for Gpt4 et al. In general for models we know its trended upward, but for sure it’d be interesting to know what they’re using now.