3 ms·
We’re pretty sure the nondeterminism is batching + mixture of experts + contention for specific experts
by QuadmasterXLII 3y ago
We’re pretty sure the nondeterminism is batching + mixture of experts + contention for specific experts
- noduerme 3y agoIf by batching you mean bad code that fails to sort or relies on hardware to best-guess how things sort, then sure, that's called a bug. Also, "we're pretty sure" is rather self-important while also admitting total, abject failure to produce a deterministic result. You shouldn't blame yourself. A lot of people had the same feeling after staking their life on the revolutionary properties of NFTs.
- n2d4 3y agoYour incorrect assumption here is that determinism comes with no tradeoffs. There are a few outsider analyses on the topic, for example, this one on sparse MoEs [1]. If OpenAI uses sparse MoE as described, then determinism would be possible but inefficient. Even if it's not sparse MoE, chances are high that the non-determinism is introduced somewhere purely as a performance optimization. The article speculates that OpenAI knows this well and hides it to protect the model internals. [1] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/