3 ms·

I have not kept up with GPT architecture, other than noticing that other people have noticed that T=0 is clearly not deterministic for these things (and that th

I have not kept up with GPT architecture, other than noticing that other people have noticed that T=0 is clearly not deterministic for these things (and that that results from a bug, not an intentional feature). This much was obvious when the supposedly genius idiots rolled out GPT-3. It's wonderful to see the whole world bend over and just take it up the ass from a bunch of people who can't figure out why their code can't produce the same result twice; but it's quite natural for there to be a big cheerleading section on HN for any new technology that's (1) brilliant in theory, (2) deeply anti-human in practice, and (3) just needs a couple more revisions before it "fixes" a bunch of stuff.

QuadmasterXLII 3y ago
We’re pretty sure the nondeterminism is batching + mixture of experts + contention for specific experts
- noduerme 3y ago
  If by batching you mean bad code that fails to sort or relies on hardware to best-guess how things sort, then sure, that's called a bug. Also, "we're pretty sure" is rather self-important while also admitting total, abject failure to produce a deterministic result. You shouldn't blame yourself. A lot of people had the same feeling after staking their life on the revolutionary properties of NFTs.
  - n2d4 3y ago
    Your incorrect assumption here is that determinism comes with no tradeoffs. There are a few outsider analyses on the topic, for example, this one on sparse MoEs [1]. If OpenAI uses sparse MoE as described, then determinism would be possible but inefficient. Even if it's not sparse MoE, chances are high that the non-determinism is introduced somewhere purely as a performance optimization. The article speculates that OpenAI knows this well and hides it to protect the model internals. [1] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
fl7305 3y ago
> people who can't figure out why their code can't produce the same result twice It is well known that the Nvidia parallel processing optimizations cause non-deterministic results. It's easy to get deterministic results as far as that goes. They've just elected not to do that, since it would run much slower.

QuadmasterXLII 3y ago

We’re pretty sure the nondeterminism is batching + mixture of experts + contention for specific experts

noduerme 3y ago
If by batching you mean bad code that fails to sort or relies on hardware to best-guess how things sort, then sure, that's called a bug. Also, "we're pretty sure" is rather self-important while also admitting total, abject failure to produce a deterministic result. You shouldn't blame yourself. A lot of people had the same feeling after staking their life on the revolutionary properties of NFTs.
- n2d4 3y ago
  Your incorrect assumption here is that determinism comes with no tradeoffs. There are a few outsider analyses on the topic, for example, this one on sparse MoEs [1]. If OpenAI uses sparse MoE as described, then determinism would be possible but inefficient. Even if it's not sparse MoE, chances are high that the non-determinism is introduced somewhere purely as a performance optimization. The article speculates that OpenAI knows this well and hides it to protect the model internals. [1] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/

fl7305 3y ago

> people who can't figure out why their code can't produce the same result twice It is well known that the Nvidia parallel processing optimizations cause non-deterministic results. It's easy to get deterministic results as far as that goes. They've just elected not to do that, since it would run much slower.