3 ms·
I mean random sampling in the sense how autoregressive language models like GPT generate sequences using token probabilities. It's not a random svg, but the tex
by sigmoid10 3y ago
I mean random sampling in the sense how autoregressive language models like GPT generate sequences using token probabilities. It's not a random svg, but the text sequence that is used to draw it suffers from inherent non-determinism in the underlying model.
- rcme 3y agoRelying on token probabilities seems like the exact opposite of random
- esafak 3y ago"Random" is a loose term. We're discussing a technical subject. Talk about the distribution the random numbers are sampled from.
- sigmoid10 3y agoThe neural network just generates a set of probabilities for all tokens. The actual next token is then sampled from this set, which is always random for T>0 (and in the case of GPT4 even for T=0, because of the way the model itself works).