3 ms·
By emitting a next token distribution with a 10% chance of 0, 10% chance of 1, etc. Also it's an LLM, not a brain.
by gzread 7mo ago
By emitting a next token distribution with a 10% chance of 0, 10% chance of 1, etc.
Also it's an LLM, not a brain.
- alextheparrot 7mo agoNo LLMs are calibrated?
- gzread 7mo agoWhat?
- throw310822 7mo agoInteresting. So you expect it to "not think" and simply produce a value corresponding to "it's the same to me", knowing that it will be translated into an actual random value. Instead, exactly as a person would do, it does think of a specific number that feels random in that particular moment.
- pmontra 7mo agoIf I care a little bit about that random number I might reach for my phone and look at the digits of the seconds of the current time. It's 31 now. Not appropriate for multiple lookups.
- throw310822 7mo agoYes, there is probably some variable context in every chat (like date and time). Could work as a good seed but I guess you should ask the LLM to really make an effort to produce a seriously random number. (Actually I've just tried, even if you ask it to make an effort, the number will be always the same).
- gzread 7mo agoIf "a random number is" is followed by all digits equally in the training dataset, then emitting a uniform distribution should minimise cross-entropy, right?
- throw310822 7mo agoEr, should it? Even of you trained an LLM exactly over this type of question, if the sequence to predict in the training data is really random, then any output is equally wrong. Even if the output is a fixed "1234" or "0000". There is no signal to train on, not even one that favours an equal distribution. On the other hand, LLMs show they know very well what a random number is and the fact it just shouldn't look like anything in particular, so they strive to come up with a number that doesn't look like anything in particular. Which happens to be always the same number given the starting conditions.