4 ms·

From what I've seen, low temp "only the most probable" output tends to be more repetetive and incoherent than medium-tempeature output, which is a bit of a para

From what I've seen, low temp "only the most probable" output tends to be more repetetive and incoherent than medium-tempeature output, which is a bit of a paradox. I wonder why that is.

geocar 4y ago
Look up at the stars: You can only see a few of them. These functions are like really twisted/complex lenses and mirrors you might imagine we have hoisted into view so that now the sky is completely white (to our eyes) and we can see every star in the sky. Now without knowing the precise shape and position of that contraption, because all those stars really flicker at different rates, you can with a very fast camera recognise them by their flicker and note that the same star is actually visible in multiple "places" in the sky. You might if it is useful to you consider a star being "close" to many different stars in the sky -- as distorted by those lenses and mirrors -- simply by how long you watch the flicker. It is in this way, that I imagine "low temperatures" producing incoherence like the fast camera, not waiting long enough to see how many better (closer!) options there might be, seeing only the brightest (closest to us! cookie!) stars, and not waiting for all the other options (cheese, baguette, etc) to make into a longer exposure. Or the kinds of things you can meaningfully say after studying the Internet for (any random) week is probably going to be more repetitive and less coherent than the kinds of things you can meaningfully say after studying the Internet for 20 years.
- p1esk 4y ago
  I did not understand your explanation.
f_devd 4y ago
The reason is that actual human-written text generally doesn't follow the most probable next word rule, but rather there are occasional 'decisions' made which make the text unique and therefore more interesting/coherent. There are other sampling methods which try to avoid this issue like "Locally Typical" and "Nucleus" sampling.

From what I've seen, low temp "only the most probable" output tends to be more repetetive and incoherent than medium-tempeature output, which is a bit of a paradox. I wonder why that is.

geocar 4y ago

Look up at the stars: You can only see a few of them. These functions are like really twisted/complex lenses and mirrors you might imagine we have hoisted into view so that now the sky is completely white (to our eyes) and we can see every star in the sky. Now without knowing the precise shape and position of that contraption, because all those stars really flicker at different rates, you can with a very fast camera recognise them by their flicker and note that the same star is actually visible in multiple "places" in the sky. You might if it is useful to you consider a star being "close" to many different stars in the sky -- as distorted by those lenses and mirrors -- simply by how long you watch the flicker. It is in this way, that I imagine "low temperatures" producing incoherence like the fast camera, not waiting long enough to see how many better (closer!) options there might be, seeing only the brightest (closest to us! cookie!) stars, and not waiting for all the other options (cheese, baguette, etc) to make into a longer exposure. Or the kinds of things you can meaningfully say after studying the Internet for (any random) week is probably going to be more repetitive and less coherent than the kinds of things you can meaningfully say after studying the Internet for 20 years.

p1esk 4y ago
I did not understand your explanation.

f_devd 4y ago

The reason is that actual human-written text generally doesn't follow the most probable next word rule, but rather there are occasional 'decisions' made which make the text unique and therefore more interesting/coherent. There are other sampling methods which try to avoid this issue like "Locally Typical" and "Nucleus" sampling.