4 ms·

Right - in terms of information theory, automatically-generated infilled data communicates zero bytes of additional information. It could be compressed away ent

Right - in terms of information theory, automatically-generated infilled data communicates zero bytes of additional information. It could be compressed away entirely losslessly and I could instead generate it myself. There is no need to waste bandwidth sharing it. Communication only has value if it is surprising. That much has been known since Shannon.

toss1 4y ago
EXACTLY what I was pointing to, more clearly expounded - thx!
andrewflnr 4y ago
I don't know if this is quite correct. The inpainted text has zero bits of information only if you already have on your end the exact, very large model they would have used. If you don't, and they only sent their prompt rather than the full text, you're in for a bit of a download. Put another way, their prompt is arguably a pointer to data buried in that model. You're raising the question of whether they should only send the pointer, or just cache the query result, in a sense, and save you the trouble of looking it up.
visarga 4y ago
Language models don't pick the top choice all the time, and even picking the top choice is some amount of information, more precisely -p_token*log(p_token). They return a distribution we sample from. Sampling adds surprise and detail that was not originally there. Even more, if you have some sort of verification going - a second model to rate the outputs of the first, a math verifier or code executer when available - then language models can surprise us with interesting solutions to our tasks.
- toss1 4y ago
  That is great when we are looking for interesting solutions. for example, it looks like GPT3-type systems might be very good search engines for obscure topics, getting them to finish paragraphs with their enormous search base. I've found a few tantalizing bits that I couldn't find on Google or DDG searches. But, when they are trying to express what WE are saying, it looks like a very lossy solution at best. I've had a deposition taken with an "AI" stenographer, and it was horrific, frequently reversing the meaning of sentences I said, or replacing an uncommon name with a common name (e.g., "John Kemeny" replaced with "Jack Kennedy"). Of course the transcript LOOKS great, it doesn't have any of the "(unintellegibile)" notations of a human transcript. It also does not go back at break points and ask for proper spellings of names, addresses, etc. like a human transcriber. This is in the context of a legal trial with consequences, and I'm horrified to see this kind of crap passing for usable products, and here we are looking to foist it off on the general public as writing tools. We're forking doomed by smart idiots looking to make a quick buck with novel "tools".

toss1 4y ago

EXACTLY what I was pointing to, more clearly expounded - thx!

andrewflnr 4y ago

I don't know if this is quite correct. The inpainted text has zero bits of information only if you already have on your end the exact, very large model they would have used. If you don't, and they only sent their prompt rather than the full text, you're in for a bit of a download. Put another way, their prompt is arguably a pointer to data buried in that model. You're raising the question of whether they should only send the pointer, or just cache the query result, in a sense, and save you the trouble of looking it up.

visarga 4y ago

Language models don't pick the top choice all the time, and even picking the top choice is some amount of information, more precisely -p_token*log(p_token). They return a distribution we sample from. Sampling adds surprise and detail that was not originally there. Even more, if you have some sort of verification going - a second model to rate the outputs of the first, a math verifier or code executer when available - then language models can surprise us with interesting solutions to our tasks.

toss1 4y ago
That is great when we are looking for interesting solutions. for example, it looks like GPT3-type systems might be very good search engines for obscure topics, getting them to finish paragraphs with their enormous search base. I've found a few tantalizing bits that I couldn't find on Google or DDG searches. But, when they are trying to express what WE are saying, it looks like a very lossy solution at best. I've had a deposition taken with an "AI" stenographer, and it was horrific, frequently reversing the meaning of sentences I said, or replacing an uncommon name with a common name (e.g., "John Kemeny" replaced with "Jack Kennedy"). Of course the transcript LOOKS great, it doesn't have any of the "(unintellegibile)" notations of a human transcript. It also does not go back at break points and ask for proper spellings of names, addresses, etc. like a human transcriber. This is in the context of a legal trial with consequences, and I'm horrified to see this kind of crap passing for usable products, and here we are looking to foist it off on the general public as writing tools. We're forking doomed by smart idiots looking to make a quick buck with novel "tools".