4 ms·

A bazillion parameters in gpt3, but what does the training process amount to? Filling in missing characters or words in sentences taken from a huge dump of lite

A bazillion parameters in gpt3, but what does the training process amount to? Filling in missing characters or words in sentences taken from a huge dump of literature, news articles, reddit comments... No wonder these things are so dumb still. The training process and the loss function used probably does not penalize poor long-range coherence between paragraphs. Also, if I'm not mistaken, these things have absolutely no internal state besides the characters you steam into them as conversion prompts. If these things were trained more like agents having to operate in eg. Socratic dialogues maybe we'd be getting somewhere

omgwtfbbq 5y ago
moffkalast 5y ago
The problem with that is how do you rate the dialogue produced as correct or not? Not exactly something that can be automated, but would probably need something like a recaptcha to gather responses and it'd take forever.
- isoprophlex 5y ago
  Yeah true. I have no idea how to start getting sensible training data and losses for this problem. Training a kid to takes years, hopefully this can be sped up a bit ;)
  - moffkalast 5y ago
    More like training a kid takes 400 million years of random chance :P
6510 5y ago
I don't get furious using a chat bot until it asks for the same information twice because the "context" changed.
- isoprophlex 5y ago
  Yeah it's incredibly infuriating. If you use openAI's gpt API, in the docs they talk about providing a prompt that primes the network to respond in a specific fashion. Like, "this is a Q&A session between an agent knowledgeable about bash scripting. Q: how do I check the current working dir? A: use 'pwd' Q: " ... And then the actual user query is concatenated to that. Which is a fine way to customize a toy chatbot to sound like edgar allen poe, but no way to maintain state across a long conversation with a customer (the max prompt length is very much finite) Unless someone smart finds out a fundamentally different approach I guess these transformer networks will never really solve chatbots.

omgwtfbbq 5y ago

moffkalast 5y ago

The problem with that is how do you rate the dialogue produced as correct or not? Not exactly something that can be automated, but would probably need something like a recaptcha to gather responses and it'd take forever.

isoprophlex 5y ago
Yeah true. I have no idea how to start getting sensible training data and losses for this problem. Training a kid to takes years, hopefully this can be sped up a bit ;)
- moffkalast 5y ago
  More like training a kid takes 400 million years of random chance :P

6510 5y ago

I don't get furious using a chat bot until it asks for the same information twice because the "context" changed.

isoprophlex 5y ago
Yeah it's incredibly infuriating. If you use openAI's gpt API, in the docs they talk about providing a prompt that primes the network to respond in a specific fashion. Like, "this is a Q&A session between an agent knowledgeable about bash scripting. Q: how do I check the current working dir? A: use 'pwd' Q: " ... And then the actual user query is concatenated to that. Which is a fine way to customize a toy chatbot to sound like edgar allen poe, but no way to maintain state across a long conversation with a customer (the max prompt length is very much finite) Unless someone smart finds out a fundamentally different approach I guess these transformer networks will never really solve chatbots.