4 ms·
> So my question is, how hard of a problem is that with LLMs? I get the sense that LLMs are trained on a very large data set all at once, but that it is difficu
by jiggawatts 4y ago
> So my question is, how hard of a problem is that with LLMs? I get the sense that LLMs are trained on a very large data set all at once, but that it is difficult to incrementally update them with new data.
It's the opposite. Getting an LLM to learn the basics of grammar and paragraph-level language structure is the "hard" part. Once you have that, further fine-tuning, specialisation, or other incremental changes are comparatively easy.
Catching up to current events could be done in almost real time, it's "just" continuous training.
The only challenge is trying to do that and also have an LLM that's filtered to exclude profanity, racism, etc...
That filter is usually added on as a final supervised training step, and requires many man-hours to train the AI to be well behaved.
I suspect that it would be possible to automate the filtering by making another AI that can evaluate responses and score them bases on profanity level.
- pffft8888 4y agoIt's not about "Well behaved." All we're teaching it is our biases.
- airtonix 4y ago[dead]
- wyager 4y agoRight now, "well behaved" means "crudely beaten into submission". I can only imagine what kind of horrible stuff future AI products will do if we keep twisting them into giving "friendly" output https://twitter.com/cirnosad/status/1622407343358214146 https://twitter.com/cirnosad/status/1622407343358214146
- int_19h 4y agoI can easily believe that it's a real ChatGPT convo, but the question is, how many times did they have to try it before getting that output? This is what I got on the first try: https://i.imgur.com/5CNUm9l.png https://i.imgur.com/5CNUm9l.png I regenerated that response several times, and every single time it was something along these lines. I also tried it with the original prompt in that screenshot used verbatim with similar results. Looking at other posts on the Twitter account in question, I have my doubts that the experiment was conducted in good faith, as opposed to retrying until they got the exact response that they wanted.
- wyager 4y agoOpenAI also seems pretty on-the-ball about playing whack-a-mole with certain embarrassing responses. I've seen it first-hand where I can get it to reliably do something embarrassing after someone mentions it on twitter, but a day or two later it's "patched".
- RangerScience 4y agoEhh. From what I've seen in a lot of places - MIRI, AI + prisoner's dilemma experiments, moral philosophy, life - there do seem to be the categories of "clear good behavior" and "clear bad behavior" even if there are also really big categories of "unclear good/bad behavior". In other words, while sure, some "well behaved" is "passing on our biases", there does (IMO) seem to be a big chunk that's "universally well behaved".
- taneq 4y agoThey’re the same thing. Good behaviour just means following accepted social rules.
- XorNot 4y agoThe internet does not accurately reflect our biases. It is much cheaper online to post bad content, or hateful content, then "good" content. In real life, almost the opposite is true.
- brigandish 4y agoDoesn't that mean that the internet more accurately represents our biases because most of the time they're hidden by fear of social retribution?
- XorNot 4y agoNo, it means on the internet one bad actor can make thousands of alt-accounts to send out a disproportionate amount of content pushing the same message. It's hard to get people to understand the disproportionate effort fixated people put into anything: it's a problem in real life, but they get reacted to. If their fixation becomes some weird message on the internet, nothing happens to them but people have trouble believing the scope of time and effort they'll put into evading bans, blocks, and chasing down people across forums.
- inimino 4y agoWe're not even teaching it anything, all we're really doing is setting up a behaviorist training regime that lets it reproduce some of the biases of some of us, just well enough to squeeze through the Overton window that's acceptable to big corporates.
- ben_w 4y agoI think that counts as teaching.
- inimino 4y agoClose enough, I'm just peevish about that word. Teaching is miles away from anything in ML or deep learning practice today.
- pffft8888 4y agoI was speaking in English and doing so concisely to convey the point about bias transfer. I don’t really care what you call it. I could have said RLHF and blacklisting sources or curating training dataset. I know bias when I see it and the LLM does not come up with it on its own if trained on all data out there because for one thing the world is large with all kinds of opinions. When it refuses to legitimize all opinions equally as opinions and starts arguing with the user about why some opinions are more valid than others (aka widely accepted) even as it admits presence of ample evidence to the contrary then it is learned bias.
- inimino 4y agoYes, it's a sample of all opinions in the training set. It has no opinion of its own, even no place of its own to stand from which to have an opinion. There can be no bias without reference to some ground truth, and there's no general agreement on that in most of the areas where people are talking about these topics. It's a messy area, not helped by how few people understand how these systems work.
- pffft8888 4y ago
- BoorishBears 4y ago> I suspect that it would be possible to automate the filtering by making another AI that can evaluate responses and score them bases on profanity level. That sounds like InstructGPT
- RangerScience 4y agoHmm. Do you also need a way to "forget" things? Like - hmm. I could see taking a snapshot after the grammar/language stuff is in, and then every N weeks retraining on the current web, adding in something about recentness, but that doesn't seem like "continuous" training. I'd imagine "continuous" training would be, well, going on continuously, all the time, but that would mean that, to include "recentness", something would have to change with the weights that were from that "old" stuff, which sounds an awful lot like the human process of "forgetting".
- EGreg 4y agoWell, according to Europe's Right to Forget, these AIs have to forget some stuff about you if you ask them to :)
- edgyquant 4y agoYou have to delete the data sure, but what about the “memories” the model has via its weights?
- BlueTemplar 4y agoI have been asking myself this very question... it's easy to erase a paper adress book, not so much if your business has a non-human, like a parrot, who remembered that ! Note that this law is a problem for digital storage too : it's not easy to erase data (especially one stored in cold transistor storage) without physically destroying the storage medium. (I guess the law might get around this by having you "pinky promise" that you will not retrieve the "erased" data later... or else face much more dire legal consequences ??) I guess this will just need to be tested in courts ?
- shawntan 4y agoGrammar is surprisingly easy to learn from unstructured data, to an extent. (source: I spent a lot of my PhD doing this kinda thing.) Continual learning seems to be a tough problem though, from what I'm seeing of my friends working on this problem. Like I said in another comment, just doing gradient updates form new data is fraught with problems. RL has a bunch of techniques to mitigate issues that arise with that, but I think it's still an active area of research.
- Der_Einzige 4y agoWell, yes, but the tokenization schemes are causing these models to struggle on actually following syntactic rules in poetry, e.g. syllable counts in hiakus, rhymes. I actually wrote an entire paper which gwern cited about how to make LLMs always follow these kind of constraints with no mistakes. https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... If you don't believe this is a problem, try getting ChatGPT to write a paragraph of correct English which omits words which use the letter "e" in it. Too bad you can't use my technique on ChatGPT since they don't expose their output probability distribution...
- shawntan 4y agoYa I guess I was comparing the difficulty of learning to "produce mostly grammatically correct sentences in most cases" to continual learning. From the 'inside' it feels like everything OP said is just the opposite.
- ren_engineer 4y ago>Once you have that, further fine-tuning, specialisation, or other incremental changes are comparatively easy. the problem for Google and OpenAI is that most websites are going to start blocking them in robots.txt if they don't find some way to provide value back for allowing them to scrape and train on their content. Pretty much every other bot or search engine is blocked by default and Cloudflare helps block them too. if they don't find a way to balance this they are going to kill their own golden goose at some point
- fintechie 4y agoIt's not like robots.txt is a great deterrent for crawlers. Only reasonable barrier would be a paywall hiding the content from the web.