3 ms·

> Like the earlier responses said, if it is retrieving information from an external data source I was under the impression it went through like a multi-month l

> Like the earlier responses said, if it is retrieving information from an external data source I was under the impression it went through like a multi-month long ultra expensive training process involving many GPUs on terabytes of data "snapshotted" point in time. Is it possible to do this at a smaller scale once a day at the end of the day with "all new content scraped from the Internet nightly"?

nicoburns 4y ago
I think the idea might be to effectively provide it with the result of a google search (or other datasource) as part of the prompt.
visarga 4y ago
Not effectively, because you probably can't return to language pre-training after multi-task fine-tuning and RLHF. Stage 1 has to go before stages 2 and 3. So they would need to fine-tune a stage 1 model and re-apply stages 2 and 3.
dragonwriter 4y ago
> Is it possible to do this at a smaller scale once a day at the end of the day with "all new content scraped from the Internet nightly"? If the model has the ability to browse the web (the tech behind ChatGPT by design does, but it is disabled in ChatGPT proper, which is, in some respects, a conservative public demo) and incorporate data in responses, this can “cover” a bit for staleness in the base model. Even if you can't retrain the model fully daily, with enough resources [0] you can have multiple training sessions running concurrently and just swap in new backend models behind the interface as they arr ready, which, combined with browsing ability. acheives something very similar. This obviously, barring an enormous advantage in underlying tech or access to relevant training data for one player, works most in favor of whoever can subsidize the biggest hardware commitment. (Given the different training stages, there may be cost efficiency advantages to, say, running less-frequent iterations of the lowest-level training stage but more frequent iterations of the higher-level ones.) [0] probably an utterly ludicrous investment for all but a handful of firms, but for a ~$trillion firm where this hits very close to their core business? Not so ludicrous.
shawntan 4y ago
How I imagine what you're thinking of is a kind of small update to an existing model by doing backpropagation training on 'new' data. There are existing problems with that, one keyword to lookup is 'catastrophic forgetting', where you update on recent stuff so much that you overfit on that and forget all the prior stuff you learned before. There are methods to mitigate that, but I would call that an 'active area of research'.
1024core 4y ago
The paper about PaLM mentioned that they used 6144 TPU chips to train it over several weeks. That's a metric shit ton of processing power (pardon my language). If Google could dedicate 2 racks of TPUs to just continuously training PaLM (note: I'm not sure if LAMDA is PaLM, but let's assume they're similar), they could do this retraining. Catch is: consider these statements: Donald Trump is the President. Joe Biden is the President. Barack Obama is the President. George W Bush is the President. George H. W. Bush is the President. Bill Clinton is the President. All of them were true at some point of time. How do you train a model to disambiguate these?
- htrp 4y ago
  Lamda is about 20% of the size of palm IIRC

nicoburns 4y ago

I think the idea might be to effectively provide it with the result of a google search (or other datasource) as part of the prompt.

visarga 4y ago

Not effectively, because you probably can't return to language pre-training after multi-task fine-tuning and RLHF. Stage 1 has to go before stages 2 and 3. So they would need to fine-tune a stage 1 model and re-apply stages 2 and 3.

dragonwriter 4y ago

> Is it possible to do this at a smaller scale once a day at the end of the day with "all new content scraped from the Internet nightly"? If the model has the ability to browse the web (the tech behind ChatGPT by design does, but it is disabled in ChatGPT proper, which is, in some respects, a conservative public demo) and incorporate data in responses, this can “cover” a bit for staleness in the base model. Even if you can't retrain the model fully daily, with enough resources [0] you can have multiple training sessions running concurrently and just swap in new backend models behind the interface as they arr ready, which, combined with browsing ability. acheives something very similar. This obviously, barring an enormous advantage in underlying tech or access to relevant training data for one player, works most in favor of whoever can subsidize the biggest hardware commitment. (Given the different training stages, there may be cost efficiency advantages to, say, running less-frequent iterations of the lowest-level training stage but more frequent iterations of the higher-level ones.) [0] probably an utterly ludicrous investment for all but a handful of firms, but for a ~$trillion firm where this hits very close to their core business? Not so ludicrous.

shawntan 4y ago

How I imagine what you're thinking of is a kind of small update to an existing model by doing backpropagation training on 'new' data. There are existing problems with that, one keyword to lookup is 'catastrophic forgetting', where you update on recent stuff so much that you overfit on that and forget all the prior stuff you learned before. There are methods to mitigate that, but I would call that an 'active area of research'.

1024core 4y ago

The paper about PaLM mentioned that they used 6144 TPU chips to train it over several weeks. That's a metric shit ton of processing power (pardon my language). If Google could dedicate 2 racks of TPUs to just continuously training PaLM (note: I'm not sure if LAMDA is PaLM, but let's assume they're similar), they could do this retraining. Catch is: consider these statements: Donald Trump is the President. Joe Biden is the President. Barack Obama is the President. George W Bush is the President. George H. W. Bush is the President. Bill Clinton is the President. All of them were true at some point of time. How do you train a model to disambiguate these?

htrp 4y ago
Lamda is about 20% of the size of palm IIRC