The paper about PaLM mentioned that they used 6144 TPU chips to train it over several weeks. That's a metric shit ton of processing power (pardon my language).

The paper about PaLM mentioned that they used 6144 TPU chips to train it over several weeks. That's a metric shit ton of processing power (pardon my language). If Google could dedicate 2 racks of TPUs to just continuously training PaLM (note: I'm not sure if LAMDA is PaLM, but let's assume they're similar), they could do this retraining. Catch is: consider these statements: Donald Trump is the President. Joe Biden is the President. Barack Obama is the President. George W Bush is the President. George H. W. Bush is the President. Bill Clinton is the President. All of them were true at some point of time. How do you train a model to disambiguate these?