4 ms·
> "Google, similar to OpenAI, didn’t provide a lot of the technical details about how it trained this next-gen model, including parameter counts (PaLM 2 is a 54
by ftxbro 3y ago
> "Google, similar to OpenAI, didn’t provide a lot of the technical details about how it trained this next-gen model, including parameter counts (PaLM 2 is a 540-billion parameter model, for what it’s worth). The only technical details Google provided here are that PaLM 2 was built on top of Google’s latest JAX and TPU v4 infrastructure."
I'm sad but not really surprised that these companies aren't publishing and bragging about all of the technical details of their model architecture, size, and training anymore.
- ftxbro 3y agoupdate: there's at least some technical report that I didn't know when I wrote the comment https://ai.google/static/documents/palm2techreport.pdf https://ai.google/static/documents/palm2techreport.pdf but it's still mostly capabilities evaluation and not explaining the details of the model
- chaxor 3y agoI think that may be wrong. PaLM was 540B, but I don't think PaLM2 is near that. Perhaps 120B like Galactica large, but afaik they don't say.
- ftxbro 3y agoI think you are right, and the quote from article we are commenting here is wrong. From the PaLM technical report: "The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute."
- fullsend 3y agoThey are still stuck in the mindset of “this is our proprietary secret and we can’t let others catch up.” That mindset is just one of the reasons their company is stagnating.
- ftxbro 3y agoThat's a tempting narrative but I think it's wrong. The used to let their researchers publish freely like Google Brain and Deep Mind, but then there was a combination of some of their employees leaving and also other companies learning from the publications. This led to the AI race, so now all race participants are shutting down their publications.