4 ms·
Estimating PaLM's Training Cost
- axg11 5y agoThere are 67 authors on the PaLM paper. Assuming average salary of $150k (probably a large underestimate for Google employees) that's $10M+/year in salary alone. Compute is a big cost factor for this project but it's not clear if it even dominates the total cost. In the long run this type of research will more than pay itself back in benefits to Google. NLP underpins everything they do. It would be interesting to see how OpenAI API (GPT-3) is doing in terms of revenue. They're going for the more direct method of seeking value from a trained large language model.
- the8472 5y agoThe model itself didn't take all those authors a full-time work year to build from scratch. They build other models and release other papers throughout the year. Of course those works build on each other, but that turns it into a much more complicated question how we account for the value and costs of prior work.
- axg11 5y agoTo counter that, there were likely other people with smaller contributions not listed as authors. Some of the leadership are likely earning 2-3x my estimate salary.
- platers 5y agoEveryone on the paper is likely making at least 300k. And double that again for taxes, healthcare, and perks.
- hansvm 5y agoDouble is a good rule of thumb for a median liquidatable income, but healthcare, food, gyms, 401k matching, ... are all bounded by reasonable constants, and taxes only add <10% from the employer's side of things. I'd be surprised if the fully loaded cost were more than an extra 50%.
- deleted 5y ago[deleted]
- marczoid 5y agoThe fact that there are 67 people on the paper doesn’t mean they all worked on it full time. In fact, I suppose less than 5 did. I am quite certain most people only spent a tiny fraction of their time on this project. (Not saying the outcome isn’t impressive, it’s just how these projects tend to go.)
- deleted 5y ago[deleted]
- modeless 5y agoLarge underestimate is an understatement. The average fully loaded cost of these employees may be an order of magnitude higher than your guess. Jeff Dean alone probably costs Google $10m per year.
- gwern 5y agoOne major caveat here: the Chinchilla scaling law indicates that most of this compute was wasted. You could've gotten PaLM's performance at a fraction of the cost - Chinchilla approaches PaLM performance but at a fifth the cost.
- _hark 5y agoDeepMind to OpenAI and everyone else[1]: > your hyperparameters are bad and you should feel bad It's amazing to me that such a big goof was missed by so many for so long. All these multimillion dollar language models and people just took the scaling laws at face value. [1]: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556
- gwern 5y agoIsn't it amazing? One of those reminders that we don't understand DL as much as we like to pretend we do. Cyclic learning rates work well elsewhere, but I can't think of any other case where switching made such a difference. I was completely shocked to read Chinchilla, and I'm still a little baffled - a cosine schedule? Really? That's it? You guys didn't make any other changes?
- timpetri 5y agoQuestion related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced across the entire internet, all our written works, etc? Is there such a thing as a 216 trillion token set? [0] https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2203.15556 [1] https://arxiv.org/abs/2204.02311 https://arxiv.org/abs/2204.02311
- gwern 5y agoHumans produce an astonishing amount of text if you consider all the source code, research data, social media websites, emails etc and project out a decade or two; there is also multimodal and RL to consider as a source of 'tokens' like visual tokens, which have ~infinite data. Text is great, but there is no reason to train only text. It's just a good starting point. But the real question you should be asking is, where would you get the compute to train a model that needs 216t tokens?
- nootropicat 5y agoGiven how good PaLM is, this is nothing. I'm sure it could make over $10M/year of profit if it was open to outside use. I would actually pay a bit just to have fun with text games. I tried with GPT3 but it was a bit too stupid and stopped being fun fast. As a separate point, this suggests to me that 100x larger models are within the current reach of Google and other megacorps.
- BooneJS 5y ago> Of course, Google didn't pay that much. They own the hardware. You’re right. Google had to: 1. Design, tape-out, manufacture, and deploy TPUv4 2. Run PaLM
- DOsinga 5y agoAlso: they are running Google Cloud at a loss which at some level means that they are selling compute below costs which would suggest that Google could be paying more.
- rhacker 5y agoThat's really neat. That's like near TNG computer cool. You know when on TNG they ask Computer for a bunch of related things and it somehow knows what they're talking about.