20 ms·
Calculating the cost of a Google DeepMind paper
- hnthr_w_y 2y agothat's not very much in the business range, it's a lot when it comes to paying us salaries.
- willis936 2y agoAny company of any size that doesn't learn the right lessons from a $10M mistake will be out of business before long.
- brainwad 2y agoThat's like staffing a single-manager team on a bad project for a year. Which I assure you happens all the time in big companies, and yet they survive.
- saikia81 2y agoThey are not saying it doesn't happen. They are saying: The companies that don't learn from these mistakes will go out of business before long.
- duggan 2y agoIn principle, for some other company, sure. Google makes ~$300b a year in profit. They could make a $10m mistake every day and barely make a dent in it.
- hnbad 2y agoI think there might be a disagreement about what "big" means. Google can easily afford to sink millions each year into pointless endeavours without going out of business and they probably have. Alphabet's annual revenue has been growing a good 10% each year since 2021[0]. That's in the range of $20-$30 billion dollars with a B. To put that into perspective, Alphabet's revenue has increased 13.38% year-over-year as of June 30, arriving at $328.284 billion dollars - i.e. it has increased by $38.74 billion in that time. A $10 million dollar mistake translates to losing 0.0258% of that number. A $10 million dollar mistake costs Alphabet 0.0258% of the amount their revenue increased year-over-year as of last month. Alphabet could have afforded to make 40 such $10 million dollar mistakes in that period and it would have only represented a loss of 1% of the year-over-year increase in revenue. Taking the year-over-year increase down by 1% (from 13.38% to 12.38%) would have required making 290 such $10 million dollar mistakes within one year. Let me repeat that because it bears emphasizing: over the past years, every year Google could have easily afforded an additional 200 such $10 million dollar mistakes without significantly impacting their increase in revenue - and even in 2022 when inflation was almost double what it was in the other year they would have still come out ahead of inflation. So in terms of numbers this is demonstrably false. Of course the existence of repeated $10 million dollar mistakes may suggest the existence of structural issues that will result in $1, $10 or $100 billion dollar problems eventually and sink the company. But that's conjecture at this point. [0]: https://www.macrotrends.net/stocks/charts/GOOG/alphabet/revenue https://www.macrotrends.net/stocks/charts/GOOG/alphabet/reve...
- vishnugupta 2y agohttps://killedbygoogle.com/ https://killedbygoogle.com/ I’m confident each one of them were multiple of $10M investments. And this is just what we know because they were launched publicly.
- Sebb767 2y agoThe point the parent made is not to not make mistakes, but to learn from them. Which they probably did not from all of them, as indicated by the sheer amount of messenger apps on this list, but there's definitely a lot to learn from this list.
- OtherShrezzing 2y agoI'm not really certain that's true at Google's size. Their annual revenue is something like a quarter trillion dollars. 25,000x larger than a $10m mistake. The equivalent wastage for a self-employed person would be allowing a few cups of Starbucks coffee per year to go cold.
- Workaccount2 2y agoThere was that time that google paid out something like $300M in fraudulent invoices.
- willis936 2y agoTo be clear: what I mean by "not learning the right lessons" is a company deciding that the issue with wasting $10M in six months is that they didn't do it 100x in parallel in three months. Then when that goes wrong they must need to do it 100x wider in parallel again in three weeks.
- BartjeD 2y agoIf this ran on google's own cloud it amounts to internal bookkeeping. The only cost is then the electricity and used capacity. Not consumer pricing. So negligible. It is rather unfortunate that this sort of paper is hard to reproduce. That is a BIG downside, because it makes the result unreliable. They invested effort and money in getting an unreliable result. But perhaps other research will corroborate. Or it may give them an edge in their business, for a while. They chose to publish. So they are interested in seeing it reproduced or improved upon.
- rty32 2y agoOpportunity cost is cost. What you could have earned by selling the resources to customers instead of using them yourself is what the resources are worth.
- g15jv2dp 2y agoThis assumes that you can sell 100% of the resources' availability 100% of the time. Whenever you have more capacity that you can sell, there's no opportunity cost in using it yourself.
- michaelt 2y agoA few months back, a lot of the most powerful GPU instances on GCP seemed to be sold out 24/7. I suppose it's possible Google's own infrastructure is partitioned from GCP infrastructure, so they have a bunch of idle GPUs even while their cloud division can sell every H100 and A100 they can get their hands on?
- dmurray 2y agoI'd expect they have both: dedicated machines that they usually use and are sometimes idle, but also the ability to run a job on GCP if it makes sense. (I doubt it's the other way round, that the Deepmind researchers could come in one day and find all their GPUs are being used by some cloud customer).
- 2y ago
- sigmoid10 2y agoThis is calculation is pretty pointless and the title is flat out wrong. It also gets lost in finer details while totally missing the bigger picture. After all, the original paper written by people either working for Google or at Google. So you can safely assume they used Google resources. That means they wouldn't have used H100s, but Google TPUs. Since they design and own these TPUs, you can also safely assume that they don't pay whatever they charge end users for them. At the scale of Google, this basically amounts to the cost of houseing/electricity, and even that could be a tax write-off. You also can't directly assume that the on paper performance of something like an H100 will be the actual utilization you can achieve, so basing any estimate in terms of $/GPU-hour will be off by default. That means Google payed way less than this amount and if you wanted to reproduce the paper yourself, you would potentially pay a lot more, depending on how many engineers you have in your team to squeeze every bit of performance per hour out of your cluster.
- michaelmior 2y agoEven if they did use H100s and paid the current premium on them, you could probably buy 100 H100s and the boxes to put them in for less than $10M.
- c-linkage 2y agoReproducibility is a key element of the scientific process How is anyone else going to reproduce the experiment if it's going to cost them $10 million because they don't work at Google and would have to rent the infrastructure?
- arcade79 2y agoA lot of misunderstandings among the commenters here. From the link: "the total compute cost it would take to replicate the paper" It's not Google's cost. Google's cost is of course entirely different. It's the cost for the author if he were to rent the resources to replicate the paper. For Google, all of it is running at a "best effort" resource tier, grabbing available resources when not requested by higher priority jobs. It's effectively free resources (except electricity consumption). If any "more important" jobs with a higher priority comes in and asks for the resources, the paper-writers jobs will just be preempted.
- mrazomor 2y agoThis assumes the common resources (CPU, RAM, etc.), not the ones required for the LLM training (GPU, TPU, etc.). It's different economy. TL; DR: It's not ~free.
- akutlay 2y agoWhy does GPU matter? Do you think GCP keeps GPU utilization at 100% at all times?
- mrazomor 2y agoWhat the OP is referring to requires overprovisioning of the high priority traffic and the sine-like utilization (without it, the benefits of the "batch" tier is close to zero -- the preemption is too high for any meaningful work when you are close to the top of the utilization hill). You get that organically when you are serving lots of users. And, there's not much GPUs etc. used for that. Training LLMs gives you a different utilization pattern. The "best effort" resources aren't as useful in that setup.
- bbminner 2y agoBecause accelerators (tpus, gpus) unlike ram/cpu are notoriously hard to timeshare and vitrualize. So if you get evicted in an environment like that, you have to reload your entire experiment state from a model checkpoint. With giant models like that, it might take dozens of minutes. As a result, I doubt that these experiments are done using "spare" resources - in that case, constant interruptions and reloading would result in these experiments finishing sometime around the heat death of the universe :)
- deleted 2y ago[deleted]
- your-username 2y ago[flagged]
- floor_ 2y agoContent aside. This is hands down my favorite blog format.
- mostthingsweb 2y agoI agree, but I'm curious if it's for the same reason. I like it because there is now flowery writing. Just direct "here are the facts".
- deleted 2y ago[deleted]
- pama 2y ago3USD/hour on the H100 is much more expensive than a reasonable amortized full ownership cost, unless one assumes the GPU is useless within 18 months, which I find a bit dramatic. The MFU can be above 40% and certainly well above the 35% in the estimate, also for small models with plain pytorch and trivial tuning [1] I didnt read the linked paper carefully but I seriously doubt the google team used vocab embedding layers with 2 D V parameters stated in the link, because this would be suboptimal by not tying the weights of the token embedding layer in the decoder architecture (even if they did double the params in these layers, it would not lead to 6 D V compute because the embedding input is indexed). To me these assumptions suggested a somewhat careless attitude towards the cost estimation and so I stopped reading the rest of this analysis carefully. My best guess is that the author is off by a large factor in the upward direction, and a true replication with H100/200 could be about 3x less expensive. [1] if the total cost estimate was relatively low, say less than 10k, then of course the lowest rental price and a random training codebase might make some sense in order to reduce administrative costs; once the cost is in the ballpark of millions of USD, it feels careless to avoid optimizing it further. There exist H100s in firesales or Ebay occasionally, which could reduce the cost even more, but the author already mentions 2USD/gpu/hour for bulk rental compute, which is better than the 3USD/gpu/hour estimate they used in the writeup.
- 152334H 2y agoYou are correct on true H100 ownership costs being far lower. As I mention in the H100 blurb, the H100 numbers are fungible and I don't mind if you halve them. MFU can certainly be improved beyond 40%, as I mention. But on the point of small models specifically: the paper uses FSDP for all models, and I believe a rigorous experiment should not vary sharding strategy due to numerical differences. FSDP2 on small models will be slow even with compilation. The paper does not tie embeddings, as stated. The readout layer does lead to 6DV because it is a linear layer of D*V, which takes 2x for a forward and 4x for a backward. I would appreciate it if you could limit your comments to factual errors in the post.
- lonk11 2y agoI think the commenter was thinking about the input embedding layer, where to get an input token embedding the model does a lookup of the embedding by index, which is constant time. And the blog post author is talking about the output layer where the model has to produce an output prediction for every possible token in the vocabulary. Each output token prediction is a dot-product between the transformer hidden state (D) and the token embedding (D) (whether shared with input or not) for all tokens in the vocabulary (V). That's where the VD comes from. It would be great to clarify this in the blog post to make it more accessible but I understand that there is a tradeoff.
- deleted 2y ago[deleted]
- tempfile 2y ago[flagged]
- rgmerk 2y agoWorth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources to produce. My wife works on high-throughout drug screens. They routinely use over $100,000 of consumables in a single screen, not counting the cost of the screening “libraries”, the cost of using some of the -$10mil of equipment in the lab for several weeks, the cost of the staff in the lab itself, and the cost of the time of the scientists who request the screens and then take the results and turn them into papers.
- ramraj07 2y agoI estimated that any paper that has mouse work and produced in a first world country (I.e. they have to do good by the animals), the minimum cost of that paper in expenses and salary would be $200,000. Average likely higher. Tens of thousands of papers a year published like this!
- esperent 2y agoTo be fair, supposing the Google paper took six months to a year to produce, it also must have cost several hundred thousand dollars in salaries and other non-compute costs.
- paxys 2y agoThese are mostly fixed costs. If you produce a hundred papers from the same team and same research, the costs aren't 100x.
- lucianbr 2y agoBut starting from the 10th paper, the value is also pretty low I imagine. How many new things can you discover from the same team and same research? That's 3 papers per year for a 30-year career. Every single year, no breaks.
- sdenton4 2y agoWell, to be sure, mouse research consistently produces amazing cures for cancer, insomnia, lost limbs, and even gravity itself. Sure, none of it translates to humans, but it's an important source of headlines for high impact journals and science columnists.
- dont_forget_me 2y agoAll that compute power just to invade privacy and show people more ads. Can this get anymore depressing?
- psychoslave 2y agoYes, sure! Imagine a world where every HN thread you engage in is fed with information that are all subtly tailored to push you into buying whatever crap the market is able to produce.
- jeffbee 2y agoI think if you wanted to think about a big expense you'd look at AlphaStar.
- 5kg 2y agoI am wondering if AlphaStar is the most expensive paper ever.
- lern_too_spel 2y ago"Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC"
- jeffbee 2y agoI think it could be. I also think it is likely that HN frequenter `dekhn` has personally spent more money on compute resources than any other living human, so maybe they will chime in on how the cost gets allocated to the research.
- dekhn 2y agoA big part of it is basically hard production quota: the ability to run jobs at a high priority on large machines for an entire quarter. The main issue was that quota was somewhat overallocated, or otherwise unable to be used (if you and another team both wanted a full TPUv3 with all its nodes and fabric). From what I can tell, ads made the money and search/ads bought machines with their allocated budget, TI used their budget to run the systems, and then funny money in the form of quota was allocated to groups. THe money was "funny" in the sense that the full reach-through costs of operating a TPU for a year looks completely different from the production allocation quota that gets handed out. I think Google was long trying to create a market economy, but it was really much more like a state-funded exercise. (I am not proud of how much CPU I wasted on protein folding/design and drug discovery, but I'm eternally thankful for Urs giving me the opportunity to try it out and also to compute the energy costs associated with the CPU use)
- ipsum2 2y ago
- brg 2y agoI found this exercise interesting, and as arcade79 pointed out it is the cost of replication not the cost to Google. Humorously I wonder the cost of of replicating Higgs-Boson verification or Gravity Wave detection would be.
- faitswulff 2y agoI wonder how many tons of CO2 that amounts to. Google Gemini estimated 125,000 tons of carbon emissions, but I don’t have the know-how to double check it.
- godelski 2y agoWorth mentioning that "GPU Poor" isn't created because those without much GPU compute can't contribute, but rather because those with massive amounts of GPU are able to perform many more experiments and set a standard, or shift the Overton window. The big danger here is just that you'll start expecting a higher "thoroughness" from everyone else. You may not expect this level, but seeing this level often makes you think what was sufficient before is far from sufficient now, and what's the cost of that lower bound? I mention this because a lot of universities and small labs are being edged out of the research space but we still want their contributions. It is easy to always ask for more experiments but the problem is, as this blog shows, those experiments can sometimes cost millions of dollars. This also isn't to say that small labs and academics aren't able to publish, but rather that 1) we want them to be able to publish __without__ the support of large corporations to preserve the independence of research[0], 2) we don't want these smaller entities to have to go through a roulette wheel in an effort to get published. Instead, when reviewing be cautious in what you ask for. You can __always__ ask for more experiments, datasets, "novelty", and so on. Instead ask if what's presented is sufficient to push forward the field in any way and when requesting the previous things be specific as to why what's in the paper doesn't answer what's needed and what experiment would answer it (a sentence or two would suffice). If not, then we'll have the death of the GPU poor and that will be the death of a lot of innovation, because the truth is, not even big companies will allocate large compute for research that is lower level (do you think state space models (mamba) started with multimillion dollar compute? Transformers?). We gotta start somewhere and all papers can be torn to shreds/are easy to critique. But you can be highly critical of a paper and that paper can still push knowledge forward. [0] Lots of papers these days are indistinguishable from ads. A lot of papers these days are products. I've even had works rejected because they are being evaluated as products not being evaluated on the merits of their research. Though this can be difficult to distinguish when evaluation is simply empirical. [1] I once got desk rejected for "prior submission." 2 months later they overturned it, realizing it was in fact an arxiv paper, for only a month later for it to be desk rejected again for "not citing relevant materials" with no further explanation.
- pas 2y agoHow does preregistration factor into this? It seems to me that it would make sense to do at least some review before the bulk of the money is spent. > But you can be highly critical of a paper and that paper can still push knowledge forward. Can you give a concrete example of this?
- hiddencost 2y agoIt's likely the cost of the researchers was about $1m/ head, with 11 names that puts the staffing costs on par with the compute costs. (A good rule of thumb is that an employee costs about twice their total compensation.)