6 ms·

Worth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources

by rgmerk 2y ago

Worth pointing out here that in other scientific domains, papers routinely require hundreds of thousands of dollars, sometimes millions of dollars, of resources to produce. My wife works on high-throughout drug screens. They routinely use over $100,000 of consumables in a single screen, not counting the cost of the screening “libraries”, the cost of using some of the -$10mil of equipment in the lab for several weeks, the cost of the staff in the lab itself, and the cost of the time of the scientists who request the screens and then take the results and turn them into papers.

ramraj07 2y ago
I estimated that any paper that has mouse work and produced in a first world country (I.e. they have to do good by the animals), the minimum cost of that paper in expenses and salary would be $200,000. Average likely higher. Tens of thousands of papers a year published like this!
- esperent 2y ago
  To be fair, supposing the Google paper took six months to a year to produce, it also must have cost several hundred thousand dollars in salaries and other non-compute costs.
- paxys 2y ago
  These are mostly fixed costs. If you produce a hundred papers from the same team and same research, the costs aren't 100x.
  - lucianbr 2y ago
    But starting from the 10th paper, the value is also pretty low I imagine. How many new things can you discover from the same team and same research? That's 3 papers per year for a 30-year career. Every single year, no breaks.
    - sdenton4 2y ago
      Well, to be sure, mouse research consistently produces amazing cures for cancer, insomnia, lost limbs, and even gravity itself. Sure, none of it translates to humans, but it's an important source of headlines for high impact journals and science columnists.
      - godelski 2y ago
        This is also true for machine learning papers. They cure cancer, discover physics, and all sorts of things. Sure, they don't actually translate to useful science, but they are highly valuable pieces of advertisements. And hey, maybe someday they might!
        dekhn 2y ago
        AlphaFold has basically paid the bills for a decade's worth of machine learning research. It's been that transformative.
        godelski 2y ago
        I'd like to point out that AlphaFold does not constitute all, nor even the majority of ML works. My comment was a bit tongue in cheek. Not every research is going to be profitable or eventually profitable, but that also doesn't mean it isn't useful. If we're willing to account for the indirect profits via learning what doesn't work (an important part of science), then this vastly diminishes the number of worthless papers (to essentially those that are fraudulent or plagiarism) But as specifically for AlphaFold, I'm going to need a citation on that. If I understand the calculus correctly, Google acquired DM in 2014 for somewhere between $525 million to $850 million, and yearly spends a similar amount each year along with forgiving a 1.5bn dollar debt[0]. So I think (VERY) conservatively we can say $2bn (I think even $4bn is likely conservative here)? While I see articles that discuss how the value could be north of $100bn[1] (which certainly surpasses a very liberal estimate of costs), I have yet to see evidence that this is actual value that has been returned to Google. I can only find information about 2022 and 2023 having profits in the ballpark of $60m. This isn't to say that AlphaFold won't offset all the costs (I actually believe it will), but I your sentence does not suggest speculation but rather actualization ("has basically paid", "been"). I think that difference matters enough that we have a dozen cliches with similar sentiment. In the same way my annoyance is not that we are investing in ML[2], but how quick we are to make promises and celebrate success[3]. Actually, my concern is that while hype is necessary, overdoing it allows charlatans[4] to more easily enter the space. And if they are able to gain a significant foothold (I believe that is happening/has happened) then this is actually destructive to those who actually wish to push forward technology. [0] https://www.quora.com/How-much-money-did-Google-spend-on-DeepMind-in-its-early-years-before-DeepMind-started-generating-revenue-from-its-products-and-services-e-g-AlphaGo https://www.quora.com/How-much-money-did-Google-spend-on-Dee... [1] https://www.bloomberg.com/news/articles/2024-05-08/deepmind-ceo-targets-100-billion-plus-ai-drug-discovery-business-with-alphafold https://www.bloomberg.com/news/articles/2024-05-08/deepmind-... [2] disclosure, I'm an ML researcher. I actually am in favor of more funding. Though different allocation. [3] I'm willing to concede that success is realistically determined by how one measures success, and that this may be arbitrary and no objective measure actually exists or is possible. [4] One needs not knowingly be a charlatan. Only that the claims made are false or inaccurate. There are many charlatans who believe in the snake oil they sell. Most of these are unwilling to acknowledge critiques. A clear example is religion. If you believe in a religion, this applies to all religious organizations except the one you are a part of. If you are not religious, well the same sentence holds true but the resultant set is one larger.
  - dontreact 2y ago
    I agree with you but it also makes me think: Google's TPUs are also fixed costs and these research experiments could have been run at times when production serving need isn't as high.
    - ec109685 2y ago
      They sell them on the spot market, so there’s someone that would consume the baseline compute.
      - dontreact 2y ago
        I imagine it would never be optimal to set a price so low that utilization is always 100% externally
        londons_explore 2y ago
        There is plenty of people who want cheap compute and are willing to wait till 3am if that's when the cheap compute is. Happens for all computing services, but for ML stuff the effect is even more pronounced because compute costs are typically a large part of the cost of many projects.
  - ramraj07 2y ago
    Nope this is literally my estimate of how much the experiments cost for the paper alone (reagents, animals, and the labor for these experiments).
- dumb1224 2y ago
  Well not everyone starts experiment anew. Many also reuse accumulated datasets. For human data even more so.
slashdave 2y ago
I assure you that the companies performing these screens expect a return on this investment. It is not for a journal paper.
- godelski 2y ago
  I used to believe this line. But then I worked for a big tech company where my manager constantly made those remarks ("the difference in industry and academia is that in industry it has to actually work"). I then improved the generalization performance (i.e. "actually work") by over 100% and they decided not to update the model they were selling. Then again, I had a small fast model and it was 90% as accurate as the new large transformer model. Though they also didn't take the lessons learned and apply them to the big model, which had similar issues but were just masked by the size. Plus, I mean, there are a lot of products that don't work. We all buy garbage and often can't buy not garbage. Though I guess you're technically correct that in either of these situations there can still be a return on investment, but maybe that shouldn't be good enough...
  - shpongled 2y ago
    The post you are replying to is talking about high throughput assays for drug development. This is something actually run in a lab, not a model. As another person working at a biotech, I can assure you that screens are not just run as busy work.
    - rgmerk 2y ago
      No they’re not busywork, but not all such screens are directly in the drug discovery pipeline.
      - godelski 2y ago
        It also entirely misses the point of my comment which was that I don't believe this sentence to be true in an at least indirect sense. I conceded that it is technically valid in that lab experiments can also be used as advertisements and generate revenue even if such screens do not directly lead to novel or improved drug discoveries.
Metacelsus 2y ago
Yeah, I'm a wet-lab biologist and my most recent paper (which is still not past peer review) has already cost about $200,000. And I just spent another $2000 today...
deleted 2y ago
[deleted]

by rgmerk 2y ago

ramraj07 2y ago

I estimated that any paper that has mouse work and produced in a first world country (I.e. they have to do good by the animals), the minimum cost of that paper in expenses and salary would be $200,000. Average likely higher. Tens of thousands of papers a year published like this!

esperent 2y ago
To be fair, supposing the Google paper took six months to a year to produce, it also must have cost several hundred thousand dollars in salaries and other non-compute costs.
paxys 2y ago
These are mostly fixed costs. If you produce a hundred papers from the same team and same research, the costs aren't 100x.
- lucianbr 2y ago
  But starting from the 10th paper, the value is also pretty low I imagine. How many new things can you discover from the same team and same research? That's 3 papers per year for a 30-year career. Every single year, no breaks.
  - sdenton4 2y ago
    Well, to be sure, mouse research consistently produces amazing cures for cancer, insomnia, lost limbs, and even gravity itself. Sure, none of it translates to humans, but it's an important source of headlines for high impact journals and science columnists.
    - godelski 2y ago
      This is also true for machine learning papers. They cure cancer, discover physics, and all sorts of things. Sure, they don't actually translate to useful science, but they are highly valuable pieces of advertisements. And hey, maybe someday they might!
      - dekhn 2y ago
        AlphaFold has basically paid the bills for a decade's worth of machine learning research. It's been that transformative.
        godelski 2y ago
        I'd like to point out that AlphaFold does not constitute all, nor even the majority of ML works. My comment was a bit tongue in cheek. Not every research is going to be profitable or eventually profitable, but that also doesn't mean it isn't useful. If we're willing to account for the indirect profits via learning what doesn't work (an important part of science), then this vastly diminishes the number of worthless papers (to essentially those that are fraudulent or plagiarism) But as specifically for AlphaFold, I'm going to need a citation on that. If I understand the calculus correctly, Google acquired DM in 2014 for somewhere between $525 million to $850 million, and yearly spends a similar amount each year along with forgiving a 1.5bn dollar debt[0]. So I think (VERY) conservatively we can say $2bn (I think even $4bn is likely conservative here)? While I see articles that discuss how the value could be north of $100bn[1] (which certainly surpasses a very liberal estimate of costs), I have yet to see evidence that this is actual value that has been returned to Google. I can only find information about 2022 and 2023 having profits in the ballpark of $60m. This isn't to say that AlphaFold won't offset all the costs (I actually believe it will), but I your sentence does not suggest speculation but rather actualization ("has basically paid", "been"). I think that difference matters enough that we have a dozen cliches with similar sentiment. In the same way my annoyance is not that we are investing in ML[2], but how quick we are to make promises and celebrate success[3]. Actually, my concern is that while hype is necessary, overdoing it allows charlatans[4] to more easily enter the space. And if they are able to gain a significant foothold (I believe that is happening/has happened) then this is actually destructive to those who actually wish to push forward technology. [0] https://www.quora.com/How-much-money-did-Google-spend-on-DeepMind-in-its-early-years-before-DeepMind-started-generating-revenue-from-its-products-and-services-e-g-AlphaGo https://www.quora.com/How-much-money-did-Google-spend-on-Dee... [1] https://www.bloomberg.com/news/articles/2024-05-08/deepmind-ceo-targets-100-billion-plus-ai-drug-discovery-business-with-alphafold https://www.bloomberg.com/news/articles/2024-05-08/deepmind-... [2] disclosure, I'm an ML researcher. I actually am in favor of more funding. Though different allocation. [3] I'm willing to concede that success is realistically determined by how one measures success, and that this may be arbitrary and no objective measure actually exists or is possible. [4] One needs not knowingly be a charlatan. Only that the claims made are false or inaccurate. There are many charlatans who believe in the snake oil they sell. Most of these are unwilling to acknowledge critiques. A clear example is religion. If you believe in a religion, this applies to all religious organizations except the one you are a part of. If you are not religious, well the same sentence holds true but the resultant set is one larger.
- dontreact 2y ago
  I agree with you but it also makes me think: Google's TPUs are also fixed costs and these research experiments could have been run at times when production serving need isn't as high.
  - ec109685 2y ago
    They sell them on the spot market, so there’s someone that would consume the baseline compute.
    - dontreact 2y ago
      I imagine it would never be optimal to set a price so low that utilization is always 100% externally
      - londons_explore 2y ago
        There is plenty of people who want cheap compute and are willing to wait till 3am if that's when the cheap compute is. Happens for all computing services, but for ML stuff the effect is even more pronounced because compute costs are typically a large part of the cost of many projects.
- ramraj07 2y ago
  Nope this is literally my estimate of how much the experiments cost for the paper alone (reagents, animals, and the labor for these experiments).
dumb1224 2y ago
Well not everyone starts experiment anew. Many also reuse accumulated datasets. For human data even more so.

slashdave 2y ago

I assure you that the companies performing these screens expect a return on this investment. It is not for a journal paper.

godelski 2y ago
I used to believe this line. But then I worked for a big tech company where my manager constantly made those remarks ("the difference in industry and academia is that in industry it has to actually work"). I then improved the generalization performance (i.e. "actually work") by over 100% and they decided not to update the model they were selling. Then again, I had a small fast model and it was 90% as accurate as the new large transformer model. Though they also didn't take the lessons learned and apply them to the big model, which had similar issues but were just masked by the size. Plus, I mean, there are a lot of products that don't work. We all buy garbage and often can't buy not garbage. Though I guess you're technically correct that in either of these situations there can still be a return on investment, but maybe that shouldn't be good enough...
- shpongled 2y ago
  The post you are replying to is talking about high throughput assays for drug development. This is something actually run in a lab, not a model. As another person working at a biotech, I can assure you that screens are not just run as busy work.
  - rgmerk 2y ago
    No they’re not busywork, but not all such screens are directly in the drug discovery pipeline.
    - godelski 2y ago
      It also entirely misses the point of my comment which was that I don't believe this sentence to be true in an at least indirect sense. I conceded that it is technically valid in that lab experiments can also be used as advertisements and generate revenue even if such screens do not directly lead to novel or improved drug discoveries.

Metacelsus 2y ago

Yeah, I'm a wet-lab biologist and my most recent paper (which is still not past peer review) has already cost about $200,000. And I just spent another $2000 today...

deleted 2y ago

[deleted]