6 ms·

I particularly like the rant on pharmaceuticals companies lack of basic research. My impression has been that medical progression have been slow for quite some

I particularly like the rant on pharmaceuticals companies lack of basic research. My impression has been that medical progression have been slow for quite some time, nice to see that there are some truth to that. In the end software and tech companies might just eat up the pharmaceutical industry as well. - It's all just code at some level. The Deepmind team did this with ; "We trained this system on publicly available data consisting of ~170,000 protein structures from the protein data bank together with large databases containing protein sequences of unknown structure. It uses approximately 128 TPUv3 cores (roughly equivalent to ~100-200 GPUs) run over a few weeks, which is a relatively modest amount of compute in the context of most large state-of-the-art models used in machine learning today." So it wasn't out of reach for academia, pharmaceuticals, or others with a bit of resources.

flobosg 6y ago
Yeah, it was a big slap in the face. But, to be fair, most of the scientific and technological advances (sequencing efforts, structural genomics projects, etc.) that generated the data used by DeepMind came from academia and, to a lesser extent, the pharma industry.
- sjg007 6y ago
  I think the lesson here is that most of the big data genomic, metabolic, pharmacologic and other research will all be driven by deep learning. The models themselves however require 100+ gpus so we are sort of back in that phase where you need large compute systems to even compete. A single lab will have issues unless they can leverage a cloud and then also get grant funding to spend that money on the cloud compute... which may be difficult b/c its basically a consumable now and you don't have any hardware leftover.
throwawayiionqz 6y ago
This is the cost of training the final architecture with all the refinements enabled by years of research. These years of research involved trying many different architectures, many of which received as much or more compute time than the final system. The price of training the final architecture is meaningless. Researching and training AlphaGo was expensive but it enabled the ideas and development of AlphaZero which is more computationally tractable. To have any chance, an academic team would need the same compute resources as what the DeepMind protein folding team used during the whole development of the architecture during the last few years, not only the resources used to train the final system. And I bet this funding is not available to most if not all academic teams.
- zaroth 6y ago
  > The price of training the final architecture is meaningless. The research is the giant shoulders you stand on, the compute cost is the price of the tool you need to do the present-day work. Both are relevant but the shoulder’s of giants are generally more accessible, particularly if we’re talking about published research and not proprietary tech. A competing team is not starting from the same place the DeepMind team started at 5 or 10 years ago.
  - zaroth 6y ago
    To expand on this, after fully reading AlQuraishi's "What Just Happened" post from a couple years ago, was this point that he made; > I don’t think we would do ourselves a service by not recognizing that what just happened presents a serious indictment of academic science. There are dozens of academic groups, with researchers likely numbering in the (low) hundreds, working on protein structure prediction. We have been working on this problem for decades, with vast expertise built up on both sides of the Atlantic and Pacific, and not insignificant computational resources when measured collectively. For DeepMind’s group of ~10 researchers, with primarily (but certainly not exclusively) ML expertise, to so thoroughly route everyone surely demonstrates the structural inefficiency of academic science. This is not Go, which had a handful of researchers working on the problem, and which had no direct applications beyond the core problem itself. Protein folding is a central problem of biochemistry, with profound implications for the biological and chemical sciences. How can a problem of such vital importance be so badly neglected? In short, academia got utterly schooled by a small group at Google spending a relatively small dollar amount on compute, using techniques that in hindsight are fairly described as "simplistic". There's no way around it.
    - dash2 6y ago
      To be fair, I'm not sure that they are "simplistic" in the sense that, e.g., writing a neural network to recognise cat pictures is now simplistic. I don't know how many people have Deepmind levels of expertise in ML, or could implement what they have done, but I doubt it is many, and they are thinly spread amongst many interesting problems.
tonfa 6y ago
> So it wasn't out of reach for academia, pharmaceuticals, or others with a bit of resources. How much does hiring a deepmind-like team cost though? (massively more than the TPU resources?) Still within reach of pharmaceutical industry I guess, but maybe not so easy for academia.
- Seanambers 6y ago
  From what I can gather, Google bought Deepmind for 500 million USD in 2014, they have outstanding debt to its parent company as of 2019 of 1.3 billion USD. And they had income around 100 million in 2019 but it's all against Google, so looks like a 2 billion +/- 0.5 operation so far, and who knows if they pay for compute. Other articles place the runrate at 500 million per year in 2019. Which means 500 million * 6 years = 3 bn + 0.5 purchase price. = 3.5 bn. So somewhere in the 2.5 - 3.5 billion range its seems likely as total cost so far. Nevertheless doesn't seem out of reach for a multinational.
  - sseagull 6y ago
    It would still be a significant amount of money for a lot of companies. Remember, we are looking in hindsight that it seemingly paid off. A few years ago, this was just an educated bet; only the richest companies with money to burn (from selling ads) would be willing to take on that kind of a risk.
  - TulliusCicero 6y ago
    That's the cost of running DeepMind as a whole, right? Which includes all the other stuff they've worked on, like games.
    - Seanambers 6y ago
      Yeah, as far as I can tell, that's the whole lot of it.
  - xbmcuser 6y ago
    Only the energy cost savings google got from Deepmind probably already makes it a very profitable acquisition https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40 https://deepmind.com/blog/article/deepmind-ai-reduces-google...
beowulfey 6y ago
I mean, credit where credit is due. Google employs some of the greatest names in artificial intelligence and the DeepMind team had a huge chunk of them working on this problem. While the resources may have been available, I don’t think any other single institution had the level of brain power.
- mrDmrTmrJ 6y ago
  Absolutely. The capability to "create" the breakthrough is extremely rare. Perhaps only DeepMind, OpenAI, and GoogleBrain can assemble these types of teams. Luckily, the capability to replicate and exploit the breakthrough is far more 'common'; though still very rare. Excited to see how follow on use of these models, by many more teams, researchers, and companies plays out over the next two decades. This is a foundational advance!
- elcritch 6y ago
  It also makes one reconsider the notion that monopolies are entirely bad. This essentially appears to be a vanity project for Google. Though of course they'll benefit from it in many ways, but it's not like they're doing this as the core product of their service. It's a pretty awesome achievement.
  - soup10 6y ago
    It's kind of like a modern day Bell Labs where they have so much excess profit from adtech that they can fund lots of "basic research" or the computer science equivalent of that.
  - Supermancho 6y ago
    > It also makes one reconsider the notion that monopolies are entirely bad. Much like political dictators, they can be exceedingly efficient and have resources (and authority) to do things in spite of opposing interests. People who faced with the narrative that countries have a monopoly on a number of aspects of life find monopolies are not a BAD THING(tm), but that they are bad for a consumer market - as a monopoly eventually blockades aspects of the market.
IfOnlyYouKnew 6y ago
In a prior(/n) life I worked on Protein folding, and participated in CASP. This was a/the "holy grail" problem of molecular biology, long thought to be an automatic Nobel. It's somewhat unfair to characterise developments prior to this as insignificant. In fact by the time I was working on it, that "automatic Nobel" was no longer assumed, because the field had made quite a bit of progress, in many tiny steps by many different groups, and the assumption was it would continue in this slog until reaching some state of sufficiency for practical applications without ever seeing the sort of singular achievement that would be worthy of praise and prize. Far more went into this breakthrough, obviously, than those TPU-hours: the development of those TPUs, for example, and assembling a team that can make use of them. The protein folding problem requires very little knowledge of biology or physics to understand and was always pre-destined for some outsider to sweep. Indeed, there was game that allowed people to solve structures by intuition alone, and, IIRC, some 13-year old Mexican kid cleaned everyone's clock some years back. Why didn't some research group do this first? Most of them just don't have the budget. We were five people, total, IIRC, and felt pretty rich because we were computer-people getting the same budget for materials as everyone at our institution, which was all wetlab, otherwise. So I was a student being paid $20/h but with a $50,000/p.a. hardware budget. How many false start does it take before you do that run with 128TPUs "for a few weeks" that works? If you blow your budget on one gigantic Google invoice, what's going to happen to you when it doesn't pan out, and the whole institute laughs at you? Etc... There are quite a few rather good things this problem has inspired over the years, though. Among them is CASP itself: the idea of instituting a yearly competition that gives unequivocal feedback on the state of the field and every group working on it is rather rare, I believe, and it's been successful. Indeed, it would seem that CASP was necessary to attract outside groups like Deepmind, i. e. deep-pocketed industry groups striving to prove themselves on a clearly defined problem. Chess, Jeopardy, CASP: maybe it would be worthwhile to explore not <solving x>, but <stating X as a problem that attracts Google/IBM/etc.-scale money> as a superior strategy in some cases. There was also folding@home, pioneering the distributed-donated-computing model, and the aforementioned gamification of the problem, and hundreds of the most intricate, custom-tailed, more-or-less insane ideas people devoted months and/or careers and/or careers of their most promising post-docs to that didn't pan out. Like cellular automata. They don't work for this, trust me. (Great hit for interactive poster sessions, though)

flobosg 6y ago

Yeah, it was a big slap in the face. But, to be fair, most of the scientific and technological advances (sequencing efforts, structural genomics projects, etc.) that generated the data used by DeepMind came from academia and, to a lesser extent, the pharma industry.

sjg007 6y ago
I think the lesson here is that most of the big data genomic, metabolic, pharmacologic and other research will all be driven by deep learning. The models themselves however require 100+ gpus so we are sort of back in that phase where you need large compute systems to even compete. A single lab will have issues unless they can leverage a cloud and then also get grant funding to spend that money on the cloud compute... which may be difficult b/c its basically a consumable now and you don't have any hardware leftover.

throwawayiionqz 6y ago

This is the cost of training the final architecture with all the refinements enabled by years of research. These years of research involved trying many different architectures, many of which received as much or more compute time than the final system. The price of training the final architecture is meaningless. Researching and training AlphaGo was expensive but it enabled the ideas and development of AlphaZero which is more computationally tractable. To have any chance, an academic team would need the same compute resources as what the DeepMind protein folding team used during the whole development of the architecture during the last few years, not only the resources used to train the final system. And I bet this funding is not available to most if not all academic teams.

zaroth 6y ago
> The price of training the final architecture is meaningless. The research is the giant shoulders you stand on, the compute cost is the price of the tool you need to do the present-day work. Both are relevant but the shoulder’s of giants are generally more accessible, particularly if we’re talking about published research and not proprietary tech. A competing team is not starting from the same place the DeepMind team started at 5 or 10 years ago.
- zaroth 6y ago
  To expand on this, after fully reading AlQuraishi's "What Just Happened" post from a couple years ago, was this point that he made; > I don’t think we would do ourselves a service by not recognizing that what just happened presents a serious indictment of academic science. There are dozens of academic groups, with researchers likely numbering in the (low) hundreds, working on protein structure prediction. We have been working on this problem for decades, with vast expertise built up on both sides of the Atlantic and Pacific, and not insignificant computational resources when measured collectively. For DeepMind’s group of ~10 researchers, with primarily (but certainly not exclusively) ML expertise, to so thoroughly route everyone surely demonstrates the structural inefficiency of academic science. This is not Go, which had a handful of researchers working on the problem, and which had no direct applications beyond the core problem itself. Protein folding is a central problem of biochemistry, with profound implications for the biological and chemical sciences. How can a problem of such vital importance be so badly neglected? In short, academia got utterly schooled by a small group at Google spending a relatively small dollar amount on compute, using techniques that in hindsight are fairly described as "simplistic". There's no way around it.
  - dash2 6y ago
    To be fair, I'm not sure that they are "simplistic" in the sense that, e.g., writing a neural network to recognise cat pictures is now simplistic. I don't know how many people have Deepmind levels of expertise in ML, or could implement what they have done, but I doubt it is many, and they are thinly spread amongst many interesting problems.

tonfa 6y ago

> So it wasn't out of reach for academia, pharmaceuticals, or others with a bit of resources. How much does hiring a deepmind-like team cost though? (massively more than the TPU resources?) Still within reach of pharmaceutical industry I guess, but maybe not so easy for academia.

Seanambers 6y ago
From what I can gather, Google bought Deepmind for 500 million USD in 2014, they have outstanding debt to its parent company as of 2019 of 1.3 billion USD. And they had income around 100 million in 2019 but it's all against Google, so looks like a 2 billion +/- 0.5 operation so far, and who knows if they pay for compute. Other articles place the runrate at 500 million per year in 2019. Which means 500 million * 6 years = 3 bn + 0.5 purchase price. = 3.5 bn. So somewhere in the 2.5 - 3.5 billion range its seems likely as total cost so far. Nevertheless doesn't seem out of reach for a multinational.
- sseagull 6y ago
  It would still be a significant amount of money for a lot of companies. Remember, we are looking in hindsight that it seemingly paid off. A few years ago, this was just an educated bet; only the richest companies with money to burn (from selling ads) would be willing to take on that kind of a risk.
- TulliusCicero 6y ago
  That's the cost of running DeepMind as a whole, right? Which includes all the other stuff they've worked on, like games.
  - Seanambers 6y ago
    Yeah, as far as I can tell, that's the whole lot of it.
- xbmcuser 6y ago
  Only the energy cost savings google got from Deepmind probably already makes it a very profitable acquisition https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40 https://deepmind.com/blog/article/deepmind-ai-reduces-google...

beowulfey 6y ago

I mean, credit where credit is due. Google employs some of the greatest names in artificial intelligence and the DeepMind team had a huge chunk of them working on this problem. While the resources may have been available, I don’t think any other single institution had the level of brain power.

mrDmrTmrJ 6y ago
Absolutely. The capability to "create" the breakthrough is extremely rare. Perhaps only DeepMind, OpenAI, and GoogleBrain can assemble these types of teams. Luckily, the capability to replicate and exploit the breakthrough is far more 'common'; though still very rare. Excited to see how follow on use of these models, by many more teams, researchers, and companies plays out over the next two decades. This is a foundational advance!
elcritch 6y ago
It also makes one reconsider the notion that monopolies are entirely bad. This essentially appears to be a vanity project for Google. Though of course they'll benefit from it in many ways, but it's not like they're doing this as the core product of their service. It's a pretty awesome achievement.
- soup10 6y ago
  It's kind of like a modern day Bell Labs where they have so much excess profit from adtech that they can fund lots of "basic research" or the computer science equivalent of that.
- Supermancho 6y ago
  > It also makes one reconsider the notion that monopolies are entirely bad. Much like political dictators, they can be exceedingly efficient and have resources (and authority) to do things in spite of opposing interests. People who faced with the narrative that countries have a monopoly on a number of aspects of life find monopolies are not a BAD THING(tm), but that they are bad for a consumer market - as a monopoly eventually blockades aspects of the market.

IfOnlyYouKnew 6y ago

In a prior(/n) life I worked on Protein folding, and participated in CASP. This was a/the "holy grail" problem of molecular biology, long thought to be an automatic Nobel. It's somewhat unfair to characterise developments prior to this as insignificant. In fact by the time I was working on it, that "automatic Nobel" was no longer assumed, because the field had made quite a bit of progress, in many tiny steps by many different groups, and the assumption was it would continue in this slog until reaching some state of sufficiency for practical applications without ever seeing the sort of singular achievement that would be worthy of praise and prize. Far more went into this breakthrough, obviously, than those TPU-hours: the development of those TPUs, for example, and assembling a team that can make use of them. The protein folding problem requires very little knowledge of biology or physics to understand and was always pre-destined for some outsider to sweep. Indeed, there was game that allowed people to solve structures by intuition alone, and, IIRC, some 13-year old Mexican kid cleaned everyone's clock some years back. Why didn't some research group do this first? Most of them just don't have the budget. We were five people, total, IIRC, and felt pretty rich because we were computer-people getting the same budget for materials as everyone at our institution, which was all wetlab, otherwise. So I was a student being paid $20/h but with a $50,000/p.a. hardware budget. How many false start does it take before you do that run with 128TPUs "for a few weeks" that works? If you blow your budget on one gigantic Google invoice, what's going to happen to you when it doesn't pan out, and the whole institute laughs at you? Etc... There are quite a few rather good things this problem has inspired over the years, though. Among them is CASP itself: the idea of instituting a yearly competition that gives unequivocal feedback on the state of the field and every group working on it is rather rare, I believe, and it's been successful. Indeed, it would seem that CASP was necessary to attract outside groups like Deepmind, i. e. deep-pocketed industry groups striving to prove themselves on a clearly defined problem. Chess, Jeopardy, CASP: maybe it would be worthwhile to explore not <solving x>, but <stating X as a problem that attracts Google/IBM/etc.-scale money> as a superior strategy in some cases. There was also folding@home, pioneering the distributed-donated-computing model, and the aforementioned gamification of the problem, and hundreds of the most intricate, custom-tailed, more-or-less insane ideas people devoted months and/or careers and/or careers of their most promising post-docs to that didn't pan out. Like cellular automata. They don't work for this, trust me. (Great hit for interactive poster sessions, though)