9 ms·

BioNTech-InstaDeep Early Warning System to Detect High-Risk SARS-CoV-2 Variants

149 points by adsodemelk 5y ago

alexwg 5y ago
Preprint: https://www.biorxiv.org/content/10.1101/2021.12.24.474095v1 https://www.biorxiv.org/content/10.1101/2021.12.24.474095v1
- est31 5y ago
  Pretty cool stuff! > When using a weekly watch-list with a size of 20 variants (less than 0.5% of the weekly average of new variant sequences), EWS flagged 12 WHO designated variants out of 13 (Fig. 4.A), with an average of 58 days of lead time (i.e two months) before these were designated as such by the WHO (Table S.4). > Our system however does not accurately pinpoint the emergence of the B.1.617.2 Delta family of variants. Delta is known to be neutralised by vaccines24 and its global prevalence can be attributed to other fitness-enhancing factors [than immune escape]. These factors, such as P681R mutation, which abrogates O-glycosylation, thus further enabling furin cleavage, are outside of the scope of our approach. > Specifically, the EWS identified Omicron as the highest immune escaping variant over more than 70,000 variants discovered between early October and late November 2021.
axg11 5y ago
Impressive if true, although I am very sceptical of the results. The real test is to put their neck on the line and predict/warn of the next significant variant. If the authors are not willing to do this, I doubt that the algorithm is useful. As a side note, the first author on this paper is the CEO of InstaDeep. I see it as a red flag that the CEO of a 150+ person company would put themselves as first author on this paper. Perhaps I'm unfairly judging and the CEO really was the lead contributor to the study.
- mlindner 5y ago
  > As a side note, the first author on this paper is the CEO of InstaDeep. I see it as a red flag that the CEO of a 150+ person company would put themselves as first author on this paper. Perhaps I'm unfairly judging and the CEO really was the lead contributor to the study. Authors on scientific papers are ordered by name, so whoever's name is first is pure coincidence.
  - axg11 5y ago
    This isn't true for the field of biology or medicine. If you take a look at the order of authors on the paper in question you can also see that the ordering is not alphabetical.
  - gpm 5y ago
    The order of authors varies by field. I believe in biology it usually starts with the person who did most of the work (or people, but if more than one it's explicitly noted), everyone who contributed in the middle, principal investigator (i.e. head of lab) last. In this case I notice that they first sorted all the instadeep people to the front, and biontech people to the end... not sure what I'm supposed to take away from the ordering, but it's not random.
    - meepmorp 5y ago
      Or sometimes PI first, depending on the scope of the PI's ego
    - orintorynchus 5y ago
      It is not random. InstaDeep has designed, developed, built, benchmarked the method. BioNTech performed experimental validation, provided expert insight into the problem domain. Hence, authors are segmented by contribution areas.
- orintorynchus 5y ago
  I am the second author of the paper. We have been putting our head on the line for the last half a year. We detected Lambda, Mu (with a caveat, that we did not consider it competitive) and Omicron - all blindly. We have been verifying all our predictions experimentally, post factum. And the method is purely data driven - with no fitting to the experiments or observations. The first author came up with the approach, participated in analysis and got his hands dirty as everyone else. While he is a CEO of a 160+ person company, this has been a labor of love for all of us, done to a large extent in the evenings, during weekends and holidays. It is indeed an unusual situation. But this was not a regular project and InstaDeep is not a regular company either.
  - whycombagator 5y ago
    Thank you for chiming in. It’s great that you detected them. I’m curious how many false positives you had during that same time? Did you detect many others that just didn’t pan out to be of significance?
    - orintorynchus 5y ago
      This is something that amazes me. Any time a true High Risk Variant appears, it is clear as day in the system. This was the case with Lambda, Mu (which we predicted to have a limited propensity to proliferate), and now with Omicron. However, there are lineages, which are prospectively dangerous, that we detect. As the classification is relative, there is no fixed threshold beyond which we would call for alert. In the evaluation, we have been looking at 20 sequences per week, as this was roughly the testing capacity of our partners. Going for ONE sequence a week makes us detect some variants a bit later, but still. The sequences and lineages we predicted to be of interest were predominantly spreading afterwards. Some were just a blip on the radar, though. We aimed at sensitivity and not missing ominous signs, rather than specificity. NB: Each week there are thousands (now 12k+) new sequence variants. Out of them we consider 20. And detect most of the variants as early on as on the first day.
  - zxexz 5y ago
    Wow, amazing to see you here. I have somewhat of a bioinformatics background, and this is the sort of work that I always found very interesting - do you think your team will produce some sort of high level architectural and process breakdown at some point?
- jdavis703 5y ago
  Because of both founder effects and potentially people taking earlier containment actions, I’m unsure that’s a realistic measure. It’s like how people said Y2K was just a conspiracy theory because most systems kept running — without realizing the early warnings and preparations is the reason Y2K wasn’t a civilizational disruption.
1cvmask 5y ago
The blog post from the InstaDeep site: https://www.instadeep.com/2022/01/biontech-and-instadeep-developed-and-successfully-tested-early-warning-system-to-detect-potential-high-risk-sars-cov-2-variants/ https://www.instadeep.com/2022/01/biontech-and-instadeep-dev...
fasteddie31003 5y ago
I'm all for stopping new variants of COVID. Why in the world do we have a treatment for COVID that intentionally creates mutations in COVID https://en.wikipedia.org/wiki/Molnupiravir https://en.wikipedia.org/wiki/Molnupiravir ? I'm not a doctor but it seems pretty clear to me that Molnupiravir creates a great environment for new variants to arise (along with cancer, but that's another issue).
- snet0 5y ago
  How does it seem clear to you? What's the mechanism through with Molnupiravir creates new CoV mutations?
  - rmellow 5y ago
    GP explicitly referenced the drug's Wikipedia page: > "The emergency use authorization was only narrowly approved (13-10) because of questions regarding efficacy and concerns that molnupiravir's mutagenic effects could create new variants that evade immunity and prolong the COVID-19 pandemic." The quote above has its own citations in the Wikipedia page posted by GP.
  - cinntaile 5y ago
    He linked to a source that contains that claim, it includes several sources for that specific claim. It's unclear why that is relevant in this thread though.
- WJW 5y ago
  From the wikipedia page you linked: > Molnupiravir is indicated for the treatment of mild-to-moderate coronavirus disease (COVID-19) in adults with positive results of direct SARS-CoV-2 viral testing, and who are at high risk for progression to severe COVID-19.[1][5] So after the risk of creating mutations in the Covid virus has been assessed and weighed against the chances of patients just outright dying due to not having this medicine available, the board full of medical professionals trained in this matter voted 13 to 10 that they thought the risk was acceptable.
  - 5y ago
solididiot 5y ago
Regardless of the specifics here, I'm pleased and envious of being reminded that some people have meaningful jobs that actually make a difference.
- adsodemelk 5y ago
  Another author of the paper here. We are recruiting in case you would like to contribute in further meaningful projects. https://www.instadeep.com/about-us/careers/ https://www.instadeep.com/about-us/careers/
- orintorynchus 5y ago
  We are actively looking for motivated colleagues. Feel free to look at the page above or drop us an email at hello[at]instadeep.com. We are also happy to make new friends and work together on exciting projects - same email as above works.
dopylitty 5y ago
Given that we still don’t know the mechanism by which variants achieve their increased fitness I’m a little suspicious of claims of using “AI” to identify new variants. Claims that spike affinity for ACE2 (or indeed changes to the spike protein at all) haven’t been conclusively proven to increase fitness as far as I know. It’s possible that this tool is simply overfit to the known variants and wouldn’t detect a new one. I’d be interested to hear what real virologists think of this.
- maxerickson 5y ago
  Is the concern that this tool would be prioritized over other information? Like what is the harm in doing the data collection and publishing about how it is going?
defaultprimate 5y ago
- schleck8 5y ago
  That's what happens when you offer a good product
  - defaultprimate 5y ago
    Or when you're one of the most coercive lobbying forces on the globe.
jcfrei 5y ago
spuz 5y ago
> More than 10,000 novel variant sequences are currently discovered every week and human experts simply cannot cope with complex data at this scale This is interesting. I think the Greek alphabet naming system would lead some people to believe the virus only mutates once every few months. Of course, the reality is that every infected individual will produce hundreds of mutations within their body. I think there's a gap in the public messaging here which if addressed could help people understand what the future direction of the pandemic might be.
- im3w1l 5y ago
  Well to be classified as a variant it would have to be phenotypically different right? And also viable enough to infect a number of people.
  - orintorynchus 5y ago
    Well, there is a lot of overlapping nomenclature. Here by variant we understand any sample, which is not identical sequence-wise to sequences seen before. Most of the observed mutations are innocuous and do not lead to a potential new lineage (what better fits the definition of variant above). However, it is difficult to tell by eye - or even with the standard computational tools - if new mutation is (a) deleterious (harms the virus' capacity to spread), (b) neutral, or (c) fitness enhancing.
- carbocation 5y ago
  Public messaging often includes the more extensive PANGO lineage (e.g., B.1.1.529 for Omicron). I think there is a limit to what can be conveyed in each news story.
- jdavis703 5y ago
  This is the kind of education that needs to be done in high school during health and biology classes. We can’t hope the general public will become epidemiologically literate on 240 character tweets and 30-second television quotes. Heck, my old high school biology teacher remains a source of coronavirus FUD, so maybe that’s not even enough.
twofornone 5y ago
>More than 10,000 novel variant sequences are currently discovered every week and human experts simply cannot cope with complex data at this scale I suspected something like this, given the frequency of viral mutation. Officials announce a dominant global strain but what proportion of positive cases are actually sequenced and evaluated for confirmation? How many undiscovered strains are actually in circulation at any given time, in geographically isolated areas? Could that explain variability in severity and/or long covid?
- deleted 5y ago
  [deleted]
liuliu 5y ago
Is the idea to develop mRNA vaccines that can boost immunity against variants that haven't existed today?
- orintorynchus 5y ago
  The idea is to introduce a bit of sanity into the current world, to calm down overreactions to every new, scary looking variant. And on the flip side - to identify the truly scary ones early on, so that proper measures can be taken. But the idea above directly follows from the work in the paper.
a45a33s 5y ago
dam53434 5y ago
dam53434 5y ago
just_steve_h 5y ago
I assume this is also part of a product development process: if the EWS flags a particular variant, BioNTech can begin crafting a new vaccine component and have it ready sooner, earning a competitive advantage in the (capitalist) vaccine marketplace.
abeppu 5y ago
This is not my area and I read the press release but not the paper -- but I cannot help noticing that they mention a sensitivity/recall number (>90%) but not a specificity or precision number. Even if you're not trying to be cynical or skeptical about this, when there are this few true positive examples available, how can one plausibly do a good job calibrating such a system?
- orintorynchus 5y ago
  That's the key bit here. Supervised learning is not applicable here and any sort of fitting to known labels is doomed to fail. The system is not calibrated, tuned or parameterized. The ML part learns in a self-supervised manner what are the spike sequence features important for a successful (proliferating) coronavirus (or - more correctly - learns low-dimensional embeddings of multi-point interactions/co-occurrences of different amino acids in spike proteins). The rest is based on either frequentist statistics, or computational biochemistry. At no point in training EWS any information about certain sequences belonging to High Risk Variant classes (Variant of Concern, Variant of Interest etc.) is fed to EWS.
71a54xd 5y ago
Is this literally a joke? This might as well be a doomsday machine driven by weak AI selecting new Greek glyphs to drive demand for new superfluous covid vaccines. I'm fully vaccinated, but more data points to come up with media fear mongering is the last thing anyone in the developed world needs right now.
- consumer451 5y ago
  One of the authors states here that the goal was to create sanity(data) for the press to reference on each new variant.
n380 5y ago
amazing, who thought machine learning will help on finding variant is dangerous or not without going to lab !
- orintorynchus 5y ago
  Not exactly "finding out", but rather estimating. It's all about prioritizing testing through detecting suspiciously "good" and unexpected outliers popping up.
wesletimk 5y ago
Very good News, I hope this will save a lot of souls
wesletimk 5y ago
bolbols 5y ago
Great, finally one real example where AI can really help with this pandemic
mrtri 5y ago

149 points by adsodemelk 5y ago

Preprint: https://www.biorxiv.org/content/10.1101/2021.12.24.474095v1 https://www.biorxiv.org/content/10.1101/2021.12.24.474095v1

est31 5y ago
Pretty cool stuff! > When using a weekly watch-list with a size of 20 variants (less than 0.5% of the weekly average of new variant sequences), EWS flagged 12 WHO designated variants out of 13 (Fig. 4.A), with an average of 58 days of lead time (i.e two months) before these were designated as such by the WHO (Table S.4). > Our system however does not accurately pinpoint the emergence of the B.1.617.2 Delta family of variants. Delta is known to be neutralised by vaccines24 and its global prevalence can be attributed to other fitness-enhancing factors [than immune escape]. These factors, such as P681R mutation, which abrogates O-glycosylation, thus further enabling furin cleavage, are outside of the scope of our approach. > Specifically, the EWS identified Omicron as the highest immune escaping variant over more than 70,000 variants discovered between early October and late November 2021.

axg11 5y ago

Impressive if true, although I am very sceptical of the results. The real test is to put their neck on the line and predict/warn of the next significant variant. If the authors are not willing to do this, I doubt that the algorithm is useful. As a side note, the first author on this paper is the CEO of InstaDeep. I see it as a red flag that the CEO of a 150+ person company would put themselves as first author on this paper. Perhaps I'm unfairly judging and the CEO really was the lead contributor to the study.

mlindner 5y ago
> As a side note, the first author on this paper is the CEO of InstaDeep. I see it as a red flag that the CEO of a 150+ person company would put themselves as first author on this paper. Perhaps I'm unfairly judging and the CEO really was the lead contributor to the study. Authors on scientific papers are ordered by name, so whoever's name is first is pure coincidence.
- axg11 5y ago
  This isn't true for the field of biology or medicine. If you take a look at the order of authors on the paper in question you can also see that the ordering is not alphabetical.
- gpm 5y ago
  The order of authors varies by field. I believe in biology it usually starts with the person who did most of the work (or people, but if more than one it's explicitly noted), everyone who contributed in the middle, principal investigator (i.e. head of lab) last. In this case I notice that they first sorted all the instadeep people to the front, and biontech people to the end... not sure what I'm supposed to take away from the ordering, but it's not random.
  - meepmorp 5y ago
    Or sometimes PI first, depending on the scope of the PI's ego
  - orintorynchus 5y ago
    It is not random. InstaDeep has designed, developed, built, benchmarked the method. BioNTech performed experimental validation, provided expert insight into the problem domain. Hence, authors are segmented by contribution areas.
orintorynchus 5y ago
I am the second author of the paper. We have been putting our head on the line for the last half a year. We detected Lambda, Mu (with a caveat, that we did not consider it competitive) and Omicron - all blindly. We have been verifying all our predictions experimentally, post factum. And the method is purely data driven - with no fitting to the experiments or observations. The first author came up with the approach, participated in analysis and got his hands dirty as everyone else. While he is a CEO of a 160+ person company, this has been a labor of love for all of us, done to a large extent in the evenings, during weekends and holidays. It is indeed an unusual situation. But this was not a regular project and InstaDeep is not a regular company either.
- whycombagator 5y ago
  Thank you for chiming in. It’s great that you detected them. I’m curious how many false positives you had during that same time? Did you detect many others that just didn’t pan out to be of significance?
  - orintorynchus 5y ago
    This is something that amazes me. Any time a true High Risk Variant appears, it is clear as day in the system. This was the case with Lambda, Mu (which we predicted to have a limited propensity to proliferate), and now with Omicron. However, there are lineages, which are prospectively dangerous, that we detect. As the classification is relative, there is no fixed threshold beyond which we would call for alert. In the evaluation, we have been looking at 20 sequences per week, as this was roughly the testing capacity of our partners. Going for ONE sequence a week makes us detect some variants a bit later, but still. The sequences and lineages we predicted to be of interest were predominantly spreading afterwards. Some were just a blip on the radar, though. We aimed at sensitivity and not missing ominous signs, rather than specificity. NB: Each week there are thousands (now 12k+) new sequence variants. Out of them we consider 20. And detect most of the variants as early on as on the first day.
- zxexz 5y ago
  Wow, amazing to see you here. I have somewhat of a bioinformatics background, and this is the sort of work that I always found very interesting - do you think your team will produce some sort of high level architectural and process breakdown at some point?
jdavis703 5y ago
Because of both founder effects and potentially people taking earlier containment actions, I’m unsure that’s a realistic measure. It’s like how people said Y2K was just a conspiracy theory because most systems kept running — without realizing the early warnings and preparations is the reason Y2K wasn’t a civilizational disruption.

1cvmask 5y ago

The blog post from the InstaDeep site: https://www.instadeep.com/2022/01/biontech-and-instadeep-developed-and-successfully-tested-early-warning-system-to-detect-potential-high-risk-sars-cov-2-variants/ https://www.instadeep.com/2022/01/biontech-and-instadeep-dev...

fasteddie31003 5y ago

I'm all for stopping new variants of COVID. Why in the world do we have a treatment for COVID that intentionally creates mutations in COVID https://en.wikipedia.org/wiki/Molnupiravir https://en.wikipedia.org/wiki/Molnupiravir ? I'm not a doctor but it seems pretty clear to me that Molnupiravir creates a great environment for new variants to arise (along with cancer, but that's another issue).

snet0 5y ago
How does it seem clear to you? What's the mechanism through with Molnupiravir creates new CoV mutations?
- rmellow 5y ago
  GP explicitly referenced the drug's Wikipedia page: > "The emergency use authorization was only narrowly approved (13-10) because of questions regarding efficacy and concerns that molnupiravir's mutagenic effects could create new variants that evade immunity and prolong the COVID-19 pandemic." The quote above has its own citations in the Wikipedia page posted by GP.
- cinntaile 5y ago
  He linked to a source that contains that claim, it includes several sources for that specific claim. It's unclear why that is relevant in this thread though.
WJW 5y ago
From the wikipedia page you linked: > Molnupiravir is indicated for the treatment of mild-to-moderate coronavirus disease (COVID-19) in adults with positive results of direct SARS-CoV-2 viral testing, and who are at high risk for progression to severe COVID-19.[1][5] So after the risk of creating mutations in the Covid virus has been assessed and weighed against the chances of patients just outright dying due to not having this medicine available, the board full of medical professionals trained in this matter voted 13 to 10 that they thought the risk was acceptable.
- 5y ago

solididiot 5y ago

Regardless of the specifics here, I'm pleased and envious of being reminded that some people have meaningful jobs that actually make a difference.

adsodemelk 5y ago
Another author of the paper here. We are recruiting in case you would like to contribute in further meaningful projects. https://www.instadeep.com/about-us/careers/ https://www.instadeep.com/about-us/careers/
orintorynchus 5y ago
We are actively looking for motivated colleagues. Feel free to look at the page above or drop us an email at hello[at]instadeep.com. We are also happy to make new friends and work together on exciting projects - same email as above works.

dopylitty 5y ago

Given that we still don’t know the mechanism by which variants achieve their increased fitness I’m a little suspicious of claims of using “AI” to identify new variants. Claims that spike affinity for ACE2 (or indeed changes to the spike protein at all) haven’t been conclusively proven to increase fitness as far as I know. It’s possible that this tool is simply overfit to the known variants and wouldn’t detect a new one. I’d be interested to hear what real virologists think of this.

maxerickson 5y ago
Is the concern that this tool would be prioritized over other information? Like what is the harm in doing the data collection and publishing about how it is going?

defaultprimate 5y ago

schleck8 5y ago
That's what happens when you offer a good product
- defaultprimate 5y ago
  Or when you're one of the most coercive lobbying forces on the globe.

jcfrei 5y ago

spuz 5y ago

> More than 10,000 novel variant sequences are currently discovered every week and human experts simply cannot cope with complex data at this scale This is interesting. I think the Greek alphabet naming system would lead some people to believe the virus only mutates once every few months. Of course, the reality is that every infected individual will produce hundreds of mutations within their body. I think there's a gap in the public messaging here which if addressed could help people understand what the future direction of the pandemic might be.

im3w1l 5y ago
Well to be classified as a variant it would have to be phenotypically different right? And also viable enough to infect a number of people.
- orintorynchus 5y ago
  Well, there is a lot of overlapping nomenclature. Here by variant we understand any sample, which is not identical sequence-wise to sequences seen before. Most of the observed mutations are innocuous and do not lead to a potential new lineage (what better fits the definition of variant above). However, it is difficult to tell by eye - or even with the standard computational tools - if new mutation is (a) deleterious (harms the virus' capacity to spread), (b) neutral, or (c) fitness enhancing.
carbocation 5y ago
Public messaging often includes the more extensive PANGO lineage (e.g., B.1.1.529 for Omicron). I think there is a limit to what can be conveyed in each news story.
jdavis703 5y ago
This is the kind of education that needs to be done in high school during health and biology classes. We can’t hope the general public will become epidemiologically literate on 240 character tweets and 30-second television quotes. Heck, my old high school biology teacher remains a source of coronavirus FUD, so maybe that’s not even enough.

twofornone 5y ago

>More than 10,000 novel variant sequences are currently discovered every week and human experts simply cannot cope with complex data at this scale I suspected something like this, given the frequency of viral mutation. Officials announce a dominant global strain but what proportion of positive cases are actually sequenced and evaluated for confirmation? How many undiscovered strains are actually in circulation at any given time, in geographically isolated areas? Could that explain variability in severity and/or long covid?

deleted 5y ago
[deleted]

liuliu 5y ago

Is the idea to develop mRNA vaccines that can boost immunity against variants that haven't existed today?

orintorynchus 5y ago
The idea is to introduce a bit of sanity into the current world, to calm down overreactions to every new, scary looking variant. And on the flip side - to identify the truly scary ones early on, so that proper measures can be taken. But the idea above directly follows from the work in the paper.

a45a33s 5y ago

dam53434 5y ago

just_steve_h 5y ago

I assume this is also part of a product development process: if the EWS flags a particular variant, BioNTech can begin crafting a new vaccine component and have it ready sooner, earning a competitive advantage in the (capitalist) vaccine marketplace.

abeppu 5y ago

This is not my area and I read the press release but not the paper -- but I cannot help noticing that they mention a sensitivity/recall number (>90%) but not a specificity or precision number. Even if you're not trying to be cynical or skeptical about this, when there are this few true positive examples available, how can one plausibly do a good job calibrating such a system?

orintorynchus 5y ago
That's the key bit here. Supervised learning is not applicable here and any sort of fitting to known labels is doomed to fail. The system is not calibrated, tuned or parameterized. The ML part learns in a self-supervised manner what are the spike sequence features important for a successful (proliferating) coronavirus (or - more correctly - learns low-dimensional embeddings of multi-point interactions/co-occurrences of different amino acids in spike proteins). The rest is based on either frequentist statistics, or computational biochemistry. At no point in training EWS any information about certain sequences belonging to High Risk Variant classes (Variant of Concern, Variant of Interest etc.) is fed to EWS.

71a54xd 5y ago

Is this literally a joke? This might as well be a doomsday machine driven by weak AI selecting new Greek glyphs to drive demand for new superfluous covid vaccines. I'm fully vaccinated, but more data points to come up with media fear mongering is the last thing anyone in the developed world needs right now.

consumer451 5y ago
One of the authors states here that the goal was to create sanity(data) for the press to reference on each new variant.

n380 5y ago

amazing, who thought machine learning will help on finding variant is dangerous or not without going to lab !

orintorynchus 5y ago
Not exactly "finding out", but rather estimating. It's all about prioritizing testing through detecting suspiciously "good" and unexpected outliers popping up.

wesletimk 5y ago

Very good News, I hope this will save a lot of souls

wesletimk 5y ago

bolbols 5y ago

Great, finally one real example where AI can really help with this pandemic

mrtri 5y ago