7 ms·

Covid-19 Open Research Dataset

219 points by shinryudbz 7y ago

clircle 7y ago
How can applying a machine learning algorithm to this data (a collection of research papers) help fight Covid? It’s possible most papers here are bogus or low quality. Garbage in, garbage out?
- m3kw9 7y ago
  Yeah I don’t see any current AI models that can combine all the knowledge from a subject and do something really profound with it. I do see you can generate more AI covid19 articles from it.
- google234123 7y ago
  I imagine they have some algorithm that takes an input consisting of the title, abstract, and authors with the information about how impactful their previous work has been (this is probably the most important factor) and outputs some ranking or a likelihood that the research will be cited/clicked on.
  - inertiatic 7y ago
    That's feasible, it's not even a hard thing to do (maybe hard to do very accurately), I worked for a company that offered exactly that service, as a small project. Still, that probably doesn't answer the question, how does that really help the fight. I suppose we'll learn what they have in mind when those questions are asked in the competition.
- ksk 7y ago
  Might be useful to cluster the papers into various topics. For e.g. a person interested in drug discovery needs a different (biochem biased) set of papers than one doing vaccine development (immunology biased). *(caveat obvious overlaps)
- gpm 7y ago
  Create a generative model that makes more covid-19 papers. Bonus points if it's possible to manipulate the model into creating papers that tell us how to create a cure or vaccine /s.
MuffinFlavored 7y ago
Where are the most up to date, most reliable case numbers? I'm tracking US day-by-day case growth. These all have different numbers: http://covid19.fyi/ http://covid19.fyi/ https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_the_United_States https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_t... https://coronavirus.1point3acres.com/ https://coronavirus.1point3acres.com/
- dantheman 7y ago
  Here are the CDCs #s - they're released @ noon for up to 4pm the previous day.
- sureshv 7y ago
  For US Data try: https://covidtracking.com/ https://covidtracking.com/
  - sroussey 7y ago
    What about by city?
- doctoboggan 7y ago
  I have been using NYT: https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html https://www.nytimes.com/interactive/2020/us/coronavirus-us-c... They update it many times a day.
- gregsadetsky 7y ago
  https://github.com/CSSEGISandData/COVID-19 https://github.com/CSSEGISandData/COVID-19 ?
  - kroeckx 7y ago
    I think https://www.coronawiki.org/ https://www.coronawiki.org/ uses that data to visualize it.
- ddeck 7y ago
  For global numbers, the WHO daily situation reports: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports/ https://www.who.int/emergencies/diseases/novel-coronavirus-2... The same data is also presented graphically here: https://www.who.int/redirect-pages/page/novel-coronavirus-(covid-19)-situation-dashboard https://www.who.int/redirect-pages/page/novel-coronavirus-(c...
m3kw9 7y ago
Without proper search assists like Elastic search/lucene etc, it’s not useful for most people trying to read it. Maybe someone can set up a site with elastic search with articles on there?
- jonny_eh 7y ago
  Maybe algolia could help?
- shardinator 7y ago
  I thought so too, I’m working on this. If anyone wants to collaborate please let me know how I can get in touch email, Twitter.
  - inertiatic 7y ago
    I'm not sure what the cost would be to have ES/Solr hosted (is there a free option?). I'm too poor/cheap to do so, if it were up to me I'd set this up with a static js page that uses something similar in the browser (eg. lunr js) and allows you to download that dataset on the spot.
    - shardinator 7y ago
      I have a few apps that I host with a cloud provider (scalingo.com) and initially I can cover the costs. If there’s a lot of traffic because it proves to be helpful there are a few groups I could ask to help with some funding. For now I’m building a custom index with mongoDB - if anyone is familiar with building ES queries, would be keen to chat about how that might be better. I'll post a link here later today.
      - inertiatic 7y ago
        A lucene based solution would probably scale better and would allow you to implement more complex behavior. You get analyzers, stopword filters, synonyms etc. out of the box, and you can express things like "virus within 5 words of lung" or "covid or virus but covid is way more important". I believe given the rather small dataset and the fact that you won't even expose more complex queries on your interface, the main benefits are the analyzers and various filters you can use. If you are already doing stemming, stopword filtering and maybe synonyms you're probably fine. (My email is in my profile if you want to discuss this in further detail)
dougb 7y ago
This is would be the perfect time for IBM to apply all Watson technologies and resources to develop new insight into Covid-19.
- heyitsguay 7y ago
  Watson is basically a brand name for IBM's data analytics consulting services. My understanding is they're not that great at it, they haven't scored any major wins outside of that Jeopardy run. I don't have any articles on hand but i seem to recall reading about some failures with a medical partner in particular, but then that's been a tough field for other big names like Google, too.
pp19dd 7y ago
It takes downloading one of these files, gunzipping them, extracting the tar and opening up a JSON file at random to really understand just how distant the title of the project is from its contents. I fully realize this is about natural language processing, but... this is beyond reach. You'd have to teach a computer to become a doctor first. Feels like where AI (and computing in general) should go in case of Covid-19 and other illnesses is analyzing and simulating how the heck we work to begin with. Take a few minutes to watch these videos of supercomputer simulations that show -- fragments -- of the fundamentals of how we exist. Multi Scale Modeling of Chromatin and Nucleosomes - https://www.youtube.com/watch?v=4Z4KwuUfh0A https://www.youtube.com/watch?v=4Z4KwuUfh0A DNA animation showing realtime DNA replication - https://www.youtube.com/watch?v=7Hk9jct2ozY https://www.youtube.com/watch?v=7Hk9jct2ozY Somewhere in these mind-boggling processes is where the disruption called Covid-19 puts a stick in our wheels. Compared to the complexity of the simulations, abstract ideas contained in these files are so macroscopic by comparison. It's both humbling and awe-inspiring.
- typon 7y ago
  Even though the Multi Scale Modeling video is extremely impressive - it is still an MD simulation that uses classical mechanics. A full atomistic quantum simulation of such a large system is out of reach for even the largest super computers. We barely know anything about biology.
  - ksk 7y ago
    They're already using MD simulation in certain areas of drug discovery (computer aided drug design). Naturally, as you mentioned we're compute-limited so the models are simpler and the questions we ask of it are also 'simple'. But that is not stopping us at all. While not exactly MD, My S.O. works in drug discovery and they use simple computational screening models to simulate the chemistry and screen out tens of thousands of possible drug candidates.
axegon_ 7y ago
Call me cynical but isn't this a bit... Redundant? I mean sure, nlp, parse some papers and make some sort of search/q&a type of thing? Fine, whatever. Nice work on organizing the data and to a fair degree taking care of the tedious pre-processing associated with any ml task but... Idk, I fail to see how this could help. Personally I'll fiddle around with it in my spare time but mostly for fun, I don't expect anything significantly useful to come out of it. On a higher level I feel like the success of the IT industry has made a lot of people feel like it is the answer to every possible question. And sure, bioinformatics is an incredible subject and I've invested a lot in studying it in my spare time in the last couple of years but __purely__ out of curiosity. But my 2 cents on the subject is that us(developers, ml engineers, etc) cannot provide the answer to the meaning of life. In this situation our safest course of action is to work from home as much as possible and avoid making decision that could potentially make the situation worse. Basically protect ourselves and those around us as much as possible. And by doing so, let people who have the adequate training, knowledge and experience take care of the situation. Sure, play around with it in your spare time, and if you come up with something - share it. But the whole "stand back boys, let us men handle it" mentality is what really bugs me. If anything history has taught us that this goes badly 990 out of 1000 times(to be more in lines with Bayesians). We all want the underdog to win the game but... Come on...
- scastillo 7y ago
  Yeah, some critics about this feeling improvised, but guess what? it IS improvisation. Remember what hickey says about improvising? (https://www.infoq.com/presentations/Design-Composition-Performance/ https://www.infoq.com/presentations/Design-Composition-Perfo...) It is art and years of preparation right there performing real-time. So that guy criticizing is only auto declaring he is incapable of improvising on this matter.. and it's ok it's only for the best ones after all. It is called a challenge for a reason.
- markholmes 7y ago
  Redundant how? You say you don’t see how this could help, but the opposite - not doing it - is certain not to help. I don’t understand your criticism and cynicism here.
gillesjacobs 7y ago
Many of the criticisms voiced in this thread stem from a lack of expertise in biomedical Natural Language Processing and text mining. Various annotated datasets and models already exist within the field which can extract potentially useful information and be used in downstream task for targetted document and information retrieval. Biomedical text mining [1] is wide a subfield with plenty of open datasets and competitions such as the bi-annual ACL BioNLP workshop [1]. - Biomedical Named Entity Recognition: extract names of proteins, drugs, diseases, symptoms, etc. and classify their biomedical category [3]. Extracting the terms of symptoms is a crucial in document discovery and modeling and knowledge-base creation. Several open datasets can be found here [4]. - Biomedical relation and event extraction: Traditionally focused on extracting protein-protein interactions, which are crucial for virtually every process in a living cell. Information about these interactions provides the foundations for new therapeutic approaches. Recently interest have been shifted to the extraction of complex relations such as biomolecular events. [2] These methods can detect and classify the causal relations between the genes and proteins in a sentence like "TNF-alpha is a rapid activator of IL-8 gene expression by...". - Document retrieval: Helping researchers and medical staff find relevant topic-specific papers by improving search with topic modeling, document similarity, named entities, etc. These are only some examples of common biomedical text mining tasks and there are plenty more. Now of course, relying on previous annotated data is an issue because the tagged categories might not relevant for many of the issues related to COVID19. However, even unsupervised modeling like using SciBERT to create topic models or document clusters of related documents can be helpful for scientific discovery. 1. https://en.wikipedia.org/wiki/Biomedical_text_mining https://en.wikipedia.org/wiki/Biomedical_text_mining 2. https://aclweb.org/aclwiki/BioNLP_Workshop https://aclweb.org/aclwiki/BioNLP_Workshop 3. https://www.hindawi.com/journals/cmmm/2015/571381/ https://www.hindawi.com/journals/cmmm/2015/571381/ 4. http://gcancer.org/clstmdata/ http://gcancer.org/clstmdata/ 5. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3321-4 https://bmcbioinformatics.biomedcentral.com/articles/10.1186...
- deleted 7y ago
  [deleted]
ISNIT 7y ago
Have you put this in https://coronavirustechhandbook.com/data https://coronavirustechhandbook.com/data?
anon1253 7y ago
It might be fun to checkout https://covid19.doctorevidence.com/ https://covid19.doctorevidence.com/ they have loaded the CORD dataset with a dashboard like interface and a query language. E.g. https://search.doctorevidence.com/search?query=ss(6f1da786-67ab-11ea-9b44-001b21bedfe8) https://search.doctorevidence.com/search?query=ss(6f1da786-6... (user/pass covid19/covid19) for a direct link, and it provides integration with the other medically relevant feeds.

219 points by shinryudbz 7y ago

How can applying a machine learning algorithm to this data (a collection of research papers) help fight Covid? It’s possible most papers here are bogus or low quality. Garbage in, garbage out?

m3kw9 7y ago
Yeah I don’t see any current AI models that can combine all the knowledge from a subject and do something really profound with it. I do see you can generate more AI covid19 articles from it.
google234123 7y ago
I imagine they have some algorithm that takes an input consisting of the title, abstract, and authors with the information about how impactful their previous work has been (this is probably the most important factor) and outputs some ranking or a likelihood that the research will be cited/clicked on.
- inertiatic 7y ago
  That's feasible, it's not even a hard thing to do (maybe hard to do very accurately), I worked for a company that offered exactly that service, as a small project. Still, that probably doesn't answer the question, how does that really help the fight. I suppose we'll learn what they have in mind when those questions are asked in the competition.
ksk 7y ago
Might be useful to cluster the papers into various topics. For e.g. a person interested in drug discovery needs a different (biochem biased) set of papers than one doing vaccine development (immunology biased). *(caveat obvious overlaps)
gpm 7y ago
Create a generative model that makes more covid-19 papers. Bonus points if it's possible to manipulate the model into creating papers that tell us how to create a cure or vaccine /s.

MuffinFlavored 7y ago

Where are the most up to date, most reliable case numbers? I'm tracking US day-by-day case growth. These all have different numbers: http://covid19.fyi/ http://covid19.fyi/ https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_the_United_States https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_t... https://coronavirus.1point3acres.com/ https://coronavirus.1point3acres.com/

dantheman 7y ago
Here are the CDCs #s - they're released @ noon for up to 4pm the previous day.
sureshv 7y ago
For US Data try: https://covidtracking.com/ https://covidtracking.com/
- sroussey 7y ago
  What about by city?
doctoboggan 7y ago
I have been using NYT: https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html https://www.nytimes.com/interactive/2020/us/coronavirus-us-c... They update it many times a day.
gregsadetsky 7y ago
https://github.com/CSSEGISandData/COVID-19 https://github.com/CSSEGISandData/COVID-19 ?
- kroeckx 7y ago
  I think https://www.coronawiki.org/ https://www.coronawiki.org/ uses that data to visualize it.
ddeck 7y ago
For global numbers, the WHO daily situation reports: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports/ https://www.who.int/emergencies/diseases/novel-coronavirus-2... The same data is also presented graphically here: https://www.who.int/redirect-pages/page/novel-coronavirus-(covid-19)-situation-dashboard https://www.who.int/redirect-pages/page/novel-coronavirus-(c...

m3kw9 7y ago

Without proper search assists like Elastic search/lucene etc, it’s not useful for most people trying to read it. Maybe someone can set up a site with elastic search with articles on there?

jonny_eh 7y ago
Maybe algolia could help?
shardinator 7y ago
I thought so too, I’m working on this. If anyone wants to collaborate please let me know how I can get in touch email, Twitter.
- inertiatic 7y ago
  I'm not sure what the cost would be to have ES/Solr hosted (is there a free option?). I'm too poor/cheap to do so, if it were up to me I'd set this up with a static js page that uses something similar in the browser (eg. lunr js) and allows you to download that dataset on the spot.
  - shardinator 7y ago
    I have a few apps that I host with a cloud provider (scalingo.com) and initially I can cover the costs. If there’s a lot of traffic because it proves to be helpful there are a few groups I could ask to help with some funding. For now I’m building a custom index with mongoDB - if anyone is familiar with building ES queries, would be keen to chat about how that might be better. I'll post a link here later today.
    - inertiatic 7y ago
      A lucene based solution would probably scale better and would allow you to implement more complex behavior. You get analyzers, stopword filters, synonyms etc. out of the box, and you can express things like "virus within 5 words of lung" or "covid or virus but covid is way more important". I believe given the rather small dataset and the fact that you won't even expose more complex queries on your interface, the main benefits are the analyzers and various filters you can use. If you are already doing stemming, stopword filtering and maybe synonyms you're probably fine. (My email is in my profile if you want to discuss this in further detail)

dougb 7y ago

This is would be the perfect time for IBM to apply all Watson technologies and resources to develop new insight into Covid-19.

heyitsguay 7y ago
Watson is basically a brand name for IBM's data analytics consulting services. My understanding is they're not that great at it, they haven't scored any major wins outside of that Jeopardy run. I don't have any articles on hand but i seem to recall reading about some failures with a medical partner in particular, but then that's been a tough field for other big names like Google, too.

pp19dd 7y ago

It takes downloading one of these files, gunzipping them, extracting the tar and opening up a JSON file at random to really understand just how distant the title of the project is from its contents. I fully realize this is about natural language processing, but... this is beyond reach. You'd have to teach a computer to become a doctor first. Feels like where AI (and computing in general) should go in case of Covid-19 and other illnesses is analyzing and simulating how the heck we work to begin with. Take a few minutes to watch these videos of supercomputer simulations that show -- fragments -- of the fundamentals of how we exist. Multi Scale Modeling of Chromatin and Nucleosomes - https://www.youtube.com/watch?v=4Z4KwuUfh0A https://www.youtube.com/watch?v=4Z4KwuUfh0A DNA animation showing realtime DNA replication - https://www.youtube.com/watch?v=7Hk9jct2ozY https://www.youtube.com/watch?v=7Hk9jct2ozY Somewhere in these mind-boggling processes is where the disruption called Covid-19 puts a stick in our wheels. Compared to the complexity of the simulations, abstract ideas contained in these files are so macroscopic by comparison. It's both humbling and awe-inspiring.

typon 7y ago
Even though the Multi Scale Modeling video is extremely impressive - it is still an MD simulation that uses classical mechanics. A full atomistic quantum simulation of such a large system is out of reach for even the largest super computers. We barely know anything about biology.
- ksk 7y ago
  They're already using MD simulation in certain areas of drug discovery (computer aided drug design). Naturally, as you mentioned we're compute-limited so the models are simpler and the questions we ask of it are also 'simple'. But that is not stopping us at all. While not exactly MD, My S.O. works in drug discovery and they use simple computational screening models to simulate the chemistry and screen out tens of thousands of possible drug candidates.

axegon_ 7y ago

Call me cynical but isn't this a bit... Redundant? I mean sure, nlp, parse some papers and make some sort of search/q&a type of thing? Fine, whatever. Nice work on organizing the data and to a fair degree taking care of the tedious pre-processing associated with any ml task but... Idk, I fail to see how this could help. Personally I'll fiddle around with it in my spare time but mostly for fun, I don't expect anything significantly useful to come out of it. On a higher level I feel like the success of the IT industry has made a lot of people feel like it is the answer to every possible question. And sure, bioinformatics is an incredible subject and I've invested a lot in studying it in my spare time in the last couple of years but __purely__ out of curiosity. But my 2 cents on the subject is that us(developers, ml engineers, etc) cannot provide the answer to the meaning of life. In this situation our safest course of action is to work from home as much as possible and avoid making decision that could potentially make the situation worse. Basically protect ourselves and those around us as much as possible. And by doing so, let people who have the adequate training, knowledge and experience take care of the situation. Sure, play around with it in your spare time, and if you come up with something - share it. But the whole "stand back boys, let us men handle it" mentality is what really bugs me. If anything history has taught us that this goes badly 990 out of 1000 times(to be more in lines with Bayesians). We all want the underdog to win the game but... Come on...

scastillo 7y ago
Yeah, some critics about this feeling improvised, but guess what? it IS improvisation. Remember what hickey says about improvising? (https://www.infoq.com/presentations/Design-Composition-Performance/ https://www.infoq.com/presentations/Design-Composition-Perfo...) It is art and years of preparation right there performing real-time. So that guy criticizing is only auto declaring he is incapable of improvising on this matter.. and it's ok it's only for the best ones after all. It is called a challenge for a reason.
markholmes 7y ago
Redundant how? You say you don’t see how this could help, but the opposite - not doing it - is certain not to help. I don’t understand your criticism and cynicism here.

gillesjacobs 7y ago

Many of the criticisms voiced in this thread stem from a lack of expertise in biomedical Natural Language Processing and text mining. Various annotated datasets and models already exist within the field which can extract potentially useful information and be used in downstream task for targetted document and information retrieval. Biomedical text mining [1] is wide a subfield with plenty of open datasets and competitions such as the bi-annual ACL BioNLP workshop [1]. - Biomedical Named Entity Recognition: extract names of proteins, drugs, diseases, symptoms, etc. and classify their biomedical category [3]. Extracting the terms of symptoms is a crucial in document discovery and modeling and knowledge-base creation. Several open datasets can be found here [4]. - Biomedical relation and event extraction: Traditionally focused on extracting protein-protein interactions, which are crucial for virtually every process in a living cell. Information about these interactions provides the foundations for new therapeutic approaches. Recently interest have been shifted to the extraction of complex relations such as biomolecular events. [2] These methods can detect and classify the causal relations between the genes and proteins in a sentence like "TNF-alpha is a rapid activator of IL-8 gene expression by...". - Document retrieval: Helping researchers and medical staff find relevant topic-specific papers by improving search with topic modeling, document similarity, named entities, etc. These are only some examples of common biomedical text mining tasks and there are plenty more. Now of course, relying on previous annotated data is an issue because the tagged categories might not relevant for many of the issues related to COVID19. However, even unsupervised modeling like using SciBERT to create topic models or document clusters of related documents can be helpful for scientific discovery. 1. https://en.wikipedia.org/wiki/Biomedical_text_mining https://en.wikipedia.org/wiki/Biomedical_text_mining 2. https://aclweb.org/aclwiki/BioNLP_Workshop https://aclweb.org/aclwiki/BioNLP_Workshop 3. https://www.hindawi.com/journals/cmmm/2015/571381/ https://www.hindawi.com/journals/cmmm/2015/571381/ 4. http://gcancer.org/clstmdata/ http://gcancer.org/clstmdata/ 5. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3321-4 https://bmcbioinformatics.biomedcentral.com/articles/10.1186...

deleted 7y ago
[deleted]

ISNIT 7y ago

Have you put this in https://coronavirustechhandbook.com/data https://coronavirustechhandbook.com/data?

anon1253 7y ago

It might be fun to checkout https://covid19.doctorevidence.com/ https://covid19.doctorevidence.com/ they have loaded the CORD dataset with a dashboard like interface and a query language. E.g. https://search.doctorevidence.com/search?query=ss(6f1da786-67ab-11ea-9b44-001b21bedfe8) https://search.doctorevidence.com/search?query=ss(6f1da786-6... (user/pass covid19/covid19) for a direct link, and it provides integration with the other medically relevant feeds.