4 ms·

OpenAlex: The Promising Alternative to Microsoft Academic Graph

82 points by tokai 5y ago

lmc 5y ago
Anyone know of any nice front-ends or visualizations for this? I guess a Python or R Notebook and network analysis libraries would be a good place to start?
- fluidcruft 5y ago
  I don't know if it's related to this database or does what you're looking for but I've recently learned that https://www.connectedpapers.com https://www.connectedpapers.com and https://www.researchrabbit.ai https://www.researchrabbit.ai exist and have started using them (particularly connectedpapers) pretty heavily.
  - lmc 5y ago
    Thanks, both seem like good options!
  - shishy 5y ago
    You can also do visualizations with https://scite.ai https://scite.ai! The edges of the graphs show exactly what the papers (nodes) said about each other :) Here's a video: https://www.youtube.com/watch?v=P5J9EgAaai4 https://www.youtube.com/watch?v=P5J9EgAaai4
    - fluidcruft 5y ago
      That's pretty cool. It made me suddenly think that it would be great if there was a way to use this safely and without going broke while performing peer-review.
- zozbot234 5y ago
  Scholia https://scholia.toolforge.org/ https://scholia.toolforge.org/ is a frontend to comparable data on Wikidata, a general-purpose knowledge graph. Edit: The OpenAlex website states that a web interface is planned for Summer 2022.
- jasonpriem 5y ago
  If you go that route, you may also want to check out these community-contributed Python and R libraries: https://docs.openalex.org/api#client-libraries https://docs.openalex.org/api#client-libraries
sva_ 5y ago
First I thought having a REST API isn't that "open", but discovered you can mirror the complete dataset (~200G): https://docs.openalex.org/download-snapshot https://docs.openalex.org/download-snapshot
- tokai 5y ago
  And the data is licensed under cc0.
- jasonpriem 5y ago
  We emphasize the API because imho a well-documented, high-throughput API is something really lacking in the ecosystem right now. Dealing with a dataset this size (200M works, 200M authors) is a pain, especially since many end users for this data don't have a lot of technical expertise. Often people have really basic questions (like the ones from the linked post) or they want to build simple applications like monitoring dashboards, recommender systems, and scholarly search engines. With the API, folks can outsource the heavy data engineering to us for free, and just do the fun parts themselves. We want to make building real-world apps on the global research graph fun and easy, the kind of thing you can do as a hackathon project, instead of with a six-figure grant. That said, I agree that it's absolutely essential the entire dataset be easy to download and mirror as well. It's called OpenAlex because it's _open_, soup to nuts (the "Alex" part is homage to the ancient Library of Alexandria). All the data is open, the code is open, and our governance is as open as we can make it. [1] [1] https://ourresearch.org/transparency https://ourresearch.org/transparency
tomthe 5y ago
OpenAlex seems to be a really useful alternative to Scopus and Web of Science. Although I found that their author-disambiguation is not as good as the one from Scopus. I would like to see that they make it easier to download database dumps. Their json-based format seems to contain a lot of redundant information. A compressed dump should not be much bigger than 20...30 GB (Theirs is 200GB, according to https://docs.openalex.org/download-snapshot/download-to-your-machine https://docs.openalex.org/download-snapshot/download-to-your... )
- jasonpriem 5y ago
  Agreed, the author disambiguation isn't quiet as good as Scopus'...they have a bit of a head start on us. But we're improving it quickly. Thanks for the suggestion about the data dump. A lot of that weight is abstracts, which come in at over 30GB just by themselves. But it's true that the JSON format has some redundancies. For now we think those are worth it, because the denormalized schema is very compatible with the API and easy for beginners to get started with. Plus you only have to download it once (for free! HT to AWS Open Data sponsorship), and after that the updates are very light. We'll certainly consider offering a smaller, normalized format in the future though, if we get more requests for it.
  - zozbot234 5y ago
    Author disambiguation can be a very hard problem in practice, especially with authors who share the same names (a relatively common occurrence in some non-Western countries) and work on similar things. It's even somewhat common to see the exact same ORCid (supposedly a unique identifier) attached to what's clearly papers published by distinct individuals. At some point, one pretty much has to guess.
    - uptime 5y ago
      Yes, I think some services attempt to retro-assign ORCIDs. I usually have to try to grab the affiliation and date to make sure I have the right person. Someday for the authors I follow I want to train a model to give me a match score with my Smiths and Lees etc.
shishy 5y ago
We use this at scite [1] to power some of our features, kudos to Jason, Heather and the rest of the team for their awesome work on maintaining this infrastructure (and the rest of their work on Unpaywall, etc.) For context, many companies, including us, used Microsoft Academic Graph but it was shuttered last year [2]. So OpenAlex came in to replace, but they also made it backwards compatible with the MAG data, making it super easy for us to migrate. Overall there are some problems with the data that we've found and reported / fixed (and continue to do so), but it feels well supported to use moving forward. I believe the OpenAlex team put out a preprint about this recently [3]. 1. https://scite.ai https://scite.ai 2. https://www.microsoft.com/en-us/research/project/academic/articles/microsoft-academic-to-expand-horizons-with-community-driven-approach/ https://www.microsoft.com/en-us/research/project/academic/ar... 3. https://arxiv.org/ftp/arxiv/papers/2205/2205.01833.pdf https://arxiv.org/ftp/arxiv/papers/2205/2205.01833.pdf
- jasonpriem 5y ago
  Thanks for the kudos! I'm very glad we've been able to help out!
- uptime 5y ago
  Nthing thanks for Unpaywall and any efforts to free people from Google Scholar etc. Arcadia seems to be funding good things.
legerdemain 5y ago
This looks very similar to what meta.org developed before it got shut down to pave the way for the Facebook renaming. I wonder if SMU inherited meta.org's archives and data pipelines.
- jessfyi 5y ago
  Meta only focused on biomedical information. This is explicitly based on Microsoft Academic Graph which was all encompassing. They did remove patents, tag papers with fewer topics/subtopics, and don't index as many conferences, but they did add a couple of nice things like ORCID. [0] [0] https://docs.openalex.org/download-snapshot/mag-format/mag-migration-guide https://docs.openalex.org/download-snapshot/mag-format/mag-m...
- jasonpriem 5y ago
  To be clear, this is a project isn't affiliated with SMU, they just did the blog post...it's from a nonprofit called OurResearch (source: I'm a cofounder). We did have some good conversations with folks at Meta before they closed up shop, but didn't end up using any of their data.
jasonpriem 5y ago
Jason here, cofounder of OurResearch, the nonprofit behind OpenAlex. I appreciate everyone's feedback! Happy to answer any questions.

82 points by tokai 5y ago

lmc 5y ago

Anyone know of any nice front-ends or visualizations for this? I guess a Python or R Notebook and network analysis libraries would be a good place to start?

fluidcruft 5y ago
I don't know if it's related to this database or does what you're looking for but I've recently learned that https://www.connectedpapers.com https://www.connectedpapers.com and https://www.researchrabbit.ai https://www.researchrabbit.ai exist and have started using them (particularly connectedpapers) pretty heavily.
- lmc 5y ago
  Thanks, both seem like good options!
- shishy 5y ago
  You can also do visualizations with https://scite.ai https://scite.ai! The edges of the graphs show exactly what the papers (nodes) said about each other :) Here's a video: https://www.youtube.com/watch?v=P5J9EgAaai4 https://www.youtube.com/watch?v=P5J9EgAaai4
  - fluidcruft 5y ago
    That's pretty cool. It made me suddenly think that it would be great if there was a way to use this safely and without going broke while performing peer-review.
zozbot234 5y ago
Scholia https://scholia.toolforge.org/ https://scholia.toolforge.org/ is a frontend to comparable data on Wikidata, a general-purpose knowledge graph. Edit: The OpenAlex website states that a web interface is planned for Summer 2022.
jasonpriem 5y ago
If you go that route, you may also want to check out these community-contributed Python and R libraries: https://docs.openalex.org/api#client-libraries https://docs.openalex.org/api#client-libraries

sva_ 5y ago

First I thought having a REST API isn't that "open", but discovered you can mirror the complete dataset (~200G): https://docs.openalex.org/download-snapshot https://docs.openalex.org/download-snapshot

tokai 5y ago
And the data is licensed under cc0.
jasonpriem 5y ago
We emphasize the API because imho a well-documented, high-throughput API is something really lacking in the ecosystem right now. Dealing with a dataset this size (200M works, 200M authors) is a pain, especially since many end users for this data don't have a lot of technical expertise. Often people have really basic questions (like the ones from the linked post) or they want to build simple applications like monitoring dashboards, recommender systems, and scholarly search engines. With the API, folks can outsource the heavy data engineering to us for free, and just do the fun parts themselves. We want to make building real-world apps on the global research graph fun and easy, the kind of thing you can do as a hackathon project, instead of with a six-figure grant. That said, I agree that it's absolutely essential the entire dataset be easy to download and mirror as well. It's called OpenAlex because it's _open_, soup to nuts (the "Alex" part is homage to the ancient Library of Alexandria). All the data is open, the code is open, and our governance is as open as we can make it. [1] [1] https://ourresearch.org/transparency https://ourresearch.org/transparency

tomthe 5y ago

OpenAlex seems to be a really useful alternative to Scopus and Web of Science. Although I found that their author-disambiguation is not as good as the one from Scopus. I would like to see that they make it easier to download database dumps. Their json-based format seems to contain a lot of redundant information. A compressed dump should not be much bigger than 20...30 GB (Theirs is 200GB, according to https://docs.openalex.org/download-snapshot/download-to-your-machine https://docs.openalex.org/download-snapshot/download-to-your... )

jasonpriem 5y ago
Agreed, the author disambiguation isn't quiet as good as Scopus'...they have a bit of a head start on us. But we're improving it quickly. Thanks for the suggestion about the data dump. A lot of that weight is abstracts, which come in at over 30GB just by themselves. But it's true that the JSON format has some redundancies. For now we think those are worth it, because the denormalized schema is very compatible with the API and easy for beginners to get started with. Plus you only have to download it once (for free! HT to AWS Open Data sponsorship), and after that the updates are very light. We'll certainly consider offering a smaller, normalized format in the future though, if we get more requests for it.
- zozbot234 5y ago
  Author disambiguation can be a very hard problem in practice, especially with authors who share the same names (a relatively common occurrence in some non-Western countries) and work on similar things. It's even somewhat common to see the exact same ORCid (supposedly a unique identifier) attached to what's clearly papers published by distinct individuals. At some point, one pretty much has to guess.
  - uptime 5y ago
    Yes, I think some services attempt to retro-assign ORCIDs. I usually have to try to grab the affiliation and date to make sure I have the right person. Someday for the authors I follow I want to train a model to give me a match score with my Smiths and Lees etc.

shishy 5y ago

We use this at scite [1] to power some of our features, kudos to Jason, Heather and the rest of the team for their awesome work on maintaining this infrastructure (and the rest of their work on Unpaywall, etc.) For context, many companies, including us, used Microsoft Academic Graph but it was shuttered last year [2]. So OpenAlex came in to replace, but they also made it backwards compatible with the MAG data, making it super easy for us to migrate. Overall there are some problems with the data that we've found and reported / fixed (and continue to do so), but it feels well supported to use moving forward. I believe the OpenAlex team put out a preprint about this recently [3]. 1. https://scite.ai https://scite.ai 2. https://www.microsoft.com/en-us/research/project/academic/articles/microsoft-academic-to-expand-horizons-with-community-driven-approach/ https://www.microsoft.com/en-us/research/project/academic/ar... 3. https://arxiv.org/ftp/arxiv/papers/2205/2205.01833.pdf https://arxiv.org/ftp/arxiv/papers/2205/2205.01833.pdf

jasonpriem 5y ago
Thanks for the kudos! I'm very glad we've been able to help out!
uptime 5y ago
Nthing thanks for Unpaywall and any efforts to free people from Google Scholar etc. Arcadia seems to be funding good things.

legerdemain 5y ago

This looks very similar to what meta.org developed before it got shut down to pave the way for the Facebook renaming. I wonder if SMU inherited meta.org's archives and data pipelines.

jessfyi 5y ago
Meta only focused on biomedical information. This is explicitly based on Microsoft Academic Graph which was all encompassing. They did remove patents, tag papers with fewer topics/subtopics, and don't index as many conferences, but they did add a couple of nice things like ORCID. [0] [0] https://docs.openalex.org/download-snapshot/mag-format/mag-migration-guide https://docs.openalex.org/download-snapshot/mag-format/mag-m...
jasonpriem 5y ago
To be clear, this is a project isn't affiliated with SMU, they just did the blog post...it's from a nonprofit called OurResearch (source: I'm a cofounder). We did have some good conversations with folks at Meta before they closed up shop, but didn't end up using any of their data.

jasonpriem 5y ago

Jason here, cofounder of OurResearch, the nonprofit behind OpenAlex. I appreciate everyone's feedback! Happy to answer any questions.