11 ms·
Giant, free index to world's research papers released online
- someonewhocar3s 5y agoStill can't construe it as ethical or right to prevent access to scientific papers. There's something about universities pooling cash for publishing, but whatever's happening is something else.
- perihelions 5y ago>"Malamud says that he did have to get copies of the 107 million articles referenced in the index in order to create it; he declined to say how," It's clearly Sci-Hub, as this 2019 article all-but-confirms: https://www.nature.com/articles/d41586-019-02142-1 https://www.nature.com/articles/d41586-019-02142-1 >"And around the same time that he heard about the Rameshwari judgment, he had come into possession (he won’t say how) of eight hard drives containing millions of journal articles from Sci-Hub [...] Malamud began to wonder whether he could legally use the Sci-Hub drives to benefit Indian students [...] Asked directly whether some of the text-mining depot’s articles come from Sci-Hub, he said he wouldn’t comment" (It'd be nice if there was a coverage source other than this scientific publisher, whose biases are obvious).
- bane 5y agoOut of interest, is sci-hub generally available to get a copy of for...research purposes?
- cinntaile 5y agoThere are torrents available. I'm unsure if I can link to the site directly without getting banned so you'll have to check the Gizmodo article for the link to the site containing the torrents. [0] It's missing some of the 2021 papers though, judging by the dates. [0] https://gizmodo.com/archivists-want-to-make-sci-hub-un-censorable-1846898276 https://gizmodo.com/archivists-want-to-make-sci-hub-un-censo...
- AlexCoventry 5y agoTaking you at your word that you're just asking out of interest, I hear (but haven't verified) that libgen mirrors scihub. http://libgen.rs/dbdumps/ http://libgen.rs/dbdumps/
- throwaway879080 5y agosee this reddit thread on Scihub rescue mission https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_mission_for_scihub_and_open_science_we_are/ https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_... there is a link there to 851 torrents , ~77TB of data (compressed zip files), and also an index (sql db dump)
- Frost1x 5y ago77 TB of compressed publications. That's pretty crazy to think about, even if you consider overhead and multimedia assets like images often included in PDFs. I remember the old "the library of congress fits on this CDROM" analogies (which wasn't entirely true) but this takes it to a whole new level. At some point, it seems like in research, it will be far easier to skip the lit review and just do the work then later if you do the same work someone else did, compare results for consistency. We may yet get through the hurdle of the reproducibility crisis due to the deluge of information. The underlying issue though is, because you couldn't find the related effort that caused you to independently attempt to replicate the effort, you may also never find all the duplicate efforts to compare for consistency.
- throwaway879080 5y agoabout half a petabyte uncompressed, i'm pondering training a language model/embeddings on this treasure trove, immediate use cases - search, related/similar papers recommendations, medical informatics. it does require OCRing the entire collection first though
- musicale 5y ago> 77 TB of compressed publications. PDF is fairly inefficient compared to formats like DVI (and consider that so many papers are produced using TeX anyway, though figures may be in various image formats.) But 77TB? You could host the whole thing in a shoebox with nine 8TB flash drives, a 10-socket powered USB hub, and a Raspberry Pi. Someone really needs to build Shoe-Hub.
- vmception 5y agoalso, I want a version that lets me browse papers. Unlike people in academia or other research fields, I don't know the paper that I want to pirate yet. from my lived experience, this request has been seen as absurd and invalidated by many because their lived experience is always having feeds of papers and abstracts they can start from. or how they see a browser extension that doesn’t do what I’m asking at all to be good enough (there’s some that automatically give you the scihub link to any paper you are looking at, which still puts me at square one, which paper?). its a pretty basic feature though.
- abcc8 5y agoUnpaywall is a browser extension you might find useful. It provides links to open access papers, with the links appearing on pubmed pages, publisher pages, etc. A significant proportion of papers are now open access so this tool is very often useful.
- m0zg 5y agoScihub has been a life saver for me once I started working on some of the more obscure areas of AI such as signal/time series processing. Anything off the beaten path is locked up behind paywalls, and I'm sorry I ain't paying $40 just to see if someone's paper sucks or not (which 95% of them do, in this particular niche, especially the ones shielded from scrutiny by paywalls).
- liketochill 5y ago95% of all papers I have read have sucked. Maybe they just weren’t what I was looking for but a lot of them I couldn’t believe got published as anything novel
- m0zg 5y agoRecent AI stuff from major labs on Arxiv is pretty good, but yeah, anything that's AI+some other field is usually pretty bad. It's usually written by someone in that other field who might be an expert in their own domain, but who knows very little about AI or even just numerical optimization in general. The fact that such "work" is accepted uncritically by publishers doesn't inspire a lot of confidence in the value they purportedly add. It's right on the surface: "awesome" results are easy to achieve in AI if you screw up your train/val split, or deliberately choose an extremely weak baseline.
- deleted 5y ago[deleted]
- krageon 5y agoOf the remaining five percent, at least in software you can be sure at least 80% (4% of the total) doesn't actually work when tested. It is beyond frustrating to deal with research, to the point that these days unless the algorithm is very well described or there is source code available I have to assume the researchers are just lying.
- nuerow 5y ago> It is beyond frustrating to deal with research, to the point that these days (...) I have to assume the researchers are just lying. This is not something new, or even from this century. The Royal Society, which was founded on 1660 and is a reference in the history of science, adopted the motto "take nobody's word for it". https://en.wikipedia.org/wiki/Royal_Society https://en.wikipedia.org/wiki/Royal_Society
- wolverine876 5y agoNature has provided straight coverage of Sci-Hub: http://www.nature.com/news/pirate-research-paper-sites-play-hide-and-seek-with-publishers-1.18876 http://www.nature.com/news/pirate-research-paper-sites-play-... As has Science: http://www.sciencemag.org/news/2016/04/whos-downloading-pirated-papers-everyone http://www.sciencemag.org/news/2016/04/whos-downloading-pira...
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- bane 5y agoFor n-gram indexes where n>1, I wonder if some of the sequence alignment techniques from bioinformatics could reconstruct the original documents?
- anitil 5y agoExactly my thought. I haven't delved in to what's here but I suspect this copyright workaround is not meant to be foolproof
- rmbyrro 5y agoAlso crossed my mind. We'd still would be left without illustrations, table organization, charts, etc, which in a considerable number of cases are of extreme relevance, though
- marginalia_nu 5y agoYou would typically require some sort of filtering in an N-gram index to prevent the index from growing larger than the corpus. Maybe something like a tf-idf-threshold, or some other heuristic. Otherwise you get a lot of useless "of that", "in an", "as a"-type word salad that is of no use to anyone, while what's actually useful sounds more like N-grams out of Tamarian.
- Luisvel 5y agoI was looking for a big and novel dataset to work on as a personal project. Can devote a couple months after work to it. What are interesting things that could be done with this now as a non-scientist, but that could be useful to share too?
- markdown 5y agoCreate a search engine, and put it online. I'd love to search the index but can't download 7 terabytes of data to do so.
- eigenman 5y agoI would be interested in seeing how different branding terms evolve in the literature; e.g., "machine learning" vs "artificial intelligence" vs "neural net" or "surrogate model" vs "digital twin" vs "response surface". There a number of terms of art that have substantial overlap, but which term ends up being used depends on the audience, which often includes grant providers. I suspect the popularity of these terms evolves according to what terms appeal the most to funding agencies.
- mandor 5y agoProvide examples of uses of words and possibly help us rephrase in a more idiomatic way. Basically, answer "how do people use this word usually?" A bit like linguee but for scientists: (at the bottom of the page): https://www.linguee.fr/francais-anglais/search?source=auto&query=hacker https://www.linguee.fr/francais-anglais/search?source=auto&q...
- exikyut 5y agoAn annoyingly broad answer, but: broadcast an opportunity in which you provide vetoed access to the server/instance holding the data (eg, with standard user accounts, private home directories and the data located read-only in a dir off of /). Would create a small storm of overhead ("please install ..." ad nauseum), but provide interesting opportunities for open-ended creativity. Also, while unusual, you could potentially extract value by lightly imposing requests ;) on others to (where viable/a good fit) at least have a go at helping you with problem-solving/busywork-type steps in the research you're doing (however experimental/uncertain). Since everyone would be looking at the same dataset, this may bring interesting/unexpected ideas to the table (and maybe even shed light on potential opportunities for collaboration down the road). For individuals who are reasonably independent and self-directed but have no experience working with huge amounts of data, this would also provide a cool chance to play around in a real-world-scale environment without the failure risks attached to eg fulfilling^Wfiguring out business requirements etc. (Now I'm reminded of https://tilde.club/ https://tilde.club/, which operates a shared Linux box a (very) large bunch of people have collective access to. It's a stab in the dark (ie, the one reference I'm aware of), but maybe the admins there would have interesting insight about managing shared access Linux boxes.)
- minihat 5y agoI wonder if this data could be used to index papers based on the type of mathematics they employ. The same applied math turns up across domains - I would love to be able to see who else is using the same math I already know on their problems.
- tomrod 5y agoBrilliant. Let us dig in!
- hashin 5y agoThat sounds super interesting! The results will depend upon the kind of information this database provides about each paper. We don't know yet the kind of processing each paper goes through and the quality of the "description" in the database. Assuming that it is good enough, this would be a really interesting project to work on.
- amitlzkpa 5y agoSign me up for that!
- pddpro 5y agoThis is an amazing idea !
- lordnacho 5y agoHey this is an interesting idea. How would it be done though? For example I've noticed certain similarities across classical statistics, Machine Learning and digital signal processing. But they all seem to come at it from different angles, and they often use different terminology. Sometimes a field will rediscover something discovered elsewhere a long time ago, eg the trapezium rule.
- ausbah 5y agoeven worse, a field will take term from another field and misuse it enough such that "wrong" definition" becomes the default definition
- 5y ago
- gopiandcode 5y ago“We have seen some initiatives run into trouble, however, when the necessary rights have not been secured to enable their sustainability,” I love the double-speak that these corrupt publishers use - it really highlights how mafia-esque the whole situation is. "Pay your protection fee, or we won't be able to /protect/ you". Intellectual slavery and copyright is just another tool of opression used to keep the wealthy in their positions of power.
- charcircuit 5y agoAnyone can create IP. It's not something only the wealthy can do.
- pugworthy 5y agoIndeed. This is my new strategy at the place I work, where I can't get acceptance of new interesting "a few years out there" ideas for our product line, but I can file for IP for such ideas. I'm actually encouraged to. It's a great outlet for the creative if you've got the organization to help out.
- desiderantes 5y agoThere are people out there gifting their IP to companies? What a crazy world.
- Cerium 5y agoUsually it is not quite a gift, where I work you get some bonus, something like $1500 for each patent with your name on it.
- pugworthy 5y agoNot to mention I can't hire lawyers, researchers, illustrators, and all the other things my company does for me. It would cost me so much more, both in dollars and time.
- wyldfire 5y agoI wonder -- there's lots of machine learning designed for making inferences and there's some for language and transfer. How easy is it to train an oracle that you can ask questions? Or could you make an ontology? Seems like a wealth of scientific papers might be a neat thing to train on. Not to emit gibberish new papers, but to make something useful. Somehow.
- jmnicholson 5y agocheck out our work in this area: https://scite.ai/search/citations https://scite.ai/search/citations We have extracted 918M citation statements from 27M full text articles.
- dash2 5y agoBack button is a bit broken there, if you want to back out of the login page.
- wyldfire 5y agoThis is interesting but AFAICT it is "merely" indexing.
- fibrennan 5y agoSo far as I understand it's a lot more than that. Their model has not only indexed but has also individually classified 900M+ citation statements by sentiment, the outputs of which produce a score to represent each paper's relative trustworthiness. To your point though it's fun to think about generative applications too. I for one would appreciate a writing assistant trained on millions of scientific papers -- like OK I'll write that last-minute proposal for you but you better believe it's going to be chock-full of dispensable lexical arcana.
- jmnicholson 5y agoYes, there is a lot more that goes into it. The details of our indexing are described in detail in a recent paper of ours: https://direct.mit.edu/qss/article/doi/10.1162/qss_a_00146/102990/Scite-A-smart-citation-index-that-displays-the https://direct.mit.edu/qss/article/doi/10.1162/qss_a_00146/1...
- jmnicholson 5y agoWe (scite.ai) have extracted 918M citation statements (3 full sentences) from 27M full-text articles (more than half are from paywalled articles from indexing agreements). We've taken this an extra step and have classified the citation statements as providing supporting or contrasting evidence (https://direct.mit.edu/qss/article/doi/10.1162/qss_a_00146/102990/Scite-A-smart-citation-index-that-displays-the https://direct.mit.edu/qss/article/doi/10.1162/qss_a_00146/1...) And, recently, have made these citation statements easily searchable to find expert analyses and opinions on nearly any topic extracted from the literature: https://scite.ai/search/citations https://scite.ai/search/citations
- pedalpete 5y agoNice work on Scite, I'm not sure if this is a different use case than I have, but my searches are listing duplicates of the same papers many times. Is there a reason to not collapse duplicates into a single result?
- jmnicholson 5y agoIs this on Citation Statement search? I think it is probably because the citation context contains two or more citations in it. We look at citations per sentence so it is duplicated there. I can see how that is confusing though.
- a3_nm 5y agoThis looks very nice! is it possible to download the dataset? (PS: comparable initiative of citation datasets that are downloadable: S2ORC https://github.com/allenai/s2orc https://github.com/allenai/s2orc and Internet Archive Refcat https://blog.archive.org/2021/10/19/internet-archive-releases-refcat-the-ia-scholar-index-of-over-1-3-billion-scholarly-citations/ https://blog.archive.org/2021/10/19/internet-archive-release...)
- jmnicholson 5y agoWe can't release the full dataset as our licensing agreements with publishers restrict it. We do have an API though that can be used: https://api.scite.ai/docs https://api.scite.ai/docs
- kragen 5y agoI wonder if this project is what Carl's friend Aaron Swartz was working on that he got prosecuted for until he committed suicide.
- 2Gkashmiri 5y agoScihub is what 100 tb ? That is like inr 80,000 + a server to hold drives say another 30,000 so in total around 110,000 or 120,000. That is around USD 1500. If I get a windfall of sorts, this is the thing I am going to spend on. Donations, eh yeah but I feel like scihub needs everyone's support.
- exikyut 5y agoI happened to check Amazon a few minutes ago while writing another comment and found 18TB drives seem to be hovering around US$350. 6*18=108TB (and US$2100), however, that does not account for redundancy. I recently learned the fun way that ZFS performance (incl read-only) absolutely tanks if the pool is full (the box was unusable until I deleted the data), so to use ZFS you'd want to add in a few TB of spare capacity. https://wintelguy.com/zfs-calc.pl https://wintelguy.com/zfs-calc.pl says 10 drives ($3500) should provide 100TB of space. Alternatively you could use dm-integrity (which is very new and does not yet have optimized I/O routines) and either layer RAID on top of that or manually duplicate the data yourself, which would let you just get exactly the number of disks. It's possible you could alternatively run Ceph with ECC, but I don't know if Ceph likes having some free space to work with. This also does not take into account a server, but this does not need to be too expensive, probably US$750 or potentially even less. I respond entirely out of curiosity; I also dream of backing up large datasets one day :D
- 2Gkashmiri 5y agomy argument is just to provide redundancy. we do not need additional local redundancy because the data is out there. it would be a problem if my copy or your copy somehow becomes the only copy. then we would be concerned. https://www.amazon.in/Western-Digital-DigitalBBGB0060HBK-BESN-External-Drive/dp/B01NCSZKWM/ref=sr_1_3?crid=2F4G1J01YL7EX&dchild=1&keywords=6tb+hard+disk&qid=1635346861&sprefix=8tb+hard+disk%2Caps%2C245&sr=8-3 https://www.amazon.in/Western-Digital-DigitalBBGB0060HBK-BES... this says INR 11,000.00 which is $137.raw 100 would need around 18 so $2500. i think there can be variations of disk sizes and amazon fluctuations but in cases like these i would have a business buy them because that way input tax of 18% would pass on to the business and conversely to me for lowering the final bill. so, $2118 would be for these drives. same for the server. a tax benefit of 18%. as i said, i would love to have this
- wolverine876 5y agoHow does this differ from Google Scholar's search index, or JSTOR's? Is it that it's open?
- allturtles 5y agoFrom the article: > And although free search engines such as Google Scholar have — with publishers’ agreement — indexed the text of paywalled literature, they only allow users to search with certain types of text queries, and restrict automated searching. That doesn’t allow large-scale computerized analysis using more specialized searches, Malamud says.
- ant6n 5y agoCan this new index used for search. A quick glance makes it seem like a huge list of ngrams, which isn't exactly easy to with with.
- 6510 5y agoWe need a generic effort to take out old ideas that bother all but a few people.
- modernerd 5y agoCarl Malamud's Aaron Swartz memorial speech is relevant and moving: https://www.youtube.com/watch?v=VllJDnMcTzM https://www.youtube.com/watch?v=VllJDnMcTzM
- ozymandias12 5y agoThanks for this. I also noticed the bottom thanks on the index: >In_memoriam Shamnad Basheer ; Aaron Swartz
- wolverine876 5y agoLooking around, I see Shamnad Basheer was found dead in a car. Is there evidence of how that happened?
- webmaven 5y agoIt looks like he was sleeping in his car with the engine running and suffocated: https://www.thehindu.com/news/national/karnataka/legal-expert-shamnad-basheer-found-dead/article28918277.ece https://www.thehindu.com/news/national/karnataka/legal-exper...
- wolverine876 5y agoI saw that article, which portrayed that statement as speculation by a non-expert, before even a post-mortem examination. I'm not saying something controversial happened, but I haven't found information on what did happen.
- noiwillnot 5y agoHow people goes about downloading/mirroring sci-hub? I assume you cannot start torrenting from your home in Texas without getting into problems. Do you get a seedbox? You are in a state like China or Russia? Getting these > 70 TB to a residential address does not seems easy.
- krageon 5y agoIf you are serious: you can download it to a seedbox, connect to it and download it to your residential address over scp. The storage is a solved problem, not cheap but then you are trying to store a significant chunk of all papers :)
- d110af5ccf 5y ago> download it to a seedbox Or if you already have a VPN provider, check their terms to see what their torrent policies are. Last I checked mine permitted such traffic in a few specific datacenters.
- exikyut 5y agoThat solves the routing problem, but not the pipe size problem. 70TB over 1Gbps: 6+ days 70TB over 500Mbps: 12+ days 70TB over 100Mbps: 2+ months 70TB over 50Mbps: 4+ months 70TB over 25Mbps: 8+ months 70TB over 10Mbps: 21+ months 70TB over 5Mbps: 3 years 70TB over 1Mbps: 17 years -- 70TB via mail: equivalent to 100Mbps (international) to 1Gbps+ (local) depending on distance, but only cost-effective if an equivalent connection is unavailable due to average weight (and price) of sufficient HDDs, eg, 3x18TB = ~$1200, combined weight approx 1kg.
- mindcrime 5y agoDoes Amazon allow torrenting? Or could you have your seedbox push the data to S3 from $WHEREVER? If so, you could possibly use Snowball[1] to get the data to your residence. [1]: https://aws.amazon.com/getting-started/projects/migrate-petabyte-scale-data/faq/ https://aws.amazon.com/getting-started/projects/migrate-peta...
- brainwipe 5y agoThis is awesome; did a quick ego search and didn't find mine in there but found some references to it. Nonetheless, an excellent resource!
- heinrichhartman 5y agoHow about learning a language model on ~~the SciHub texts~~ some large private repository of articles, that can auto-complete a paper from the title ...
- tasogare 5y agoThere is enough irrelevant and dumb papers written by humans not to add more generated artificially.
- plg 5y agoIs there a search interface?
- capableweb 5y agoDirect link: https://archive.org/details/GeneralIndex https://archive.org/details/GeneralIndex If you want to perform searches against the index: https://archive.org/search.php?query=creator%3A%22Public+Resource%22 https://archive.org/search.php?query=creator%3A%22Public+Res... Great to see a large "Signatories to the Declaration of Support" and that it's hosted by the Internet Archive. Keep fighting the good fights my friends! :)
- johnnybaptist 5y agoAny advice on how to search against this index? I can't seem to figure out the right search queries, everything I try gives no results. For example, I'm trying to find articles with "neuroscience" or "brain-computer interface" among the keywords.
- IshKebab 5y agoThis doesn't look especially useful. You can search for small sentence fragments and get the paper title I guess? But you can already do that on Google Scholar. You can't read the papers or see their citations or cited-bys. What's it for? Poking Elsevier?
- echopurity 5y ago[dead]
- RNCTX 5y agoReminder that these journal publishers have egregious and exploitive contracts with universities, and should not be commended for turning any blind eyes to such databases. The state university I attended publishes their subscription expenditures, link to the document here [1]. Besides the outrageous prices the other thing that jumps off the pages is how many of them have managed to hide the amount of tax money they receive from a state institution with contract terms. 1. https://digital.library.unt.edu/ark:/67531/metadc1132742/ https://digital.library.unt.edu/ark:/67531/metadc1132742/