11 ms·
Mendeley encrypts users' database after Zotero provides an importer
- carbocation 8y agoLuckily, my Linux desktop was still running an old version of Mendeley so I was able to make the switch. My reason for wanting to switch was that Zotero has Google Docs integration. After making the switch, I was pleased to pay a modest amount for storage of my PDFs, which makes me feel like a customer instead of a product.
- leemailll 8y agoIf your institution does offer f1000 subscription, I suggest you take a look at f1000 workspace.
- carbocation 8y agoThanks for the recommendation. The features look interesting, and it supports my basic requirement (adding citations to Google Docs).
- black_puppydog 8y agoApart from it being domain specific (according to wikipedia) it is also not open source, so recommending this as a reaction to the Mendeley news is... words fail me... :|
- leemailll 8y agoWhat do you mean "domain specific"? Personal subscription is also available. And of the popular online reference managers, I don't think open source by itself certify anything related to the core functions of a reference manager. In my opinion, the really established open source ref manager is BibTeX, however, its user base is quite limited even inside academy to specific fields.
- black_puppydog 8y agoWikipedia on f1000 workspace: > It includes a Word plug-in, from which you're able to directly search PubMed [...] That sounds like it's at least focusing on medicine as a field (and generally fields in which Word documents are an acceptable means of dissemination...) Regarding open source: Being open source is certainly not core to the functionality of a reference manager. But it does protect you from exactly the kind of behaviour shown by Mendeley here. So recommending a "solution" for the situation that has exactly the same drawback as the Mendeley (i.e. vendor lock-in) is short-sighted in the extreme.
- andrepd 8y agoHmm, I thought this would be a lesson to illustrate what happens when you depend on proprietary software...
- ramraj07 8y agoThis is literally the worst thing you can do. Even a tenured professor doesn't seem to stay in the same institution for their lifetime nowadays so why would you go to a service that even you personally cannot pay for yourself (reasonably)?
- bocklund 8y agoIt's $10/month for premium? https://f1000workspace.com/#plans https://f1000workspace.com/#plans
- leemailll 8y agoAnd online storage space is unlimited https://guides.lib.unc.edu/comparecitationmanagers https://guides.lib.unc.edu/comparecitationmanagers
- leemailll 8y agoF1000 offers tools to transfer an account to another email address. Why this makes you think it is "the worst thing you can do"?
- boyband6666 8y agoI gave this a try, but sadly it came up short. For writing a single paper of MSc Thesis, it might work. However it just doesn't seem suited for handling a large number of references for a variety of projects, which get called on frequently as standard references in papers, reference material, and much more. If it helps I have just under 2000 papers in my Zotero. It isn't massive by research standards, but it is substantial.
- leemailll 8y agoTry to group into different projects in F1000. One thing I still don't like about F1000 is the reference manager need to go online for entries, so it kind of slow on first call. By the way, i don't how you cite when writing, but for me, I mostly handle that part after I finish the writing. At the places citations needed in the manuscript, I often just put a mark saying #somebody's paper on somethin#, and then do the job all the once. There always some really prolific guys make the citation search taking too much time to break to flow.
- justinclift 8y agoReading through that, it seems like Mendeley uses SQLite for it's local database. There aren't that many SQLite encryption libraries around (generalising), so it's probably using either SQLCipher or SEE. https://www.zetetic.net/sqlcipher https://www.zetetic.net/sqlcipher https://www.sqlite.org/see/doc/trunk/www/index.wiki https://www.sqlite.org/see/doc/trunk/www/index.wiki There's a reasonable chance the page size and passphrase is stored or cached on the machine. If someone (not me) takes the time and effort to trace through the application, it should be feasible to figure out what's going on. :)
- shittyadmin 8y agoYeah, this sounds like an interesting reverse engineering challenge, do you need any special accounts to use Mendeley or can it be freely installed by anyone? It'd likely just be a hard-coded key, you wouldn't want to restrict people from copying their own databases around... probably.
- dunham 8y agoI did this a while back with dropbox (I was curious about their database structure). I ended up adapting some code that scans an OSX process for AES key schedules and prints any keys that it finds (caveat - sqlite3 rolls its own AES and the key schedule ends up byte-swapped in memory). I also had to write some Go code to decrypt the sqlite3 database file. There is code to read an encrypted database buried in sqlite3's revision history, but it's an old enough version of sqlite3 that it didn't understand dropbox's schema.
- Lord_Nightmare 8y agoThey're using SEE, see https://eighty-twenty.org/2018/06/13/mendeley-encrypted-db https://eighty-twenty.org/2018/06/13/mendeley-encrypted-db
- shittyadmin 8y agoUnfortunately here's the catch - while it's probably trivial to reverse engineer enough to extract the key for the SEE database, it means that Zotero would need an SEE license to be able to handle the file still. As an alternative it should be quite possible to produce a hook using Frida or similar that would disable database encryption on the Mendeley side so it can be imported through the usual channels. Not pretty though.
- pjc50 8y agoAh, Elsevier. It's probably not that difficult to reverse-engineer and extract the decryption keys, but doing so opens you up to DMCA risks.
- lozenge 8y agoI don't think the DMCA would apply, as presumably you own the copyright of whatever's in the database. You would need EFF or other support to prove it though...
- black_puppydog 8y agoOn the other hand, it's gonna be hard to even detect if you just use it to import your collection to Zotero.
- pygy_ 8y agoPortability is explicitly mentioned as an exception to the "no RE" clauses of the DMCA, so I wouldn't expect any charge to stick, but I don't expect that fact to deter Elsevier from suing anyway, out of spite.
- mathymichael 8y ago"Ah, Elsevier" was my exact reaction as well. Nothing more frustrating than lazy business models undermining progress.
- zwaps 8y agoZotero has improved a lot, while Mendeley has repeatedly regressed. Mendeley used to be quite a good program, but recently you can not export annotated PDFs meaningfully. For example, sending a folder of annotated PDFs to a co-author during a literature review is impossible. This is obviously the case since Elsevier does not want you to trade research papers, whether you have lawfull access or not. The updates that took away features were silent. What happened to me some time ago was these updates occured during a high-stress phase with a short deadline until conference submissions (if you are a researcher, you know what I mean). I had used Mendeley for years to annotate and categorize literature. I was now in need to send my categorized PDFs to a central repository for my co-authors to evaluate and add to. After some update, without me noticing it, it was no longer possible to export folders of PDFs or PDFs in general! I had everything in Mendeley, weeks of work. I was completely f'ed - deadline approaching. I had to re-aquire all PDFs and go through all annotations by hand. ELSEVIER IS SIMPLY ANTI SCIENCE. Collaboration is a key in science. Sharing results, research and literature is crucial. Mendeley makes this impossible. It does NOT allow you to fully access your own work! So in conclusion, USE ZOTERO. It's good now, better than before. You can use a PDF reader with annotations to open and save the PDF and Zotero will keep those annotations. You can export Bibliographies, including notes AND files. You can not do that with Mendeley. So again, as a researcher, I emplore you to drop Mendeley completely, as I have done. Thank you.
- walrus01 8y agoElsevier is to science as Oracle is to database software licensing.
- mdip 8y agoThank you for this. As someone who does not make a habit of hanging around academia, I have very little experience with Elsevier short of what I read here. However, as someone who worked in a large IT organization for a huge company and wrote software for license compliance tracking, I completely understand. Grab me a pitch fork, I'll march.
- AnaniasAnanas 8y ago
- Vinnl 8y agoNote that this has been happening a while, given that there's submissions [1] [2] from 7 months ago. [1] https://news.ycombinator.com/item?id=17302019 https://news.ycombinator.com/item?id=17302019 [2] https://news.ycombinator.com/item?id=17433880 https://news.ycombinator.com/item?id=17433880
- mwexler 8y agoFor those wondering, here's what I gathered as some context. Zotero = Your personal research assistant. Zotero is a free, easy-to-use tool to help you collect, organize, cite, and share research. https://www.zotero.org/ https://www.zotero.org/ Mendeley = Reference Management Software, produced by Elsevier who also happens to be the publisher of many peer-reviewed journals. Elsevier come under fire for it's high costs and gateway actions to restrict access to information they've published in journals and host in archives. This most recent action of making the database of references in Mendeley difficult to export is a continuation of their attempt to protect what they, and some legal systems, would see as their IP. Others disagree. The battle continues...
- carbocation 8y agoGood idea re: giving context as to what these programs are. For most of us who do research, these programs are essentially interchangeable "reference managers." Here's what I mean: I read a scientific paper or look up a citation. I add that with a click or two to my reference manager. It also stores the PDF for me. In the future, I can easily re-read the PDF. I can annotate it and the annotations will be stored. Critically, when writing my own paper, I can import those citations and trivially change the format to whatever the publisher wants without any effort. To me, across most features, these two programs do exactly the same thing. When picking a citation manager, it's more about which one I trust will be around for the long haul and will not interfere with my research.
- zwaps 8y agoThe only difference is that Mendeley has a built-in PDF viewer with annotation. Of course since you can no longer share these annotations, nor export the pdf, that's all pointless. Mendeley opens PDFs externally, but many PDF readers can create and save annotations.
- carbocation 8y agoIn your second paragraph I think you mean to lead with "Zotero" instead of "Mendeley." And yes, I agree with that distinction. In practice, it hasn't been much of a problem (even though I do my work across OS X and Linux).
- mdip 8y agoIf I'm understanding things correctly from the reading (which, let's be honest, that's an assumption on its own), the tl;dr seems to be that Mendeley heard about an open source product working on a set of importers that would, potentially, result in Mendeley customers migrating to this free alternative[0]. To provide cover for this change, they invoked user security. And hey, user security is a pretty good reason to encrypt a database, but if security were the only aim, they could have made the data accessible for export purposes (after all, it's accessible for the user to read). At this point in history I'm not even sure it's worth going into all of the reasons this is a terrible thing to do on their part. At best, they'll scare off some users from migrating away briefly and buy themselves some time to figure out what they need to do to produce something that's better than the competition they're attempting thwart. More likely, they'll simply fail (quickly or slowly). It's that last point that really matters to me. If I'm buying an application that I'm going to rely on, I want comfort that the company will continue to maintain the product -- or in the case of open-source, confidence that the product is still regularly maintained (or in languages/frameworks that make it practical for me to maintain myself). Aside from the fact that I wouldn't want that feature 'as it has been designed, today', I wouldn't want to rely on a product that's being designed by individuals who aren't well versed in the perils that DRM-like behavior causes. After all, that's what this is -- it's an attempt to keep the user from transforming their own data for the purposes of locking that customer into the product while attempting to position the lock-in as a security feature. And it failed on both accounts, Zotero worked around the encryption (which means the bad guys can to) and if people looking for a product like this care about that capability, they'll likely refine it to the point that it's as capable encrypted as it is decrypted. Every time I've been asked to implement a DRM solution I've either outright refused or paired it down so much that the "R" didn't really exist. They're horribly complex to write, off the shelf solutions are already cracked and time would be better spent giving customers something they're willing to pay for. The thing that makes me roll my eyes every time I read one of these is "there are still engineers out there who think this is a good idea?" I can't wrap my head around what causes this kind of thinking -- it's not a zero-sum game, there's room for more than one product in the marketplace -- even if one of them is free and open-source. Trying to beat the competition by sabotage rather than by making a better product doesn't work ... at least not for very long. [0] And they also do my least favorite "other marketing thing" -- "Create a Free Account" and no hint of what the thing costs in any obvious place. So maybe this other product is free? I doubt it -- Like most companies, I'm assuming they're perfectly happy collecting your personal information for free (in order to send marketing to you to get you to pay for whatever service is tied to the free account). It's the little deceptive things that drive me crazy -- you didn't trick me, you only managed to make me suspicious of your product. [1] I've gotten lucky ... usually budget and time doesn't allow for wasting energy on customer-hostile features that won't achieve the ends they're aiming for. The only case where something "DRM-like" was in an application I wrote was to be the opposite of customer-hostile. We had an app that provided information to users based on their permissions, determined by them logging in. I pushed to make the automation (sending us the login information) opt-in, with manual provisioning options. It worked well, since if they failed to true up after months of warnings, the application would deactivate (due to the size of the customers, months of warning was somewhere near a year and could also be disabled with a flag).
- masnick 8y agoFor anyone who is interested in the different options for reference management, I've been sporadically maintaining a webpage about the various options since 2015: https://maxmasnick.com/projects/reference-managers/ https://maxmasnick.com/projects/reference-managers/ I re-un-recommended Mendeley back in June 2018 when this news first broke. (I was initially too sketched out by Elsevier to recommend Mendeley, and this just confirms these suspicions.) tl;dr Zotero is a pretty good bet for most people, especially since they added support for citation management in Google Docs at the end of last year (very important for academic writers). PaperPile (https://paperpile.com https://paperpile.com) and Papers (https://www.papersapp.com https://www.papersapp.com) are also worth checking out.
- ramraj07 8y agoWhich one gives the best seamless mobile integration experience? Papers for iOS wasn't bad (if you let it finish syncing with Dropbox ) but curious about this!
- masnick 8y agoProbably Papers is your best bet at this point. Zotero will let you send PDFs to/from a tablet with the ZotFile (http://zotfile.com http://zotfile.com) plugin. This works well if your workflow is (1) find PDF on your computer; (2) read on tablet. But if you want to do anything else -- even choosing a PDF to read from your tablet without touching your computer -- then Zotero won't work. With that said, I know Zotero is working on mobile apps...not sure how far they have come though. If you're Mac/iOS only, also check out https://www.sonnysoftware.com/bookends/bookends.html https://www.sonnysoftware.com/bookends/bookends.html.
- spott 8y agoDo you have any experience with bookends?
- masnick 8y agoYes, it's got some really nice features but it is a little clunky/complex compared to Zotero. I would try out Papers first before looking at Bookends...Bookends can work great, but it does take some work to set up. Also, Bookends annoyingly checks to see if you are running the same license on multiple devices on your network, and if you have more than once instance running it forces you to close it down. This may seem superficially reasonable, but for someone with both a desktop and a laptop it's quite annoying. I emailed the developer about it and they didn't seem to get why this was a problem.
- __jl__ 8y agoFor Zotero users: Try the add on zotfile (zotfile.com). It allows you to extract PDF annotations and save them as notes in Zotero (among other things). Full disclosure: I am the developer.
- esalman 8y agoI love Zotero. Thanks.
- masnick 8y agozotfile is a must-have for Zotero! Thank you for making it.
- burtonator 8y agoHey.. I tried to reach out to you at your personal email address regarding collaboration between Zotfile and Polar and I never heard back. https://getpolarized.io/ https://getpolarized.io/ The idea was to have Polar also work with Zotfile since a lot of people use Zotfile.
- boyband6666 8y agoGenuine question - why make zotfile and not adding things to Zotero directly? After all it is open source. Zotfile adds a lot of useful things that feel like they should be in the main program, not optional extras. I'm not a programmer however, so this may be very naive question!
- 12fro 8y agoAnother very grateful Zotfile user who wants to say thank you for your hard work. I use it every day.
- minosniu 8y agoSecond this. I use zotfile to relocate all my PDFs into a single folder, which is Dropbox-synced. This works like a breeze for 2000+ and mounting papers.
- masteranza 8y agoYep, if you're still using Mendeley, you should move to Zotero immediately. I have used Papers app for some time, but they've worked on the desktop app way too long after their merge with ReadCube and now I'm happy with my free Zotero. I've just tested ReadCube Papers with a simple pdf and it's nice and fast, but cannot resolve the metadata (even the suggestions are wrong) while Zotero can do it unambiguously on the first try.
- jjoonathan 8y agoMendeley also snitches evidence of your SciHub habit to Elsevier. It probably won't amount to anything, but do you really want to bet against Elsevier becoming exceptionally greedy, desperate, or both?
- aw3c2 8y agoPlease provide some citations for that claim.
- eythian 8y agoThey would, but they can't export them.
- SubiculumCode 8y agoI presume that the source of the file is in the metadata..it is often placed in references. Since those are uploaded to Mendeley in their cloud storage service, it would seem that thry have the data..now it comes down to eulas
- jjoonathan 8y agoIt uploads your PDFs to Elsevier servers for storage. By design. Asking how that could lead to Elsevier obtaining evidence of paper piracy is like asking how using gmail could lead to google using your email to do targeted advertising. You are giving them everything they need to do it, so of course they might do it! The position that needs defending is the supposition that they can't.
- aw3c2 8y agoThat is quite far fetched. You made it sound like they specifically identify and monitor sci-hub usage somehow.
- phren0logy 8y agoTo anyone looking for an alternative, I'd just throw out there that I've been very happy with ReadCube. It's proprietary subscription software, but that's a trade-off I can live with for a great browser plugin, mobile apps with good annotation, cloud storage of annotated articles, seamless syncing across devices, and a high-quality Word citation plugin that works on the Mac. This company bought Papers, the Mac app, and it looks like they are finally getting ready to release the desktop client based on that acquisition.
- krull10 8y agoI've been looking for a replacement for Papers for years; it just doesn't work well anymore with the lack of attention they've given it since the acquisition. For me subscription pricing is a no-go. It's sad, I would happily pay for a new standalone Papers...
- veddox 8y agoI used ReadCube several years ago when it was still really new (and free). Great program, very pretty interface - but it didn't work on Linux. When I switched from Windows, I had to let it go.
- phren0logy 8y agoI mostly use the web version, and that seems to function well everywhere these days. Not sure about the desktop version, but after ages and ages they are finally getting the desktop application that incorporates their Papers acquisition, so it might be worth looking again if you like it. Hopefully, that does a better job on Linux, but that's always a gamble.
- pytyper2 8y agoThese science website corporate types keep outsmarting all you scientists.
- boromi 8y agoAnyone else here use Jabref? I like it's simplicity.
- erreJulian 8y agoOnly if I need to tweak a .bib file, but I use Zotero with BetterBibTex (https://retorque.re/zotero-better-bibtex/ https://retorque.re/zotero-better-bibtex/) and it generates a .bib file of my entire database.
- veddox 8y agoMendeley is (or at least, was) a great software. It works on Linux and did exactly what I needed. Half a year ago, I suddenly couldn't open my PDFs anymore from within the desktop app. Web interface still worked, mobile still worked, just the desktop version was broken. (Which happens to be the only one I seriously use.) When I wrote to them several months later inquiring into the issue, they told me they were still working on the fix. (It's still not working on my machine.) Now this. Perhaps I really should look into Zotero...
- iicc 8y agoGDPR?
- TelmoMenezes 8y agoOh come on...
- iicc 8y agoEU citizens have an explicit right to a direct transfer in to Zotero. https://gdpr-info.eu/art-20-gdpr/ https://gdpr-info.eu/art-20-gdpr/ "Right to data portability" >The data subject shall have the right to receive the personal data concerning him or her, which he or she has provided to a controller, in a structured, commonly used and machine-readable format and have the right to transmit those data to another controller without hindrance from the controller to which the personal data have been provided, where: > the processing is based on consent pursuant to point (a) of Article 6(1) or point (a) of Article 9(2) or on a contract pursuant to point (b) of Article 6(1); and the processing is carried out by automated means. >In exercising his or her right to data portability pursuant to paragraph 1, the data subject shall have the right to have the personal data transmitted directly from one controller to another, where technically feasible.
- fantasticfears 8y agoGood point. I'm trying to get my dump from them.
- guitarbill 8y agoIronically, GDPR is what Mendeley used to justify the encryption [0]. Obviously, complete rubbish. It would be interesting to see what happens if people start asking for their data in a portable format though. [0] https://twitter.com/mendeley_com/status/1006915998841221120 https://twitter.com/mendeley_com/status/1006915998841221120
- bocklund 8y agoChanges in 1.19 encrypted the database, but I think 1.19.2 restored this feature see: https://www.mendeley.com/release-notes/v1_19_2 https://www.mendeley.com/release-notes/v1_19_2
- falien 8y agoThat just added the ability to save a copy of the encrypted database. It did not remove the encryption.
- lvs 8y agoEntertainingly, it was precisely that update that induced me to drop Mendeley and switch to Zotero. After updating to that version, every login attempt at the app failed. I guess it hadn't fully updated the database because import into Zotero worked flawlessly. I was pleasantly surprised by how much better Zotero has gotten in the last few years. The browser and MS Word plugins actually work pretty well, although several of the citation style templates seem out-of-date.
- erreJulian 8y agoI´ve been using Zotero for a long time now and I can’t recommend it enough. Along with several plugins, such as ZotFile [1] and ZoteroBetterBibTex [2] I’ve been able to build a very flexible workflow that allows me to work either in a traditional processor, in Latex and even in Markdown+pandoc on iOS. Recently they also added support for Google Docs [3] and it does seems to be working quite well. It enjoys the benefits of open software, such as an engaging community and support. [1] http://zotfile.com http://zotfile.com [2] https://retorque.re/zotero-better-bibtex/ https://retorque.re/zotero-better-bibtex/ [3] https://www.zotero.org/support/google_docs https://www.zotero.org/support/google_docs
- plg 8y agopaperpile is sweet I just wish they would let iOS users add a paper to their database
- laughingman2 8y agoI hate to be under the thrall of Elsevier software. But the shitty tactics would force me to switch to Zotero anyways. I just wish Zotero packed in a consistent cross-platform pdf annotation tool and a mobile app. I wouldn't mind paying for these.
- ismail 8y agoI dropped Mendely a year ago. Because it kept crashing and they could not resolve the issue. The bug has not closed since then. I was unable to extract my highlights. I tried mucking about in their SQLite dB. Tried zoetro but prefer using highlights app ad It saves the highlights on the PDF. And you can then export to markdown etc. Not sure how zoetro saves highlights but I am never going to take that risk again. A years worth of research down the drain.
- aasasd 8y agoI wonder if they're not actually required by GDPR to provide access to data export in an open format―though it might, again, only apply to online storage. And of course, it would only be required in the EU.
- burtonator 8y agoI'm the author of Polar. An app very similar to both Zotero and Mendeley: https://getpolarized.io/ https://getpolarized.io/ Both Mendeley and Zotero are better than Polar (for now) with bibliography management but I believe we're better at both at document annotation and knowledge management. We're also Open Source and launched on Hacker News a few months back. https://news.ycombinator.com/item?id=18219960 https://news.ycombinator.com/item?id=18219960 We're still rather new so not a lot of people have heard of us yet but moving forward very aggressively. I'd also like to thank the Zotero guys for posting their notes about the Mendeley encryption issue. I think this is a completely unacceptable situation and antithetical to what we should expect in the scientific community. The Polar on disk repo is exactly the opposite. We store all your data on disk and your annotations are in JSON format so they're easily hackable. This is part of the design. We MAY add end to end repository encryption at some point but it will be in the users control. We're not doing it to lock the user out or to prevent export. The end to end encryption is so that you can store your repo in the cloud and not have to worry about your data being viewed by anyone other than you.
- zwaps 8y agoThank you for your work. I will try your software soon. What I really like so far is the reading progress, which is actually very, very important for me. So far, I have to do this with manual tags in Zotero. By the way, since this is a work tool, I (and I imagine others) are fully willing and able to pay you for support or storage or other features. But since Mendeley, me and people I know have become very sensitive about lock-in, encryption and collaboration barriers associated with pro/non-pro accounts. It seems you are determined to take the right steps, I just want to emphasize the importance for your business. The backlash against Mendeley isn't random. If I build my pipeline on your product, my lifelihood and hours upon hours of crucial work depend on it. This is not a casual software, where switching is inconvenient but not a big deal. Mendeley has f'ed me in a crucial deadline situation, even though I was paying them lots of money. That company is dead to me, and I will badmouth and try to destroy their business any chance I get.
- snowwindwaves 8y agoI have been using Qiqqa for 4 years and it is perfect for me. It allows me to perform full text search on my PDF library, performs OCR when necessary, and supports tagging, annotations, and bibtex. http://www.qiqqa.com http://www.qiqqa.com It is wonderful software. It provides me with my own personal google for my PDFs. I used to use google desktop.
- shittyadmin 8y agoSome quick RE work: sqlite_rekey_v2 is at 00CA1C13 on the Windows version of 1.19.3 Hooking it with Frida or similar should allow you to drop the 3rd parameter and set pKey to NULL allowing you to create a database which can be read by standard sqlite. Keys however appear to be account-specific - I haven't looked into the full algorithm yet, but it's fairly pointless as you'd need a copy of sqlite3 with SEE support (which costs $2600 or so) to decrypt it using that method anyways, I tried briefly but was unable to come across a stray copy of sqlite3 with SEE enabled, the above solution is probably better for now anyways.
- asdff 8y agoI wish there was one reference manager that wasn't so clunky and convoluted, and I've about tried them all now.
- fori1to10 8y agoThe only reason I keep using Mendeley sometimes is because it is nice for reading on the phone, and it syncs annotations back and forth without issues. If I can set something as smooth with Zotero, that would be it.