12 ms·
Elsevier embeds a hash in the PDF metadata that is unique for each download
- wallaBBB 5y agoPatents (the way they are used today) are often mentioned on HN as something pretty damaging to progress, but these efforts of maximum monetization of the scientific publishing industry are no less of an evil. If you can donate to SciHub, and if you're publishing look for open alternatives to Elsevier's claw.
- MayeulC 5y agoAt least patents expire after a while, and they are public. They mostly slow down commercial application of new discoveries. Copyright doesn't really expire in some places, or after a very long time. I believe this is way more damaging to progress. Especially for publication, since science works better in a tight feedback loop (and it doesn't work as well if... authors die before others can reply to their papers).
- PicassoCTs 5y agoMost likely to grab funds from librarys who "lost" the document.
- leemailll 5y agoa hash in meta might be news, but such behavior is not unique to Elsevier. Years ago journal based on Highwire platform add the date, the institution, or maybe ip address to the side boundary of pdfs.
- deanc 5y agoI worked on an a service that did this to prevent unauthorised distribution of something we sold in zip files. You can add a lot of identifiable data to pretty much any file format if you try hard enough.
- atoav 5y agoI would download from multiple accounts, and then compare the resulting file, normalize the differences.
- y4mi 5y agoYou wouldn't unless you knew they're doing that. And you likely wouldn't even have the option because you'd have to pay multiple licences just to diff them with franc's example.
- deleted 5y ago[deleted]
- deanc 5y agoSure, this is not foolproof - but you'd be surprised how many people were caught.
- hardwaresofton 5y agoI’ve often thought of building a similar service… on a scale of one to NSA how questionable did it feel to work on? When I think about it doing it well might mean journalists or scientists get caught doing something I morally agree with
- deanc 5y ago1. This was in order to track pirates of the software and revoke their license. Nothing out the ordinary :)
- Bancakes 5y agoWhat happened to the SciHub seeding efforts? Do they still need help with torrents?
- Ericson2314 5y agoWell, better update sci hub to download twice and then normalize any differences!
- iforgotpassword 5y agoCame here to suggest as much :-) While you can apparently just strip it from the metadata properly like suggested on twitter, maybe a "low level" approach like comparing the files on a binary level and setting any bytes that differ to 0 would be more robust. It would still work if they move the hash out of the meta data into the document itself. Only downside is that this requires the hash to be fixed size.
- jrib 5y agoI think this is a clever solution. I'm not seeing why it requires the hash to be a fixed size though. Why can't the algorithm just keep whatever is common between the two files? Or just take the shorter side of the diff and zero it out?
- roblabla 5y agoIf the PDF contains encrypted blocks, this won't work, as simply zeroing out the bits will break the file. Even worse, if the PDF contains compressed chunks (which happens often enough), such a naive approach won't work either - the chunks would have to be uncompressed before comparison. Such a tool would need a lot of smarts to work reliably enough. At this point, I feel like a metadata stripper that understands the various watermarking methods may be easier to write.
- iforgotpassword 5y agoHm, makes sense. But if I were in their shoes, moving the fingerprint out of the metadata would be the first thing to do then, so no easy solution then I guess.
- simiones 5y ago> normalize any differences! Elsevier could easily make this completely non-trivial. For a very silly example, imagine how this would work if each user got a PDF using a different font. How do you automatically normalize the difference between Comic Sans and Times New Roman? Sure, it's trivial to write a tool that understands what "fonts" are and does this (especially for PDF), but you can't do it with a simple binary tool. And of course Elsevier can do something entirely more complex.
- firstSpeaker 5y agoAt some point the whole state of academic research/papers and publishing should be overhauled. My SO left Academic work and started in the industry because of the mess that exist in the academic research industry. The funding and keeping yourself funded as a researcher is a depressing subject to think about.
- YeBanKo 5y agoWhy is getting yourself funded is such a depressing idea? And what is an alternative?
- wly_cdgr 5y agoSo just take pics of the pages and convert the pics back to a PDF
- dbsmith83 5y agoEasier to just strip out the metadata
- squarefoot 5y agoA motivated publisher could embed codes by altering in subtle ways the differences in distances or color between adjacent characters, so that they would survive most color or grey scale conversions; a seemingly innocuous frame drawn around a photo could be either larger or smaller by say one millimeter, representing de facto a bit, therefore using enough pages they could identify a book among billions. Unfortunately there's no way to be 100% sure that a complex document doesn't contain some form of embedded code.
- json_dirs 5y agointeresting -- i don't know enough about PDFs, but I saw something in a PDF /OpenAction that sounded like this. wonder if anyone who knows more could see? https://twitter.com/json_dirs/status/1486274161655771136?s=20 https://twitter.com/json_dirs/status/1486274161655771136?s=2...
- deleted 5y ago[deleted]
- asdff 5y agoYou could try and break this by adding something like some random noise or jitter maybe, or slightly transforming the proportions of the pages perhaps, or shifting colors in a stochastic way would probably complicate their efforts. The frame around the photo will no longer be exactly 1.1231 mm and will throw off their embedded code reading systems. The colors won't be the same hex codes they are expecting and won't be shifted evenly. Spacing is now all off between the characters.
- 5y ago
- jinseokim 5y agoBe aware that deleting metadata is never enough: There are too many ways to hide some fingerprints on the PDF document.
- post-it 5y agoAll you need to do is download two documents and diff them, and delete anything that doesn't overlap.
- yjftsjthsd-h 5y agoYou'd need to download from different identities; if I was them I'd be injecting user, IP, organization, date, and a signed hash thereof (tamper evidence if someone does something like change a digit in the IP)
- deleted 5y ago[deleted]
- post-it 5y agoThe signed hash doesn't matter because you only need to de-identify the document, not pass it off as someone else's. If the organization finds a document with all of the identifying information removed, they know that someone fucked with their DRM but they don't know who.
- yjftsjthsd-h 5y agoMy thought was that if the publisher is trying to hunt people sharing copies, and they have such a copy, it would be useful to be confident that the metadata you embedded is actually accurate; sure, it's obvious if, say, the IP field is zeroed out, but what if they just changed the last octet to 7, and that results in you spending weeks leaning on an ISP to give you the identity of the wrong person? Granted, that's probably more care than Elsevier is likely to take, but the point is that they're passing data through hostile hands, so it'd be sensible to do something for integrity checking.
- isoprophlex 5y agoI've often seen banners stating "Downloaded on such and such date, by this and that university" on papers downloaded from sci-hub. I'm hopeful this hidden metadata won't hurt them either.
- camel_Snake 5y agothis isn't uncommon outside of scihub/libgen as well. I use Google Scholar to search for citations of scientific articles and they'll often have links to pdf hard-copies hosted by university servers. I see Penn State University a lot. Polish universities too, but I'm blocked from access as I dont' have a polish IP
- nemoniac 5y agoThe author of the tweet also supplies a tool for cleaning the metadata in pdfs. https://gist.github.com/sneakers-the-rat/172e8679b824a3871decd262ed3f59c6 https://gist.github.com/sneakers-the-rat/172e8679b824a3871de...
- hdjjhhvvhga 5y agoThis is a very basic tool. It may work against this particular scheme by Elsevier if it's not too sophisticated. A few years ago I was working for another content provider to mark the PDFs, my job was to make sure the watermarks are everywhere and that removing them is difficult while being practically transparent to the user. We also employed steganography etc. so that rendering the PDFs as images and combining them again into PDF would preserve the watermark. We were very clear to the user the watermarks are there and, as far as I can tell, there was no single copy protected in this way uploaded to the public Internet.
- sam_lowry_ 5y agoI wonder how you coped with the ethical side of such a job.
- hdjjhhvvhga 5y agoHere's the story. A middle-size publisher contacted me about the problem they had: someone was distributing their PDFs online using various services, but mainly Scribd. At that time (it was maybe 2012?) it worked like this: the attacker released a document, notified his group, and they started downloading. The publisher immediately filed the DMCA copyright infringement notification and it was handled within 3-4 days - more than enough for people to download the "release". It was going on for a long time and Scribd refused to remove the account of the attacker.[0] After a couple of months they finally gave in and deleted it but of course a new account was immediately created. It was not a mouse-and-cat game, but more like mouse-and-turtle. So when they asked my help I told them, you don't have a copyright problem, you have a SEO problem - since people looking for their publications could find them easily online with the stolen copies sometimes appearing higher on SERP than their own. They said they would address that and they somehow came to terms with the fact that the past documents are lost, but they would like to avoid this situation in the future. I checked some of their publications. They were moderately priced and the content was quite interesting. Many of their customers were very supportive - actually they were sending reports when they noticed the pirated copies. And they weren't saying "Why should I pay for your books when I can get them online?" but rather "Please take care of protecting your content since we want you to survive and publish more books". So, to answer your question, I was very happy to work for them, and I would do it again. [0] I was told they changed their approach later and started to collaborate with publishers.
- durnygbur 5y agoElsevier, Axel Springer, et al. - proudly publishing research sponsored by EU and national grants... and hounding anyone who doesn't pay 20 EUR for every PDF! Anyways TIL about mat2 and dangerzone.
- cblconfederate 5y agoBut sci-hub does not embed anything. Checkmate elsevier I m considering that elbakyan has enough credibility to start her own series of open source journals
- dodgerdan 5y agoProbably a uid rather than a hash. And the token has corresponding metadata like ip, account number, date/time etc.
- eecc 5y agoMy “don’t even call me” list has two entries: * anything military * Elsevier
- throwaway984393 5y agoI like how you're fine with being called by a publisher that charges more than Elsevier and provides less open access papers
- NmAmDa 5y agoI wonder if there are any offenders that exceeds Elsevier in this regard.
- jimmygrapes 5y agoI nominate ISO and/or ANSI
- iqanq 5y agoIt makes sense to do something like this if you want to detect who is exfiltrating data.
- sharikous 5y agoEven if I don't like the practice of charging so much for scientific papers, I must say that watermarks are the best form of DRM. You own the file, you can print it, you can save it. That's a stark difference from most schemes going around these days
- jmnicolas 5y agoExcept when the watermark is right in the middle of the page and harms reading flow. I had the case a few months ago very frustrating. The PDF was encrypted, I couldn't remove the watermark.
- SSLy 5y agoThere are open source apps that ignore PDF "encryption"
- newaccount74 5y agoqpdf --decrypt infile.pdf outfile.pdf
- criddell 5y agoWhere do they get the key from?
- jmnicolas 5y agoSince I wanted to read it asap, I didn't invest too much time trying to defeat it but if you have something reliable I'm all ears for next time. From what I understand Adobe Acrobat might have done the job, but it's an expensive software and I refuse to run cracks anymore.
- mherrmann 5y agoI've often wondered why such schemes are not more ubiquitous, for instance for porn movies. Does anybody have an idea?
- steerablesafe 5y agoIt probably doesn't worth the cost. Metadata can be trivially stripped, and altering the encoded video and audio streams for each user for fingerprinting is costly.
- zibzab 5y agoSomeone should check this against the sci-hub library. Could make a nice research project :)
- miki123211 5y agoOver here (Poland) almost everyone does something of that sort. Our ebook market is so fragmented that there's no DRM solution that all (or even most) e-readers work with. If you add in smartphones and car stereos (for audiobooks), the situation gets even worse. Therefore, most publishers use watermarking instead of DRM, usually giving you Epub, PDF and Mobi, which you can read on any device you want. The most common form of watermarking, at least when epub is concerned, is a 1px by 1px div containing a nonsensical hex or base64-encoded string. There are rumors of watermarks in cover images and even the text itself, though. Apparently, sometimes spaces get removed or extraneous spaces are added, lines get split up a little differently, or some common spelling or typing mistakes are made. Considering the huge number of alteration points you have, let's say 10000 per book, even doing two alterations lets you uniquely watermark 10000^2 copies. Similar things are done to audiobooks, whether by modifying the audio itself in imperceptible ways, or by modifying the internal structure of the mp3 files. From what I've heard, messing with how frames are laid out and what's in-between them is a common tactic.
- darkwater 5y agoBut how do they enforce this watermarking on the "pirate" afterwards? Only for big enough cases?
- s_dev 5y agoProbably a maintained whitelist.
- matheusmoreira 5y agoThey probably disable the account of whoever leaked the copyrighted material.
- newaccount74 5y agoThey could just call whoever leaked the stuff and ask them to stop.
- aqme28 5y agoWhat do they do with them? Are there criminal or civil cases with those watermarks? In the US courts I believe those secret watermarks would be made public at trial, so what’s the point?
- steerablesafe 5y agoTo verify that a watermark removal method works: 1. Download the content with N accounts, preferably from different networks. 2. Run your watermark removal tool on each downloaded data independently. 3. Check if the processed outputs are bit-for-bit identical. Have fun writing watermark removal tools.
- amelius 5y agoWhat if the files are still different? https://en.wikipedia.org/wiki/Steganography https://en.wikipedia.org/wiki/Steganography
- steerablesafe 5y agoThen your tool is incomplete. Steganography works by encoding information into side channels, like precise positioning of words in the PDF, fonts, etc... Your tool can be as crude as running the document through pdftotext and only keeping the text output. It can just throw away all these possible side channels that are not relevant to the actual content. Others mentioned other more sophisticated side channels, as rewording sentences, but I'm sure authors would not welcome that.
- nisegami 5y agoIf the bits are identical, where does the hidden information live?
- hiq 5y agoDoesn't work if the watermark has a time-based component.
- steerablesafe 5y agoI don't see why.
- torstenvl 5y agoOf course it does. If two files are identical then by definition there is nothing in them that can be used to tell them apart.
- NKosmatos 5y agoMany years ago I had a discussion with a friend about something similar, in the sense of tracking PDF downloads. He wanted to sell educational material in PDFs and he wanted to be able to track pirated copies and who shared his paid PDF and that sort of thing. My proposal was something similar, but using steganography inside the PDF content instead of using metadata that can be (relatively easy) stripped. Each time someone bought the PDF, some trivial data (email, date, IP...) would be embedded in the first and last page of the PDF (title page & blank page) at specific coordinates with very small font size and almost white color (same as the page), so that later someone could read them if needed. Yeah, a silly solution and perhaps that's why it didn't move any further past the proposal phase. Hope I'm not giving any ideas to Elsevier and all the other greedy publishers with this ;-)
- chiefalchemist 5y agoI see mention of removal tools. Is this something that could be baked into a browser so it happens automatically during the download process? Or some other way to make it automatic, as it's more likely to happen. As, along the same lines, could the original be "scrubbed" for storage so there's no "paper trail" of you having received it?
- Eddy_Viscosity2 5y agoPrint and then scan the paper for the old-school slightly blurry, often misaligned, feel that papers should have.
- rob_c 5y agoOk, that's just being a bad actor...
- herodotus 5y agoI think that, on MacOS, if you print the PDF to a PDF and save that, the metadata will not be transferred.
- lr1970 5y agoIf a watermarked PDF ended up on the internet it does not necessarily mean that the person who purchased the said PDF leaked it themself or did anything wrong for that matter. Computers are hacked, stolen all the time. At Universities machines in the lab are oftentimes shared and document dumps exist on shared partitions. In a court of law (at least in the USA) the burden of proof is on the plaintiff. It could be expensive and difficult to prove that the PDF purchaser did upload it themselves and broke the law. Similar to the music and movie pirates - going mostly after big fish.
- yohannparis 5y agoA researcher explains that if you want a paper, you can directly email them, and they will happily send you one. Avoiding all this non-sense.