6 ms·
Elsevier embeds a hash in the PDF metadata that is unique for each download (2022)
- 1vuio0pswjnm7 2y agoPrevious discussion: Elsevier embeds a hash in the PDF metadata that is unique for each download (twitter.com/json_dirs) 343 points by sohkamyung on Jan 26, 2022 https://news.ycombinator.com/item?id=30082138 https://news.ycombinator.com/item?id=30082138
- 3lit3krew 2y agoDoesn't everybody publish their articles online now too?
- hobofan 2y ago[2022]
- vsuperpower2020 2y agoIt sounds like it's good practice to get two copies of a document from two different sources, and to compare their hashes before publishing them. You can embed data in anything, so this would include images, audio files, PDF files, or programs. At least for Elsevier it's pretty obvious they're using a key to track you.
- stefek99 2y agoSOP = standard operating procedure. I would be surprised if it was otherwise.
- tecleandor 2y agoComing from the DICOM world, for me it's always Service-Object Pair
- DaSHacka 2y agoFrom the locksmithing world, I thought of System Of Progression at first Before, of course, realizing no way someone on HN would be talking about obscure locksmith masterkey terminology
- hristov 2y ago1. Download the PDF from Elsevier. 2. Open it in your own PDF viewer. 3. Press print. 4. In the printer selection box, select "Print to PDF".
- Xelbair 2y agoif you print PDF file to PDF it creates a copy of a file.
- deleted 2y ago[deleted]
- withinboredom 2y agoOn all OS's? That seems like a nice optimization, but I doubt it is true everywhere. There's always someone who never optimizes anything.
- dsissitka 2y agoDoesn't seem to. david@desktop:~/test$ ls -hl total 8.9M -rw-r--r--. 1 david david 2.2M Jun 10 06:02 chromium.pdf -rw-r--r--. 1 david david 4.4M Jun 10 06:02 firefox.pdf -rw-r--r--. 1 david david 2.3M Jun 10 06:01 input.pdf david@desktop:~/test$
- BSDobelix 2y agoOn Firefox if you print you can choose "safe to pdf" then it has a different checksum (compared to the downloaded one), if you say "print using the system dialog" then "print to file" the checksum is again different (compared to the other two). Linux with Firefox 126
- ExoticPearTree 2y agoMost likely it uses different libraries for PDF print, like the browser one is different than the system one.
- fallingknife 2y ago
- nonrandomstring 2y agoMetadata is very low hanging fruit for document watermarking. Typically the PDF renderer will use spacing, kerning, invisible characters, and all sorts of steganography to make each copy unique. What would be the point of a hash? More likely the hash is a MAC, that's been salted with some secret plus the unique copy. That would help the publisher identify a laundered copy. With two or more copies its possible to re-anonymise. That's actually something I wonder whether summarising language models would be good at. Of course they may also make steganographic alterations to diagrams. Because PDFs are such dirty documents I almost always convert them to plain text, usually with no loss of semantics.
- amelius 2y agoYes, but don't give them any ideas.
- nicce 2y agoEven plain text has its risks: https://news.ycombinator.com/item?id=33621562 https://news.ycombinator.com/item?id=33621562
- sodality2 2y agoIf they modify academic papers’ actual source content they would lose all of their trust. Imagine they doctor some data just to create more diffs
- nicce 2y agoThey modify it all the time by providing different access formats and with editors. But it does not change the meaning. Whitespaces do not alter the meaning.
- Y_Y 2y agoWhitespaces do not alter the meaning. Whitespaces do not alter the meaning. White spaces do not alter them eaning. M e a n i n g
- hcrean 2y agoImagine working this hard to track down people sharing ideas, that you didn't work to produce or fund in the first place, in order to punish them for not giving you a financial cut... Companies like this are just holding humanity back.
- dandanua 2y agoIt's not actually hard, but I agree with the point. BTW, they also charge authors ridiculous amount of money for additional services, e.g. printing colored images. In a far (and optimistic) future such companies will be studied as examples of social parasites in schools and universities.
- giancarlostoro 2y agoStarting with Oracle I am sure.
- geysersam 2y agoPrinting colored images! What does a single issue of their printed editions cost? Hundreds of dollars? The color printing cost is a rounding error.
- qup 2y agoI think the universities themselves will be studied as social parasites, too.
- vsuperpower2020 2y agoI think they'll be studied as universal basic income. There's so many college graduates with no skills who basically unemployable, but are still collecting a decent salary to teach things that no one wants to pay for in the first place. Our economy can evidently support a lot of people who aren't working.
- LastTrain 2y agoIt already is studied and there is even a term for it: rent seeking. https://en.m.wikipedia.org/wiki/Rent-seeking https://en.m.wikipedia.org/wiki/Rent-seeking
- huhtenberg 2y agoWith all possible steganographic and watermarking options this looks ... I don't know ... eye-poppingly dumb?
- gpvos 2y agoWhatever method they choose, you can just diff a couple of PDFs to figure out what they did, and then write some code to detect and revert or mask that.
- deleted 2y ago[deleted]
- red_admiral 2y agoI'm sure they use other watermarking techniques as well.
- henriquenunez 2y agoWe need to create a tool to allow the sharing and unlocking of humanity’s knowledge
- cutemonster 2y agoYou haven't heard about Sci-Hub? https://en.m.wikipedia.org/wiki/Sci-Hub https://en.m.wikipedia.org/wiki/Sci-Hub I suppose it'd be nice if they automatically removed such fingerprints
- BSDobelix 2y agoOpen society's?
- lotsofpulp 2y agoThe internet?
- BSDobelix 2y agoProudly licensed from Red Star OS ;)
- omeid2 2y agoFor the uninitiated: https://insinuator.net/2015/07/redstar-os-watermarking/ https://insinuator.net/2015/07/redstar-os-watermarking/
- yencabulator 2y agoHah! > The watermark ends with the string “EOF”.
- henriquenunez 2y agoI knew it!
- kkfx 2y agoNothing strange, I have a small script to cleanup pdfs in general (reducing their size as well), essentially pdftops -paper A4 -expand -level3 file.pdf ps2pdf14 -dEmbedAllFonts=true \ -dUseFlateCompression=true \ -dOptimize=true \ -dProcessColorModel=/DeviceRGB \ -r72 \ -dDownsampleGrayImages=true \ -dGrayImageResolution=150 \ -dAutoFilterGrayImages=false \ -dGrayImageDownsampleType=/Bicubic \ -dDownsampleMonoImages=true \ -dMonoImageResolution=150 \ -dMonoImageDownsampleType=/Subsample \ -dDownsampleColorImages=true \ -dColorImageResolution=150 \ -dAutoFilterColorImages=false \ -dColorImageDownsampleType=/Bicubic \ -dPDFSETTINGS=/ebook \ -dNOSAFER \ -dALLOWPSTRANSPARENCY \ -dShowAnnots=false \ file.pdf file.pdf that's is. After if needed we can add extra metadata. It's not specially designed to remove certain kind of tracking but simple and useful enough in most cases.
- barfbagginus 2y agoEksevier must die, and we must spare no effort to kill it. Work to kill elsevier. Use the library Genesis Use the QubesOS PDF sanitizer or similar Integrate a sanitizer into the Zotero ref manager Contribute to libgen software projects Use the sci-hub Tell your peeps, tell your peers, tell your org Sabotage peeps peers and orgs that won't help
- TaylorAlexander 2y agoSci-hub is so good. I’m a professional robotics engineer working on open source robotics funded by philanthropy. I don’t have extra money to pay extortion fees to information brokers. That’s a wild concept when the internet serves data effectively for free and all I want to do is help people all over the world grow organic vegetables with automation. Sci-hub gives me access to research done by other people like me who want other people to see their work. I stand on the shoulders of giants. We absolutely must dismantle the systems that keep human beings from valuable knowledge. Our future depends on it.
- thsksbd 2y agoSci hub is so good. Im a university partner and have accesses to a top research uni's library. I use Sci hun because logging into Elsevier is a massive PIA.
- barfbagginus 2y agoThe scary part is that sci hub is in danger! It's been under massive legal attack in a few jurisdictions, and as far as I know, has stopped scraping. Many, many papers from after 2021 that are not available. It has become a major issue, cutting me off at times from new results important to my work or study. It's kind of a scary time now, and I feel the strong need to regroup and find a way to get sci-hub back to scraping. We have to imagine there are millions of people being cut off from millions of people-years of research for every year this embargo continues. It's crucial that we bring it back up! Anyhow, thanks for being a sci-hub user and endorser! I love it when my peers in academia use it!
- 2y ago
- ptman 2y agoWatermarking is much preferable to DRM
- yencabulator 2y agoBeing robbed is much preferable to being killed, but neither happening would be a lot better.
- vendiddy 2y agoMaybe a quick website that allowed folks to upload a PDF and get back an anonymous PDF would be great. That way folks don't have to be proficient in command line!
- cpach 2y agoHow would one design a general tool for that…? If multiple publishers uses their own proprietary methods for embedding hidden digital watermarks in the PDF, it might not be so easy to detect them all.
- sylence 2y agoI wrote docleaner (https://github.com/TUD-CERT/docleaner https://github.com/TUD-CERT/docleaner) for a very similar purpose, specifically to remove unintended PII from PDFs that are created from e.g. Word. I don't host it anywhere publicly, though.
- Archelaos 2y agoIs this practice legal in the EU?
- adamm255 2y agoElsevier is a Dutch company, owned by a British company, so I don't think they'd have any issues.
- ur-whale 2y agoThe fact that elsevier and other similar entities ar allowed to continue to exist, i.e. that people are actually willing to give these parasites money needs to be analyzed so a root cause can be found and the dreaded thing can be put to pasture for good. Leads: - academics are ultimately lazy and do not care to fix the system - academics are so self-engrossed with their research that, like much of what they do that isn't directly pertaining to their work, the quality of what they do is horrible. - there exist a system of incentives that feed something back to people helping perpetuate such a parasitic system. Other ideas?
- matwood 2y agoIt’s a mix depending on discipline. A lot of tech for example has moved to OA journals and those work just fine for researcher advancement. But displacing Nature just isn’t going to happen in the foreseeable future when things like tenure ride on publishing there.
- robblbobbl 2y agoRIP ripped ebooks
- wanderingmind 2y agoA noob question. Do we need to so many steps or can the hash be simple removed by printing the pdf to another pdf and sharibg the printed pdf? Granted the quality might suffer and size might increase, but if that works that looks like a simple option. If that doesn't work, any way to integrate all these steps and make it possible through pandoc?
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- jbn 2y agoLeaving these here: https://github.com/kanzure/pdfparanoia https://github.com/kanzure/pdfparanoia and https://github.com/firstlookmedia/pdf-redact-tools https://github.com/firstlookmedia/pdf-redact-tools as they are relevant to this topic.
- squarefoot 2y agoA FOSS tool used for .pdf files cleaning from potential malware, which also can delete metadata, is DangerZone. Probably overkill for simple metadata cleaning, but worth mentioning nonetheless. https://dangerzone.rocks/ https://dangerzone.rocks/
- jojobas 2y agoIt can be not just metadata. There are plenty of ways to hide 128 bits of ID in a long enough pdf. White rectangles on white background, slightly non-black coloured pixels here and there, you name it. Many can even survive a print-and-scan.
- squarefoot 2y agoIn theory turning everything into an image then compressing using lossy formats aiming at very low size should mask subtle visual artifacts, but yeah, steganography could be implemented in many other ways, such as slightly different fonts (sizes, line spacing, orientation, etc). I wonder if AI, could be used to analyze and find those details by comparing potentially watermarked identical documents.
- rsanek 2y agothis tool is mentioned in the second post of the linked thread.
- squarefoot 2y agoMissed it at first. Whoops...
- sam_goody 2y agoAt some point long ago, an online music shop decided to embed the credit card details of the buyer into the MP3 metadata. If the original user kept the song to himself, "no harm done". If not, his credit details would be all over Napster and eMule. I don't remember the details now, as that was way back when the 'Net was the Wild West, but it was all the rage on IRC....
- account42 2y ago> If the original user kept the song to himself, "no harm done". Yeah no. So many ways this can go wrong without the user intentionally sharing the files.
- bayindirh 2y agoIt's good for them. Their blindness makes thair grave digging process more efficient.
- bastard_op 2y agoYet another way to weaponize their paywall on knowledge and continued existence as nothing more than a greedy middle-man. Sad that more organizations and academics refuse to buck the system and continue to support Elsever as a gatekeeper. Hopefully zlib and other public knowledge repositories will add to their upload process a stripper for this PDF metadata automatically.
- BrandoElFollito 2y agoElsevier is not to blame, this is fair business. It is like you were blaming homeopathy companies for selling overpriced sugar and chalk. The blame is on Academia who decided that they will be based on paying a company to send them knowledge, paying a company to get knowledge from them, providing them competent people for free and making their decisions dependent on all of this. This is whining about their pain when whipping themselves at the same time. This system was put in place in ancient times and changed in the meantime, the first interested somehow did not notice that. Note: I used to be in Academia and left after my PhD, among others because of these practices and medieval organizations. I loved the time I spent with brilliant people and the hours of teaching.
- FireBeyond 2y ago> It is like you were blaming homeopathy companies for selling overpriced sugar and chalk.[1] Yes, I do. Because they make claims that stay just this side of the line of active deception[2] about the benefits of the products they produce, benefits they know are non-existent. [1] Sugar and chalk might have more effect on the body than many homeopathic remedies. Most are structurally 'distilled water', according to studies with gas chromatography. [2] and have, many times, actively crossed that line into outright deception.
- BrandoElFollito 2y agore [1]: you are right - they are the holders of the active substance (which is non-existent) so their effects are going to be the only ones. There are more such substances, starting with water. re [2]: would you have some examples? For the record, I hate homeopathy with all my soul and actively fought it for years because of their unscientific claims. There were claims that were UFO-like (quantum field effects on substance energy - pick your own random words) or backed by idiots who made a science out of it (starting with Nature which published Bonaventure's idiocies and shown that they are a crappy journal like some many others). But I do not have anything handy that would show deception (as in lying)
- deleted 2y ago[deleted]
- beryilma 2y agoYou should see what ISO and ANSI do when you try to get a PDF of one of their standards. PDF with DRM, installing a DRM application on your computer that some claim cannot be uninstalled from your machine. You are also allowed to keep only a SINGLE copy of the PDF and only a single printout. It's just crazy.