4 ms·

Well, better update sci hub to download twice and then normalize any differences!

iforgotpassword 5y ago
Came here to suggest as much :-) While you can apparently just strip it from the metadata properly like suggested on twitter, maybe a "low level" approach like comparing the files on a binary level and setting any bytes that differ to 0 would be more robust. It would still work if they move the hash out of the meta data into the document itself. Only downside is that this requires the hash to be fixed size.
- jrib 5y ago
  I think this is a clever solution. I'm not seeing why it requires the hash to be a fixed size though. Why can't the algorithm just keep whatever is common between the two files? Or just take the shorter side of the diff and zero it out?
- roblabla 5y ago
  If the PDF contains encrypted blocks, this won't work, as simply zeroing out the bits will break the file. Even worse, if the PDF contains compressed chunks (which happens often enough), such a naive approach won't work either - the chunks would have to be uncompressed before comparison. Such a tool would need a lot of smarts to work reliably enough. At this point, I feel like a metadata stripper that understands the various watermarking methods may be easier to write.
  - iforgotpassword 5y ago
    Hm, makes sense. But if I were in their shoes, moving the fingerprint out of the metadata would be the first thing to do then, so no easy solution then I guess.
simiones 5y ago
> normalize any differences! Elsevier could easily make this completely non-trivial. For a very silly example, imagine how this would work if each user got a PDF using a different font. How do you automatically normalize the difference between Comic Sans and Times New Roman? Sure, it's trivial to write a tool that understands what "fonts" are and does this (especially for PDF), but you can't do it with a simple binary tool. And of course Elsevier can do something entirely more complex.
- steerablesafe 5y ago
  You just pick a font and always replace with that.
  - amelius 5y ago
    They could change "the" into "a" in different places. This is the kind of stuff done when documents that shouldn't be leaked are handed to politicians.
    - asddubs 5y ago
      Then they'd be messing with the actual written content of papers not authored by them, though
      - amelius 5y ago
        They are editors, right?
        steerablesafe 5y ago
        Editing once, with the approval of the authors is one thing. "Automatic" editing on each download is an other.
    - Isthatablackgsd 5y ago
      > They could change "the" into "a" in different places. Bad bad bad bad, again bad idea. That changes the structure and the meaning of the content. A single word replacement can change the context of the entire sentence which can change the content of the entire paper. Changing it can create unintended effects which can make Elsevier to thrash their reputation and universities will move on to different scientific/academic journals site. If Elsevier tries this method with peer-reviewed papers, then it have to go through the reviews again to ensure that the original and the revision have the equivalent expression which is difficult to do. Authors chose those words and structure to convey their expression in those papers. They chose it for a reason and Elsevier is not going to risk their reputation to change the authors papers without affecting the content of the paper.
steerablesafe 5y ago
I don't recommend normalizing differences, as in using both sources as input to produce one "cleaned" output. It could leave watermarks that happen to be the same in both sources. I recommend having a tool that works on a single source, then verify that it produces the same output from multiple sources. Also when downloading multiple times, try to do that from different public IPs and accounts.
MayeulC 5y ago
If we go that far, why not just upload the pre-print to sci-hub if it's close enough?

Came here to suggest as much :-) While you can apparently just strip it from the metadata properly like suggested on twitter, maybe a "low level" approach like comparing the files on a binary level and setting any bytes that differ to 0 would be more robust. It would still work if they move the hash out of the meta data into the document itself. Only downside is that this requires the hash to be fixed size.

jrib 5y ago
I think this is a clever solution. I'm not seeing why it requires the hash to be a fixed size though. Why can't the algorithm just keep whatever is common between the two files? Or just take the shorter side of the diff and zero it out?
roblabla 5y ago
If the PDF contains encrypted blocks, this won't work, as simply zeroing out the bits will break the file. Even worse, if the PDF contains compressed chunks (which happens often enough), such a naive approach won't work either - the chunks would have to be uncompressed before comparison. Such a tool would need a lot of smarts to work reliably enough. At this point, I feel like a metadata stripper that understands the various watermarking methods may be easier to write.
- iforgotpassword 5y ago
  Hm, makes sense. But if I were in their shoes, moving the fingerprint out of the metadata would be the first thing to do then, so no easy solution then I guess.

simiones 5y ago

> normalize any differences! Elsevier could easily make this completely non-trivial. For a very silly example, imagine how this would work if each user got a PDF using a different font. How do you automatically normalize the difference between Comic Sans and Times New Roman? Sure, it's trivial to write a tool that understands what "fonts" are and does this (especially for PDF), but you can't do it with a simple binary tool. And of course Elsevier can do something entirely more complex.

steerablesafe 5y ago
You just pick a font and always replace with that.
- amelius 5y ago
  They could change "the" into "a" in different places. This is the kind of stuff done when documents that shouldn't be leaked are handed to politicians.
  - asddubs 5y ago
    Then they'd be messing with the actual written content of papers not authored by them, though
    - amelius 5y ago
      They are editors, right?
      - steerablesafe 5y ago
        Editing once, with the approval of the authors is one thing. "Automatic" editing on each download is an other.
  - Isthatablackgsd 5y ago
    > They could change "the" into "a" in different places. Bad bad bad bad, again bad idea. That changes the structure and the meaning of the content. A single word replacement can change the context of the entire sentence which can change the content of the entire paper. Changing it can create unintended effects which can make Elsevier to thrash their reputation and universities will move on to different scientific/academic journals site. If Elsevier tries this method with peer-reviewed papers, then it have to go through the reviews again to ensure that the original and the revision have the equivalent expression which is difficult to do. Authors chose those words and structure to convey their expression in those papers. They chose it for a reason and Elsevier is not going to risk their reputation to change the authors papers without affecting the content of the paper.

steerablesafe 5y ago

I don't recommend normalizing differences, as in using both sources as input to produce one "cleaned" output. It could leave watermarks that happen to be the same in both sources. I recommend having a tool that works on a single source, then verify that it produces the same output from multiple sources. Also when downloading multiple times, try to do that from different public IPs and accounts.

MayeulC 5y ago

If we go that far, why not just upload the pre-print to sci-hub if it's close enough?