6 ms·

I want to share a story in a somewhat related topic: anti web-scraping techniques The most devious version I ever seen of this, I was baffled, astonished and

I want to share a story in a somewhat related topic: anti web-scraping techniques The most devious version I ever seen of this, I was baffled, astonished and completely helpless: This website I was trying to scrap generated a new font (as in a .woff file) on every request, the font had the position of the letters randomly moved around (for example, the 'J' would be in place of the 'F' character in the .woff and so on) and the text produced by the website would be encoded to match that specific font. So every time you loaded the website you got a completely different font with a completely different text, but for the user the text would look fine because the font mapped it to the original characters. If you tried to copy-and-paste the text from the website you would get some random garbled text. The only way I could think of to scrap that would have been to OCR the .woff font files, but OCR could easily prevent mass-scraping due to sheer processing costs.

DaiPlusPlus 2y ago
> easily prevent mass-scraping due to sheer processing costs. my 2018 iPad Pro does OCR on images in Safari instantly. People only think OCR is slow because Adobe Acrobat still uses the same single-threaded OCR algo it’s had for decades now; then consider how blazing a GPU-based impl would be…
- DanielHB 2y ago
  I dunno, I never measured it. If you are scraping billions of small social media posts I would expect it to add up and make it unviable.
- jakjak123 2y ago
  It pre processes your photo library while charging
  - ChadNauseam 2y ago
    The GP mentioned it working for pictures viewed in safari
    - hhh 2y ago
      it works for any photo anywhere in the OS, same for macOS
    - jakjak123 2y ago
      Yeah, i was thinking more about why it looks like it works so fast when you browse your photo library
teraflop 2y ago
That seems like it ought to be straightforward to defeat without OCR. If you know that a particular glyph looks like the letter J, then you just need to parse the WOFF file, find that glyph's data, and find the character that maps to it. It's definitely annoying enough to deter a casual scraper, but there's nothing conceptually difficult about it. You do need to determine the "correct" character code for each glyph, but there are lots of ways to do that, on a spectrum from manual to automated. And you only need to do it once.
sebstefan 2y ago
If it's just swapping letters then rather than trying to dive into the WOFF you could just get the garbled data and treat it as a cesar cypher, I guess. A few dozen rotations and you're through It's kind of annoying and prone to break but I'd rather have that than whatever Facebook is doing where every class name, ID & identifiable tags in the markup gets randomly generated every once in a while
- wbl 2y ago
  Could be an arbitrary permutation or worse have multiple equivalent characters. Fonts can do a lot.
- Apofis 2y ago
  That likely wouldn't work, doesn't mean the letters were simply rotated. Would probably be just random.
flerchin 2y ago
LOL the replies are hilarious. You've sniped several nerds today. Neat story.
- DanielHB 2y ago
  I know right? I just scraped another website instead. I am actually surprised no one went: "actually that technique is called 'chicken ostrich sandwich' and was first employed in babylon in 2000BC"
  - viciousvoxel 2y ago
    Actually that technique is called a "Caesar cipher" and it has been employed since at least the 1st c. BCE.
wildpeaks 2y ago
A downside is it makes the site unusable for screen readers and SEO, plus it adds backend costs (compared to a plain backend that serves static files) if it's generated dynamically, although one can pre-generate a bunch of variants and randomly pick one at runtime (which could be handled by the load balancer) to minimize the costs.
- ksp-atlas 2y ago
  Yeah, my immediate thought was this would be bad for screen readers, plus OCR could easily defeat this
voldacar 2y ago
So it's a Caesar cipher, which is trivial to break. You don't need OCR or any computationally intensive solution.
- NoMoreNicksLeft 2y ago
  You need OCR unless you're going to personally sit there and break it by hand so you can feed the tr/// translation yourself every time you need to scrape. And it's a bit more tedious than the puzzles we did as kids, likely the punctuation and lowercase/uppercase were mixed into the slop.
  - connicpu 2y ago
    If there's a part that doesn't change, eg a footer or something, you can get a head start and have it figure out the rest by deduction with a spellchecker
    - NoMoreNicksLeft 2y ago
      You might manage to cobble together frequency analysis too, but that would be challenging. If the ciphertext is very small, or is marketspeak without any sense to its message, then that's going to fall flat. And all this assumes just ascii rather than say a (even limited) unicode font. These assholes could be doing that just to have curly quotes or whatever.
deleted 2y ago
[deleted]

DaiPlusPlus 2y ago

> easily prevent mass-scraping due to sheer processing costs. my 2018 iPad Pro does OCR on images in Safari instantly. People only think OCR is slow because Adobe Acrobat still uses the same single-threaded OCR algo it’s had for decades now; then consider how blazing a GPU-based impl would be…

DanielHB 2y ago
I dunno, I never measured it. If you are scraping billions of small social media posts I would expect it to add up and make it unviable.
jakjak123 2y ago
It pre processes your photo library while charging
- ChadNauseam 2y ago
  The GP mentioned it working for pictures viewed in safari
  - hhh 2y ago
    it works for any photo anywhere in the OS, same for macOS
  - jakjak123 2y ago
    Yeah, i was thinking more about why it looks like it works so fast when you browse your photo library

teraflop 2y ago

That seems like it ought to be straightforward to defeat without OCR. If you know that a particular glyph looks like the letter J, then you just need to parse the WOFF file, find that glyph's data, and find the character that maps to it. It's definitely annoying enough to deter a casual scraper, but there's nothing conceptually difficult about it. You do need to determine the "correct" character code for each glyph, but there are lots of ways to do that, on a spectrum from manual to automated. And you only need to do it once.

sebstefan 2y ago

If it's just swapping letters then rather than trying to dive into the WOFF you could just get the garbled data and treat it as a cesar cypher, I guess. A few dozen rotations and you're through It's kind of annoying and prone to break but I'd rather have that than whatever Facebook is doing where every class name, ID & identifiable tags in the markup gets randomly generated every once in a while

wbl 2y ago
Could be an arbitrary permutation or worse have multiple equivalent characters. Fonts can do a lot.
Apofis 2y ago
That likely wouldn't work, doesn't mean the letters were simply rotated. Would probably be just random.

flerchin 2y ago

LOL the replies are hilarious. You've sniped several nerds today. Neat story.

DanielHB 2y ago
I know right? I just scraped another website instead. I am actually surprised no one went: "actually that technique is called 'chicken ostrich sandwich' and was first employed in babylon in 2000BC"
- viciousvoxel 2y ago
  Actually that technique is called a "Caesar cipher" and it has been employed since at least the 1st c. BCE.

wildpeaks 2y ago

A downside is it makes the site unusable for screen readers and SEO, plus it adds backend costs (compared to a plain backend that serves static files) if it's generated dynamically, although one can pre-generate a bunch of variants and randomly pick one at runtime (which could be handled by the load balancer) to minimize the costs.

ksp-atlas 2y ago
Yeah, my immediate thought was this would be bad for screen readers, plus OCR could easily defeat this

voldacar 2y ago

So it's a Caesar cipher, which is trivial to break. You don't need OCR or any computationally intensive solution.

NoMoreNicksLeft 2y ago
You need OCR unless you're going to personally sit there and break it by hand so you can feed the tr/// translation yourself every time you need to scrape. And it's a bit more tedious than the puzzles we did as kids, likely the punctuation and lowercase/uppercase were mixed into the slop.
- connicpu 2y ago
  If there's a part that doesn't change, eg a footer or something, you can get a head start and have it figure out the rest by deduction with a spellchecker
  - NoMoreNicksLeft 2y ago
    You might manage to cobble together frequency analysis too, but that would be challenging. If the ciphertext is very small, or is marketspeak without any sense to its message, then that's going to fall flat. And all this assumes just ascii rather than say a (even limited) unicode font. These assholes could be doing that just to have curly quotes or whatever.

deleted 2y ago

[deleted]