3 ms·
If they're willing to flip off search engines and do all page rendering in JS, there's no limit to what they could do to make a scraper's life hell. It'd be re
by ConstantineXVI 14y ago
If they're willing to flip off search engines and do all page rendering in JS, there's no limit to what they could do to make a scraper's life hell.
It'd be relatively simple to generate non-sensible fonts (where code points don't map as expected) where the browser still renders legible tweets but scrapers get unintelligible gibberish. The special font would essentially be a one-time pad, generate new font mappings every few requests and your HTML is essentially uncrackable to scrapers that don't have the horsepower to OCR every tweet (at which point you may as well pay for Gnip anyway).
- chimeracoder 14y agoTrue, but it would be much simpler to hack the Twitter native applications to do the dirty work for you, depending on your purposes. That, or just use the oauth_secret from the official Twitter applications; those were cracked ages ago, and even if they changed them (thereby breaking all their old applications), they're super-easy to crack again.
- ConstantineXVI 14y agoHardening the webpage would still decimate the amount of people with the skill/motivation to pry out tweets. HTML scraping is much more accessible than poking around native, source-less apps. (And if it was still a problem, the same tactics as above would work equally well in native apps.)
- zachalexander 14y ago>If they're willing to flip off search engines and do all page rendering in JS Isn't that a pretty big if?