3 ms·
I know it's a solved problem now, but: > One that couldn't easily be changed, either, because everyone's clients relied on the exact structure of the markup fo
by derefr 10y ago
I know it's a solved problem now, but:
> One that couldn't easily be changed, either, because everyone's clients relied on the exact structure of the markup for screen scraping.
This doesn't seem like that hard a problem. Presumably the robots all sent their own distinct UA strings (as scrapers only get written to imitate browser UA strings if you ban non-browser UA strings.) So why not just filter your logs to find all the known scrapers, serve anything with those exact UA strings "legacy" table-based responses, and serve everyone else the new layout?
- chasingtheflow 10y agoThen you have to maintain two layouts in perpetuity.