3 ms·
God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must
by oneeyedpigeon 11y ago
God, page scraping is horrible at the best of times but, as anyone who's ever viewed source on one of their pages must be thinking, scraping Google markup must be hell-on-earth.
- userbinator 11y agoIf you have an HTML parser, it's just a matter of selecting the right DOM nodes.
- oneeyedpigeon 11y agoYes, but if your markup is spaghetti, "selecting the right DOM nodes" becomes a lot more difficult.
- audiohihack 11y agoDef not that easy with Google. It is not uncommon to see 20+ serp variations on a given day if you are crawling at high volume, changing user agents, etc. The whole process is fairly terrible to parse consistently.
- Ixiaus 11y agoIt was indeed pretty rough it wouldn't surprise me if Google moves to js generated dom elements to combat rank trackers, at the time it was fine because they want to service non-js browsers but that might change. Parsing it wasn't hard but it wasn't fun...
- slig 11y agoWouldn't it be easier if they generated the dom elements via JS? That would imply that they're getting a JSON or something like, parsing it and creating the DOM.
- Ixiaus 11y agoNo because then you'd have to use a headless browser that can execute js. That increases time and cost when scraping, though it wouldn't surprise me if it ends up going that way.