I've built so many bespoke crawlers for this exact purpose over the years and I always thought it would be excellent to have a platform purpose built for this -

I've built so many bespoke crawlers for this exact purpose over the years and I always thought it would be excellent to have a platform purpose built for this - even if it just assisted simple things like authentication/login and object identification in api's/feeds/portals. I would be really curious to see what kind of tooling you guys use internally to build these crawlers. There has to be a better option than headless webkit and xpath querying.