And it's actually getting easier with every new shiny web API. Want to make sure only the latest Chrome can retrieve the content of your website? Why not run a

And it's actually getting easier with every new shiny web API. Want to make sure only the latest Chrome can retrieve the content of your website? Why not run a Webassembly computation that will yield the correct URL to fetch. Or what about a Web Worker? There are endless possibilities, and the only sane way to scrape / index the web in 2017 is a full-fledged browser.