Right off the bat: No. The reason for that is that crawling using a proper browser (i.e. Chrome) is a lot more resource-intensive than using a dedicated tool wh

Right off the bat: No. The reason for that is that crawling using a proper browser (i.e. Chrome) is a lot more resource-intensive than using a dedicated tool which only gets the top resource and maybe tries to parse some additional resources. With these kinds of tools you're limited by available bandwidth and IO speeds if you want to store things. If you're looking at a browser, you'll be limited more by things like memory consumption and CPU time, so you'd need a bigger box or more of them to drive the same amount of traffic. There is also not the same amount of ready made applications which take care of crawling, storing and maybe even indexing your data, so not something you can do without actually implementing a lot of things yourself. Of course, that is only talking about wide-scale scanning. If you're only looking to scrape a single target, for whatever reason, then having an instrumented headless browser will greatly simplify things. Headless Chrome should be more efficient than running it in a (virtual) framebuffer. Plus the whole setup for a powerful crawler is reduced to "install Chrome, start it, point $crawler at the API endpoint". My guess is that we might see turnkey crawling / automation tools appear where you supply a list of URLs and the library + Chrome does the rest. Then, browser-based large scale scanning will be within everyone's reach, only limited by their resources. Background: I created https://urlscan.io https://urlscan.io which will simply visit a website and record HTTP interactions (annotating the resources with some helpful meta data). I've been preaching the power of headless instrumented browsers for the better part of a year now ;)