So what I'm suggesting is slightly different. This outputs a series of clicks and keyboard events. It doesn't handle the logic necessary for most JS-heavy web-

So what I'm suggesting is slightly different. This outputs a series of clicks and keyboard events. It doesn't handle the logic necessary for most JS-heavy web-apps. Scraping an infinite scroll is a good example. A naive algorithm is - scroll, wait, scrape, scroll, wait, scrape... We have a slightly more sophisticated algorithm to deal with this. Also, what happens when you encounter a page with multiple iframes? I'm essentially talking about a library which works with higher-level abstractions than just click and type, perhaps 'ScrapeInfiniteScroll' - and similar operations.