Selenium can absolutely be used for this, the caveat being that it is much slower than using a regular HTML parser. In my experience, it's best to milk as much

Selenium can absolutely be used for this, the caveat being that it is much slower than using a regular HTML parser. In my experience, it's best to milk as much as you possibly can out of a site's plain HTML and APIs, only resorting to Selenium where it's absolutely necessary.