There is some solutions to run a SPA in real browser even in a headless environment The trick is to emulate x11 with xvfb and control it with selenium web driv

There is some solutions to run a SPA in real browser even in a headless environment The trick is to emulate x11 with xvfb and control it with selenium web driver. Phantom isn't the only choice, just the one most people talk about As for non js heavy website, it's fairly trivial to find a library that will parse the dom for you, pretty every language have one