You mention selenium, i use Selenium IDE Its a browser plugin that allows you to record and playback anything that you do in your browser. https://selenium.de

You mention selenium, i use Selenium IDE Its a browser plugin that allows you to record and playback anything that you do in your browser. https://selenium.dev/selenium-ide/ https://selenium.dev/selenium-ide/ There is a right-click action to record hovering over a section if it is hidden by css, its a well-known gotcha in wordpress admin listings and other sites that love to hide until you hover over a section. I've still to look into whether the file format that is saved from selenium ide could be used within a virtual browser like phantomjs or the like. For direct scraping, i tend to go straight to xpath with dom. Its far more reliable than trying to get stuff out via regular expressions, that i only use when i need to get a certain part of the node value from a text node. I used to use php for that, but i wrote my own programming language that uses a library called htmlquery (its a go library) https://github.com/antchfx/htmlquery https://github.com/antchfx/htmlquery If i have to login, i fake each request with the correct headers. Plenty of libraries around that help with that for every language.