4 ms·
+1. Thanks for the hints about Selenium. My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like N
by joering2 14y ago
+1. Thanks for the hints about Selenium.
My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with all up to date variables, because engineers will be concerned of removing that code not to break some functionality .
* given you have rights to scrap.
not that there are no tools to debug the site; but I found websites like mentioned plus youtube, and bunch others just not fiddling too much with js.
- codewright 14y ago> even your smartest regex can have a brain fart If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.
- bdcravens 14y agoI've found that regex is very brittle when you don't control what comes across. DOM traversal is far more reliable.
- deleted 14y ago[deleted]
- derrida 14y agoAgree... basically search methods that specify a branch or leaf locally rather than the entire tree structure can more often resist layout changes. Regex for HTML is a bad idea ... http://stackoverflow.com/questions/590747/using-regular-expressions-to-parse-html-why-not#590789 http://stackoverflow.com/questions/590747/using-regular-expr...
- kybernetyk 14y agoThis. You usually traverse the DOM. Either you use some XQuery /XPath magic or a library like beautiful soup.
- codewright 14y agoSizzle for life.