11 ms·
Using regex in python to scrape some data from a website works just fine for me... shrugs
by ythn 9y ago
Using regex in python to scrape some data from a website works just fine for me... shrugs
- vermooten 9y agoyep me too, no problem.
- monk_e_boy 9y agobeautiful soup is a better tool. Regex is fine for grabbing something in a page that you have looked at yourself. Parsing millions of pages you don't have this option, you need something robust, a tool that is flexible, that doesn't barf out too many errors, that is quick.
- mackerson 9y agoBS4, requests and dataset are my go-to tools for simple scraping. Back when Bethesda pulled the paid mod nonsense on the Steam store I wrote a simple scraper to track and digest the changes people were making to their Skyrim reviews so I could tease out who was most offended, core fans with a lot of hours logged on the game or people who were just piling on the outrage bandwagon. I should dig back into that data some time. Anyway, got the whole thing done in about 200 lines. Not bad to scrape 50,000 pages. Fun times.
- Analemma_ 9y agoIt’s an acceptable choice for scraping, because with scraping you usually go into it with the expectation that it will break and need updates sometimes as the scraped site changes. The warning bells are more for things like validating user input on your own site, where there might be security implications of an impossible parsing task.
- DonHopkins 9y agoAnd I use a handgun to hunt houseflies in my motel room after a week long crystal meth binge. Works just fine for me.
- scarface74 9y agoIs there ever a good reason to use regex to do a web scraper instead of using a proper prebuilt parser and walking the DOM?
- bryanrasmussen 9y agoregex is your hammer and you just by chance happen to have a really nail shaped task to do?
- mercer 9y agoPerformance, perhaps?