4 ms·
As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream cha
by sph 4y ago
As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control.
Scrapers are very simple, effective and probably one of the least fun things to build.
- throwaway-blaze 4y agoScrapers are fun to build, just not fun to maintain :)
- mouse-linguist 4y agoDo you have recommendations for platforms that monitor scraping fleets? I occasionally code scrapers for quick data aggregation, but have trouble running anything long-term because it can be a chore to monitor. I've been looking into various options for self-hosting some sort of monitor/dashboard that can send alerts but haven't found anything satisfying yet.
- breno 4y agoWe released estela for this and other purposes, check it out, maybe it will suit your needs: https://github.com/bitmakerla/estela https://github.com/bitmakerla/estela Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. It would be great to have feedback to consider it when prioritizing some over others :-)
- real-dino 4y agoI'd create a simple health check based on the integrity of the data your retrieve. Think about giving it a score based on how the data is shaped. If it's missing prices for example, then it immediately goes down to zero, doesn't update the database and sends an alert.
- hbrn 4y agoI would recommend against building a score and always stick to booleans whenever possible. Having a numeric score means you're trying to extract nice-to-have data points, and in my experience this always leads messy codebases and false assumptions. If you allow "slightly unhealthy" scrapers in production, almost inevitably the state of your scrapers will always be "slightly unhealthy". Save yourself the trouble and always treat it as either "it works" or "it doesn't work", no in between. Your first iterations will probably break every day, but eventually you'll get to a happy place.
- deleted 4y ago[deleted]
- adriancooney 4y agoApologies for the shameless self-promotion here but it was this very problem that I built puppeteer-heap-snapshot. It decouples the HTML from the scraper and instead we inspect the booted app’s memory. Not near as performant but a lot more reliable. I wrote about it here: https://www.adriancooney.ie/blog/web-scraping-via-javascript-heap-snapshots https://www.adriancooney.ie/blog/web-scraping-via-javascript...
- jamescampbell 4y agoHard agree here. Stopped taking scraping projects because the sites always change. No matter how general you design it.