6 ms·
Ask HN: Web scraping in production?
Are any of you maintaining any web scrapers in production?
I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.
Curious what applications people have in industry and what sorts of techniques people use for reliability.
- sargstuff 1y agoexcel web scraping[0] (vs. using python[1] and/or odbc/delimited files) A few 2025 use cases [2],[3]: Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc ---- creative take on topic: modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384 Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html 3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html ---- [0] : excel scraping : https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk [1] : python scraping : https://www.youtube.com/watch?v=Oo8-nEuDBkk https://www.youtube.com/watch?v=Oo8-nEuDBkk [2] : https://dataforest.ai/blog/top-web-scraping-use-cases https://dataforest.ai/blog/top-web-scraping-use-cases [3] : https://www.parsehub.com/blog/web-scraping-examples/ https://www.parsehub.com/blog/web-scraping-examples/
- arkmm 1y agoNeat - didn't realize there were affordances for scraping in Excel (but in hindsight I shouldn't be surprised). I didn't follow the connection between modern looming and scraping though?
- sargstuff 1y agohint: silk spider webs & fabric threads. Guess 3d printing should have been clarified as linear, fused deposition. Melted plastic line gets scraped along plate/material. The 3d printed web reference, in this instance, being the in-fill pattern. [0] Robotic metal pinching / incremental sheet forming might be bit more clearer example. [1] ----- [0] : in-fill pattern : https://jlc3dp.com/blog/choosing-the-right-infill-structure-for-3d-printing https://jlc3dp.com/blog/choosing-the-right-infill-structure-... [1] : https://www.youtube.com/watch?v=Jc16Ob-yoDs https://www.youtube.com/watch?v=Jc16Ob-yoDs
- 9d 1y agoScraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years.
- sargstuff 1y agoIMHO, "untyped" format/delimited file yes. Directly placing/'compiling' in appropriate topological construct/environment works wonders. aka environment of database, spreadsheet, "reports" with information beyond raw data, etc