4 ms·
Many instances of us building scrapers are cases where a partner has data or has tools which are only built into the UI or the UI ones are much more capable. R
by robryan 10y ago
Many instances of us building scrapers are cases where a partner has data or has tools which are only built into the UI or the UI ones are much more capable.
Rather than waiting potentially years for their IT team to make the required changes we can build a scraper in a matter of days.
- hackits 10y agoI can attest to this. From personal experience I found websites that would ignore scrappers and just allow me to access their data on their public web-site easier to deal with code wise and time wise. I make the request, you give me the data I need and then I can piss off. Web-sites that make it a cluster &&*& to get access to the data do two things. They setup a challenge to break their idiotic `are you a bot?` and secondly it is trivial in most situation just to spin up a vm, and run chrome with selenium and a python script. Granted I don't use AJAX API or anything like that. Instead I've found developer who nativly have a JSON string along side the data within the HTML to easiest to parse. Reasons why I've setup bots/scrappers 1) My local city rental market is very competitive and I hate wasting time email landlords who have all-ready signed up a lease. 2) House prices 3) Car prices 4) Stock prices 5) Banking 6) Water Rates 7) Health insurance comparison 8) Automated billing and payment systems.