4 ms·
fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascrip
by toolslive 6y ago
fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago.
These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing.
Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)
- A4ET8a8uTh0 6y agoThis. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.
- iagovar 6y agoSuch as? I've never encounter anything I wasn't able to overcome.
- thinkingkong 6y agoUsually starts from simple to difficult. User agent stuff, IP address detection, aggressive rate limiting, captcha checking, browser fingerprinting, etc.
- A4ET8a8uTh0 6y agoIt can be overcome and, admittedly, I am new to this so for me that means way more time spent trying to make it work. The odd one that I got stuck on for a while was a presented list where individual record held pertinent details, but the list had, seemingly randomly, items that looked like records on the surface, but were not ( so those had to be identified and ignored ). Small things like that. Still, I would love to learn more about your approach if you would be willing to share.
- toolslive 6y agoplease click on all the traffic lights you see below.
- dewey 6y agoOnce recaptcha is in the mix it'll get tricky pretty quickly. Everything else is easy to overcome most of the time.
- amenod 6y agoRecaptcha comes to mind. That said, there are quite a few services which battle these systems for you nowadays (such as scraperapi - not affiliated, not a user). They are not always successful, but they have an advantage of maaany residential proxies (no doubt totally ethically obtained /s, but that's another story).
- ttoomm28 6y agoDatadome, Incapsula
- almost 6y agoI've found that a lot of the time that's not needed. Often you'll find the data as a JSON blob in the page and can just read it directly from there. Or find that there's an API endpoint that the javascript reads.
- monkeybutton 6y agoI find this method works best. Skip looking at the page and instead watch all the network requests as the page loads.
- WrtCdEvrydy 6y agoZAP HUD Proxy is the best option here... Load the page on it and it shows you all the request being made along with the payload as it happens. Find the one you need, copy the data, endpoint and HTTP verb and recreate it in your language of choice :D
- lemagedurage 6y agoIt's not all bad, many modern sites just expose a JSON API that can be used. It really depends on how protective and large the company behind it is.