5 ms·
More notes on writing web scrapers
- cushychicken 5y agoAuthor here. Let's talk web scrapers.
- sockopen 5y agoCopying a request as cURL, PowerShell, fetch etc… right from inside the DevTools is a blessing.
- cushychicken 5y agoIsn't it though? Does create a bit of work when you have to figure out which parts of cURL need to be ported to Python, and which can be safely omitted. Copying a cURL request adds in a lot of headers - many of which I still don't properly know the purpose of. I'll get there eventually. : ) In the meantime - thank god for whoever wrote "Copy as cURL request"!
- dnnssl2 5y agoIs there such thing as an unscrapable site? I tried to open driver.uber.com with Pyppeteer and it fails. I’m guessing it’s due to redirects, so what have you seen solve this problem?
- cushychicken 5y agoWorkday is my 800 pound scraping gorilla. Never heard of Pyppeteer.
- marginalia_nu 5y agoI'd note, regarding storing data, that HTML compresses extremely well. It will shrink to <10% of its size. If you want to save it, you would be a fool if you saved it uncompressed.
- cushychicken 5y agoGuess I'm a bit of a fool then because I was definitely saving uncompressed haha. Maybe I'll give it a try at some point, but right now, I don't have any need for it. It'd be work without a clear payoff. I'm starting a new job at the end of March, and I'd rather concentrate on some features that help me run the site between now and then.