3 ms·

Author here. Let's talk web scrapers.

• sockopen 5y ago
  Copying a request as cURL, PowerShell, fetch etc… right from inside the DevTools is a blessing.
  • cushychicken 5y ago
    Isn't it though? Does create a bit of work when you have to figure out which parts of cURL need to be ported to Python, and which can be safely omitted. Copying a cURL request adds in a lot of headers - many of which I still don't properly know the purpose of. I'll get there eventually. : ) In the meantime - thank god for whoever wrote "Copy as cURL request"!
• dnnssl2 5y ago
  Is there such thing as an unscrapable site? I tried to open driver.uber.com with Pyppeteer and it fails. I’m guessing it’s due to redirects, so what have you seen solve this problem?
  • cushychicken 5y ago
    Workday is my 800 pound scraping gorilla. Never heard of Pyppeteer.