5 ms·

> HTTP is an API of sorts… True, but it is an API that they can't easily deprecate on a whim.

• viraptor 2y ago
  You're one CAPTCHA added to the flow from this access being effectively deprecated.
  • xp84 2y ago
    In the AI era, is this as true?
    • viraptor 2y ago
      Depends on your development and per-action cost. And on the possible latency. It also changes your whole stack from "send a request" to "emulate each step in a browser while taking screenshots at (hopefully) the right event/delay" - that's a huge difference.
• reddalo 2y ago
  But they can quickly change the "structure" of that API.
  • chillfox 2y ago
    If you build your scraper to find data on the page based on the shape of the data itself instead of the structure of the page then it will be resilient to most changes that don't materially change what data is displayed on the page. So, prefer regex over css selectors, and css selectors over xpath, where possible. And don't select based on nesting or position if possible.
  • vladstudio 2y ago
    I believe this is where smarter tools like Kadoa [0] can be of help. It detects data structure changes for existing workflows, and adapts to them. [0] https://www.kadoa.com/ https://www.kadoa.com/
  • account42 2y ago
    A small problem compared to not having an API key and being stonewalled as to why.
  • michaelcampbell 2y ago
    I worked for a very early billpay company where you could pay your bills online to vendors, even if the vendor didn't support it. We used API's where we could, but where we couldn't... We had a whole team dedicated to keeping up the changes vendors would make to their websites that we scraped for info. The team was called, of course, "Scrape and P(r)ay".
  • xeromal 2y ago
    Isn't that similar to deleting your API key except you can at least fix the structure of the selenium one.