4 ms·

I've used a fair share of (painful) APIs over the years and I have one simple plea: can we please stop being a-holes? Can we stop scraping websites that have AP

by drej 7y ago

I've used a fair share of (painful) APIs over the years and I have one simple plea: can we please stop being a-holes? Can we stop scraping websites that have APIs? They are already offering machine readable data and maybe they have a reason for not providing everything. Scraping their sites circumvents this API and not only abuses their systems, but also makes your code super brittle - any change to their site breaks your code - if you code wasn't broken already as you were banned by the provider.

simonw 7y ago
If people are resorting to scraping then clearly the official API isn't fit for their purposes. As an example, here are two key features that are missing from the Twitter API at the moment: - Bookmarks. You can privately bookmark tweets on the Twitter website and apps. There is no way to access the list of tweets you have bookmarked in the API. - Threads. The concept of threads - where a tweet has replies from the same author get special treatment in terms of display - is key to how Twitter is used today. The official API doesn't support them, in that there is no way to look at a tweet and see that there exists a threaded tweet reply. There is no good commercial reason for excluding either of these features from the public API, other than that Twitter have made a strategic decision not to invest resources in expanding the API to keep up with new features they are adding to the platform. Given that, is it any surprise that people are resorting to scraping?
- simonw 7y ago
  You say "maybe they have a reason for not providing everything" - I cannot think of a reason not to provide me with API access to my own private bookmarks other than "we decided to invest our engineering resources elsewhere". Which isn't a bad reason! But it's not a good argument for people not to scrape their own data.
  - numpad0 7y ago
    Everything Twitter used to and not anymore enable third party profit off “their valuable” data. It’s economically/strategically unsustainable or so they think.
  - dewey 7y ago
    > But it's not a good argument for people not to scrape their own data. Why would you want to scrape your own data if you an already request all your data and get a whole archive? https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive https://help.twitter.com/en/managing-your-account/how-to-dow...
    - simonw 7y ago
      Because then you have to trigger and download a GB+ file every time you want to programmatically access your latest bookmarks.
- scared2 7y ago
  Or they don't know how to effectively use the API, maybe.
  - simonw 7y ago
    I doubt it. People who write scrapers are talented engineers. Talented engineers understand the limitations of scraping compared to using an official API, and know to try to get things done with the official API first. The Twint README calls out reasons for going beyond the API at the start - things like Twitter's increasingly strict rate limits and the limit of only 3,200 historic tweets for a user account.
    - paulgb 7y ago
      I've been maintaining a browser extension[1] that has used various versions of the unofficial API over a period of six years -- from literally navigating the site as the user and grabbing HTML, to the various incarnations of JSON and HTML hybrid (SSR) APIs that their web and mobile clients have used internally over that time. Believe me, I would LOVE if the official API supported threads. I have tried several times to make it work, but the official APIs are stuck in a circa-2012 idea of how Twitter works. Replies just aren't a thing to it. [1] https://github.com/paulgb/Treeverse https://github.com/paulgb/Treeverse
    - armitron 7y ago
      People who write robust, performant, maintainable systems that don’t collapse under their own weight are talented engineers. Most scrapers including this one lack these characteristics.
      - scared2 6y ago
        I totally agree, writing scrapers doesn't necessarily make one a talented engineer. You may often find it easier to write a scrapers than going through the API documentation or series of authentication procedures.
    - scared2 6y ago
- fireattack 7y ago
  API can only get first 3200 tweets from a user is also a dealbreaker.
deleted 7y ago
[deleted]
throwaway4585 7y ago
>maybe they have a reason for not providing everything Not my problem. You make your http resources accessible, I'll download them however I want.
anigbrowl 7y ago
I think it's the social media companies that are being assholes by privatizing conversations and selling access to the graph. The idea of a platform is to set up a protocol and some hardware, get people to talk to each other on it, and then gradually construct a mall around the outer edges of the platform. It's a shitty business model because it relies on knowing more about he users than they know themselves and then selling that information to advertisers.

by drej 7y ago

simonw 7y ago

If people are resorting to scraping then clearly the official API isn't fit for their purposes. As an example, here are two key features that are missing from the Twitter API at the moment: - Bookmarks. You can privately bookmark tweets on the Twitter website and apps. There is no way to access the list of tweets you have bookmarked in the API. - Threads. The concept of threads - where a tweet has replies from the same author get special treatment in terms of display - is key to how Twitter is used today. The official API doesn't support them, in that there is no way to look at a tweet and see that there exists a threaded tweet reply. There is no good commercial reason for excluding either of these features from the public API, other than that Twitter have made a strategic decision not to invest resources in expanding the API to keep up with new features they are adding to the platform. Given that, is it any surprise that people are resorting to scraping?

simonw 7y ago
You say "maybe they have a reason for not providing everything" - I cannot think of a reason not to provide me with API access to my own private bookmarks other than "we decided to invest our engineering resources elsewhere". Which isn't a bad reason! But it's not a good argument for people not to scrape their own data.
- numpad0 7y ago
  Everything Twitter used to and not anymore enable third party profit off “their valuable” data. It’s economically/strategically unsustainable or so they think.
- dewey 7y ago
  > But it's not a good argument for people not to scrape their own data. Why would you want to scrape your own data if you an already request all your data and get a whole archive? https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive https://help.twitter.com/en/managing-your-account/how-to-dow...
  - simonw 7y ago
    Because then you have to trigger and download a GB+ file every time you want to programmatically access your latest bookmarks.
scared2 7y ago
Or they don't know how to effectively use the API, maybe.
- simonw 7y ago
  I doubt it. People who write scrapers are talented engineers. Talented engineers understand the limitations of scraping compared to using an official API, and know to try to get things done with the official API first. The Twint README calls out reasons for going beyond the API at the start - things like Twitter's increasingly strict rate limits and the limit of only 3,200 historic tweets for a user account.
  - paulgb 7y ago
    I've been maintaining a browser extension[1] that has used various versions of the unofficial API over a period of six years -- from literally navigating the site as the user and grabbing HTML, to the various incarnations of JSON and HTML hybrid (SSR) APIs that their web and mobile clients have used internally over that time. Believe me, I would LOVE if the official API supported threads. I have tried several times to make it work, but the official APIs are stuck in a circa-2012 idea of how Twitter works. Replies just aren't a thing to it. [1] https://github.com/paulgb/Treeverse https://github.com/paulgb/Treeverse
  - armitron 7y ago
    People who write robust, performant, maintainable systems that don’t collapse under their own weight are talented engineers. Most scrapers including this one lack these characteristics.
    - scared2 6y ago
      I totally agree, writing scrapers doesn't necessarily make one a talented engineer. You may often find it easier to write a scrapers than going through the API documentation or series of authentication procedures.
  - scared2 6y ago
fireattack 7y ago
API can only get first 3200 tweets from a user is also a dealbreaker.

deleted 7y ago

[deleted]

throwaway4585 7y ago

>maybe they have a reason for not providing everything Not my problem. You make your http resources accessible, I'll download them however I want.

anigbrowl 7y ago

I think it's the social media companies that are being assholes by privatizing conversations and selling access to the graph. The idea of a platform is to set up a protocol and some hardware, get people to talk to each other on it, and then gradually construct a mall around the outer edges of the platform. It's a shitty business model because it relies on knowing more about he users than they know themselves and then selling that information to advertisers.