10 ms·

Well they killed the API, what did they think would happen? It's easier to control access and rate limit with a proper API.

by zerd 3y ago

oefrha 3y ago
You think companies massively scraping right now would respect API rate limit?
- guax 3y ago
  API rate limits are more easily enforceable. If they keep scraping there are methods to detect and thwart behaviour. I don't think twitter has the appropriate talent and work environment to allow proper solutions to be implemented. It's all knee jerk reaction to whatever Elon decides.
  - oefrha 3y ago
    It's more easily enforced, except when you don't give them enough they just go back to scraping. Or create a million fake developer accounts and pool the free quota if that's possible. These are not hypotheticals, loads of companies have done both against all kinds of APIs over the years, Twitter included.
UberFly 3y ago
"control access and rate limit" Isn't that basically what they did with the API changes?
- spaceman_2020 3y ago
  But they were too stingy with the tiers and too greedy with their prices. Even for minor use cases where you need to make, say, 100 API calls a day, you’ll need to pay $100/month. Which just leads people to scrape.
  - hcks 3y ago
    Why is it twitter that is “stingy” and not the people scrapping so they don’t have to pay?
    - o1y32 3y ago
      Did you actually read that comment? I think the point is very clear -- given a reasonable price, people may would want to use the API instead of scaping the data themselves. If you instead ask for exorbitant amount of money, it only forces people to scrape, because there is no business model that would make it possible to pay.
    - spaceman_2020 3y ago
      I'm not going to pay $100 just to fetch 3000 records for a hobby project. I'll either skip the project, or I'll just abuse my scraping tool. If they'd made some more reasonable pricing tiers, I would have been happy to pay. Fetching something as simple as the total follower count from an API shouldn't be more (exorbitantly) more expensive than fetching data from, say, GPT-4. No reasonable person can make an argument for $10c/call pricing.
smetj 3y ago
Indeed, spot on.
worrycue 3y ago
I despise Musk as much as anyone else and charging for API access has hurt a lot of valuable use cases like improving accessibility but … how about not massive scraping a site that doesn’t want you to?
- bmitc 3y ago
  Does Twitter have that same approach with user data?
- watwut 3y ago
  And yet people do. Kind of predicting what various people react including scammers, bots, scrapper and what not is, like, job of a management in a company like this.
- ukFxqnLa2sBSBf6 3y ago
  Oh shit you solved the problem
- TechBro8615 3y ago
  This isn't going to make them stop either. Musk is about to see a spike in account creations using the method of lowest resistance. I expect "sign in with apple" will disappear as an option soon, given its requirement of supporting "hide my email" that makes it trivial to create multiple twitter profiles from one apple ID.
  - nrb 3y ago
    This works in his favor though, more accounts means higher ad rev and MAU for valuation.
    - masklinn 3y ago
      Higher ad rev only until the advertisers realise your users don’t buy anything and ads are wasted and your ARPU drops through the floor.
- masklinn 3y ago
AnthonyMouse 3y ago
The actual problem seems to be that a large number entities now want a full copy of the entire site. But why not just... provide it? Charge however much for a box of hard drives containing every publicly-available tweet, mailed to the address of buyer's choosing. Then the startups get their stupid tweets and you don't have any load problems on your servers.
- thatguy0900 3y ago
  What do you even charge for that? We might never make a repository of human made content with no Ai postings in it ever again. Seems like selling the golden goose to me
  - zapdrive 3y ago
    > We might never make a repository of human made content with no Ai postings in it ever again. Wow, never thought of it that way before. Kinda hit me hard for some reason.
    - thatguy0900 3y ago
      Honestly I think that's why reddit is closing itself up too. Everyone sitting on a website like this might be sitting on a Ai training goldmine that can never be replicated.
      - yuuuuuuuu 3y ago
        One that's slowly ageing away though.
      - TeMPOraL 3y ago
        Too little too late. Anything pre-ChatGPT is already scrapped, packaged and mirrored around the Internet; anything post ChatGPT launch is increasingly mixed up with LLM-generated output. And it's not that the most recent data has any extra value. You don't need most recent knowledge to train LLMs. They're not good for reproducing facts anyway. Training up their "cognitive abilities" doesn't need fresh data, it needs just human-generated data.
mrweasel 3y ago
Isn't the firehose API still available?

Well they killed the API, what did they think would happen? It's easier to control access and rate limit with a proper API.

by zerd 3y ago

Well they killed the API, what did they think would happen? It's easier to control access and rate limit with a proper API.

oefrha 3y ago

You think companies massively scraping right now would respect API rate limit?

guax 3y ago
API rate limits are more easily enforceable. If they keep scraping there are methods to detect and thwart behaviour. I don't think twitter has the appropriate talent and work environment to allow proper solutions to be implemented. It's all knee jerk reaction to whatever Elon decides.
- oefrha 3y ago
  It's more easily enforced, except when you don't give them enough they just go back to scraping. Or create a million fake developer accounts and pool the free quota if that's possible. These are not hypotheticals, loads of companies have done both against all kinds of APIs over the years, Twitter included.

UberFly 3y ago

"control access and rate limit" Isn't that basically what they did with the API changes?

spaceman_2020 3y ago
But they were too stingy with the tiers and too greedy with their prices. Even for minor use cases where you need to make, say, 100 API calls a day, you’ll need to pay $100/month. Which just leads people to scrape.
- hcks 3y ago
  Why is it twitter that is “stingy” and not the people scrapping so they don’t have to pay?
  - o1y32 3y ago
    Did you actually read that comment? I think the point is very clear -- given a reasonable price, people may would want to use the API instead of scaping the data themselves. If you instead ask for exorbitant amount of money, it only forces people to scrape, because there is no business model that would make it possible to pay.
  - spaceman_2020 3y ago
    I'm not going to pay $100 just to fetch 3000 records for a hobby project. I'll either skip the project, or I'll just abuse my scraping tool. If they'd made some more reasonable pricing tiers, I would have been happy to pay. Fetching something as simple as the total follower count from an API shouldn't be more (exorbitantly) more expensive than fetching data from, say, GPT-4. No reasonable person can make an argument for $10c/call pricing.

smetj 3y ago

Indeed, spot on.

worrycue 3y ago

I despise Musk as much as anyone else and charging for API access has hurt a lot of valuable use cases like improving accessibility but … how about not massive scraping a site that doesn’t want you to?

bmitc 3y ago
Does Twitter have that same approach with user data?
watwut 3y ago
And yet people do. Kind of predicting what various people react including scammers, bots, scrapper and what not is, like, job of a management in a company like this.
ukFxqnLa2sBSBf6 3y ago
Oh shit you solved the problem
TechBro8615 3y ago
This isn't going to make them stop either. Musk is about to see a spike in account creations using the method of lowest resistance. I expect "sign in with apple" will disappear as an option soon, given its requirement of supporting "hide my email" that makes it trivial to create multiple twitter profiles from one apple ID.
- nrb 3y ago
  This works in his favor though, more accounts means higher ad rev and MAU for valuation.
  - masklinn 3y ago
    Higher ad rev only until the advertisers realise your users don’t buy anything and ads are wasted and your ARPU drops through the floor.
masklinn 3y ago

AnthonyMouse 3y ago

The actual problem seems to be that a large number entities now want a full copy of the entire site. But why not just... provide it? Charge however much for a box of hard drives containing every publicly-available tweet, mailed to the address of buyer's choosing. Then the startups get their stupid tweets and you don't have any load problems on your servers.

thatguy0900 3y ago
What do you even charge for that? We might never make a repository of human made content with no Ai postings in it ever again. Seems like selling the golden goose to me
- zapdrive 3y ago
  > We might never make a repository of human made content with no Ai postings in it ever again. Wow, never thought of it that way before. Kinda hit me hard for some reason.
  - thatguy0900 3y ago
    Honestly I think that's why reddit is closing itself up too. Everyone sitting on a website like this might be sitting on a Ai training goldmine that can never be replicated.
    - yuuuuuuuu 3y ago
      One that's slowly ageing away though.
    - TeMPOraL 3y ago
      Too little too late. Anything pre-ChatGPT is already scrapped, packaged and mirrored around the Internet; anything post ChatGPT launch is increasingly mixed up with LLM-generated output. And it's not that the most recent data has any extra value. You don't need most recent knowledge to train LLMs. They're not good for reproducing facts anyway. Training up their "cognitive abilities" doesn't need fresh data, it needs just human-generated data.

mrweasel 3y ago

Isn't the firehose API still available?