4 ms·

I have a bridge to sell anyone who believes this is true. AI companies have been a boon to businesses that want to lock down user data and now have an excuse. I

I have a bridge to sell anyone who believes this is true. AI companies have been a boon to businesses that want to lock down user data and now have an excuse. It may be true to the extent that Musk is legitimately angry that Twitter isn't getting a piece of that AI VC money (I'm sure he is). But: A) Twitter would probably move in this direction even if AI companies didn't exist, this is an excuse. Nothing about Musk's Twitter has indicated that he cares about Open data access or anonymous access to the site, and this follows a general trend of closing down the platform to non-monetizable users. Musk has abundantly shown in the past that he would prefer everyone browsing Twitter be logged into an account. B) "it's temporary" -- how? You don't have a way to stop this other than forcing login. That situation is not going to change next week. To call this "temporary emergency measures" is so funny; there is no engineering solution for this and you're not going to be able to successfully sue companies for scraping Twitter. Put a captcha in front of it? Sure, let me know how that goes. You going to wait and see if the AI market collapses in the next month? If this does turn out to be temporary, it'll only be because of migrations off of Twitter and because of user criticism, because Musk is impulsive and bends easily under pressure. But nothing about the situation Musk is complaining about is going to change next week.

m00x 3y ago
FWIW: I've been scraping the shit out of social media for my AI training. I also do Amazon, AliExpress, etc. Libs like puppeteer is so good these days that it's impossible to tell real users from fake traffic. Most of the blocks are just IP blocks.
- qingcharles 3y ago
  Right. And the IP blocks only add a small cost to the scraping because it forces people to use residential IPs which can't sanely be blocked.
agentgumshoe 3y ago
Not to mention Elon is great mates with the VC douchebags that are busy hyping and profiting off the AI hype train.
versteegen 3y ago
> Nothing about Musk's Twitter has indicated that he cares about Open data access or anonymous access to the site Not so, he tasked George Hotz with getting rid of that horrible popup which prevented you from scrolling down much if you weren't logged in, which was added soon before he bought Twitter. When that was removed I rejoiced. But now Twitter's gone 100x in the opposite direction.
- thissitesucks__ 3y ago
  [flagged]
  - postalrat 3y ago
    Let me guess. He got bored and moved on before making any breakthrough.
    - thissitesucks__ 3y ago
      [dead]
- danShumway 3y ago
  I don't know; was that Musk's idea, or was that Hotz's idea? I vaguely think this was a change that Hotz wanted that Musk went along with. To be fair, Musk will regularly pay lip-service to the idea of Open communication. I guess that's not literally nothing, but most large site policies have been in the direction of locking down content. If there ever was a version of Musk that cared about Open access, it's been a while since that version of him saw the light of day. It's very consistent with his overall behavior to believe that he views Twitter content as being primarily his property rather than a community resource, and that he thinks that scrapers/AI companies/researchers are literally stealing from him if they derive any value at all from data that Twitter hosts.
utbabya 3y ago
Pretty sure they'd get into troubles with blue check mark buyers implicit promises if it's not temporary.
random_cynic 3y ago
> there is no engineering solution for this and you're not going to be able to successfully sue companies for scraping Twitter There absolutely is, if you try instead of whining on internet. People at Vercel have already developed new anti-bot + fingerprinting + rate limiting techniques which look quite promising. I dare say within a year, new tools will be powerful enough to do this easily.
- danShumway 3y ago
  > I dare say within a year, new tools will be powerful enough to do this easily. I see where you're coming from, but if Twitter is in a position where it can't roll out those protections right now, given its current head counts, etc... it's not going to be in a position where it can roll out those protections next week. Probably not next month. So it's less that no one could block companies from scraping Twitter (although anti-scraping mechanisms are probably always going to be a cat-and-mouse game, so I'm not sure that there is ever going to be a perfect easy solution). It's more that if Twitter can't do it right now, nothing is going to magically change any time soon about the situation it has found itself in. And waiting a year (even waiting 6 months) for tools to become available before rolling back this rate limiting would be incredibly self-destructive for Twitter. The way I see it, they're basically guaranteeing that they will need to roll back these changes before they have a solution to whatever specific problem or irritation Musk is fixated on. They're not going to gain additional engineering capabilities in the next week. And how long does Musk plan to leave rate-limiting in place? A social media site where people can't look at content is just broken.
cassepipe 3y ago
It's been a week now, you prediction held. Still can't access without an account.

m00x 3y ago

FWIW: I've been scraping the shit out of social media for my AI training. I also do Amazon, AliExpress, etc. Libs like puppeteer is so good these days that it's impossible to tell real users from fake traffic. Most of the blocks are just IP blocks.

qingcharles 3y ago
Right. And the IP blocks only add a small cost to the scraping because it forces people to use residential IPs which can't sanely be blocked.

agentgumshoe 3y ago

Not to mention Elon is great mates with the VC douchebags that are busy hyping and profiting off the AI hype train.

versteegen 3y ago

> Nothing about Musk's Twitter has indicated that he cares about Open data access or anonymous access to the site Not so, he tasked George Hotz with getting rid of that horrible popup which prevented you from scrolling down much if you weren't logged in, which was added soon before he bought Twitter. When that was removed I rejoiced. But now Twitter's gone 100x in the opposite direction.

thissitesucks__ 3y ago
[flagged]
- postalrat 3y ago
  Let me guess. He got bored and moved on before making any breakthrough.
  - thissitesucks__ 3y ago
    [dead]
danShumway 3y ago
I don't know; was that Musk's idea, or was that Hotz's idea? I vaguely think this was a change that Hotz wanted that Musk went along with. To be fair, Musk will regularly pay lip-service to the idea of Open communication. I guess that's not literally nothing, but most large site policies have been in the direction of locking down content. If there ever was a version of Musk that cared about Open access, it's been a while since that version of him saw the light of day. It's very consistent with his overall behavior to believe that he views Twitter content as being primarily his property rather than a community resource, and that he thinks that scrapers/AI companies/researchers are literally stealing from him if they derive any value at all from data that Twitter hosts.

utbabya 3y ago

Pretty sure they'd get into troubles with blue check mark buyers implicit promises if it's not temporary.

random_cynic 3y ago

> there is no engineering solution for this and you're not going to be able to successfully sue companies for scraping Twitter There absolutely is, if you try instead of whining on internet. People at Vercel have already developed new anti-bot + fingerprinting + rate limiting techniques which look quite promising. I dare say within a year, new tools will be powerful enough to do this easily.

danShumway 3y ago
> I dare say within a year, new tools will be powerful enough to do this easily. I see where you're coming from, but if Twitter is in a position where it can't roll out those protections right now, given its current head counts, etc... it's not going to be in a position where it can roll out those protections next week. Probably not next month. So it's less that no one could block companies from scraping Twitter (although anti-scraping mechanisms are probably always going to be a cat-and-mouse game, so I'm not sure that there is ever going to be a perfect easy solution). It's more that if Twitter can't do it right now, nothing is going to magically change any time soon about the situation it has found itself in. And waiting a year (even waiting 6 months) for tools to become available before rolling back this rate limiting would be incredibly self-destructive for Twitter. The way I see it, they're basically guaranteeing that they will need to roll back these changes before they have a solution to whatever specific problem or irritation Musk is fixated on. They're not going to gain additional engineering capabilities in the next week. And how long does Musk plan to leave rate-limiting in place? A social media site where people can't look at content is just broken.

cassepipe 3y ago

It's been a week now, you prediction held. Still can't access without an account.