3 ms·

Why would an AI company start scraping twitter html, instead of using an already existing archive? Something similar to archive.org could earn money from that.

Why would an AI company start scraping twitter html, instead of using an already existing archive? Something similar to archive.org could earn money from that. If all you want is the content, there's no reason to suck it through a straw. I'd expect those that require real time data, such as stock market bots or sentiment data providers, to scrape twitter (if they don't provide the data by other means, for example the "firehose", which is another great way to earn money). None of this makes much sense. Also, it's much more complicated than it seems. The web works because the data is public. You cannot think of it as "my data". (Especially not twitter, since it is really their users'!) Twitter is not higher quality data than any other web page. If we accept that thinking, every home page would require a login to see that specific company's phone number of opening hours. Those pieces of data are also valuable, in the right circumstances! And then the web would either not exist or the account system required would be so wide spread that accounts would carry no value and the system would become useless.

kungito 3y ago
If there was more value in requiring login than there is in having this public and easily accessible, it would be behind a login form. The current internet has 99% of the time nothing to do with values someone imagined in the 80s
- fieldcny 3y ago
  What do you think the webs value is today? What do you think the webs value was imagined to be in the “80s”?
- xorcist 3y ago
  What you describe is the Facebook business model. Which seems to be a valid model, but twitter was not built around it and such a pivot would break all business moats around the company. There was no web in the 80s so not sure what values you refer to, or how they are relevant to today's businesses.
- Zak 3y ago
  Value to whom? Twitter is more valuable to its users, to journalists who embed tweets in stories, and to web users at large who follow links and search results if it does not require a login to view posts. Of course, none of those people own Twitter, and it may well be more valuable to its owners if it does require a login.
ben_w 3y ago
> Why would an AI company start scraping twitter html, instead of using an already existing archive? I can think of a few possible reasons. They might want more up-to-date info, or they might have no real developers and the scraper was created by a business guru who prompted ChatGPT and didn't understand the code that came out. Given what else Musk has asserted about Twitter, and how often former or current Twitter devs have contradicted him, it may not even be what Musk said. > Twitter is not higher quality data than any other web page Eh, depends how much you can infer from retweet, favourites, etc. Won't be the only such site, but it's probably better training data than blog posts are these days. But yeah, I absolutely agree that Twitter doing this caused a lot of damage to any orgs, corporate or government, which wanted to be public, anything from restaurants announcing special offers to governments issuing hurricane warnings. Twitter isn't big enough to assume everyone has an account, like Facebook is.
MagicMoonlight 3y ago
Because they want an advantage over competitors who are using the archives already…
mbrochh 3y ago
If every AI company pointed their scrapers to archive.org, that site would go down immediately as well. This is just kicking the can down the road. We have a major structural problem now. We want data to be free and machine readable, but no startup (and even a giant like Twitter) can afford the server cost to withstand all those machines.

kungito 3y ago

If there was more value in requiring login than there is in having this public and easily accessible, it would be behind a login form. The current internet has 99% of the time nothing to do with values someone imagined in the 80s

fieldcny 3y ago
What do you think the webs value is today? What do you think the webs value was imagined to be in the “80s”?
xorcist 3y ago
What you describe is the Facebook business model. Which seems to be a valid model, but twitter was not built around it and such a pivot would break all business moats around the company. There was no web in the 80s so not sure what values you refer to, or how they are relevant to today's businesses.
Zak 3y ago
Value to whom? Twitter is more valuable to its users, to journalists who embed tweets in stories, and to web users at large who follow links and search results if it does not require a login to view posts. Of course, none of those people own Twitter, and it may well be more valuable to its owners if it does require a login.

ben_w 3y ago

> Why would an AI company start scraping twitter html, instead of using an already existing archive? I can think of a few possible reasons. They might want more up-to-date info, or they might have no real developers and the scraper was created by a business guru who prompted ChatGPT and didn't understand the code that came out. Given what else Musk has asserted about Twitter, and how often former or current Twitter devs have contradicted him, it may not even be what Musk said. > Twitter is not higher quality data than any other web page Eh, depends how much you can infer from retweet, favourites, etc. Won't be the only such site, but it's probably better training data than blog posts are these days. But yeah, I absolutely agree that Twitter doing this caused a lot of damage to any orgs, corporate or government, which wanted to be public, anything from restaurants announcing special offers to governments issuing hurricane warnings. Twitter isn't big enough to assume everyone has an account, like Facebook is.

MagicMoonlight 3y ago

Because they want an advantage over competitors who are using the archives already…

mbrochh 3y ago

If every AI company pointed their scrapers to archive.org, that site would go down immediately as well. This is just kicking the can down the road. We have a major structural problem now. We want data to be free and machine readable, but no startup (and even a giant like Twitter) can afford the server cost to withstand all those machines.