3 ms·

The web has really closed in 2023. StackOverflow, Reddit, Github, Twitter all put the brakes on scraping and API access. The trigger was preventing AI training

The web has really closed in 2023. StackOverflow, Reddit, Github, Twitter all put the brakes on scraping and API access. The trigger was preventing AI training combined with a push to increase profitabilty due to commercial realities (tech recession, new ownership at Stack and X, Reddit wants to IPO). I believe a long term effect will be a rise in the marketplace for proprietary data. Search engines, AI tools, anyone who needs the data will have to pay for the firehose or API access. (That, in turn, may cause antitrust issues if only the richest companies can afford access to that data.)

xign 3y ago
To be fair, GitHub still has vast majority of features available as public anonymous API end points other than code search. It's just that they are rate-limited a lot more aggressively. And making a GitHub account is free and not terrible intrusive. Even then, all APIs are rate-limited even if you are logged in anyway as they don't want you to kill the servers (that's why tools like https://gitstar-ranking.com/ https://gitstar-ranking.com/ have troubles because they are relying on free accounts to scrape all repos' GitHub stars). As for the actual important data (source code), you can simply clone the repo anonymously using HTTPS endpoints, and do your code search there (e.g. there are third-party websites like https://grep.app/ https://grep.app/ that can do this). The issue with say Reddit/Twitter is that the source data (posts/comments/upvotes and tweets) is no longer available via APIs, making it impossible / hard to build third-party tools. With GitHub, the source data is still easily accessible, with search (the auxiliary layer) disabled for non-logged-in users. I think they are quite different situations and mixing them together is not useful.
pixl97 3y ago
It's been going on longer than this, but this year has had a lot further locking down... https://theoatmeal.com/comics/reaching_people https://theoatmeal.com/comics/reaching_people
Hizonner 3y ago
... and everybody will just think it's normal for "StackOverflow, Reddit, Github, Twitter" to sell access to content that other people created and own... If something is truly proprietary, then the compensation should go to its owner, not to some random Git server operator or whatever. And the owner, not the server operator, should be setting the price and terms. If the server needs to make money, it can charge for the basic service itself. On the other hand, if somebody created something to give it away, on a platform that at that time provided a channel effective for giving things away and advertised itself as suitable for giving things away, then the platform changing the rules midstream is morally piracy of that person's work. Probably a very large fraction of the material on those platforms would never have been put there in the first place if the actual owners had expected random restrictions and arbitrary access charges. If they want to radically rewrite the rules like that, then they need to not sell access to any preexisting content unless the original creator explicitly opts in. But we had Reddit, for instance, actively reinstating posts that had been mass-deleted by their authors specifically to prevent that kind of abuse.
xtracto 3y ago
Yes and no. ExpertsExchange was notable for having "walled" responses and was similar to StackOverflow. In fact, I remember when StackOverflow was launched, it was compared to expertsexchange but "open". It is time to move from centralized services to something more distributed and "micro transaction based". I know a lot of people in HN will dislike this butn All the ones you mentioned (Q&A forum, News link + commenting aggregator, Git Hosting + Approval Flow + Wiki + Ticketing system ) can and should be implemented in a completely distributed manner: (using Kademilla/Bittorrent technologies, IPFS and CryptoTokens for paying micro transactions). Going 100% distributed is the only way these sort of things are going to stay "open" and free of corporate greed (we have seen time and time again, original founders may start the project with good intentions, but in the long term, the product gets sold and bean counters take the lead).

xign 3y ago

To be fair, GitHub still has vast majority of features available as public anonymous API end points other than code search. It's just that they are rate-limited a lot more aggressively. And making a GitHub account is free and not terrible intrusive. Even then, all APIs are rate-limited even if you are logged in anyway as they don't want you to kill the servers (that's why tools like https://gitstar-ranking.com/ https://gitstar-ranking.com/ have troubles because they are relying on free accounts to scrape all repos' GitHub stars). As for the actual important data (source code), you can simply clone the repo anonymously using HTTPS endpoints, and do your code search there (e.g. there are third-party websites like https://grep.app/ https://grep.app/ that can do this). The issue with say Reddit/Twitter is that the source data (posts/comments/upvotes and tweets) is no longer available via APIs, making it impossible / hard to build third-party tools. With GitHub, the source data is still easily accessible, with search (the auxiliary layer) disabled for non-logged-in users. I think they are quite different situations and mixing them together is not useful.

pixl97 3y ago

It's been going on longer than this, but this year has had a lot further locking down... https://theoatmeal.com/comics/reaching_people https://theoatmeal.com/comics/reaching_people

Hizonner 3y ago

... and everybody will just think it's normal for "StackOverflow, Reddit, Github, Twitter" to sell access to content that other people created and own... If something is truly proprietary, then the compensation should go to its owner, not to some random Git server operator or whatever. And the owner, not the server operator, should be setting the price and terms. If the server needs to make money, it can charge for the basic service itself. On the other hand, if somebody created something to give it away, on a platform that at that time provided a channel effective for giving things away and advertised itself as suitable for giving things away, then the platform changing the rules midstream is morally piracy of that person's work. Probably a very large fraction of the material on those platforms would never have been put there in the first place if the actual owners had expected random restrictions and arbitrary access charges. If they want to radically rewrite the rules like that, then they need to not sell access to any preexisting content unless the original creator explicitly opts in. But we had Reddit, for instance, actively reinstating posts that had been mass-deleted by their authors specifically to prevent that kind of abuse.

xtracto 3y ago

Yes and no. ExpertsExchange was notable for having "walled" responses and was similar to StackOverflow. In fact, I remember when StackOverflow was launched, it was compared to expertsexchange but "open". It is time to move from centralized services to something more distributed and "micro transaction based". I know a lot of people in HN will dislike this butn All the ones you mentioned (Q&A forum, News link + commenting aggregator, Git Hosting + Approval Flow + Wiki + Ticketing system ) can and should be implemented in a completely distributed manner: (using Kademilla/Bittorrent technologies, IPFS and CryptoTokens for paying micro transactions). Going 100% distributed is the only way these sort of things are going to stay "open" and free of corporate greed (we have seen time and time again, original founders may start the project with good intentions, but in the long term, the product gets sold and bean counters take the lead).