3 ms·

I assume this is because of LLM scraping.

nijave 9d ago
Presumably but I wish they'd also focus on optimizing code/making pages fully cacheable instead of rate-limits and blocking
- swatcoder 9d ago
  Making requests is inherently cheaper than delivering responses, even with caches. Efficiency improvements can buy a little time on a given resources but won't solve the problem of bot saturation now that everybody can spawn a custom bot in about 12 seconds and is being encouraged to do so. Rate limits, blocking, and pay-per-use are the only roads out and even those might not last as models get better at hacking and masquerading. The internet we want to use LLM's with is simply not one that can support LLM's, and with LLM's not going anywhere, the whole experience of the internet is going to be forced into some radically less open and more expensive paradigm. Policies like this just represent the beginning of the transition.
- jayd16 9d ago
  The limits are for API requests, no? Or is this just an unrelated performance complaint?
  - nijave 9d ago
    Last I checked, they just blanket added `cache-control: max-age=0` to everything. Loading https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab is showing 14 API calls so presumably significantly more DB queries for a public page
Frieren 9d ago
We need a new non-commercial version of the internet. Free of bots, free of ads, ... you pay to access social media optimized to be interesting enough to be worth paying instead of addictive enough to keep you scrolling to show you more ads. It may look impossible right now. But what is impossible for real is to continue as we are. The damage that internet does to society is increasing by the day while its value is reduced (economic value, social value).
- phoe-krk 9d ago
  > We need a new non-commercial version of the internet. Except it's non-commercial, therefore valuable, therefore commercialized, therefore commercial. You'd need a force strong enough to prevent it from falling prey to this tragedy of the commons, and that force would need to be stronger than the incentives to commercialize it. And that's where plenty of contemporary scraping-based salaries lay.
- immortalist 9d ago
  Impossible to create
  - OtherShrezzing 9d ago
    I'm not so certain. I used to buy a broadsheet newspaper, which was full of ads. Now I pay a few hundred a year, and get the same newspaper online, with no ads at all. So, the precursor to online media has already gone through this paradigm shift.
    - pixl97 9d ago
      I mean, what you're saying is "If I pay for a product ads go away" Which is partially true, but it only shifts the distribution of the problem. Once your service gains enough popularity network effects cause it to gain value. You have to worry about high priced buyouts of the entire service (great for the site owner, terrible for the users).
  - Zambyte 9d ago
    It's not, it just requires creativity. One example I can think of is limiting connectivity by distance between nodes. Something like Meshtastic seems unlikely to ever be commercialized in the same way that the Internet has been. Sure, you would lose some useful applications of the Internet, but you would gain other things.
  - armadyl 9d ago
    Also for better or for worse the ad-centric model to some extent has allowed more people to access information. Gating everything behind paid (but with no ads) likely would hurt a significant amount of lower income users.

Presumably but I wish they'd also focus on optimizing code/making pages fully cacheable instead of rate-limits and blocking

swatcoder 9d ago
Making requests is inherently cheaper than delivering responses, even with caches. Efficiency improvements can buy a little time on a given resources but won't solve the problem of bot saturation now that everybody can spawn a custom bot in about 12 seconds and is being encouraged to do so. Rate limits, blocking, and pay-per-use are the only roads out and even those might not last as models get better at hacking and masquerading. The internet we want to use LLM's with is simply not one that can support LLM's, and with LLM's not going anywhere, the whole experience of the internet is going to be forced into some radically less open and more expensive paradigm. Policies like this just represent the beginning of the transition.
jayd16 9d ago
The limits are for API requests, no? Or is this just an unrelated performance complaint?
- nijave 9d ago
  Last I checked, they just blanket added `cache-control: max-age=0` to everything. Loading https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab is showing 14 API calls so presumably significantly more DB queries for a public page

Frieren 9d ago

We need a new non-commercial version of the internet. Free of bots, free of ads, ... you pay to access social media optimized to be interesting enough to be worth paying instead of addictive enough to keep you scrolling to show you more ads. It may look impossible right now. But what is impossible for real is to continue as we are. The damage that internet does to society is increasing by the day while its value is reduced (economic value, social value).

phoe-krk 9d ago
> We need a new non-commercial version of the internet. Except it's non-commercial, therefore valuable, therefore commercialized, therefore commercial. You'd need a force strong enough to prevent it from falling prey to this tragedy of the commons, and that force would need to be stronger than the incentives to commercialize it. And that's where plenty of contemporary scraping-based salaries lay.
immortalist 9d ago
Impossible to create
- OtherShrezzing 9d ago
  I'm not so certain. I used to buy a broadsheet newspaper, which was full of ads. Now I pay a few hundred a year, and get the same newspaper online, with no ads at all. So, the precursor to online media has already gone through this paradigm shift.
  - pixl97 9d ago
    I mean, what you're saying is "If I pay for a product ads go away" Which is partially true, but it only shifts the distribution of the problem. Once your service gains enough popularity network effects cause it to gain value. You have to worry about high priced buyouts of the entire service (great for the site owner, terrible for the users).
- Zambyte 9d ago
  It's not, it just requires creativity. One example I can think of is limiting connectivity by distance between nodes. Something like Meshtastic seems unlikely to ever be commercialized in the same way that the Internet has been. Sure, you would lose some useful applications of the Internet, but you would gain other things.
- armadyl 9d ago
  Also for better or for worse the ad-centric model to some extent has allowed more people to access information. Gating everything behind paid (but with no ads) likely would hurt a significant amount of lower income users.