4 ms·

I tried codeberg, used it a year, then early this year in all their wisdom codeberg decided to show adverserial random text instead of my repo, reporteldly to m

I tried codeberg, used it a year, then early this year in all their wisdom codeberg decided to show adverserial random text instead of my repo, reporteldly to mess up llm training to user agents they weren't sure were human. Codeberg had one job, serve my repo, it didn't do that, when brought up, I was told it was a feature not a bug, they could maybe whitelist me but that wasn't my problem, it was that random people got totally blocked or from accessing the repo. I moved back to github.

grayhatter 3mo ago
That's the reason I left as well. I complained, was told I need to drop the attitude, or leave. so I left. must be harder than I think running a src forge
- duttish 3mo ago
  Today it appears you have two options 1) Anubis or similar 2) Accept that 90-95% of your traffic is LLM scrapers fetching every file for every commit for every repo etc etc. There's some comments above with specific lists.
  - grayhatter 3mo ago
    I built bot detection into the web framework I wrote. It's blocking well over >90% of the abuse. The secret 3rd option is write some code.
hackthemack 3mo ago
Is your complaint about Anubis? I see the "checking if you are human" kind of text too when going to codeberg, but it is usually only a second or two. It is because I run a bit of obfuscation and resistance to browser finger printing. But you find seeing the text for a couple of seconds too annoying to use codeberg? Maybe it is more than 2 seconds for you? Everyone has their views on what is acceptable in the world and what they will put up with. Just, to my point of view, I think codeberg is trying to fight the good fight in keeping llms from crawling their website.
- harrisi 3mo ago
  I believe they're referencing iocaine[0], which I believe I've seen on codeberg before. I may be wrong though. [0]: https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/
  - kps 3mo ago
    [0] Gives me HTTP 418
    - intothemild 3mo ago
      A rare event you see a wild "HTTP I'm a little Teapot"
    - harrisi 3mo ago
      Interesting. Well, for information about the project: https://lib.rs/crates/iocaine https://lib.rs/crates/iocaine The site linked is excluded from the wayback machine.
- badsectoracula 3mo ago
  Codeberg has some extra "anti-scraping" measures than just Anubis which, judging from the community issues[0] sometimes ends up with false positives that cause people to get garbage (example issue[1], potential page with the garbage i found from Google[2]). Interestingly, someone mentioned that you may get the garbage when searching Codeberg using Google's `site:codeberg.org` [0] https://codeberg.org/Codeberg/Community/issues https://codeberg.org/Codeberg/Community/issues [1] https://codeberg.org/Codeberg/Community/issues/2603 https://codeberg.org/Codeberg/Community/issues/2603 [2] https://codeberg.org/Codeberg-e.V./requests/!~codeberger~!.gentles/demolierenden-aufsch%C3%BCrfender/ https://codeberg.org/Codeberg-e.V./requests/!~codeberger~!.g...
- whalesalad 3mo ago
  Seeing the Anubis weeb interstitial is super annoying.
deleted 3mo ago
[deleted]
pluto_modadic 3mo ago
codeberg's one job gets interfered with by freeloaders hammering expensive views (like git blame) inconsiderately.
- saghm 3mo ago
  Couldn't they just rate limit them? Are they literally using a new IP for every request?
  - anematode 3mo ago
    There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
    - sroussey 3mo ago
      We used to call this snow shoeing.
  - kstrauser 3mo ago
    In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
- kstrauser 3mo ago
  You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it: * For every Git commit, fetch the version of every file in the repository at that commit. * See git blame for every file at every commit. * Attempt to download the archive of each repo at every commit. * Run every possible pull request search filter combination. * Run every possible issue search filter combination. * Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before. Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.
ShinyLeftPad 3mo ago
Sounds like a feature. I didn't think Codeberg would have the balls to do something like it. I think I'm sold.

grayhatter 3mo ago

That's the reason I left as well. I complained, was told I need to drop the attitude, or leave. so I left. must be harder than I think running a src forge

duttish 3mo ago
Today it appears you have two options 1) Anubis or similar 2) Accept that 90-95% of your traffic is LLM scrapers fetching every file for every commit for every repo etc etc. There's some comments above with specific lists.
- grayhatter 3mo ago
  I built bot detection into the web framework I wrote. It's blocking well over >90% of the abuse. The secret 3rd option is write some code.

hackthemack 3mo ago

Is your complaint about Anubis? I see the "checking if you are human" kind of text too when going to codeberg, but it is usually only a second or two. It is because I run a bit of obfuscation and resistance to browser finger printing. But you find seeing the text for a couple of seconds too annoying to use codeberg? Maybe it is more than 2 seconds for you? Everyone has their views on what is acceptable in the world and what they will put up with. Just, to my point of view, I think codeberg is trying to fight the good fight in keeping llms from crawling their website.

harrisi 3mo ago
I believe they're referencing iocaine[0], which I believe I've seen on codeberg before. I may be wrong though. [0]: https://iocaine.madhouse-project.org/ https://iocaine.madhouse-project.org/
- kps 3mo ago
  [0] Gives me HTTP 418
  - intothemild 3mo ago
    A rare event you see a wild "HTTP I'm a little Teapot"
  - harrisi 3mo ago
    Interesting. Well, for information about the project: https://lib.rs/crates/iocaine https://lib.rs/crates/iocaine The site linked is excluded from the wayback machine.
badsectoracula 3mo ago
Codeberg has some extra "anti-scraping" measures than just Anubis which, judging from the community issues[0] sometimes ends up with false positives that cause people to get garbage (example issue[1], potential page with the garbage i found from Google[2]). Interestingly, someone mentioned that you may get the garbage when searching Codeberg using Google's `site:codeberg.org` [0] https://codeberg.org/Codeberg/Community/issues https://codeberg.org/Codeberg/Community/issues [1] https://codeberg.org/Codeberg/Community/issues/2603 https://codeberg.org/Codeberg/Community/issues/2603 [2] https://codeberg.org/Codeberg-e.V./requests/!~codeberger~!.gentles/demolierenden-aufsch%C3%BCrfender/ https://codeberg.org/Codeberg-e.V./requests/!~codeberger~!.g...
whalesalad 3mo ago
Seeing the Anubis weeb interstitial is super annoying.

deleted 3mo ago

[deleted]

pluto_modadic 3mo ago

codeberg's one job gets interfered with by freeloaders hammering expensive views (like git blame) inconsiderately.

saghm 3mo ago
Couldn't they just rate limit them? Are they literally using a new IP for every request?
- anematode 3mo ago
  There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
  - sroussey 3mo ago
    We used to call this snow shoeing.
- kstrauser 3mo ago
  In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.
kstrauser 3mo ago
You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it: * For every Git commit, fetch the version of every file in the repository at that commit. * See git blame for every file at every commit. * Attempt to download the archive of each repo at every commit. * Run every possible pull request search filter combination. * Run every possible issue search filter combination. * Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before. Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.

ShinyLeftPad 3mo ago

Sounds like a feature. I didn't think Codeberg would have the balls to do something like it. I think I'm sold.