3 ms·

I was also under attack recently [0]. The little Forgejo instance where I host my code (of several open source packages so it needs to be open) was run into the

I was also under attack recently [0]. The little Forgejo instance where I host my code (of several open source packages so it needs to be open) was run into the ground and the disk was filled with generated zip archives. I'm not the only one who has suffered the same fate. For me, the attacks subsided (for now) when I banned Alibaba Cloud's IP range. If you are hosting a Forgejo instance, I strongly recommend setting DISABLE_DOWNLOAD_SOURCE_ARCHIVES to true. The crawlers will still peg your CPU but at least your disk won't be filled with zip files. [0] https://blog.nytsoi.net/2025/03/01/obliterated-by-ai https://blog.nytsoi.net/2025/03/01/obliterated-by-ai

zoobab 2y ago
"disk was filled with generated zip archives" That's bad software design to generate ZIP files on the fly.
- abound 2y ago
  They could very well just be temp files, I know the Go standard library will write large multi-part file uploads to temp disk, for example. It'd be better to totally stream it of course, but that's not always an option for one reason or another.
- Ndymium 2y ago
  They're deleted by default every 24 hours and that time is configurable. Not useful when you get 60 requests per second though.
  - diggan 2y ago
    > They're deleted by default every 24 hours Hm, so it's a cache then? Requesting the same tarball 100 times shouldn't create 100 zip files if they're cached, and if they aren't cached they shouldn't fill up the disk.
    - Ndymium 2y ago
      They are a cache, but you can generate them for each branch, tag, and commit in at least three different formats... Now imagine you have a repo with several thousand commits.
      - diggan 2y ago
        Yeah, fair point. Although I think the only uniqueness is commits here, tarballs generated from branches and tags are ultimately the same as the ones generated by the commit that those reference. But I still agree with your overall point.
devit 2y ago
Or perhaps switch to well-engineered software actually properly designed to be served on the public Internet. Clearly generating zip files, writing them fully to disk and then sending them to the client all at once is a completely awful and unusable design, compared to the proper design of incrementally generating and transmitting them to the client with minimal memory consumption and no disk usage at all. The fact that such an absurd design is present is a sign that most likely the developers completely disregarded efficiency when making the software, and it's thus probably full of similar catastrophic issues. For example, from a cursory look at the Forgejo source code, it appears that it spawns "git" processes to perform all git operations rather than using a dedicated library and while I haven't checked, I wouldn't be surprised if those operations were extremely far from the most efficient way of performing a given operation. It's not surprising that the CPU is pegged at 100% load and the server is unavailable when running such extremely poor software.
- Ndymium 2y ago
  Just noting that the archives are written to disk on purpose, as they are cached for 24 hours (by default). But when you have a several thousand commit repository, and the bots tend to generate all the archive formats for every commit… But Forgejo is not the only piece of software that can have CPU intensive endpoints. If I can't fence those off with robots.txt, should I just not be allowed to have them in the open? And if I forced people to have an account to view my packages, then surely I'd have close to 0 users for them.
  - devit 2y ago
    Well then such a cache needs obviously to have limit to the disk space it uses and some sort of cache replacement policy, since if one can generate a zip file for each tag, that means that the total disk space of the cache is O(n^2) where n is the disk usage of the git repositories (imagine a single repository where each commit is tagged and adds a new file of constant size), so unless one's total disk space is a million/billion times larger than the disk space used by the repositories, it's guaranteed to fill the disk without such a limit.

zoobab 2y ago

"disk was filled with generated zip archives" That's bad software design to generate ZIP files on the fly.

abound 2y ago
They could very well just be temp files, I know the Go standard library will write large multi-part file uploads to temp disk, for example. It'd be better to totally stream it of course, but that's not always an option for one reason or another.
Ndymium 2y ago
They're deleted by default every 24 hours and that time is configurable. Not useful when you get 60 requests per second though.
- diggan 2y ago
  > They're deleted by default every 24 hours Hm, so it's a cache then? Requesting the same tarball 100 times shouldn't create 100 zip files if they're cached, and if they aren't cached they shouldn't fill up the disk.
  - Ndymium 2y ago
    They are a cache, but you can generate them for each branch, tag, and commit in at least three different formats... Now imagine you have a repo with several thousand commits.
    - diggan 2y ago
      Yeah, fair point. Although I think the only uniqueness is commits here, tarballs generated from branches and tags are ultimately the same as the ones generated by the commit that those reference. But I still agree with your overall point.

devit 2y ago

Or perhaps switch to well-engineered software actually properly designed to be served on the public Internet. Clearly generating zip files, writing them fully to disk and then sending them to the client all at once is a completely awful and unusable design, compared to the proper design of incrementally generating and transmitting them to the client with minimal memory consumption and no disk usage at all. The fact that such an absurd design is present is a sign that most likely the developers completely disregarded efficiency when making the software, and it's thus probably full of similar catastrophic issues. For example, from a cursory look at the Forgejo source code, it appears that it spawns "git" processes to perform all git operations rather than using a dedicated library and while I haven't checked, I wouldn't be surprised if those operations were extremely far from the most efficient way of performing a given operation. It's not surprising that the CPU is pegged at 100% load and the server is unavailable when running such extremely poor software.

Ndymium 2y ago
Just noting that the archives are written to disk on purpose, as they are cached for 24 hours (by default). But when you have a several thousand commit repository, and the bots tend to generate all the archive formats for every commit… But Forgejo is not the only piece of software that can have CPU intensive endpoints. If I can't fence those off with robots.txt, should I just not be allowed to have them in the open? And if I forced people to have an account to view my packages, then surely I'd have close to 0 users for them.
- devit 2y ago
  Well then such a cache needs obviously to have limit to the disk space it uses and some sort of cache replacement policy, since if one can generate a zip file for each tag, that means that the total disk space of the cache is O(n^2) where n is the disk usage of the git repositories (imagine a single repository where each commit is tagged and adds a new file of constant size), so unless one's total disk space is a million/billion times larger than the disk space used by the repositories, it's guaranteed to fill the disk without such a limit.