7 ms·

The GMP library's repository is under attack by a single GitHub user

98 points by bramhaag 3y ago

bramhaag 3y ago
From their website: "NEWS 2023-06-16: The GMP server is under attack by several hundred IP addresses owned by Microsoft cooperation. We do not know if this is made with malice, if it is some sort of mistake, or if some of their cloud customer is running the attack. The attack targets the GMP repo, with thousands of identical requests, most of which makes our server compress the data. We're firewalling off all of Microsoft's IP addresses as an emergency response. This is a blunt response, but it is the only response which solves the problem quickly, allowing legitimate site usage to work again." "UPDATE 2023-06-18: We got a reply from somebody with an impressive title at Github. This person explains that Microsoft and Github have investigated this, and they blame a Github user and the poor GMP infrastructure. It is very interesting that they have done nothing to stop the traffic; we need to keep defending our server by firewalling off more Microsoft IP ranges as I write this 30 hours after Github's response. It is also curious that they blame the victim. (Our infrastructure is pretty resilient with powerful server-class hardware and great connectivity to the Internet.)"
- blueflow 3y ago
  Link?
  - notwhereyouare 3y ago
    If you click the submission link, you can see the update in the blue box
retrocryptid 3y ago
Microsoft employee is dismissive of world outside Microsoft. Film at 11. Which is to say... what's MSFT's motivation to worry about this?
- deleted 3y ago
  [deleted]
- jsnell 3y ago
  They've clearly worried enough about it to have somebody investigate the initial report and rootcause it. (And the motivation for that wasn't just some kind of a huge public outcry on the first report, because as far as I can tell that sank without a trace at least on HN). If the traffic from that customer is within the typical norms of git usage, rather than clearly abusive, then what further action would you expect [0]? And honestly, whoever is maintaining the GMP server isn't appearing particularly competent based on the messages they've written so far. That they'd misidentify or mischaracterize non-attack traffic as abusive seems pretty plausible. [0] Ok, I'll admit that there is one thing that GitHub should do even if these particular requests are not abusive, just as a best practice when operating a system for running untrusted code that can connect to the Internet. The egress IPs from any single GitHub customer should be sticky, rather than them having a massive pool of IPs shared by all the users. But if that's not how their systems are already architected, making it happen is probably a multi-quarter effort.
  - xyse53 3y ago
    Initially, I wanted to reply to you and say it's unfair to judge the person as incompetent; I still think it's a bit unfair because they develop a specialized library and their choice of hosting shouldn't need to be modernized at every step. However, reading through their response thread [0] I do think they aren't appearing particularly mature. It feels like they are refusing to characterize what is actually going on and digging in and escalating this defensive mindset. [0] https://gmplib.org/list-archives/gmp-devel/2023-June/thread.html https://gmplib.org/list-archives/gmp-devel/2023-June/thread....
    - retrocryptid 3y ago
      So... don't ascribe to incompetence what can easily be explained by malice?
    - retrocryptid 3y ago
notamy 3y ago
From the description, it sounds like someone(s) is automatically downloading GMP as a part of a CI process. GHA being the source would explain why it's coming from "several hundred IP addresses owned by Microsoft cooperation (sic)," as well as (maybe) why GH puts the blame on the GMP project for not being able to handle the load. Edit: This seems to be the responsible project/issue: https://github.com/BtbN/FFmpeg-Builds/issues/278 https://github.com/BtbN/FFmpeg-Builds/issues/278
- Aardwolf 3y ago
  Github could cache it then to improve this?
  - rwmj 3y ago
    The title is a bit confusing. It seems the GMP project has a self-hosted Mercurial repo (https://gmplib.org/devel/repo-usage https://gmplib.org/devel/repo-usage) which is being hit by a user at Microsoft that they've somehow tied back to a single github account. I'm sure it's going to be CI or mirroring or some other automated process.
    - cxr 3y ago
      Why is this a response to the person you sent this reply to?
      - rwmj 3y ago
        Github can't easily cache a remotely hosted repo that some random CI test (essentially a script) accesses, at least not without some very hairy proxying.
        cxr 3y ago
        Is GitHub's CI infrastructure (and cloud compute, generally) not hairy? (A basic attempt at straightforward proxying seems like it would be the _least_ hairy aspect of the stack, besides.)
samwillis 3y ago
So another classic case of someone's constant integration system DoS'ing an open source project. We've all become so blinded by big tech and how they operate we fail to see how our actions can affect smaller projects we depend on CI providers really should do more to prevent and mitigate these when they happen. They should have outbound firewalls, and the ability to request a rate limit on IPs. Having to resort to a ban hammer when people depend on your tool is a shame.
- jeltz 3y ago
  And in general just making caching stuff easier. I feel like it is unnecessarily complicated for example to cache apt-get in Gitlab which I assume makes most people not do it. https://gitlab.com/gitlab-org/gitlab-runner/-/issues/991#note_126864314 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/991#not...
- deleted 3y ago
  [deleted]
progval 3y ago
Initial email mentioning it on GMP's mailing list: https://gmplib.org/list-archives/gmp-discuss/2023-June/006906.html https://gmplib.org/list-archives/gmp-discuss/2023-June/00690... Reply from Github: https://gmplib.org/list-archives/gmp-devel/2023-June/006162.html https://gmplib.org/list-archives/gmp-devel/2023-June/006162....
- CtrlAltDelete51 3y ago
  That seems like a perfectly reasonable response. They even provided the project so GMP could reach out.
  - WastingMyTime89 3y ago
    That’s in line with the response I’m used to get from cybersecurity teams at the large company I work for. It is neither reasonable nor appropriate in tone but I have got used to it being apparently the best they can do/can be bothered to write.
  - burkaman 3y ago
    It's totally reasonable to say that it isn't malicious and they aren't going to stop it, but it's unreasonable and unhelpful to add "seems like your servers just suck". They also missed or failed to mention that there are 700 forks of the project running the same thing at the same time.
    - 542458 3y ago
      I’m a bit confused as to why 700 automated clone requests per day would be an unreasonable amount of traffic for a project like GMP. That doesn’t sound like much traffic to me, especially since they reportedly have a 24 core CPU with 256GB of RAM. Is cloning a mercurial repo highly server intensive?
      - matheusmoreira 3y ago
        I'm confused as well. Seems like a negligible amount of traffic at first glance. The administrator mentioned compression, could it really have such a big impact?
htpmnxc 3y ago
[flagged]
666satanhimself 3y ago
[dead]
judofyr 3y ago
More context: https://github.com/BtbN/FFmpeg-Builds/issues/278 https://github.com/BtbN/FFmpeg-Builds/issues/278 The FFmpeg-Builds repo has a GitHub Actions Workflow which clones the Mercurial repo. However, this runs as a (daily?) cronjob. In addition, this repo has 700 forks and now all of them are running the same workflow. This is out of control for the original author of the repo as there’s no way to change those 700 forks… EDIT: Also relevant is that there’s no official GMP mirror at any of the major code hosting sites (GitHub, GitLab) etc.
- tremon 3y ago
  [flagged]
- deleted 3y ago
  [deleted]
- jeroenhd 3y ago
  I'm confused by this statement: > GMP is at this point the only dependency that does not offer a sane way to clone its repository. The latest release is from years ago and https://gmplib.org/download/gmp/gmp-6.2.1.tar.lz https://gmplib.org/download/gmp/gmp-6.2.1.tar.lz should contain every file necessary. Why would you need access to the commit history in a build script? In fact, why would you need to download a fresh copy of a dependency that almost never gets updated from upstream? Surely Github can do some kind of caching. I wonder what will happen now that their build servers can't even reach the Mercurial repo anymore. I suppose the problem should quickly correct itself soon with all the failing builds.
  - tedunangst 3y ago
    The idea that you build software from a fixed archive is passé. Everything must be a git clone. Or if they don't use git, you muddle along with hg.
    - rwmj 3y ago
      Yes and no. The other side of the argument is that a tarball offers a point in time snapshot of the code that is smaller, and easier to download since there's no need to interact with a complex internet service. Also easier to cache, mirror and back up a single file - major distros appreciate this because we have to mirror sources into build systems and provide sources on demand to satisfy the GPL. There are of course disadvantages to the tarball approach (like, no history, harder to patch) but sometimes the simpler technology can be more robust.
aseipp 3y ago
The repository in question, based on the email from Mike Blacker, sent to gmp-devel[1]: https://github.com/BtbN/FFmpeg-Builds https://github.com/BtbN/FFmpeg-Builds There's a bunch of build variants here, for various platforms, in different ways. Classic "you have to create the cartesian product of all variables" approach. Here is the single build matrix for a single "Build FFMPeg" job: https://github.com/BtbN/FFmpeg-Builds/actions/runs/5303511170 https://github.com/BtbN/FFmpeg-Builds/actions/runs/530351117... You can look through the code and see that each of these will trigger a download. The developer also added an interesting commit today, stopping forks from having the CI work instantly. Why? Because by default they would all share the same cron entry, and therefore launch all at the same time every night. So now new forks will need to go in and tweak/spread the Cron timer around if they want builds of their own https://github.com/BtbN/FFmpeg-Builds/commit/78191a73a6a1959d13a62a4d6adb9ddeffa18522 https://github.com/BtbN/FFmpeg-Builds/commit/78191a73a6a1959... But there are already over 700 forks of this repository, so the current path is unsustainable. So in short: - This isn't an "attack", and it's unclear of the direct volume, but anyway, it doesn't matter because ultimately gmplib.org had strain caused by it. - GitHub correctly identified the responsible user when asked, and it did not in any way seem malicious, but certainly excessive. - The build/CI infrastructure for this project definitely hammers the project more than it probably should, exact specifics aside. - The owner of the project put in place a small (future) mitigation for future forks, but for now IP banlists will probably remain, I'd guess? - Please try to cache things more aggressively in your build jobs. [1] https://gmplib.org/list-archives/gmp-devel/2023-June/006162.html https://gmplib.org/list-archives/gmp-devel/2023-June/006162....
- folkrav 3y ago
  > It already is cached by the docker image. The problem why it's SO MANY requests is because hundreds of people have forked this repository, and now at the same time (due to the workflow cron) start an image build. 99% of the days, it'll just use the cached image and not cause any network requests whatsoever. > If it builds new images, it'll be at max 10 requests to fetch the latest snapshot in parallel from my end (that's how many parallel jobs Github allows for free), which shouldn't be an issue. I've already switched GMP back to use an outdated release tarball because of how annoyingly flakey their mercurial server is. Straight from the repo author. Looking at the workflow, not sure how it could really fetch less than it already does.
runeks 3y ago
Sounds like GitHub actions should be caching clones of foreign Git repos. People already trust GH CI enough to run their code, so why not trust their cache as well?
- coder543 3y ago
  FWIW, it’s not a git repo, it’s a mercurial repo.
  - capitainenemo 3y ago
    And even if it was a git repo, seems this would still be a problem if everything wasn't on github. I guess the solution is to put all code on github...
    - capitainenemo 3y ago
      BTW, for anyone reading this later, I asked some mercurial folks and putting an http cache in front of clones does indeed work just fine, although using the repo bundles is more performant (but might not be something you could ask folks to do). Given this repo seems fairly low activity (last commit 7 months ago) I doubt there'd be much of a problem with caching reads. Although given how they are using this repo, they'd probably be better off with a shallow clone, or a subversion proxy, or a snapshot tarball...likely just need a set of files..
- teraflop 3y ago
  In this case, it's a foreign Mercurial repository, not Git. And remember that users' builds can contain arbitrary custom shell scripts that perform "git clone" or "hg clone" commands. It could be very, very difficult for Github to build an automated system that meddles with the behavior of those scripts to introduce caching, while guaranteeing not to break anything.
maxmcd 3y ago
I thought people sometimes used forks just like another form of star. Save the repo for later review. I have so rarely seen any commits or changes when someone forks my public github repos. It's weird to think about github actions crons in the context of this pattern. A casual "I'll mark this to check out later" could result in lots of billable computation.
- beardog 3y ago
  Disabling jobs for new forks until a commit is made in the fork or an option is toggled on the settings page would be a fix, although maybe i'm missing something. I often had experiences where people would fork my repos and I would get hopeful that someone might contribute, but it turns out that they just treat forking as bookmarking. Github running actions for forks wrongly assumes that users use the fork feature for its original purpose.
orf 3y ago
Why does forking cause a GitHub actions schedule to run? This is pretty unexpected. I wonder what % of actions builds are useless runs like this? Edit: to the shadowbanned guy who hasn’t realised he’s been shadowbanned for years, that’s not correct.
- bootloop 3y ago
  It's the reason you get random build failure notifications and warning emails that your (free) credits are running out, etc.. Took me a while to figure out what happened the first time I ran into this.
- eddythompson80 3y ago
  It’s a cron workflow
  - orf 3y ago
    Yes?
    - eddythompson80 3y ago
      Why would you expect a cron task to work differently because a repo is a fork or not? Maybe Actions/Workflows shouldn’t be enabled by default on forks, which used to be the case but that has its problems too. You can disable actions on your forks.
      - orf 3y ago
        Why would I expect pressing the fork button would suddenly enable redundant and, in aggregare, expensive cronjobs? Most people fork to make a pull request and then never delete the fork after. Why are any scheduled actions required for that?
        eddythompson80 3y ago
        It depends on how you use the fork and what you use it for. That’s up to the project to decide, not GitHub. Most workflows are triggered on an update-event, like git push or create a tag or branch or pull request etc. You want them to trigger on your fork separately from the main repo because (a) the secrets aren’t shared between forks and main repo (b) you want to catch linters, automated tests, build issues and all the other things workflows are used for before you create a pull request if possible. Cron tasks are the odd exception. You can use them to update dependencies in your repo, or track an external dependency and rebuild your stuff with it, or nightly integration tests with external APIs, etc. Maybe they could enable all actions but cron actions, but then that’s just as equally confusing. I’d rather GitHub stay out of deciding what’s “expensive” and what’s not. Either enable all actions on forks by default, or disable them by default. They decided to enable them by default and give each user however many minutes of free actions time. I’m sure there is a better upsell opportunity there too.
nmlryc 3y ago
The whole thread is curated yet again (as all Microsoft criticism threads are). Shame on everyone who does not appreciate the volunteer work on gmp and ridicules their servers.
- jeroenhd 3y ago
  Microsoft isn't to blame here, other than being kind of a dick diminishing some other project's server. The same kind of attack could just as easily come from Gitlab or any other service that allows easy CI/CD cloning. The problem started with someone making a poorly built automated build script for another project, and that build script getting forked hundreds of times, triggering a huge load all by the lack of spread in the upstream build script. This is nothing compared to an actual DDoS attack, which Microsoft was partially blamed for in the previous email. I can see why someone over at MS would be annoyed to find out the supposed DDoS was just 700 requests.
deleted 3y ago
[deleted]
agilob 3y ago
>(Our infrastructure is pretty resilient with powerful server-class hardware and great connectivity to the Internet.) You should totally move to MS Azure!
zX41ZdbW 3y ago
> I've already switched GMP back to use an outdated release tarball because of how annoyingly flakey their mercurial server is. We had a similar problem. Step one - don't allow submodules from outside of github.com: https://github.com/ClickHouse/ClickHouse/blob/master/utils/check-style/check-style#L246 https://github.com/ClickHouse/ClickHouse/blob/master/utils/c... If there is a dependency from external git repository, it can be forked to github.com. But we still have a problem with: - Docker Hub; - Cargo. Sometimes they are offline. Also we should not use them to avoid supply-chain attacks. The solution probably should be: - copy all docker images as tarballs; - or use ECR. Still not sure what to do with Cargo: https://github.com/ClickHouse/ClickHouse/issues/48575 https://github.com/ClickHouse/ClickHouse/issues/48575
rhdunn 3y ago
IIUC, you can add "on" triggers to the GitHub Actions to only build on the original repository. That would prevent this issue. Making the change in this repository would require having the forks updated, so maybe the owner can then reach out to the owners of the forks asking them to update.

98 points by bramhaag 3y ago

From their website: "NEWS 2023-06-16: The GMP server is under attack by several hundred IP addresses owned by Microsoft cooperation. We do not know if this is made with malice, if it is some sort of mistake, or if some of their cloud customer is running the attack. The attack targets the GMP repo, with thousands of identical requests, most of which makes our server compress the data. We're firewalling off all of Microsoft's IP addresses as an emergency response. This is a blunt response, but it is the only response which solves the problem quickly, allowing legitimate site usage to work again." "UPDATE 2023-06-18: We got a reply from somebody with an impressive title at Github. This person explains that Microsoft and Github have investigated this, and they blame a Github user and the poor GMP infrastructure. It is very interesting that they have done nothing to stop the traffic; we need to keep defending our server by firewalling off more Microsoft IP ranges as I write this 30 hours after Github's response. It is also curious that they blame the victim. (Our infrastructure is pretty resilient with powerful server-class hardware and great connectivity to the Internet.)"

blueflow 3y ago
Link?
- notwhereyouare 3y ago
  If you click the submission link, you can see the update in the blue box

retrocryptid 3y ago

Microsoft employee is dismissive of world outside Microsoft. Film at 11. Which is to say... what's MSFT's motivation to worry about this?

deleted 3y ago
[deleted]
jsnell 3y ago
They've clearly worried enough about it to have somebody investigate the initial report and rootcause it. (And the motivation for that wasn't just some kind of a huge public outcry on the first report, because as far as I can tell that sank without a trace at least on HN). If the traffic from that customer is within the typical norms of git usage, rather than clearly abusive, then what further action would you expect [0]? And honestly, whoever is maintaining the GMP server isn't appearing particularly competent based on the messages they've written so far. That they'd misidentify or mischaracterize non-attack traffic as abusive seems pretty plausible. [0] Ok, I'll admit that there is one thing that GitHub should do even if these particular requests are not abusive, just as a best practice when operating a system for running untrusted code that can connect to the Internet. The egress IPs from any single GitHub customer should be sticky, rather than them having a massive pool of IPs shared by all the users. But if that's not how their systems are already architected, making it happen is probably a multi-quarter effort.
- xyse53 3y ago
  Initially, I wanted to reply to you and say it's unfair to judge the person as incompetent; I still think it's a bit unfair because they develop a specialized library and their choice of hosting shouldn't need to be modernized at every step. However, reading through their response thread [0] I do think they aren't appearing particularly mature. It feels like they are refusing to characterize what is actually going on and digging in and escalating this defensive mindset. [0] https://gmplib.org/list-archives/gmp-devel/2023-June/thread.html https://gmplib.org/list-archives/gmp-devel/2023-June/thread....
  - retrocryptid 3y ago
    So... don't ascribe to incompetence what can easily be explained by malice?
  - retrocryptid 3y ago

notamy 3y ago

From the description, it sounds like someone(s) is automatically downloading GMP as a part of a CI process. GHA being the source would explain why it's coming from "several hundred IP addresses owned by Microsoft cooperation (sic)," as well as (maybe) why GH puts the blame on the GMP project for not being able to handle the load. Edit: This seems to be the responsible project/issue: https://github.com/BtbN/FFmpeg-Builds/issues/278 https://github.com/BtbN/FFmpeg-Builds/issues/278

Aardwolf 3y ago
Github could cache it then to improve this?
- rwmj 3y ago
  The title is a bit confusing. It seems the GMP project has a self-hosted Mercurial repo (https://gmplib.org/devel/repo-usage https://gmplib.org/devel/repo-usage) which is being hit by a user at Microsoft that they've somehow tied back to a single github account. I'm sure it's going to be CI or mirroring or some other automated process.
  - cxr 3y ago
    Why is this a response to the person you sent this reply to?
    - rwmj 3y ago
      Github can't easily cache a remotely hosted repo that some random CI test (essentially a script) accesses, at least not without some very hairy proxying.
      - cxr 3y ago
        Is GitHub's CI infrastructure (and cloud compute, generally) not hairy? (A basic attempt at straightforward proxying seems like it would be the _least_ hairy aspect of the stack, besides.)

samwillis 3y ago

So another classic case of someone's constant integration system DoS'ing an open source project. We've all become so blinded by big tech and how they operate we fail to see how our actions can affect smaller projects we depend on CI providers really should do more to prevent and mitigate these when they happen. They should have outbound firewalls, and the ability to request a rate limit on IPs. Having to resort to a ban hammer when people depend on your tool is a shame.

jeltz 3y ago
And in general just making caching stuff easier. I feel like it is unnecessarily complicated for example to cache apt-get in Gitlab which I assume makes most people not do it. https://gitlab.com/gitlab-org/gitlab-runner/-/issues/991#note_126864314 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/991#not...
deleted 3y ago
[deleted]

progval 3y ago

Initial email mentioning it on GMP's mailing list: https://gmplib.org/list-archives/gmp-discuss/2023-June/006906.html https://gmplib.org/list-archives/gmp-discuss/2023-June/00690... Reply from Github: https://gmplib.org/list-archives/gmp-devel/2023-June/006162.html https://gmplib.org/list-archives/gmp-devel/2023-June/006162....

CtrlAltDelete51 3y ago
That seems like a perfectly reasonable response. They even provided the project so GMP could reach out.
- WastingMyTime89 3y ago
  That’s in line with the response I’m used to get from cybersecurity teams at the large company I work for. It is neither reasonable nor appropriate in tone but I have got used to it being apparently the best they can do/can be bothered to write.
- burkaman 3y ago
  It's totally reasonable to say that it isn't malicious and they aren't going to stop it, but it's unreasonable and unhelpful to add "seems like your servers just suck". They also missed or failed to mention that there are 700 forks of the project running the same thing at the same time.
  - 542458 3y ago
    I’m a bit confused as to why 700 automated clone requests per day would be an unreasonable amount of traffic for a project like GMP. That doesn’t sound like much traffic to me, especially since they reportedly have a 24 core CPU with 256GB of RAM. Is cloning a mercurial repo highly server intensive?
    - matheusmoreira 3y ago
      I'm confused as well. Seems like a negligible amount of traffic at first glance. The administrator mentioned compression, could it really have such a big impact?

htpmnxc 3y ago

[flagged]

666satanhimself 3y ago

[dead]

judofyr 3y ago

More context: https://github.com/BtbN/FFmpeg-Builds/issues/278 https://github.com/BtbN/FFmpeg-Builds/issues/278 The FFmpeg-Builds repo has a GitHub Actions Workflow which clones the Mercurial repo. However, this runs as a (daily?) cronjob. In addition, this repo has 700 forks and now all of them are running the same workflow. This is out of control for the original author of the repo as there’s no way to change those 700 forks… EDIT: Also relevant is that there’s no official GMP mirror at any of the major code hosting sites (GitHub, GitLab) etc.

tremon 3y ago
[flagged]
deleted 3y ago
[deleted]
jeroenhd 3y ago
I'm confused by this statement: > GMP is at this point the only dependency that does not offer a sane way to clone its repository. The latest release is from years ago and https://gmplib.org/download/gmp/gmp-6.2.1.tar.lz https://gmplib.org/download/gmp/gmp-6.2.1.tar.lz should contain every file necessary. Why would you need access to the commit history in a build script? In fact, why would you need to download a fresh copy of a dependency that almost never gets updated from upstream? Surely Github can do some kind of caching. I wonder what will happen now that their build servers can't even reach the Mercurial repo anymore. I suppose the problem should quickly correct itself soon with all the failing builds.
- tedunangst 3y ago
  The idea that you build software from a fixed archive is passé. Everything must be a git clone. Or if they don't use git, you muddle along with hg.
  - rwmj 3y ago
    Yes and no. The other side of the argument is that a tarball offers a point in time snapshot of the code that is smaller, and easier to download since there's no need to interact with a complex internet service. Also easier to cache, mirror and back up a single file - major distros appreciate this because we have to mirror sources into build systems and provide sources on demand to satisfy the GPL. There are of course disadvantages to the tarball approach (like, no history, harder to patch) but sometimes the simpler technology can be more robust.

aseipp 3y ago

The repository in question, based on the email from Mike Blacker, sent to gmp-devel[1]: https://github.com/BtbN/FFmpeg-Builds https://github.com/BtbN/FFmpeg-Builds There's a bunch of build variants here, for various platforms, in different ways. Classic "you have to create the cartesian product of all variables" approach. Here is the single build matrix for a single "Build FFMPeg" job: https://github.com/BtbN/FFmpeg-Builds/actions/runs/5303511170 https://github.com/BtbN/FFmpeg-Builds/actions/runs/530351117... You can look through the code and see that each of these will trigger a download. The developer also added an interesting commit today, stopping forks from having the CI work instantly. Why? Because by default they would all share the same cron entry, and therefore launch all at the same time every night. So now new forks will need to go in and tweak/spread the Cron timer around if they want builds of their own https://github.com/BtbN/FFmpeg-Builds/commit/78191a73a6a1959d13a62a4d6adb9ddeffa18522 https://github.com/BtbN/FFmpeg-Builds/commit/78191a73a6a1959... But there are already over 700 forks of this repository, so the current path is unsustainable. So in short: - This isn't an "attack", and it's unclear of the direct volume, but anyway, it doesn't matter because ultimately gmplib.org had strain caused by it. - GitHub correctly identified the responsible user when asked, and it did not in any way seem malicious, but certainly excessive. - The build/CI infrastructure for this project definitely hammers the project more than it probably should, exact specifics aside. - The owner of the project put in place a small (future) mitigation for future forks, but for now IP banlists will probably remain, I'd guess? - Please try to cache things more aggressively in your build jobs. [1] https://gmplib.org/list-archives/gmp-devel/2023-June/006162.html https://gmplib.org/list-archives/gmp-devel/2023-June/006162....

folkrav 3y ago
> It already is cached by the docker image. The problem why it's SO MANY requests is because hundreds of people have forked this repository, and now at the same time (due to the workflow cron) start an image build. 99% of the days, it'll just use the cached image and not cause any network requests whatsoever. > If it builds new images, it'll be at max 10 requests to fetch the latest snapshot in parallel from my end (that's how many parallel jobs Github allows for free), which shouldn't be an issue. I've already switched GMP back to use an outdated release tarball because of how annoyingly flakey their mercurial server is. Straight from the repo author. Looking at the workflow, not sure how it could really fetch less than it already does.

runeks 3y ago

Sounds like GitHub actions should be caching clones of foreign Git repos. People already trust GH CI enough to run their code, so why not trust their cache as well?

coder543 3y ago
FWIW, it’s not a git repo, it’s a mercurial repo.
- capitainenemo 3y ago
  And even if it was a git repo, seems this would still be a problem if everything wasn't on github. I guess the solution is to put all code on github...
  - capitainenemo 3y ago
    BTW, for anyone reading this later, I asked some mercurial folks and putting an http cache in front of clones does indeed work just fine, although using the repo bundles is more performant (but might not be something you could ask folks to do). Given this repo seems fairly low activity (last commit 7 months ago) I doubt there'd be much of a problem with caching reads. Although given how they are using this repo, they'd probably be better off with a shallow clone, or a subversion proxy, or a snapshot tarball...likely just need a set of files..
teraflop 3y ago
In this case, it's a foreign Mercurial repository, not Git. And remember that users' builds can contain arbitrary custom shell scripts that perform "git clone" or "hg clone" commands. It could be very, very difficult for Github to build an automated system that meddles with the behavior of those scripts to introduce caching, while guaranteeing not to break anything.

maxmcd 3y ago

I thought people sometimes used forks just like another form of star. Save the repo for later review. I have so rarely seen any commits or changes when someone forks my public github repos. It's weird to think about github actions crons in the context of this pattern. A casual "I'll mark this to check out later" could result in lots of billable computation.

beardog 3y ago
Disabling jobs for new forks until a commit is made in the fork or an option is toggled on the settings page would be a fix, although maybe i'm missing something. I often had experiences where people would fork my repos and I would get hopeful that someone might contribute, but it turns out that they just treat forking as bookmarking. Github running actions for forks wrongly assumes that users use the fork feature for its original purpose.

orf 3y ago

Why does forking cause a GitHub actions schedule to run? This is pretty unexpected. I wonder what % of actions builds are useless runs like this? Edit: to the shadowbanned guy who hasn’t realised he’s been shadowbanned for years, that’s not correct.

bootloop 3y ago
It's the reason you get random build failure notifications and warning emails that your (free) credits are running out, etc.. Took me a while to figure out what happened the first time I ran into this.
eddythompson80 3y ago
It’s a cron workflow
- orf 3y ago
  Yes?
  - eddythompson80 3y ago
    Why would you expect a cron task to work differently because a repo is a fork or not? Maybe Actions/Workflows shouldn’t be enabled by default on forks, which used to be the case but that has its problems too. You can disable actions on your forks.
    - orf 3y ago
      Why would I expect pressing the fork button would suddenly enable redundant and, in aggregare, expensive cronjobs? Most people fork to make a pull request and then never delete the fork after. Why are any scheduled actions required for that?
      - eddythompson80 3y ago
        It depends on how you use the fork and what you use it for. That’s up to the project to decide, not GitHub. Most workflows are triggered on an update-event, like git push or create a tag or branch or pull request etc. You want them to trigger on your fork separately from the main repo because (a) the secrets aren’t shared between forks and main repo (b) you want to catch linters, automated tests, build issues and all the other things workflows are used for before you create a pull request if possible. Cron tasks are the odd exception. You can use them to update dependencies in your repo, or track an external dependency and rebuild your stuff with it, or nightly integration tests with external APIs, etc. Maybe they could enable all actions but cron actions, but then that’s just as equally confusing. I’d rather GitHub stay out of deciding what’s “expensive” and what’s not. Either enable all actions on forks by default, or disable them by default. They decided to enable them by default and give each user however many minutes of free actions time. I’m sure there is a better upsell opportunity there too.

nmlryc 3y ago

The whole thread is curated yet again (as all Microsoft criticism threads are). Shame on everyone who does not appreciate the volunteer work on gmp and ridicules their servers.

jeroenhd 3y ago
Microsoft isn't to blame here, other than being kind of a dick diminishing some other project's server. The same kind of attack could just as easily come from Gitlab or any other service that allows easy CI/CD cloning. The problem started with someone making a poorly built automated build script for another project, and that build script getting forked hundreds of times, triggering a huge load all by the lack of spread in the upstream build script. This is nothing compared to an actual DDoS attack, which Microsoft was partially blamed for in the previous email. I can see why someone over at MS would be annoyed to find out the supposed DDoS was just 700 requests.

deleted 3y ago

[deleted]

agilob 3y ago

>(Our infrastructure is pretty resilient with powerful server-class hardware and great connectivity to the Internet.) You should totally move to MS Azure!

zX41ZdbW 3y ago

> I've already switched GMP back to use an outdated release tarball because of how annoyingly flakey their mercurial server is. We had a similar problem. Step one - don't allow submodules from outside of github.com: https://github.com/ClickHouse/ClickHouse/blob/master/utils/check-style/check-style#L246 https://github.com/ClickHouse/ClickHouse/blob/master/utils/c... If there is a dependency from external git repository, it can be forked to github.com. But we still have a problem with: - Docker Hub; - Cargo. Sometimes they are offline. Also we should not use them to avoid supply-chain attacks. The solution probably should be: - copy all docker images as tarballs; - or use ECR. Still not sure what to do with Cargo: https://github.com/ClickHouse/ClickHouse/issues/48575 https://github.com/ClickHouse/ClickHouse/issues/48575

rhdunn 3y ago

IIUC, you can add "on" triggers to the GitHub Actions to only build on the original repository. That would prevent this issue. Making the change in this repository would require having the forks updated, so maybe the owner can then reach out to the owners of the forks asking them to update.