7 ms·

Thanks for the added context. If I may - what were the significant scale challenges for self hosted solution?

bayindirh 1y ago
I guess it's the CI/CD infrastructure. Pipeline and time requirement grows exponentially as the code supports more operating systems and configurations. I used a GitLab + GitLab Runner (docker) pipeline for my Ph.D. project which did some verification after every push (since the code was scientific), and even that took 10 minutes to complete even if it was pretty basic. Debian's some packages need more than three hours in their own CI/CD pipeline. Something like Mozilla Firefox, which is tested against regressions, performance, etc. (see https://www.arewefastyet.com https://www.arewefastyet.com) needs serious infrastructure and compute time to build in n different configurations (stable / testing / nightly + all the operating systems it supports) and then test at that scale. This needs essentially a server farm, to complete in reasonable time. An infrastructure of that size needs at least two competent people to keep it connected to all relevant cogs and running at full performance, too. So yes, it's a significant effort.
- notpushkin 1y ago
  I think the CI/CD infra stays intact here though? (and even then, I imagine GitHub Actions bill would be enormous for a project like Firefox)
  - bayindirh 1y ago
    I think it can be done half/half. Do some, well-defined builds at GitHub and pull in for testing. Another comment tells that some users needed 10+ minutes to get a lock to pass their tests through CI, so maybe some sanity tests can be offloaded to GitHub actions. I'm not claiming that my comment was 100% accurate, but they plan to move some of the CI to GitHub, at least.
    - TheDong 1y ago
      > but they plan to move some of the CI to GitHub, at least Really? I've seen no indication of that anywhere, and I'd be amazed if they did. They're not using github PRs, and github actions really fights against other development workflows... not to mention they already have invested a lot in TaskCluster, and specialized it to their needs. Where are you getting that from?
      - bayindirh 1y ago
        It was an, apparently very wrong, educated guess. Nothing more.
  - saghm 1y ago
    If the CI/CD is the most intensive part, it seems reasonable to move all of the other parts to a free provider to focus on the part that would be harder and more expensive to move. Even if they don't ever move any of the CI/CD over, I feel like I can understand the rationale for reducing the scope to just that rather than the source hosting. I've worked on plenty of projects with way less traffic than Firefox over the years that used GitHub for source hosting but alternate CI/CD; GitHub didn't even have built in CI for a while, so that was the only way to use it. Given the frequency I see comments on this site about Mozilla trying to do far too much rather than just focusing their efforts on core stuff like Firefox, I'm honestly a bit surprised that there aren't more people agreeing with this decision. Even with the other issues I have with Mozilla lately (like the whole debacle over the privacy policy changes and the extremely bizarre follow-up about what the definition of "selling user data" is), I don't see it as hypocritical to use GitHub while maintaining a stance that open solutions are better than closed ones because I think trying to make an open browser in the current era is a large and complicated goal for it to be worth it to set a high bar for taking on additional fights. Insisting on spending effort on maintaining their own version control servers feels like a effort that they don't need to be taking on right now, and I'd much rather than Mozilla pick their battles carefully like this more often than less. Trying to fight for more open source hosting at this point is a large enough battle that maybe it would make more sense for a separate organization focused on that to be leading the front in that regard; providing an alternative to Chrome is a big enough struggle that it's not crazy for them to decide that GitHub's dominance has to be someone else's problem.
    - notpushkin 1y ago
      Yeah, I agree that everything that helps reduce maintenance overhead is good for Mozilla (although I believe there’s more low-hanging fruits that could be addressed before that). I would love to see Mozilla moving to Codeberg.org (though I’d ask if they’re okay with it first) or something like that. Using GitHub is okay-ish? Personally, I frown upon it, but again I agree – it’s not the most important issue right now.
- arp242 1y ago
  > I guess it's the CI/CD infrastructure Your guess is wrong as Firefox doesn't use GitHub for any of that, and AFAIK there are no plans to either. The blog post linked in the top comment goes in to this in some detail, but in brief: git log, clone, diff, showing files, blame, etc. is CPU expensive. You can see this locally on large repo if you try something like "git log path/to/dir". Add to this all the standard requirements of running any server that needs to be 1) fast, and 2) highly available. And why bother when there's a free service available for you?
  - bayindirh 1y ago
    It was a guess and I never claimed it was 100% correct, and I'm happy to stand corrected. No hard feelings there.
    - tempaccount420 1y ago
      "I guess..." != "I'm guessing..."
      - bayindirh 1y ago
        That's new to me. Can you expand on that a little?
- jgraham 1y ago
  This is all true, but as the sibling says, not really related to the change discussed here. Firefox does indeed have a large CI system and ends up running thousands of jobs on each push to main (formerly mozilla-central), covering builds, linting, multiple testsuites, performance testing, etc. all across multiple platforms and configurations. In addition there are "try" pushes for work in progress patches, and various other kinds of non-CI tasks (e.g. fuzzing). That is all run on our taskcluster system and I don't believe there are any plans to change that.
jgraham 1y ago
Again, I can only comment from the perspective of a user; I haven't worked on the VCS infrastructure. The obvious generic challenges are availability and security: Firefox has contributors around the globe and if the VCS server goes down then it's hard to get work done (yes, you can work locally, but you can't land patches or ship fixes to users). Firefox is also a pretty high value target, and an attacker with access to the VCS server would be a problem. To be clear I'm not claiming that there were specific problems related to these things; just that they represent challenges that Mozilla has to deal with when self hosting. The other obvious problem at scale is performance. With a large repo both read and write performance are concerns. Cloning the repo is the first step that new contributors need to take, and if that's slow then it can be a dealbreaker for many people, especially on less reliable internet. Out hg backend was using replication to help with this [1], but you can see from the link how much complexity that adds. Firefox has enough contributors that write contention also becomes a problem; for example pushing to the "try" repo (to run local patches through CI) often ended up taking tens of minutes waiting for a lock. This was (recently) mostly hidden from end users by pushing patches through a custom "lando" system that asynchronously queues the actual VCS push rather than blocking the user locally, but that's more of a mitigation than a real solution (lando is still required with the GitHub backend because it becomes the places where custom VCS rules which previously lived directly in the hg server, but which don't map onto GitHub features, are enforced). [1] https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/replication.html#hgmo-replication https://mozilla-version-control-tools.readthedocs.io/en/late...
- monegator 1y ago
  why github and not codeberg? badwidth? $$$ from microsoft? (traffic, free training for copilot, ..)
  - Macha 1y ago
    I'm not sure codeberg has managed two 9s of uptime while I've been using it. Manageable when it's just a public mirror for occasional publishing of my small hobby projects, but I wouldn't recommend it for Firefox sized projects
PaulDavisThe1st 1y ago
If you provide an http based front end to git, one of the significant (newish) challenges of self hosting is dealing with AI bots/scrapers.

I guess it's the CI/CD infrastructure. Pipeline and time requirement grows exponentially as the code supports more operating systems and configurations. I used a GitLab + GitLab Runner (docker) pipeline for my Ph.D. project which did some verification after every push (since the code was scientific), and even that took 10 minutes to complete even if it was pretty basic. Debian's some packages need more than three hours in their own CI/CD pipeline. Something like Mozilla Firefox, which is tested against regressions, performance, etc. (see https://www.arewefastyet.com https://www.arewefastyet.com) needs serious infrastructure and compute time to build in n different configurations (stable / testing / nightly + all the operating systems it supports) and then test at that scale. This needs essentially a server farm, to complete in reasonable time. An infrastructure of that size needs at least two competent people to keep it connected to all relevant cogs and running at full performance, too. So yes, it's a significant effort.

notpushkin 1y ago
I think the CI/CD infra stays intact here though? (and even then, I imagine GitHub Actions bill would be enormous for a project like Firefox)
- bayindirh 1y ago
  I think it can be done half/half. Do some, well-defined builds at GitHub and pull in for testing. Another comment tells that some users needed 10+ minutes to get a lock to pass their tests through CI, so maybe some sanity tests can be offloaded to GitHub actions. I'm not claiming that my comment was 100% accurate, but they plan to move some of the CI to GitHub, at least.
  - TheDong 1y ago
    > but they plan to move some of the CI to GitHub, at least Really? I've seen no indication of that anywhere, and I'd be amazed if they did. They're not using github PRs, and github actions really fights against other development workflows... not to mention they already have invested a lot in TaskCluster, and specialized it to their needs. Where are you getting that from?
    - bayindirh 1y ago
      It was an, apparently very wrong, educated guess. Nothing more.
- saghm 1y ago
  If the CI/CD is the most intensive part, it seems reasonable to move all of the other parts to a free provider to focus on the part that would be harder and more expensive to move. Even if they don't ever move any of the CI/CD over, I feel like I can understand the rationale for reducing the scope to just that rather than the source hosting. I've worked on plenty of projects with way less traffic than Firefox over the years that used GitHub for source hosting but alternate CI/CD; GitHub didn't even have built in CI for a while, so that was the only way to use it. Given the frequency I see comments on this site about Mozilla trying to do far too much rather than just focusing their efforts on core stuff like Firefox, I'm honestly a bit surprised that there aren't more people agreeing with this decision. Even with the other issues I have with Mozilla lately (like the whole debacle over the privacy policy changes and the extremely bizarre follow-up about what the definition of "selling user data" is), I don't see it as hypocritical to use GitHub while maintaining a stance that open solutions are better than closed ones because I think trying to make an open browser in the current era is a large and complicated goal for it to be worth it to set a high bar for taking on additional fights. Insisting on spending effort on maintaining their own version control servers feels like a effort that they don't need to be taking on right now, and I'd much rather than Mozilla pick their battles carefully like this more often than less. Trying to fight for more open source hosting at this point is a large enough battle that maybe it would make more sense for a separate organization focused on that to be leading the front in that regard; providing an alternative to Chrome is a big enough struggle that it's not crazy for them to decide that GitHub's dominance has to be someone else's problem.
  - notpushkin 1y ago
    Yeah, I agree that everything that helps reduce maintenance overhead is good for Mozilla (although I believe there’s more low-hanging fruits that could be addressed before that). I would love to see Mozilla moving to Codeberg.org (though I’d ask if they’re okay with it first) or something like that. Using GitHub is okay-ish? Personally, I frown upon it, but again I agree – it’s not the most important issue right now.
arp242 1y ago
> I guess it's the CI/CD infrastructure Your guess is wrong as Firefox doesn't use GitHub for any of that, and AFAIK there are no plans to either. The blog post linked in the top comment goes in to this in some detail, but in brief: git log, clone, diff, showing files, blame, etc. is CPU expensive. You can see this locally on large repo if you try something like "git log path/to/dir". Add to this all the standard requirements of running any server that needs to be 1) fast, and 2) highly available. And why bother when there's a free service available for you?
- bayindirh 1y ago
  It was a guess and I never claimed it was 100% correct, and I'm happy to stand corrected. No hard feelings there.
  - tempaccount420 1y ago
    "I guess..." != "I'm guessing..."
    - bayindirh 1y ago
      That's new to me. Can you expand on that a little?
jgraham 1y ago
This is all true, but as the sibling says, not really related to the change discussed here. Firefox does indeed have a large CI system and ends up running thousands of jobs on each push to main (formerly mozilla-central), covering builds, linting, multiple testsuites, performance testing, etc. all across multiple platforms and configurations. In addition there are "try" pushes for work in progress patches, and various other kinds of non-CI tasks (e.g. fuzzing). That is all run on our taskcluster system and I don't believe there are any plans to change that.

jgraham 1y ago

Again, I can only comment from the perspective of a user; I haven't worked on the VCS infrastructure. The obvious generic challenges are availability and security: Firefox has contributors around the globe and if the VCS server goes down then it's hard to get work done (yes, you can work locally, but you can't land patches or ship fixes to users). Firefox is also a pretty high value target, and an attacker with access to the VCS server would be a problem. To be clear I'm not claiming that there were specific problems related to these things; just that they represent challenges that Mozilla has to deal with when self hosting. The other obvious problem at scale is performance. With a large repo both read and write performance are concerns. Cloning the repo is the first step that new contributors need to take, and if that's slow then it can be a dealbreaker for many people, especially on less reliable internet. Out hg backend was using replication to help with this [1], but you can see from the link how much complexity that adds. Firefox has enough contributors that write contention also becomes a problem; for example pushing to the "try" repo (to run local patches through CI) often ended up taking tens of minutes waiting for a lock. This was (recently) mostly hidden from end users by pushing patches through a custom "lando" system that asynchronously queues the actual VCS push rather than blocking the user locally, but that's more of a mitigation than a real solution (lando is still required with the GitHub backend because it becomes the places where custom VCS rules which previously lived directly in the hg server, but which don't map onto GitHub features, are enforced). [1] https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/replication.html#hgmo-replication https://mozilla-version-control-tools.readthedocs.io/en/late...

monegator 1y ago
why github and not codeberg? badwidth? $$$ from microsoft? (traffic, free training for copilot, ..)
- Macha 1y ago
  I'm not sure codeberg has managed two 9s of uptime while I've been using it. Manageable when it's just a public mirror for occasional publishing of my small hobby projects, but I wouldn't recommend it for Firefox sized projects

PaulDavisThe1st 1y ago

If you provide an http based front end to git, one of the significant (newish) challenges of self hosting is dealing with AI bots/scrapers.