10 ms·

Debian Git Monorepo

222 points by pabs3 3y ago

dijit 3y ago
I was always surprised that Perforce didn't have anything for open source, Git is basically eating their lunch because monorepo's suck so much with git; so people do things in a more fragmented and git-friendly way. Not that I really want to see a proprietary product succeed, but it's somewhat surprising that: A) We don't have anything better B) Perforce isn't trying to gain adoption by giving its software for free to distro maintainers C) People keep coming up with different paradigms that attempt to solve problems monorepos solve inherently.
- maccard 3y ago
  Perforce doesn't need to because nothing really threatens it. They're not interested in becoming more known, because they don't need to be. I can't imagine Debian of all projects using a proprietary VCS though.
- xprn 3y ago
  I would actually think that monorepos suck with Github and others like that. Looking at the Linux kernel for example, that seems to me like a well executed monorepo outside of Github. From what I’ve heard and read, they specifically use Git and mailing lists instead of Github or others because Github wouldn’t work for what Linux is (aside from as a mirror). There might very well be some context I’m missing, but that’s what I understand from that side at least.
  - fragmede 3y ago
    they (well, Linus) invented git for their purposes, and not the other way around. They use git and mailing lists because that's how they work, and that's how they worked before git existed. git was built for them.
  - jorvi 3y ago
    Github / Gitlab would work great and reduce friction for kernel contribution. The refusal to switch or provide the alternate path is mostly inertia.
    - skydhash 3y ago
      Sometimes friction is necessary for quality. More often than not, I’ve seen github repos where issue is treated as a project manager and there are lots of drive-by PRs.
    - zer00eyz 3y ago
      I think you have a story eyed view of github... You might want to go read Linus own thoughts on the matter in the linux GitHub backup repo Git is a tool built for a project of the kernels scope, scale and organization. Github is a thin web interface over top of that, it cuts some corners here and there and gets opinionated about how you should manage code (pull requests). Think of it this way: most git hub projects end up with a monotonic output... the kernel isnt that. Between the current version someone is using, the next version that is being developed and the older versions getting back ports there's a lot going on there. Much more than GitHub and a pull request would cover.
      - jorvi 3y ago
        Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. People see a bit of code they want to improve, they improve it, and then instead of just opening a PR (and perhaps iterating on it), now they have to learn a weird ancient e-mail workflow and will probably get chewed out for messing it up. No one is saying to take away mail-in patches but it is positively archaic.
        zer00eyz 3y ago
        >> Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. Is it friction? Or is it a filter? You might remember being a kid and there was the sign in front of the ride that said "you must be at least this high to ride".... The kernel dev process isnt for casuals. It's designed that way. There's a lot of folks out there who have popular projects on GitHub who are over the endless stream of BS from AI generated pull requests. You should really dig in deep to what goes on with the kernel, the work flow, why it is that way and why GitHub is outright incapable of supporting kernel dev (there are reasons).... Your going to look at git in a very different way and many of githubs features are gonna feel on par with linkedin adding twitch style videos and zoom adding mail features...
- codesnik 3y ago
  from what I've heard, git become much friendlier with monorepos in recent years because of microsoft involvement. Windows repo, pretty sizeable, is now using git, and it supposedly works even without strange things like vfs for git.
  - redleader55 3y ago
    It's not only Windows that uses Git at Microsoft, but Sharepoint and Office (which includes the on-prem version of SharePoint). In terms of repo size Windows and Office are similar. I was part of the team that migrated Sharepoint from a Perforce clone to Git and helped build the tooling to allow Office to move as well. VFS for Git [1] and Scalar [2] are really good pieces of software. [1] - https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit [2] - https://github.com/microsoft/scalar https://github.com/microsoft/scalar
    - silverwind 3y ago
      I hope one of these "large file/many files" extensions becomes part of git core eventually. They are too much of a hassle to deal with and many servers don't even support them.
- 4ad 3y ago
  > Perforce isn't trying to gain adoption by giving its software for free to distro maintainers Perforce has been giving its software for free to OSS projects since forever. FreeBSD used to use Perforce, for example. OSS people and projects don't want Perforce, not the other way around.
  - bluGill 3y ago
    Freebsd developers used perforce but the official repo was cvs and somehow the two synced.
- wongarsu 3y ago
  For better or for worse, Perforce seems to see itself mainly in the niche of having binary or media files in source control. They are busy promoting themselves in game dev, automotive, etc, where this is a common pain point. Maybe they should do more to highlight their strengths with regards to monorepos, but apparently their marketing hasn't identified that as a strength, or thinks it isn't a worthwhile market.
  - Xeamek 3y ago
    Games I understand, but what makes automotive a place that uses a lot of binary files?
    - francocalvo 3y ago
      Probably tons of 3D models
nolist_policy 3y ago
April fool aside, what I really want is a Debian git monorepo of submodules, where each submodule points to the upstream git repository.
- daghamm 3y ago
  What do you think of the openbsd model? It has a mono repo for the os and a ports repo for everything else (I think)
  - ninjin 3y ago
    Yes and no. OpenBSD uses four primary repositories: ports, src, www, and xenocara. CVS change announcements are merged for all but ports and go out on two mailing lists: source-changes@ and ports-changes@. Patches are discussed on tech@ and ports@ (sometimes also on bugs@). Also, some key software such as OpenSSH, OpenSMTPD, etc. is developed out-of-tree and lifted into the main train when appropriate. I am not a developer, but I do follow development somewhat closely.
- XorNot 3y ago
  You should check out dgit[1] which pretty much does exactly what you're looking for. It's an excellent way (with a little bit of pbuilder scripting) to patch Debian packages in place with sensible version numbering. [1] https://wiki.debian.org/DgitFAQ https://wiki.debian.org/DgitFAQ
- tazjin 3y ago
  Why use submodules when you can properly vendor the upstream git, and export/import commits without breaking hashes on either side? https://github.com/josh-project/josh https://github.com/josh-project/josh We've been using josh at TVL for years and it's just amazing.
  - jcelerier 3y ago
    How does that work with upstreams that often needs patches?
ctrw 3y ago
I wish that half the executive initiatives I've seen ended the same way as this post.
netol 3y ago
I wonder how many xz-like backdoors could be in there.
G3rn0ti 3y ago
I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?
- frizlab 3y ago
  Glad to see I’m not the only one thinking that.
- paulgdp 3y ago
  NixOS uses a monorepo and I think everyone loves it. I love being able to easily grep through all the packages source code, and there's regularly PRs that harmonizes conventions across many packages. Nixpkgs doesn't include the packaged software source code, so it's a lot more practical than what Debian is doing. Creating a whole distribution often requires changes synchronized across many packages, so it really makes things simpler. https://github.com/NixOS/nixpkgs https://github.com/NixOS/nixpkgs I think it's important to add that they both have the biggest number of packaged software and the most up-to-date software of all distributions even though they have far fewer maintainers than Debian. https://repology.org/repositories/graphs https://repology.org/repositories/graphs
  - bjornsing 3y ago
    > Nixpkgs doesn't include the packaged software source code, so it's a lot more practical than what Debian is doing. This is the key difference. OpenWRT is a “monorepo” too, but it downloads tar balls etc of the upstream software. Makes sense. Putting the source code of the upstream software in a monorepo sounds like a nightmare…
  - thegeekpirate 3y ago
    > I think it's important to add that they both have the biggest number of packaged software and the most up-to-date software of all distributions even though they have far fewer maintainers than Debian. Only because of https://www.reddit.com/r/NixOS/comments/zp95a2/comment/j5ko9kp/ https://www.reddit.com/r/NixOS/comments/zp95a2/comment/j5ko9... I also had plenty of issues with either packages not being available or not building, last I tried. At least with the AUR, it's generally only the latter you have to worry about ;)
bobim 3y ago
That would be ok if git gave the option to track branches in subfolders and leave other branches alone. I had a look at submodules but I'm not clever enough to understand how it could help me.
- jFriedensreich 3y ago
  Do you mean worktrees? https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree
  - bobim 3y ago
    Thanks, this is the pointer I was waiting for!
whizzter 3y ago
Annoyed reading this one the 2nd but also a tad fooled because I thought of this in the context of a nuclear-option remedy to the XZ fiasco that's still lingering in all our heads.
jFriedensreich 3y ago
git clone --sparse --filter=blob:none --depth=1
alkonaut 3y ago
I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB? To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third party code today, which would be the code merged into this hypothetical monorepo? Why?
- richardwhiuk 3y ago
  Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.
  - eichin 3y ago
    it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)
    - richardwhiuk 3y ago
      I meant what upstream Debian includes.
- tazjin 3y ago
  This includes the sources of all the Debian packages. In general this is a good idea, and it is close to how nixpkgs works (which is, in some sense, a "distributed monorepo"), but git, the Debian tooling and so on wouldn't actually be able to support a good workflow here.
  - Zambyte 3y ago
    > it is close to how nixpkgs works Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition. The Nix equivalent to this joke would be storing all of the package code directly in the nixpkgs repo, instead of storing links to the source code. Debian would ba able to support the same flow as Nix just fine. You can just use source packages[0] [0] https://wiki.debian.org/Packaging/SourcePackage https://wiki.debian.org/Packaging/SourcePackage
deleted 3y ago
[deleted]
miah_ 3y ago
This needs a "April Fools" in the Headline so people don't waste time or energy reading it.
- keybored 3y ago
  Definitely unfair to post this on the second of April without a disclaimer.
  - diggan 3y ago
    > 2024-04-01 07:41 Seems to have been posted the 1st of April.
    - keybored 3y ago
      Post this to HN…
      - diggan 3y ago
        Ah :) Seems it was also submitted 1st of April: https://news.ycombinator.com/from?site=liw.fi https://news.ycombinator.com/from?site=liw.fi I guess someone else posted it again, and the spam-filter ignored the previous entry as it didn't get any traction then.
  - squigz 3y ago
    I think ideally we'd have month/year timestamps for every link submitted, but that's probably not going to happen, so you end up with instances like this, or months-old news being taken as new, etc. I think the better solution, which I'm trying to do myself, is noting the date on the article before reading
throwaway2037 3y ago
Putting aside that this is an April Fool's joke, I like the last part: > This time, I’m cruel to Git: can it handle a repository of this size? In 2009 it could not. In 2024 it can. That really hits home for me. Really: Think about this repo for a moment. 500 GB. 15 miiiiiiiiiiiiiiilion files. It is crazy to think that a "vanilla" git repo can handle it. Applause for the Git team!
- sitzkrieg 3y ago
  i used to work in git analytics and saw the most unimaginable repos. 500gb repos with one commit. repos where the source was in the commit message. repos with millions of commits consisting of bogus utf8, and so on. repos where they dont use branches, only remotes, etc etc. every usecase annihilated some old startup mvp code somewhere it felt like we wrote our own in memory client but even before that, stock git client would always finish, it was just a matter of how long you were willing to wait. maybe we were lucky overall
  - thaliaarchi 3y ago
    Do you still have a record of any of those weird case repos? Perhaps reported as issues somewhere? I’m working on a tool like git filter-repo and Reposurgeon combined and would love stress tests and edge cases like these.
- jayofdoom 3y ago
  I find the Gentoo ebuild repository, mirrored at http://github.com/gentoo/gentoo http://github.com/gentoo/gentoo often is a stress test for git clients due to the sheer number of commits. It's interesting to see the definition of scale stretched in all kinds of different directions.
zoobab 3y ago
You could easily do it with Subversion (SVN), and only clone the directory you need.
Ericson2314 3y ago
April fools or not, Debian actually does need this. NixOS's influence looms large.
wmanley 3y ago
Disappointed to read that this is an April fools. I honestly believe that practical application of software freedom is held back by tooling, and that through the use of their monorepo and build system Google can actually benefit more than normal users from software freedom. https://blog.williammanley.net/2020/05/25/unlock-software-freedom-one-by-using-better-tools.html https://blog.williammanley.net/2020/05/25/unlock-software-fr...
MenhirMike 3y ago
Curious, what challenges would git still have to deal with to work with big repos? 500 GB isn't all that much data in 2024, as long as you can resume a broken network connection on fetch (like using HTTP range requests if using http, and when using git LFS), and while 15 million files is a pain in most file systems because of all the overhead, a relatively modern NVMe SSD should be able to cope with it just fine, but I don't know enough about the innermost internals of git on how it manages files. Is there something that's inherently slow/expensive that doesn't scale well with the number of files? (Basically, I'm trying to find a non-rhetorical answer to the question of "Why not?")
- m_niedoba 3y ago
  Git does pretty well on large repos, when you use sparse checkout and Git LFS. It makes Git behave like a centralized VC system. https://www.anchorpoint.app/blog/scaling-git-to-1tb-of-files-with-gitlab-and-anchorpoint-using-git-lfs https://www.anchorpoint.app/blog/scaling-git-to-1tb-of-files...
deleted 3y ago
[deleted]
dTal 3y ago
It's worth noting that Void Linux (and maybe others) does indeed keep all its (largely declarative) package recipes and build tooling in a single git monorepo, although it does not keep copies of the upstream source because (package, version, url, checksum) is assumed to be idempotent, and some of the stated advantages are real: >Simpler collaboration: every package uses the same process, and the same tools, and it’ll be easier than ever to help with other people’s packages. >Enables distribution-wide changes in general: With all the source code for everything in one tree, in on repository, it’s feasible to make changes to Debian that affect many packages. For example, back in the day Debian took seven years to migrate /usr/doc to /usr/share/doc, and that can now be done in one commit.

222 points by pabs3 3y ago

I was always surprised that Perforce didn't have anything for open source, Git is basically eating their lunch because monorepo's suck so much with git; so people do things in a more fragmented and git-friendly way. Not that I really want to see a proprietary product succeed, but it's somewhat surprising that: A) We don't have anything better B) Perforce isn't trying to gain adoption by giving its software for free to distro maintainers C) People keep coming up with different paradigms that attempt to solve problems monorepos solve inherently.

maccard 3y ago
Perforce doesn't need to because nothing really threatens it. They're not interested in becoming more known, because they don't need to be. I can't imagine Debian of all projects using a proprietary VCS though.
xprn 3y ago
I would actually think that monorepos suck with Github and others like that. Looking at the Linux kernel for example, that seems to me like a well executed monorepo outside of Github. From what I’ve heard and read, they specifically use Git and mailing lists instead of Github or others because Github wouldn’t work for what Linux is (aside from as a mirror). There might very well be some context I’m missing, but that’s what I understand from that side at least.
- fragmede 3y ago
  they (well, Linus) invented git for their purposes, and not the other way around. They use git and mailing lists because that's how they work, and that's how they worked before git existed. git was built for them.
- jorvi 3y ago
  Github / Gitlab would work great and reduce friction for kernel contribution. The refusal to switch or provide the alternate path is mostly inertia.
  - skydhash 3y ago
    Sometimes friction is necessary for quality. More often than not, I’ve seen github repos where issue is treated as a project manager and there are lots of drive-by PRs.
  - zer00eyz 3y ago
    I think you have a story eyed view of github... You might want to go read Linus own thoughts on the matter in the linux GitHub backup repo Git is a tool built for a project of the kernels scope, scale and organization. Github is a thin web interface over top of that, it cuts some corners here and there and gets opinionated about how you should manage code (pull requests). Think of it this way: most git hub projects end up with a monotonic output... the kernel isnt that. Between the current version someone is using, the next version that is being developed and the older versions getting back ports there's a lot going on there. Much more than GitHub and a pull request would cover.
    - jorvi 3y ago
      Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. People see a bit of code they want to improve, they improve it, and then instead of just opening a PR (and perhaps iterating on it), now they have to learn a weird ancient e-mail workflow and will probably get chewed out for messing it up. No one is saying to take away mail-in patches but it is positively archaic.
      - zer00eyz 3y ago
        >> Likewise, I think you have a starry-eyed view on how much friction it creates which equals less contribution. Is it friction? Or is it a filter? You might remember being a kid and there was the sign in front of the ride that said "you must be at least this high to ride".... The kernel dev process isnt for casuals. It's designed that way. There's a lot of folks out there who have popular projects on GitHub who are over the endless stream of BS from AI generated pull requests. You should really dig in deep to what goes on with the kernel, the work flow, why it is that way and why GitHub is outright incapable of supporting kernel dev (there are reasons).... Your going to look at git in a very different way and many of githubs features are gonna feel on par with linkedin adding twitch style videos and zoom adding mail features...
codesnik 3y ago
from what I've heard, git become much friendlier with monorepos in recent years because of microsoft involvement. Windows repo, pretty sizeable, is now using git, and it supposedly works even without strange things like vfs for git.
- redleader55 3y ago
  It's not only Windows that uses Git at Microsoft, but Sharepoint and Office (which includes the on-prem version of SharePoint). In terms of repo size Windows and Office are similar. I was part of the team that migrated Sharepoint from a Perforce clone to Git and helped build the tooling to allow Office to move as well. VFS for Git [1] and Scalar [2] are really good pieces of software. [1] - https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit [2] - https://github.com/microsoft/scalar https://github.com/microsoft/scalar
  - silverwind 3y ago
    I hope one of these "large file/many files" extensions becomes part of git core eventually. They are too much of a hassle to deal with and many servers don't even support them.
4ad 3y ago
> Perforce isn't trying to gain adoption by giving its software for free to distro maintainers Perforce has been giving its software for free to OSS projects since forever. FreeBSD used to use Perforce, for example. OSS people and projects don't want Perforce, not the other way around.
- bluGill 3y ago
  Freebsd developers used perforce but the official repo was cvs and somehow the two synced.
wongarsu 3y ago
For better or for worse, Perforce seems to see itself mainly in the niche of having binary or media files in source control. They are busy promoting themselves in game dev, automotive, etc, where this is a common pain point. Maybe they should do more to highlight their strengths with regards to monorepos, but apparently their marketing hasn't identified that as a strength, or thinks it isn't a worthwhile market.
- Xeamek 3y ago
  Games I understand, but what makes automotive a place that uses a lot of binary files?
  - francocalvo 3y ago
    Probably tons of 3D models

nolist_policy 3y ago

April fool aside, what I really want is a Debian git monorepo of submodules, where each submodule points to the upstream git repository.

daghamm 3y ago
What do you think of the openbsd model? It has a mono repo for the os and a ports repo for everything else (I think)
- ninjin 3y ago
  Yes and no. OpenBSD uses four primary repositories: ports, src, www, and xenocara. CVS change announcements are merged for all but ports and go out on two mailing lists: source-changes@ and ports-changes@. Patches are discussed on tech@ and ports@ (sometimes also on bugs@). Also, some key software such as OpenSSH, OpenSMTPD, etc. is developed out-of-tree and lifted into the main train when appropriate. I am not a developer, but I do follow development somewhat closely.
XorNot 3y ago
You should check out dgit[1] which pretty much does exactly what you're looking for. It's an excellent way (with a little bit of pbuilder scripting) to patch Debian packages in place with sensible version numbering. [1] https://wiki.debian.org/DgitFAQ https://wiki.debian.org/DgitFAQ
tazjin 3y ago
Why use submodules when you can properly vendor the upstream git, and export/import commits without breaking hashes on either side? https://github.com/josh-project/josh https://github.com/josh-project/josh We've been using josh at TVL for years and it's just amazing.
- jcelerier 3y ago
  How does that work with upstreams that often needs patches?

ctrw 3y ago

I wish that half the executive initiatives I've seen ended the same way as this post.

netol 3y ago

I wonder how many xz-like backdoors could be in there.

G3rn0ti 3y ago

I know it‘s an April‘s fool. But I don’t get this recent trend of people arguing in favor of a monorepo (at my work place, too). It’s a nightmare to handle and use. What gives?

frizlab 3y ago
Glad to see I’m not the only one thinking that.
paulgdp 3y ago
NixOS uses a monorepo and I think everyone loves it. I love being able to easily grep through all the packages source code, and there's regularly PRs that harmonizes conventions across many packages. Nixpkgs doesn't include the packaged software source code, so it's a lot more practical than what Debian is doing. Creating a whole distribution often requires changes synchronized across many packages, so it really makes things simpler. https://github.com/NixOS/nixpkgs https://github.com/NixOS/nixpkgs I think it's important to add that they both have the biggest number of packaged software and the most up-to-date software of all distributions even though they have far fewer maintainers than Debian. https://repology.org/repositories/graphs https://repology.org/repositories/graphs
- bjornsing 3y ago
  > Nixpkgs doesn't include the packaged software source code, so it's a lot more practical than what Debian is doing. This is the key difference. OpenWRT is a “monorepo” too, but it downloads tar balls etc of the upstream software. Makes sense. Putting the source code of the upstream software in a monorepo sounds like a nightmare…
- thegeekpirate 3y ago
  > I think it's important to add that they both have the biggest number of packaged software and the most up-to-date software of all distributions even though they have far fewer maintainers than Debian. Only because of https://www.reddit.com/r/NixOS/comments/zp95a2/comment/j5ko9kp/ https://www.reddit.com/r/NixOS/comments/zp95a2/comment/j5ko9... I also had plenty of issues with either packages not being available or not building, last I tried. At least with the AUR, it's generally only the latter you have to worry about ;)

bobim 3y ago

That would be ok if git gave the option to track branches in subfolders and leave other branches alone. I had a look at submodules but I'm not clever enough to understand how it could help me.

jFriedensreich 3y ago
Do you mean worktrees? https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree
- bobim 3y ago
  Thanks, this is the pointer I was waiting for!

whizzter 3y ago

Annoyed reading this one the 2nd but also a tad fooled because I thought of this in the context of a nuclear-option remedy to the XZ fiasco that's still lingering in all our heads.

jFriedensreich 3y ago

git clone --sparse --filter=blob:none --depth=1

alkonaut 3y ago

I didn't lift an eyebrow at the headline which is apparently an april fools joke. But what is it that makes it so absurd that it works as a joke, which I'm missing? The size seems absurd (500GB) but why would the Debian source be 500GB? To me it's in no way obvious that "Debian" would include any third party packages repackaged by the distro maintainers. Are Debian people maintaining N source repositories for third party code today, which would be the code merged into this hypothetical monorepo? Why?

richardwhiuk 3y ago
Debian does include the third party packages. This is either a mirror of the git repository, or a tarball of the upstream code.
- eichin 3y ago
  it's simpler than that: it's just running `dpkg-source -x` on every `*.dsc` file it can find. So that's "all of the source used to build all of the debs" (using dists/stable/main/source/Sources.xz to get the list.) (I think then just a single commit of all of it; the fun bit would be doing this for each release and looking at the size of the diffs...)
  - richardwhiuk 3y ago
    I meant what upstream Debian includes.
tazjin 3y ago
This includes the sources of all the Debian packages. In general this is a good idea, and it is close to how nixpkgs works (which is, in some sense, a "distributed monorepo"), but git, the Debian tooling and so on wouldn't actually be able to support a good workflow here.
- Zambyte 3y ago
  > it is close to how nixpkgs works Not really...? Nix distributes package definitions that describe how to build and install the package. This is close to just about every other package manager, besides the fact that most don't have / use the description on how to build the package, and instead they just download a binary based on the package definition. The Nix equivalent to this joke would be storing all of the package code directly in the nixpkgs repo, instead of storing links to the source code. Debian would ba able to support the same flow as Nix just fine. You can just use source packages[0] [0] https://wiki.debian.org/Packaging/SourcePackage https://wiki.debian.org/Packaging/SourcePackage

deleted 3y ago

[deleted]

miah_ 3y ago

This needs a "April Fools" in the Headline so people don't waste time or energy reading it.

keybored 3y ago
Definitely unfair to post this on the second of April without a disclaimer.
- diggan 3y ago
  > 2024-04-01 07:41 Seems to have been posted the 1st of April.
  - keybored 3y ago
    Post this to HN…
    - diggan 3y ago
      Ah :) Seems it was also submitted 1st of April: https://news.ycombinator.com/from?site=liw.fi https://news.ycombinator.com/from?site=liw.fi I guess someone else posted it again, and the spam-filter ignored the previous entry as it didn't get any traction then.
- squigz 3y ago
  I think ideally we'd have month/year timestamps for every link submitted, but that's probably not going to happen, so you end up with instances like this, or months-old news being taken as new, etc. I think the better solution, which I'm trying to do myself, is noting the date on the article before reading

throwaway2037 3y ago

Putting aside that this is an April Fool's joke, I like the last part: > This time, I’m cruel to Git: can it handle a repository of this size? In 2009 it could not. In 2024 it can. That really hits home for me. Really: Think about this repo for a moment. 500 GB. 15 miiiiiiiiiiiiiiilion files. It is crazy to think that a "vanilla" git repo can handle it. Applause for the Git team!

sitzkrieg 3y ago
i used to work in git analytics and saw the most unimaginable repos. 500gb repos with one commit. repos where the source was in the commit message. repos with millions of commits consisting of bogus utf8, and so on. repos where they dont use branches, only remotes, etc etc. every usecase annihilated some old startup mvp code somewhere it felt like we wrote our own in memory client but even before that, stock git client would always finish, it was just a matter of how long you were willing to wait. maybe we were lucky overall
- thaliaarchi 3y ago
  Do you still have a record of any of those weird case repos? Perhaps reported as issues somewhere? I’m working on a tool like git filter-repo and Reposurgeon combined and would love stress tests and edge cases like these.
jayofdoom 3y ago
I find the Gentoo ebuild repository, mirrored at http://github.com/gentoo/gentoo http://github.com/gentoo/gentoo often is a stress test for git clients due to the sheer number of commits. It's interesting to see the definition of scale stretched in all kinds of different directions.

zoobab 3y ago

You could easily do it with Subversion (SVN), and only clone the directory you need.

Ericson2314 3y ago

April fools or not, Debian actually does need this. NixOS's influence looms large.

wmanley 3y ago

Disappointed to read that this is an April fools. I honestly believe that practical application of software freedom is held back by tooling, and that through the use of their monorepo and build system Google can actually benefit more than normal users from software freedom. https://blog.williammanley.net/2020/05/25/unlock-software-freedom-one-by-using-better-tools.html https://blog.williammanley.net/2020/05/25/unlock-software-fr...

MenhirMike 3y ago

Curious, what challenges would git still have to deal with to work with big repos? 500 GB isn't all that much data in 2024, as long as you can resume a broken network connection on fetch (like using HTTP range requests if using http, and when using git LFS), and while 15 million files is a pain in most file systems because of all the overhead, a relatively modern NVMe SSD should be able to cope with it just fine, but I don't know enough about the innermost internals of git on how it manages files. Is there something that's inherently slow/expensive that doesn't scale well with the number of files? (Basically, I'm trying to find a non-rhetorical answer to the question of "Why not?")

m_niedoba 3y ago
Git does pretty well on large repos, when you use sparse checkout and Git LFS. It makes Git behave like a centralized VC system. https://www.anchorpoint.app/blog/scaling-git-to-1tb-of-files-with-gitlab-and-anchorpoint-using-git-lfs https://www.anchorpoint.app/blog/scaling-git-to-1tb-of-files...

deleted 3y ago

[deleted]

dTal 3y ago

It's worth noting that Void Linux (and maybe others) does indeed keep all its (largely declarative) package recipes and build tooling in a single git monorepo, although it does not keep copies of the upstream source because (package, version, url, checksum) is assumed to be idempotent, and some of the stated advantages are real: >Simpler collaboration: every package uses the same process, and the same tools, and it’ll be easier than ever to help with other people’s packages. >Enables distribution-wide changes in general: With all the source code for everything in one tree, in on repository, it’s feasible to make changes to Debian that affect many packages. For example, back in the day Debian took seven years to migrate /usr/doc to /usr/share/doc, and that can now be done in one commit.