15 ms·

Package managers keep using Git as a database, it never works out

784 points by birdculture 10mo ago

eviks 10mo ago
Indeed, the seductive nature of bad tools lying close to your hand - no need to lift your butt to get them!
twoodfin 10mo ago
What made git special & powerful from the start was its data model: Like the network databases of old, but embedded in a Merkle tree for independent evolution and verifiability. Scaling that data model beyond projects the size of the Linux kernel was not critical for the original implementation. I do wonder if there are fundamental limits to scaling the model for use cases beyond “source code management for modest-sized, long-lived projects”.
- amluto 10mo ago
  Most of the problems mentioned in the article are not problems with using a content-addressed tree like git or even with using precisely git’s schema. The problems are with git’s protocol and GitHub’s implementation thereof. Consider vcpkg. It’s entirely reasonable to download a tree named by its hash to represent a locked package. Git knows how to store exactly this, but git does not know how to transfer it efficiently.
  - mananaysiempre 10mo ago
    > Git knows how to store [a hash-addressed tree], but git does not know how to transfer it efficiently. Naïvely, I’d expect shallow clones to be this, so I was quite surprised by a mention of GitHub asking people not to use them. Perhaps Git tries too hard to make a good packfile?.. Meanwhile, what Nixpkgs does (and why “release tarballs” were mentioned as a potential culprit in the discussion linked from TFA) is request a gzipped tarball of a particular commit’s files from a GitHub-specific endpoint over HTTP rather than use the Git protocol. So that’s already more or less what you want, except even the tarball is 46 MB at this point :( Either way, I don’t think the current problems with Nixpkgs actually support TFA’s thesis.
quaintdev 10mo ago
I host my own code repository using Forgejo. It's not public. In fact, it's behind mutual tls like all the service I host. Reason? I don't want to deal with bots and other security risks that come with opening port to the world. Turns out Go module will not accept package hosted on my Forgejo instance because it asks for certificate. There are ways to make go get use ssh but even with that approach the repository needs to be accessible over https. In the end, I cloned the repository and used it in my project using replace directive. It's really annoying.
- xyzzy_plugh 10mo ago
  > There are ways to make go get use ssh but even with that approach the repository needs to be accessible over https. No, that's false. You don't need anything to be accessible over HTTP. But even if it did, and you had to use mTLS, there's a whole bunch of ways to solve this. How do you solve this for any other software that doesn't present client certs? You use a local proxy.
- agwa 10mo ago
  If you add .git to the end of your module path and set $GOPRIVATE to the hostname of your Forgejo instance, then Go will not make any HTTPS requests itself and instead delegate to the git command, which can be configured to authenticate with client certificates. See https://go.dev/ref/mod#vcs-find https://go.dev/ref/mod#vcs-find
- irusensei 10mo ago
  Have a look at Tailscale DNS and certs. Its gives you a valid cert through lets encrypt without exposing your services to the internet.
- baobun 10mo ago
  If you add the instance TLS cert (CA) to your trust store then go will happily download over https. It can be finicky depending on how you run go but I can confirm it works.
Zambyte 10mo ago
The issues with using Git for Nix seem to entirely be issues with using GitHub for Nix, no?
- femiagbabiaka 10mo ago
  Yeah, it's inclusion in here is baffling because none of the listed issues have anything to do with the particular issue nixpkgs is having.
- Rucadi 10mo ago
  I also got the same feeling from that, in fact, I would go as far as to say that nixpkgs and nix-commands integration with git works quite well and is not an issue. So the phrase the article says "Package managers keep falling for this. And it keeps not working out" I feel that's untrue. The most issue I have with this really is "flakes" integration where the whole recipe folder is copied into the store (which doesn't happen with non-flakes commands), but that's a tooling problem not an intrinsic problem of using git
ekjhgkejhgk 10mo ago
Do the easy thing while it works, and when it stops working, fix the problem. Julia does the same thing, and from the Rust numbers on the article, Julia has about 1/7th the number of packages that Rust does[1] (95k/13k = 7.3). It works fine, Julia has some heuristics to not re-download it too often. But more importantly, there's a simple path to improve. The top Registry.toml [1] has a path to each package, and once donwloading everything proves unsustainable you can just download that one file and use it to download the rest as needed. I don't think this is a difficult problem. [1] https://github.com/JuliaRegistries/General/blob/master/Registry.toml https://github.com/JuliaRegistries/General/blob/master/Regis...
- zahlman 10mo ago
  > 00000000-1111-2222-3333-444444444444 = { name = "REPLTreeViews", path = "R/REPLTreeViews" } ... Should it be concerning that someone was apparently able to engineer an ID like that?
  - adestefan 10mo ago
    It’s as random as any other UUID.
    - Severian 10mo ago
      Incorrect, only some UUIDs are random, specifically v4 and v7 (v7 uses time as well). https://en.wikipedia.org/wiki/Universally_unique_identifier https://en.wikipedia.org/wiki/Universally_unique_identifier > 00000000-1111-2222-3333-444444444444 This would technically be version 2, which would be built from the date-time and MAC address, and DCE security version. But overall, if you allow any yahoo to pick a UUID, its not really a UUID, its just some random string that looks like one.
      - ekjhgkejhgk 10mo ago
        > if you allow any yahoo to pick a UUID, its not really a UUID universally unique identifier (UUID) > 00000000-1111-2222-3333-444444444444 It's unique. Anyway we're talking about a package that doesn't matter. It's abandoned. Furthermore it's also broken, because it uses REPL without importing it. You can't even precompile it. https://github.com/pfitzseb/REPLTreeViews.jl/blob/969f04ce641d137ad9c4231f490a362a0fbdf8a2/src/REPLTreeViews.jl#L21 https://github.com/pfitzseb/REPLTreeViews.jl/blob/969f04ce64...
bencornia 10mo ago
> Grab’s engineering team went from 18 minutes for go get to 12 seconds after deploying a module proxy. That’s not a typo. Eighteen minutes down to twelve seconds. > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file. I have also had inconsistent performance with go get. Never enough to look closely at it. I wonder if I was running into the same issue?
- zahlman 10mo ago
  > needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Python used to have this problem as well (technically still does, but a large majority of things are available as a wheel and PyPI generally publishes a separate .metadata file for those wheels), but at least it was only a question of downloading and unpacking an archive file, not cloning an entire repo. Sheesh. Why would Go need to do that, though? Isn't the go.mod file in a specific place relative to the package root in the repo?
  - klooney 10mo ago
    Go's lock files arrived at around the same time as the proxy, before then you didn't have transitive dependencies pre baked.
- fireflash38 10mo ago
  How long ago were you having issues? That was changed in go 1.13.
c-linkage 10mo ago
This seems like a tragedy of the commons -- GitHub is free after all, and it has all of these great properties, so why not? -- but this kind of decision making occurs whenever externalities are present. My favorite hill to die on (externality) is user time. Most software houses spend so much time focusing on how expensive engineering time is that they neglect user time. Software houses optimize for feature delivery and not user interaction time. Yet if I spent one hour making my app one second faster for my million users, I can save 277 user hour per year. But since user hours are an externality, such optimization never gets done. Externalities lead to users downloading extra gigabytes of data (wasted time) and waiting for software, all of which is waste that the developer isn't responsible for and doesn't care about.
- ekjhgkejhgk 10mo ago
  I wouldn't call it tragedy of the commons, because it's not a commons. It's owned by microsoft. They're calculating that it's worth it for them, so I say take as much as you can. Commons would be if it's owned by nobody and everyone benefits from its existence.
  - TeMPOraL 10mo ago
    Still, because reality doesn't respect boundaries of human-made categories, and because people never define their categories exhaustively, we can safely assume that something almost-but-not-quite like a commons, is subject to an almost-but-not-quite tragedy of the commons.
    - reactordev 10mo ago
      An A- is still an A kind of thinking. I like this approach as not everything perfectly fits the mold.
    - ttiurani 10mo ago
      The whole notion of the "tragedy of the commons" needs to be put to rest. It's an armchair thought experiment that was disproven at the latest in the 90s by Elinor Ostrom with actual empirical evidence of commons. The "tragedy", if you absolutely need to find one, is only for unrestricted, free-for-all commons, which is obviously a bad idea.
miyuru 10mo ago
Funnily enough, I clicked the homebrew GitHub link in the post, only to get a rate limited error page from GitHub.
mikkupikku 10mo ago
People who put off learning SQL for later end up using anything other than a database as their database.
- redog 10mo ago
  SQL killed the set theory star
- groundzeros2015 10mo ago
  Is sql over ssh a thing?
  - yawaramin 10mo ago
    https://litestream.io/ https://litestream.io/
    - groundzeros2015 10mo ago
      A proprietary cloud subscription doesn’t seem like the right fit for this
      - yawaramin 10mo ago
        As opposed to a proprietary cloud git hosting platform?
steeleduncan 10mo ago
The other conclusion to draw is "Git is a fantastic choice of database for starting your package manager, almost all popular package managers began that way."
- saidinesh5 10mo ago
  I think the conclusion is more that package definitions can still be maintained on git/GitHub but the package manager clients should probably rely on a cache/db/a more efficient intermediate layer. Mostly to avoid downloading the whole repo/resolve deltas from the history for the few packages most applications tend to depend on. Especially in today's CI/CD World.
  - reactordev 10mo ago
    This is exactly the right approach. I did this for my package manager. It relies on a git repo branch for stable. There are yaml definitions of the packages including urls to their repo, dependencies, etc. Preflight scripts. Post install checks. And the big one, the signatures for verification. No binaries, rpms, debs, ar, or zip files. What’s actually installed lives in a small SQLite database and searching for software does a vector search on each packages yaml description. Semver included. This was inspired by brew/portage/dpkg for my hobby os.
    - pl4nty 10mo ago
      yaml definitions compiled to sqlite sounds pretty similar to winget, which is scaling very well. vector search is a cool idea - how much storage does it use per package?
  - pseufaux 10mo ago
    This is how WinGet works. It has a small SQLite db it downloads from a hosted url. The DB contains some minimal metadata and a url path to access the full metadata. This way WinGet only has to make API calls for packages it's actually interacting with. As a package manager, it has plenty of problems still, but it's a simple, elegant solution for the git as a DB issue.
ori_b 10mo ago
Alternatively: Downloading the entire state of all packages when you care about just one, it never works out. O(1) beats O(n) as n gets large.
- gruez 10mo ago
  Seems to still work out for apt?
  - ajb 10mo ago
    Not in the same sense. An analogy might be: apt is like fetching a git repo in which all the packages are submodules, so lazily fetched. Some of the package managers in the article seem to be using a monorepo for all packages - including the content. Others seem to have different issues - go wasn't including enough information in the top level, so all the submodules had to be fetched anyway. vcpkg was doing something with tree hashes which meant they weren't really addressible.
  - collinmanderson 10mo ago
    I consider apt kinds slow. I wish it were much faster.
born-jre 10mo ago
lol I see this as I plan on using Git for my thing store. https://github.com/blue-monads/potatoverse https://github.com/blue-monads/potatoverse
gjvc 10mo ago
sqlite seems to be ideal for a package manager
- sigwinch 10mo ago
  I feel like the rqlite people would have a lot to say about how to coordinate your installations, especially for the high-bandwidth non-desktop installs. https://news.ycombinator.com/item?id=45257349 https://news.ycombinator.com/item?id=45257349
- mirekrusin 10mo ago
  ...or scm [0] [0] https://fossil-scm.org https://fossil-scm.org
hk1337 10mo ago
I like Go but it’s dependency management is weird and seems to be centered around GitHub a lot.
- Hendrikto 10mo ago
  There is nothing tying Go to GitHub.
- rewgs 10mo ago
  Not at all. It can grab git repos (as well as work with other VCSs). There's just a lot of stuff on GitHub, hence your impression.
- andreashaerter 10mo ago
  It's mostly tradition rather than a hard requirement. Go has long supported vanity import paths: https://pkg.go.dev/cmd/go#hdr-Remote_import_paths https://pkg.go.dev/cmd/go#hdr-Remote_import_paths For example, we use Hugo to provide independent Go package URLs even though the code is hosted on GitHub. That makes migrating away from GitHub trivial if we ever choose to do so (Repo: https://github.com/foundata/hugo-theme-govanity https://github.com/foundata/hugo-theme-govanity; Example: https://golang.foundata.com/hugo-theme-dev/ https://golang.foundata.com/hugo-theme-dev/). Usage works as expected: go get golang.foundata.com/hugo-theme-dev Edit: Formatting
hogrug 10mo ago
The facts are interesting but the conclusion a bit strange. These package managers have succeeded because git is better for the low trust model and GitHub has been hosting infra for free that no one in their right mind would provide for the average DB. If it didn't work we would not have these massive ecosystems upsetting GitHub's freemium model, but anything at scale is naturally going to have consequences and features that aren't so compatible with the use case.
ifh-hn 10mo ago
So what's the answer then? That's the question I wanted answered after reading this article. With no experience with git or package management, would using a local client sqlite database and something similar on the server do?
- encom 10mo ago
  I quite like Gentoo's rsync based package manager. I believe they've used that since the beginning. It works well.
  - MarsIronPI 10mo ago
    To be clear though, the rsync trees come from a central Git repo (though it's not hosted on GitHub). And syncing from Git actually makes syncing faster.
- AaronFriel 10mo ago
  OCI artifacts, using the same protocol as container registries. It's a protocol designed for versioning (tagging) content addressable blobs, associating metadata with them, and it's CDN friendly. Homebrew uses OCI as its backend now, and I think every package manager should. It has the right primitives you expect from a registry to scale.
aniou 10mo ago
As side note. Maybe someone knows, why rust devs chose an already used name for language changes proposal? "RFC" was already taken and well-established and I simply refuse to accept that someone wasn't aware about Request For Comments - and if it was true and clash was created deliberately, then it was rude and arrogant. Every, ...king time, when I read something like "RFC 2789 introduced a sparse HTTP protocol." my brain suffers from a short-circuit. BTW: RFC 2789 is a "Mail Monitoring MIB".
- adastra22 10mo ago
  There are many, many RFC collections. Including many that predate the IETF. Some even predate computers.
  - aniou 10mo ago
    But they were in different domains. Here, we have a strong clash because Rust is positioning itself as secure system and internet language and computer and internet standard are already defined by RFC-s. So, it may be not uncommon, when someone would tell about Rust mechanisms, defined by particular RFC in context of handling particular protocol, defined by... well... RFC too. But not by rust-one. Not so smart, when we realize, that one of aspects of secure and reliable system is elimination of ambiguities.
- Conan_Kudo 10mo ago
  Ask them, don't ask us. They have a public interface, you can ask them to change the name to something unique.
frumplestlatz 10mo ago
Since ~2002, Macports has used svn or git, but users, by default, rsync the complete port definitions + a server-generated index + a signature. The index is used for all lookups; it can also be generated or incrementally updated client-side to accommodate local changes. This has worked fine for literally decades, starting back when bandwidth and CPU power was far more limited. The problem isn’t using SCM, and the solutions have been known for a very long time.
gethly 10mo ago
If we stopped using VCS to fetch source files, we would lose the ability to get the exact commit(understand as version that has nothing to do with the underlying VCS) of these files. Git, Mercurial, SVN.., github, bitbucket...it does not matter. Absolutely nobody will be building downloadable versions of their source files, hosted on who knows how "prestigious" domains, by copying them to another location just to serve the --->exact same content<--- that github and alike already provide. This entire blog is just a waste of time for anyone reading it.
- throwway120385 10mo ago
  Or you could just ship a tarball and an sha checksum.
  - gethly 10mo ago
    you could, in case you want to make only certain releases publicly available. but then, who wants to do that manual labour? we're talking mainstream here, not specific use cases.
- layer8 10mo ago
  And yet, that's pretty much how the Java world works (Maven repositories).
- forrestthewoods 10mo ago
  > This entire blog is just a waste of time for anyone reading it. Well that’s an extremely rude thing to say. Personally I thought it was really interesting to read about a bunch of different projects all running into the same wall with Git. I also didn’t realize that Git had issues with sparse checkouts. Or maybe author meant shallow? I forget.
encom 10mo ago
>[Homebrew] Auto-updates now run every 24 hours instead of every 5 minutes[...] That is such an insane default, I'm at a loss for words.
- croemer 10mo ago
  You mean the 5 minutes is insane, right?
- justsomehnguy 10mo ago
  Every time I things like I really want to punch people over TCP/IP. UDP wouldn't suffice.
dboon 10mo ago
I’m building Cargo/UV for C. Good article. I thought about this problem very deeply. Unfortunately, when you’re starting out, the idea of running a registry is a really tough sell. Now, on top of the very hard engineering problem of writing the code and making a world class tool, plus the social one of getting it adopted, I need to worry about funding and maintaining something that serves potentially a world of traffic? The git solution is intoxicating through this lense. Fundamentally, the issue is the sparse checkouts mentioned by the author. You’d really like to use git to version package manifests, so that anyone with any package version can get the EXACT package they built with. But this doesn’t work, because you need arbitrary commits. You either need a full checkout, or you need to somehow track the commit a package version is in without knowing what hash git will generate before you do it. You have to push the package update and then push a second commit recording that. Obviously infeasible, obviously a nightmare. Conan’s solution is I think just about the only way. It trades the perfect reproduction for conditional logic in the manifest. Instead of 3.12 pointing to a commit, every 3.x points to the same manifest, and there’s just a little logic to set that specific config field added in 3.12. If the logic gets too much, they let you map version ranges to manifests for a package. So if 3.13 rewrites the entire manifest, just remap it. I have not found another package manager that uses git as a backend that isn’t a terrible and slow tool. Conan may not be as rigorous as Nix because of this decision but it is quite pragmatic and useful. The real solution is to use a database, of course, but unless someone wants to wire me ten thousand dollars plus server costs in perpetuity, what’s a guy supposed to do?
- adrianN 10mo ago
  Before you managed to build a popular tool it is unlikely that you need to serve many users. Directly going for something that can serve the world is probably premature
  - EPWN3D 10mo ago
    The point is not "design to serve the world". The point is "use the right technology for your problem space".
  - dboon 10mo ago
kibwen 10mo ago
I think there's a form of survivorship bias at work here. To use the example of Cargo, if Rust had never caught on, and thereby gotten popular enough to inflate the git-based index beyond reason, then it would never have been a problem to use git as the backing protocol for the index. Likewise, we can imagine innumerable smaller projects that successfully use git as a distributed delta-updating data distribution protocol, and never happen to outgrow it. The point being, if you're not sure whether your project will ever need to scale, then it may not make sense to reinvent the wheel when git is right there (and then invent the solution for hosting that git repo, when Github is right there), letting you spend time instead on other, more immediate problems.
- stickfigure 10mo ago
  Right, this post may encourage premature optimization. Cargo, Homebrew, et al chose an easy, good-enough solution which allowed them to grow until they hit scaling limits. This is a good problem to have. I am sure there's value having a vision for what your scaling path might be in the future, so this discussion is a good one. But it doesn't automatically mean that git is a bad place to start.
- 8note 10mo ago
  im surprised nobody has made a common db for package managers, so cargo could use it without having to think about it
  - kibwen 10mo ago
    I mean, it's sort of the other way around. Cargo was built to be able to natively understand git-based dependencies, in the sense that you can bypass a crate registry and instead just point it directly at a git repo. That means that Cargo already had to have the ability to clone git repos, and so when it came to decide how to implement the index (which looks pretty similar to a git repo if you squint), choosing to use git required them to add literally no new dependencies and almost no new code. Let's also keep in mind that the use case mentioned in the OP is specifically about the index, which is just the datastructure that informs the version resolver how to resolve versions. When it came time to replace the git-based index, Cargo didn't replace it with a specialized database, it replaced it with HTTP endpoints (which are probably just backed by an off-the-shelf database). It's not clear what sort of specialized database would be useful to abstract this for other package managers.
nacozarina 10mo ago
successful things often have humble origins, it’s a feature not a bug for every project that managed to out-grow ext4/git there were a hundred that were well-served and never needed to over-invest in something else
PunchyHamster 10mo ago
The article conclusion is just... not good. There are many benefits to using Git as backend, you can point your project to every single commit as a version which makes testing any fixes or changes in libs super easy, it has built in integrity control and technically (sadly not in practice) you could just sign commits and use that to verify whether package is authentic. It being unoptimal bandwidth wise is frankly just a technical hurdle to get over it, with benefits well worth the drawback
0xbadcafebee 10mo ago
YOLO software engineering, the hallmark of the 21st century
cesarb 10mo ago
One of these is not like the others... > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. This article is mixing two separate issues. One is using git as the master database storing the index of packages and their versions. The other is fetching the code of each package through git. They are orthogonal; you can have a package index using git but the packages being zip/tar/etc archives, you can have a package index not using git but each package is cloned from a git repository, you can have both the index and the packages being git repositories, you can have neither using git, you can even not have a package index at all (AFAIK that's the case for Go).
- bobpaw 10mo ago
  I think the article takes issue not with fetching the code, but with fetching the go.mod file that contains index and dependency information. That’s why part of the solution was to host go.mod files separately.
- jayd16 10mo ago
  Even with git, it should be possible to grab the single file needed without the rest of the repo, but i'ts still trying to round a square peg.
  - skywhopper 10mo ago
    Honestly I think the article is a bit ahistorical on this one. ‘go get’ pulls the source code into a local cache so it can build it, not just to fetch the go.mod file. If they were having slow CI builds because they didn’t or couldn’t maintain a filesystem cache, that’s annoying, but not really a fault in the design. Anyway, Go improved the design and added an easy way to do faster, local proxies. Not sure what the critique is here. The Go community hit a pain point and the Go team created an elegant solution for it.
    - everforward 10mo ago
      I was thinking this too. I think it might be talking about operations like “go mod tidy” or update operations where it updates your go.mod/sum but doesn’t actually build the code. I would guess enterprise tools do a lot of checking whether there are updates without actually doing any building.
sghiassy 10mo ago
Use the git clone —shallow option and you’ll only download the most recent commits. Yeesh
- deleted 10mo ago
  [deleted]
- ZenoArrow 10mo ago
  Did you read the article? It references the server-side overhead for shallow clones.
  - sghiassy 10mo ago
    The “server’ cost you’re referencing is the CI system running git shallow then brew update on GitHubs CI servers.
    - cben 10mo ago
      That's not how I understood it. Full clones are big but simple — the server just sends all the packfiles. A first shallow clone needs some server work, but that's cachable, OK. But then on subsequent interactions between a git client that made a shallow clone various time ago and the git server, it's AFAIU actually expensive for the git server to compute the portion this particular client doesn't yet have. Intuitively, and very hand-wavingly, I suspect things could be improved by: (1) clients relaxing "exact depth" requests to "give me approximately N days of stuff, over-sending being OK", and server relaxing "minimal traffic" to roughly map time ranges to whole packfiles — CPU/traffic tradeoff. (2) allowing servers to under-send too (makes (1) tradeoffs easier), by client asking for missing parts right away and/or later — needs on-demand fetch ability to be transparent to user. With "promisor" mechanism in "partial clones" this sounds more realistic? (3) storing history/trees/blobs in entirely separate packfiles(?) I suspect recent years work on bitmaps & MIDX move in this direction, only less naively? I'm not saying Git can scale as well as a DB, but I do feel we sat on an effectively frozen Git format & protocol for a ~decade, and are now exploring more of the design space so hope future will be less clear-cut... And specifically, partial clones remove the hard "fully offline vs. centralized" dichotomy we long clinged to. Assuming you stay online (necessary anyway if you consider HTTP/DB), things that used to be up-front UX decisions can now be matters of perf tuning! * The most dramatic win is if you had to fetch info from every package's separate repo, like Go did. Then, a central DB/caching proxy can build global indexes, unlocking huge wins, no question. It's like "1+N" issues. However, most examples other than Go in the article talk of a single Git repo already storing a global view (still leaving opportunity for custom indexing and querying).
dleslie 10mo ago
GitHub is intoxicatingly free hosting, but Git itself is a terrible database. Why not maintain an _actual_ database on GitHub, with tagged releases? Sqlite data is paged and so you can get away with only fetching the pages you need to resolve your query. https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...
- jarofgreen 10mo ago
  This seems to be about hosting an Sqlite database on a static website like GitHub Pages - this can be a great plan, there is also Datasette in a browser now: https://github.com/simonw/datasette-lite https://github.com/simonw/datasette-lite But that's different from how you collect the data in a git repository in the first place - or are you suggesting just putting a Sqlite file in a git repository? If so I can think of one big reason against that.
  - dleslie 10mo ago
    Yes, I'm suggesting hosting it on GitHub, leveraging their git lfs support. Just treat it like a binary blob and periodically update with a tagged release.
    - jarofgreen 10mo ago
      It's not clear if you are suggesting accepting contributions to the SQLite file via PR from people (but accepting contributions is generally the point of why people put these on projects on GitHub). But if you are I wouldn't recommend it. PR's won't be able to show diff's. Worse, as soon as multiple people send a PR at once you'll have a really painful merge to resolve, and GitHub's tools won't help you at all. And you can't edit the files in GitHub's web UI. I recommend one file per record, JSON, YAML, whatever non-binary format you want. But then you get: * PR's with diff's that show you what's being changed * Files that technical people can edit directly in GitHub's web editor * If 2 people make PR's on different records at once it's an easy merge with no conflicts * If 2 people make PR's on the same record at once ... ok, you might now have a merge conflict to resolve but it's in an easy text file and GitHub UI will let you see what it is. You can of course then compile these data files into a SQLite file that can be served in a static website nicely - in fact if you see my other comments on this post I have a tool that does this. And on that note, sorry, I've done a few projects in this space so I have views :-)
xpressvideoz 10mo ago
The article lists Git-based wiki engines as a bad usage of Git. Can anybody recommend alternatives? I want something that can be self-hosted, is easily modified by text editors, and has individual page history, preferably with Markdown.
cbondurant 10mo ago
Admittedly, I try and stay away from database design whenever possible at work. (Everything database is legacy for us) But the way the term is being used here kinda makes me wonder, do modern sql databases have enough security features and permissions management systems in place that you could just directly expose your database to the world with a "guest" user that can only make incredibly specific queries? Cut out the middle man, directly serve the query response to the package manager client. (I do immediately see issues stemming from the fact that you cant leverage features like edge caching this way, but I'm not really asking if its a good solution, im more asking if its possible at all)
- brendoncarroll 10mo ago
  I personally think that this is the future, especially since such an architecture allows for E2E encryption of the entire database. The protocol should just be a transaction layer for coordinating changes of opaque blobs. All of the complexity lives on the client. That makes a lot of sense for a package manager because it's something lots of people want to run, but no one really wants to host.
- bob1029 10mo ago
  There are still no realistic ways to expose a hosted SQL solution to the public without really unhappy things occurring. It doesn't matter which vendor you pick. Anything where you are opening a TCP connection to a hosted SQL server is a non-starter. You could hypothetically have so many read replicas that no one could blow anyone else up, but this would get to be very expensive at scale. Something involving SQLite is probably the most viable option.
  - deleted 10mo ago
    [deleted]
  - IshKebab 10mo ago
    Feels like there's an opening in the market there. Why can't you expose an SQL server to the public? Also Stackoverflow exposes a SQL interface so it isn't totally impossible.
dromologist 10mo ago
We wanted to pull updated code in our undockerized instances when they were instantiated, so we decided to pull the code from GitHub. Worked out pretty well though after a thousand trials we got a 502 and now we're one step closer to being forced into a CD pipeline.
keithgroves 10mo ago
When building https:/enact.tools we considered this. I'm glad we didn't go this route.
jarofgreen 10mo ago
It's not just package manager who do this - a lot of smaller projects crowd source data in git repositories. Most of these don't reach the scale where the technical limitations become a problem. Personally my view is that the main problem when they do this is that it gets much harder for non-technical people to contribute. At least that doesn't apply to package managers, where it's all technical people contributing. There are a few other small problems - but it's interesting to see that so many other projects do this. I ended up working on an open source software library to help in these cases: https://www.datatig.com/ https://www.datatig.com/ Here's a write up of an introduction talk about it: https://www.datatig.com/2024/12/24/talk.html https://www.datatig.com/2024/12/24/talk.html I'll add the scale point to future versions of this talk with a link to this post.
- Hasnep 10mo ago
  Oh, this would have been great for a project I was working on a while ago! I'll have to keep it in mind for the future. Thanks for sharing
pizlonator 10mo ago
What is the alternative? "Use a database" isn't actionable advice because it's not specific enough
- yawaramin 10mo ago
  Use an SQLite database file, stream out delta updates to clients using zipped plaintext or Litestream or something.
holyknight 10mo ago
It’s basically the same thing that always happens when you choose a technology because it’s convenient rather than a great fit for your problem. Sooner or later, you’ll hit a wall. Just because you can cook a salmon in your dishwasher doesn’t mean you should.
BlueTemplar 10mo ago
Wait, isn't fossil based on sqlite ? Or does fossil itself still have the same issues ?
dwardu 10mo ago
Worst thing is when you’re in a an office and your pc along with other pcs pulls from git unauthenticated, then you get hit with api limits
mukundesh 10mo ago
Though not Github, worth mentioning Huggingface, which is also using git, but managing large files with their(?) xet protocol. https://huggingface.co/docs/hub/en/xet/index https://huggingface.co/docs/hub/en/xet/index
Ericson2314 10mo ago
The Nixpkgs example is not like the others, because it is source code. I don't get what is so bad about shallow clones either. Why should they be so performance sensative?
- ajb 10mo ago
  In a compressed format, later commits would be added as a delta of some kind, to avoid increasing the size by the whole tree size each time. To make shallow clones efficient you'd need to rewrite the compressed form such that earlier commits are instead deltas on later ones, or something equivalent.
- deleted 10mo ago
  [deleted]
- __MatrixMan__ 10mo ago
  It also seems like it's not git that's emitting scary creaks and groans, but rather GitHub. As much as it would be a bummer to forgo some of GitHub's nice-to-have features, I expect we could survive without some of it.
  - mindslight 10mo ago
    Furthermore, the issues given for nixpkgs are actually demonstrating the success of using git as the database! Those 20k forks are all people maintaining their own version of nixpkgs on Github, right? Each their own independent tree that users can just go ahead and modify for their own whims and purposes, without having to overcome the activation energy of creating their own package repository. If 83GB (4MB/fork) is "too big" then responsibility for that rests solely on the elective centralization encouraged by Github. I suspect if you could go and total up the cumulative storage used by the nixpkgs source tree distributed on computers spread throughout the world, that is many orders of magnitude larger.
    - __MatrixMan__ 10mo ago
      Agreed, nix really makes it easy to go from solving the problem for yourself to solving it for everybody. Not much else is easy, but when it comes to building an open source community, that criterion is a pretty powerful one.
teiferer 10mo ago
And this my friends is the reason why (only) focusing on CPU cycles and memory hierarchies is insufficient when thinking of the performance of a system. Yes they are important. But no level of low-level optimization will get you out of the hole that a wrong choice of algorithm and/or data structure may have dug you into.
iamwil 10mo ago
This sounds like a missing piece of software in the OSS world. If you have the inclination, you should write it.
- yawaramin 10mo ago
  People have: https://0install.net/ https://0install.net/
weiwenhao 10mo ago
For package management software that is rarely used, free is the biggest motivation.
drzaiusx11 10mo ago
One of the first things I did at my current place of employment was to detangle the mess of gemfile git dependencies and get them to adopt semver and an actual package repo. There were so many footguns with git dependencies in ruby we were getting taken down by friendly fire on the daily...
drzaiusx11 10mo ago
I'd add git gemfile dependencies to the list of languages called out here as well. It supports git repos, but in general it's a bad idea unless you are diligent with git tag use and disallow git tag mutability, which also assumes you have complete control of your git dependencies...
newswangerd 10mo ago
It’s always humbling when you go on the front page of HN and see an article titled “the thing you’re doing right now is a bad idea and here’s why” This has happened to me a few times now. The last one was a fantastic article about how PG Notify locks the whole database. In this particular case it just doesn’t make a ton of sense to change course. Im a solo dev building a thing that may never take off, so using git for plug-in distribution is just a no brainer right now. That said, I’ll hold on to this article in case I’m lucky enough to be in a position where scale becomes an issue for me.
- baobun 10mo ago
  The good news is you can easier avoid some of the pitfalls now even as you stick with it. Some good points in comments. I don't know if you rely on github.com but IMO vendor lock-in there might be a bigger issue which you can avoid.
  - newswangerd 10mo ago
    Yeah, I'm implementing a couple of things to make my life easier in the future. I don't use any github APIs and I'm setting up my clients to load the plugin repo URLs from my server so I can change them later if I need to. I want all of the resources my clients need to come from my domain name so I can move it around if I need to.
ekjhgkejhgk 10mo ago
Uncertain if this is OT, but given that the CCC is politically inspired organization, I hope not: One thing that still seems absent is awareness of the complete takeover of "gadgets" in schools. Schools these days, as early as primary school, shove screens in front of children. They're expected to look at them, and "use" them for various activities, including practicing handwriting. I wish I was joking [1]. I see two problems with this. First is that these devices are engineered to be addictive by way of constant notifications/distractions, and learning is something that requires long sustained focus. There's a lot of data showing that under certain common circumstances, you do worse learning from a screen than from paper. Second is implicitly it trains children to expect that anything has to be done through a screen connected to a closed point-and-click platform. (Uninformed) people will say "people who work with computers make money, so I want my child to have an ipad". But interacting with a closed platform like an ipad is removing the possibilities and putting the interaction "on rails". You don't learn to think, explore and learn from mistakes, instead you learn to use the app that's put in front of you. This in turn reinforces the "computer says no" [2] approach to understanding the world. I think this is a matter of civil rights and freedom, but sadly I don't often see "civil rights" organizations talk about this. I think I heard Stallman say something along these lines once, but other than that I don't see campaigns anywhere. [1] https://www.letterjoin.co.uk/ https://www.letterjoin.co.uk/ [2] https://youtu.be/eE9vO-DTNZc https://youtu.be/eE9vO-DTNZc
- AceJohnny2 10mo ago
  It looks like you commented on the wrong post, although I don't immediately see a front-page post about the ongoing Chaos Computer Congress.
  - kzrdude 10mo ago
    it's here https://news.ycombinator.com/item?id=46386211 https://news.ycombinator.com/item?id=46386211 (and it was last on the front page at the moment)
    - ekjhgkejhgk 10mo ago
grumbel 10mo ago
Do we have distributed databases that regular users can clone, modify and merge?
stephenlf 10mo ago
Omarchy
jama211 10mo ago
“It never works out” - hmm, seems like it worked out just fine, worked great to get the operation of the ground and when scale became an issue it was solvable by moving to something else. It served its purpose, sounds like it worked out to me.
- swiftcoder 10mo ago
  You appear to have glossed over the two projects in the list that are stuck due to architectural decisions, and don't have any route to migrate off of git-as-database?
  - hombre_fatal 10mo ago
    Be more specific because I just see a list of workarounds deployed once they had the scale to warrant them, supporting the OP’s claim.
    - swiftcoder 10mo ago
      Read the vcpkg section, it explicitly states that they have no horizontal on a solution. The nix section also doesn’t explain any potential solution.
      - deleted 10mo ago
        [deleted]
  - baobun 10mo ago
    The issues with nixpkgs stem from that it is a monorepo for all packages and doubling as an index. The issues are only fundamental with that architecture. Using a separate repo for each package, like the Arch User Repos, does not have the same problems. Nixpkgs certainly could be architected like that and submodules would be a graceful migration path. I'm not aware of discussion of this but guess that what's preventing it might be that github.com tooling makes it very painful to manage thousands of repos for a single project. So I think it can be a lesson not to that using git as a database is bad but that using github.com as a database is. PRs as database transactions is clunky and GitHub Actions isn't really ACID.
mikepurvis 10mo ago
The nix cli almost exclusively pulls GitHub as zipballs. Not perfect but certainly far faster than a real git clone.
- pxc 10mo ago
  That it supports fetching via Git as well as various via forge-specific tarballs, even for flakes, is pretty nice. It means that if your org uses Nix, you can fall back to distribution via Git as a solution that doesn't require you to stand up any new infra or tie you to any particular vendor, but once you get rolling it's an easy optimization to switch to downloading snapshots. The most pain probably just becomes from the hugeness of Nixpkgs, but I remain an advocate for the huge monorepo of build recipes.
  - mikepurvis 10mo ago
    Yes agreed. It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates, but I suspect the folks who would have to build and maintain that are all on gigabit internet connections and don’t feel the complexity is worth it.
    - pxc 10mo ago
      > It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates I think the snix¹ folks are working on something like this for the binary caches— the greater granularity of the content-addressing offers morally the same kind of optimization as delta RPMs: you can download less of what you don't need to re-download. But I'm not aware of any current efforts to let people download the Nixpkgs tree itself more efficiently. Somehow caching Git deltas would be cool. But I'd expect that kind of optimization to come from a company that runs a Git forge, if it's generally viable, and to benefit many projects other than Nix and Nixpkgs. -- 1: https://snix.dev/ https://snix.dev/
      - mikepurvis 10mo ago
        Yes indeed. That said nix typically throws away the .git dir so it would require some work to adapt a solution to nix that operates at the git repo level. The ideal for nix would be “I have all content at commit X and need the deltas for content at commit Y” and i suspect nix would be fairly unique in being able to benefit from that. To the point that it might actually make sense to just implement the fact git repo syncs and have a local client serving those tarballs to the nix daemon.
didip 10mo ago
So… What we need is a globally distributed git seeders of all open source github content, then? Seems possible if every git client is also a torrent client.
the__alchemist 10mo ago
The Cargo example at the top is striking. Whenever I publish a crate, and it blocks me until I write `--allow-dirty`, I am reminded that there is a conflation between Cargo/crates.io and Git that should not exist. I will write `--allow-dirty` because I think these are two separate functionalities that should not be coupled. Crates.io should not know about or care about my project's Git usage or lack thereof.
- cesarb 10mo ago
  > The Cargo example at the top is striking. Whenever I publish a crate, and it blocks me until I write `--allow-dirty`, I am reminded that there is a conflation between Cargo/crates.io and Git that should not exist. I will write `--allow-dirty` because I think these are two separate functionalities that should not be coupled. That's completely unrelated. The --allow-dirty flag is to bypass a local safety check which prevents you from accidentally publishing a crate with changes which haven't been committed to your local git repository. It has no relation at all to the use of git for the index of packages. > Crates.io should not know about or care about my project's Git usage or lack thereof. There are good reasons to know or care. The first one, is to provide a link from the crates.io page to your canonical version control repository. The second one, is to add a file containing the original commit identifier (commit hash in case of git) which was used to generate the package, to simplify auditing that the contents of the package match what's on the version control repository (to help defend against supply chain attacks). Both are optional.
  - the__alchemist 10mo ago
    Those are great points, and reinforce the concept that there is conflation between Cargo and Git/commits. Commits and Cargo IMO should be separate concepts. Cargo should not be checking my Git history prior to publishing.
deleted 10mo ago
[deleted]
aidenn0 10mo ago
As far as I know, Nixpkgs doesn't use git as a package database. The packages definitions are stored and developed in git, but the channels certainly are not.
mcny 10mo ago
I want to take a quick detour here if anyone is knowledgeable about this topic. > The hosting problems are symptoms. The underlying issue is that git inherits filesystem limitations, and filesystems make terrible databases. Does this mean mbox is inherently superior to maildir? I really like the idea of maildir because there is nothing to compact but if we assume we never delete emails (on the local machine anyways), does that mean mbox or similar is preferable over maildir?
- juped 10mo ago
  No, of course not.
notorandit 10mo ago
Repsy
pxc 10mo ago
Loved this article. Just enough detail to make the broad scope compatible with a reasonable length, and well-argued. I feel sometimes like package management is a relatively second-class topic in computer science (or at least among many working programmers). But a package manager's behavior can be the difference between a grotesque, repulsive experience and a delightful, beautiful one. And there aren't quite yet any package managers that do well everything that we collectively have learned how to do well, which makes it an interesting space imo. Re: Nixpkgs, interestingly, pre-flakes Nix distributes all of the needed Nix expressions as tarballs, which does play nice with CDNs. It also distributes an index of the tree as a SQLite database to obviate some of the "too many files/directories" problem with enumerating files. (In the meantime, Nixpkgs has also started bucketing package directories by name prefix, too.) So maybe there was a lesson learned here that would be useful to re-learn. On the other hand, IIRC if you use the GitHub fetcher rather than the Git one, including for fetching flakes, Nix will download tarballs from GitHub instead of doing clones. Regardless, downloading and unpacking Nixpkgs has become kinda slow. :-\
themk 10mo ago
I think git is overkill, and probably a database is as well. I quite like the hackage index, which is an append-only tar file. Incremental updates are trivial using HTTP range requests making hosting it trivial as well.
deleted 10mo ago
[deleted]
leoh 10mo ago
The conclusion reached in this essay is 100% wrong. See " The reftable backend What it is, where it's headed, and why should you care?" >With release 2.45, Git has gained support for the “reftable” backend to read and write references in a Git repository. While this was a significant milestone for Git, it wasn‘t the end of GitLab’s journey to improve scalability in repositories with many references. In this talk you will learn what the reftable backend is, what work we did to improve it even further and why you should care. https://www.youtube.com/watch?v=0UkonBcLeAo https://www.youtube.com/watch?v=0UkonBcLeAo Also see Scalar, which Microsoft used to scale their 300GiB Windows repository, https://github.com/microsoft/scalar https://github.com/microsoft/scalar.
deleted 10mo ago
[deleted]
skywhopper 10mo ago
Not sure I can agree with the takeaway. It works well at first, but doesn’t scale, so folks found workarounds. That’s how literally every working system grows. There are always bottlenecks eventually. And you address them when they become an issue, not five years earlier.
juped 10mo ago
These are actually all problems with using Github as an ersatz CDN.
bandrami 10mo ago
Maybe I'm misreading the article but isn't every example about the downside of using github as a database host, not the downside of using git as a database? Like, yes, you should host your own database. This doesn't seem like an argument against that database being git.
khc 10mo ago
seems like the issue isn't with using git as a database, but using github as a distribution mechanism?
zzo38computer 10mo ago
Git commits will have a hash and each file will have a hash, which means that locking is unnecessary for read access. (This is also true of fossil, although fossil does have locking since it uses SQLite.) The other stuff mentioned in the article seems to be valid criticisms.
wg0 10mo ago
Why not use SQLite then as database for package managers? A local copy could be replicated easily with delta fetch.
kuahyeow 10mo ago
GitLab employee here. We have completed the move away from Gollum years ago (see https://gitlab.com/groups/gitlab-org/-/epics/2381 https://gitlab.com/groups/gitlab-org/-/epics/2381). It looks like that doc https://docs.gitlab.com/development/wikis/ https://docs.gitlab.com/development/wikis/ was outdated - since fixed to no longer mention Gollum.
venturecruelty 10mo ago
This is why I don't use programming languages that do that.
unit149 10mo ago
[dead]
skylurk 10mo ago
Sounds like it worked pretty well several times? But yeah it does not scale forever.
krbaccord941x 10mo ago
I understand article is concerning RFC2789, in cloning whole indexes for lang indexes, but /cargo/src shallow-clones need another layer, where tertiary compilation or decompression takes place in mutex libraries, whether its SSL certificate is dependent on HTTP fetch.
shellkr 10mo ago
I am not sure this is necessarily a git issue as it is mostly a GitHub issue. just look at the Aur of Arch Linux which works perfectly.
nottorp 10mo ago
> Auto-updates now run every 24 hours instead of every 5 minutes What the... why would you run an autoupdate every 5 minutes?
tarun_anand 10mo ago
why dont we have a P2P transfer platform for this (modulo security)
ferfumarma 10mo ago
I love this write-up. As a non-expert user of package managers I can quickly understand a set of patterns that have been deeply considered and carefully articulated. Thanks for taking the time to write up your observations!
whytevuhuni 10mo ago
No mention of Guix. Has its situation improved? I remember waiting almost an hour on “guix pull” to catch up with its git repo on a fresh install.
rudolph9 10mo ago
It’s worth considering if these package managers would have taken off if they didn’t use git. You get a bunch for free, why not use it while you’re small?
rldjbpin 10mo ago
the title and core argument do not seem to align much. subject is git, but most discourse is around github. the role discussed is for serving packages, while the title refers to it as "database". regardless of the semantics, git is not ideal for serving files. this has been more apparent in the ai world, where extensions such as git lfs has allowed larger file size. but as seen elsewhere, network effects trump over any design issues. we can always introduce an "lfs" for better shallow fetching (cached? compressed?) and this would resolve a majority of the op's grievences.
rurban 10mo ago
You can add Debian to the list of pain. They'll find out soon enough
schnatterer 10mo ago
> Even GitOps tools that embrace git as a source of truth have to work around its limitations I'd say this only applies to huge scale (or monorepos, as mentioned in the article). Another workaround might be gitless gitops via OCI.
cben 10mo ago
Obligatory link to the gold intro to So. Many. Aspects. of pkg manager design: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... Even if its section on "Central Package Registry" isn't very deep.
account42 10mo ago
> Windows restricts paths to 260 characters, a constraint dating back to DOS. It doesn't if you do it properly.

784 points by birdculture 10mo ago

eviks 10mo ago

Indeed, the seductive nature of bad tools lying close to your hand - no need to lift your butt to get them!

What made git special & powerful from the start was its data model: Like the network databases of old, but embedded in a Merkle tree for independent evolution and verifiability. Scaling that data model beyond projects the size of the Linux kernel was not critical for the original implementation. I do wonder if there are fundamental limits to scaling the model for use cases beyond “source code management for modest-sized, long-lived projects”.

amluto 10mo ago
Most of the problems mentioned in the article are not problems with using a content-addressed tree like git or even with using precisely git’s schema. The problems are with git’s protocol and GitHub’s implementation thereof. Consider vcpkg. It’s entirely reasonable to download a tree named by its hash to represent a locked package. Git knows how to store exactly this, but git does not know how to transfer it efficiently.
- mananaysiempre 10mo ago
  > Git knows how to store [a hash-addressed tree], but git does not know how to transfer it efficiently. Naïvely, I’d expect shallow clones to be this, so I was quite surprised by a mention of GitHub asking people not to use them. Perhaps Git tries too hard to make a good packfile?.. Meanwhile, what Nixpkgs does (and why “release tarballs” were mentioned as a potential culprit in the discussion linked from TFA) is request a gzipped tarball of a particular commit’s files from a GitHub-specific endpoint over HTTP rather than use the Git protocol. So that’s already more or less what you want, except even the tarball is 46 MB at this point :( Either way, I don’t think the current problems with Nixpkgs actually support TFA’s thesis.

quaintdev 10mo ago

I host my own code repository using Forgejo. It's not public. In fact, it's behind mutual tls like all the service I host. Reason? I don't want to deal with bots and other security risks that come with opening port to the world. Turns out Go module will not accept package hosted on my Forgejo instance because it asks for certificate. There are ways to make go get use ssh but even with that approach the repository needs to be accessible over https. In the end, I cloned the repository and used it in my project using replace directive. It's really annoying.

xyzzy_plugh 10mo ago
> There are ways to make go get use ssh but even with that approach the repository needs to be accessible over https. No, that's false. You don't need anything to be accessible over HTTP. But even if it did, and you had to use mTLS, there's a whole bunch of ways to solve this. How do you solve this for any other software that doesn't present client certs? You use a local proxy.
agwa 10mo ago
If you add .git to the end of your module path and set $GOPRIVATE to the hostname of your Forgejo instance, then Go will not make any HTTPS requests itself and instead delegate to the git command, which can be configured to authenticate with client certificates. See https://go.dev/ref/mod#vcs-find https://go.dev/ref/mod#vcs-find
irusensei 10mo ago
Have a look at Tailscale DNS and certs. Its gives you a valid cert through lets encrypt without exposing your services to the internet.
baobun 10mo ago
If you add the instance TLS cert (CA) to your trust store then go will happily download over https. It can be finicky depending on how you run go but I can confirm it works.

Zambyte 10mo ago

The issues with using Git for Nix seem to entirely be issues with using GitHub for Nix, no?

femiagbabiaka 10mo ago
Yeah, it's inclusion in here is baffling because none of the listed issues have anything to do with the particular issue nixpkgs is having.
Rucadi 10mo ago
I also got the same feeling from that, in fact, I would go as far as to say that nixpkgs and nix-commands integration with git works quite well and is not an issue. So the phrase the article says "Package managers keep falling for this. And it keeps not working out" I feel that's untrue. The most issue I have with this really is "flakes" integration where the whole recipe folder is copied into the store (which doesn't happen with non-flakes commands), but that's a tooling problem not an intrinsic problem of using git

ekjhgkejhgk 10mo ago

Do the easy thing while it works, and when it stops working, fix the problem. Julia does the same thing, and from the Rust numbers on the article, Julia has about 1/7th the number of packages that Rust does[1] (95k/13k = 7.3). It works fine, Julia has some heuristics to not re-download it too often. But more importantly, there's a simple path to improve. The top Registry.toml [1] has a path to each package, and once donwloading everything proves unsustainable you can just download that one file and use it to download the rest as needed. I don't think this is a difficult problem. [1] https://github.com/JuliaRegistries/General/blob/master/Registry.toml https://github.com/JuliaRegistries/General/blob/master/Regis...

zahlman 10mo ago
> 00000000-1111-2222-3333-444444444444 = { name = "REPLTreeViews", path = "R/REPLTreeViews" } ... Should it be concerning that someone was apparently able to engineer an ID like that?
- adestefan 10mo ago
  It’s as random as any other UUID.
  - Severian 10mo ago
    Incorrect, only some UUIDs are random, specifically v4 and v7 (v7 uses time as well). https://en.wikipedia.org/wiki/Universally_unique_identifier https://en.wikipedia.org/wiki/Universally_unique_identifier > 00000000-1111-2222-3333-444444444444 This would technically be version 2, which would be built from the date-time and MAC address, and DCE security version. But overall, if you allow any yahoo to pick a UUID, its not really a UUID, its just some random string that looks like one.
    - ekjhgkejhgk 10mo ago
      > if you allow any yahoo to pick a UUID, its not really a UUID universally unique identifier (UUID) > 00000000-1111-2222-3333-444444444444 It's unique. Anyway we're talking about a package that doesn't matter. It's abandoned. Furthermore it's also broken, because it uses REPL without importing it. You can't even precompile it. https://github.com/pfitzseb/REPLTreeViews.jl/blob/969f04ce641d137ad9c4231f490a362a0fbdf8a2/src/REPLTreeViews.jl#L21 https://github.com/pfitzseb/REPLTreeViews.jl/blob/969f04ce64...

bencornia 10mo ago

> Grab’s engineering team went from 18 minutes for go get to 12 seconds after deploying a module proxy. That’s not a typo. Eighteen minutes down to twelve seconds. > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Cloning entire repositories to get a single file. I have also had inconsistent performance with go get. Never enough to look closely at it. I wonder if I was running into the same issue?

zahlman 10mo ago
> needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. Python used to have this problem as well (technically still does, but a large majority of things are available as a wheel and PyPI generally publishes a separate .metadata file for those wheels), but at least it was only a question of downloading and unpacking an archive file, not cloning an entire repo. Sheesh. Why would Go need to do that, though? Isn't the go.mod file in a specific place relative to the package root in the repo?
- klooney 10mo ago
  Go's lock files arrived at around the same time as the proxy, before then you didn't have transitive dependencies pre baked.
fireflash38 10mo ago
How long ago were you having issues? That was changed in go 1.13.

c-linkage 10mo ago

This seems like a tragedy of the commons -- GitHub is free after all, and it has all of these great properties, so why not? -- but this kind of decision making occurs whenever externalities are present. My favorite hill to die on (externality) is user time. Most software houses spend so much time focusing on how expensive engineering time is that they neglect user time. Software houses optimize for feature delivery and not user interaction time. Yet if I spent one hour making my app one second faster for my million users, I can save 277 user hour per year. But since user hours are an externality, such optimization never gets done. Externalities lead to users downloading extra gigabytes of data (wasted time) and waiting for software, all of which is waste that the developer isn't responsible for and doesn't care about.

ekjhgkejhgk 10mo ago
I wouldn't call it tragedy of the commons, because it's not a commons. It's owned by microsoft. They're calculating that it's worth it for them, so I say take as much as you can. Commons would be if it's owned by nobody and everyone benefits from its existence.
- TeMPOraL 10mo ago
  Still, because reality doesn't respect boundaries of human-made categories, and because people never define their categories exhaustively, we can safely assume that something almost-but-not-quite like a commons, is subject to an almost-but-not-quite tragedy of the commons.
  - reactordev 10mo ago
    An A- is still an A kind of thinking. I like this approach as not everything perfectly fits the mold.
  - ttiurani 10mo ago
    The whole notion of the "tragedy of the commons" needs to be put to rest. It's an armchair thought experiment that was disproven at the latest in the 90s by Elinor Ostrom with actual empirical evidence of commons. The "tragedy", if you absolutely need to find one, is only for unrestricted, free-for-all commons, which is obviously a bad idea.

miyuru 10mo ago

Funnily enough, I clicked the homebrew GitHub link in the post, only to get a rate limited error page from GitHub.

mikkupikku 10mo ago

People who put off learning SQL for later end up using anything other than a database as their database.

redog 10mo ago
SQL killed the set theory star
groundzeros2015 10mo ago
Is sql over ssh a thing?
- yawaramin 10mo ago
  https://litestream.io/ https://litestream.io/
  - groundzeros2015 10mo ago
    A proprietary cloud subscription doesn’t seem like the right fit for this
    - yawaramin 10mo ago
      As opposed to a proprietary cloud git hosting platform?

steeleduncan 10mo ago

The other conclusion to draw is "Git is a fantastic choice of database for starting your package manager, almost all popular package managers began that way."

saidinesh5 10mo ago
I think the conclusion is more that package definitions can still be maintained on git/GitHub but the package manager clients should probably rely on a cache/db/a more efficient intermediate layer. Mostly to avoid downloading the whole repo/resolve deltas from the history for the few packages most applications tend to depend on. Especially in today's CI/CD World.
- reactordev 10mo ago
  This is exactly the right approach. I did this for my package manager. It relies on a git repo branch for stable. There are yaml definitions of the packages including urls to their repo, dependencies, etc. Preflight scripts. Post install checks. And the big one, the signatures for verification. No binaries, rpms, debs, ar, or zip files. What’s actually installed lives in a small SQLite database and searching for software does a vector search on each packages yaml description. Semver included. This was inspired by brew/portage/dpkg for my hobby os.
  - pl4nty 10mo ago
    yaml definitions compiled to sqlite sounds pretty similar to winget, which is scaling very well. vector search is a cool idea - how much storage does it use per package?
- pseufaux 10mo ago
  This is how WinGet works. It has a small SQLite db it downloads from a hosted url. The DB contains some minimal metadata and a url path to access the full metadata. This way WinGet only has to make API calls for packages it's actually interacting with. As a package manager, it has plenty of problems still, but it's a simple, elegant solution for the git as a DB issue.

ori_b 10mo ago

Alternatively: Downloading the entire state of all packages when you care about just one, it never works out. O(1) beats O(n) as n gets large.

gruez 10mo ago
Seems to still work out for apt?
- ajb 10mo ago
  Not in the same sense. An analogy might be: apt is like fetching a git repo in which all the packages are submodules, so lazily fetched. Some of the package managers in the article seem to be using a monorepo for all packages - including the content. Others seem to have different issues - go wasn't including enough information in the top level, so all the submodules had to be fetched anyway. vcpkg was doing something with tree hashes which meant they weren't really addressible.
- collinmanderson 10mo ago
  I consider apt kinds slow. I wish it were much faster.

born-jre 10mo ago

lol I see this as I plan on using Git for my thing store. https://github.com/blue-monads/potatoverse https://github.com/blue-monads/potatoverse

gjvc 10mo ago

sqlite seems to be ideal for a package manager

sigwinch 10mo ago
I feel like the rqlite people would have a lot to say about how to coordinate your installations, especially for the high-bandwidth non-desktop installs. https://news.ycombinator.com/item?id=45257349 https://news.ycombinator.com/item?id=45257349
mirekrusin 10mo ago
...or scm [0] [0] https://fossil-scm.org https://fossil-scm.org

hk1337 10mo ago

I like Go but it’s dependency management is weird and seems to be centered around GitHub a lot.

Hendrikto 10mo ago
There is nothing tying Go to GitHub.
rewgs 10mo ago
Not at all. It can grab git repos (as well as work with other VCSs). There's just a lot of stuff on GitHub, hence your impression.
andreashaerter 10mo ago
It's mostly tradition rather than a hard requirement. Go has long supported vanity import paths: https://pkg.go.dev/cmd/go#hdr-Remote_import_paths https://pkg.go.dev/cmd/go#hdr-Remote_import_paths For example, we use Hugo to provide independent Go package URLs even though the code is hosted on GitHub. That makes migrating away from GitHub trivial if we ever choose to do so (Repo: https://github.com/foundata/hugo-theme-govanity https://github.com/foundata/hugo-theme-govanity; Example: https://golang.foundata.com/hugo-theme-dev/ https://golang.foundata.com/hugo-theme-dev/). Usage works as expected: go get golang.foundata.com/hugo-theme-dev Edit: Formatting

hogrug 10mo ago

The facts are interesting but the conclusion a bit strange. These package managers have succeeded because git is better for the low trust model and GitHub has been hosting infra for free that no one in their right mind would provide for the average DB. If it didn't work we would not have these massive ecosystems upsetting GitHub's freemium model, but anything at scale is naturally going to have consequences and features that aren't so compatible with the use case.

ifh-hn 10mo ago

So what's the answer then? That's the question I wanted answered after reading this article. With no experience with git or package management, would using a local client sqlite database and something similar on the server do?

encom 10mo ago
I quite like Gentoo's rsync based package manager. I believe they've used that since the beginning. It works well.
- MarsIronPI 10mo ago
  To be clear though, the rsync trees come from a central Git repo (though it's not hosted on GitHub). And syncing from Git actually makes syncing faster.
AaronFriel 10mo ago
OCI artifacts, using the same protocol as container registries. It's a protocol designed for versioning (tagging) content addressable blobs, associating metadata with them, and it's CDN friendly. Homebrew uses OCI as its backend now, and I think every package manager should. It has the right primitives you expect from a registry to scale.

aniou 10mo ago

As side note. Maybe someone knows, why rust devs chose an already used name for language changes proposal? "RFC" was already taken and well-established and I simply refuse to accept that someone wasn't aware about Request For Comments - and if it was true and clash was created deliberately, then it was rude and arrogant. Every, ...king time, when I read something like "RFC 2789 introduced a sparse HTTP protocol." my brain suffers from a short-circuit. BTW: RFC 2789 is a "Mail Monitoring MIB".

adastra22 10mo ago
There are many, many RFC collections. Including many that predate the IETF. Some even predate computers.
- aniou 10mo ago
  But they were in different domains. Here, we have a strong clash because Rust is positioning itself as secure system and internet language and computer and internet standard are already defined by RFC-s. So, it may be not uncommon, when someone would tell about Rust mechanisms, defined by particular RFC in context of handling particular protocol, defined by... well... RFC too. But not by rust-one. Not so smart, when we realize, that one of aspects of secure and reliable system is elimination of ambiguities.
Conan_Kudo 10mo ago
Ask them, don't ask us. They have a public interface, you can ask them to change the name to something unique.

frumplestlatz 10mo ago

Since ~2002, Macports has used svn or git, but users, by default, rsync the complete port definitions + a server-generated index + a signature. The index is used for all lookups; it can also be generated or incrementally updated client-side to accommodate local changes. This has worked fine for literally decades, starting back when bandwidth and CPU power was far more limited. The problem isn’t using SCM, and the solutions have been known for a very long time.

gethly 10mo ago

If we stopped using VCS to fetch source files, we would lose the ability to get the exact commit(understand as version that has nothing to do with the underlying VCS) of these files. Git, Mercurial, SVN.., github, bitbucket...it does not matter. Absolutely nobody will be building downloadable versions of their source files, hosted on who knows how "prestigious" domains, by copying them to another location just to serve the --->exact same content<--- that github and alike already provide. This entire blog is just a waste of time for anyone reading it.

throwway120385 10mo ago
Or you could just ship a tarball and an sha checksum.
- gethly 10mo ago
  you could, in case you want to make only certain releases publicly available. but then, who wants to do that manual labour? we're talking mainstream here, not specific use cases.
layer8 10mo ago
And yet, that's pretty much how the Java world works (Maven repositories).
forrestthewoods 10mo ago
> This entire blog is just a waste of time for anyone reading it. Well that’s an extremely rude thing to say. Personally I thought it was really interesting to read about a bunch of different projects all running into the same wall with Git. I also didn’t realize that Git had issues with sparse checkouts. Or maybe author meant shallow? I forget.

encom 10mo ago

>[Homebrew] Auto-updates now run every 24 hours instead of every 5 minutes[...] That is such an insane default, I'm at a loss for words.

croemer 10mo ago
You mean the 5 minutes is insane, right?
justsomehnguy 10mo ago
Every time I things like I really want to punch people over TCP/IP. UDP wouldn't suffice.

dboon 10mo ago

I’m building Cargo/UV for C. Good article. I thought about this problem very deeply. Unfortunately, when you’re starting out, the idea of running a registry is a really tough sell. Now, on top of the very hard engineering problem of writing the code and making a world class tool, plus the social one of getting it adopted, I need to worry about funding and maintaining something that serves potentially a world of traffic? The git solution is intoxicating through this lense. Fundamentally, the issue is the sparse checkouts mentioned by the author. You’d really like to use git to version package manifests, so that anyone with any package version can get the EXACT package they built with. But this doesn’t work, because you need arbitrary commits. You either need a full checkout, or you need to somehow track the commit a package version is in without knowing what hash git will generate before you do it. You have to push the package update and then push a second commit recording that. Obviously infeasible, obviously a nightmare. Conan’s solution is I think just about the only way. It trades the perfect reproduction for conditional logic in the manifest. Instead of 3.12 pointing to a commit, every 3.x points to the same manifest, and there’s just a little logic to set that specific config field added in 3.12. If the logic gets too much, they let you map version ranges to manifests for a package. So if 3.13 rewrites the entire manifest, just remap it. I have not found another package manager that uses git as a backend that isn’t a terrible and slow tool. Conan may not be as rigorous as Nix because of this decision but it is quite pragmatic and useful. The real solution is to use a database, of course, but unless someone wants to wire me ten thousand dollars plus server costs in perpetuity, what’s a guy supposed to do?

adrianN 10mo ago
Before you managed to build a popular tool it is unlikely that you need to serve many users. Directly going for something that can serve the world is probably premature
- EPWN3D 10mo ago
  The point is not "design to serve the world". The point is "use the right technology for your problem space".
- dboon 10mo ago

kibwen 10mo ago

I think there's a form of survivorship bias at work here. To use the example of Cargo, if Rust had never caught on, and thereby gotten popular enough to inflate the git-based index beyond reason, then it would never have been a problem to use git as the backing protocol for the index. Likewise, we can imagine innumerable smaller projects that successfully use git as a distributed delta-updating data distribution protocol, and never happen to outgrow it. The point being, if you're not sure whether your project will ever need to scale, then it may not make sense to reinvent the wheel when git is right there (and then invent the solution for hosting that git repo, when Github is right there), letting you spend time instead on other, more immediate problems.

stickfigure 10mo ago
Right, this post may encourage premature optimization. Cargo, Homebrew, et al chose an easy, good-enough solution which allowed them to grow until they hit scaling limits. This is a good problem to have. I am sure there's value having a vision for what your scaling path might be in the future, so this discussion is a good one. But it doesn't automatically mean that git is a bad place to start.
8note 10mo ago
im surprised nobody has made a common db for package managers, so cargo could use it without having to think about it
- kibwen 10mo ago
  I mean, it's sort of the other way around. Cargo was built to be able to natively understand git-based dependencies, in the sense that you can bypass a crate registry and instead just point it directly at a git repo. That means that Cargo already had to have the ability to clone git repos, and so when it came to decide how to implement the index (which looks pretty similar to a git repo if you squint), choosing to use git required them to add literally no new dependencies and almost no new code. Let's also keep in mind that the use case mentioned in the OP is specifically about the index, which is just the datastructure that informs the version resolver how to resolve versions. When it came time to replace the git-based index, Cargo didn't replace it with a specialized database, it replaced it with HTTP endpoints (which are probably just backed by an off-the-shelf database). It's not clear what sort of specialized database would be useful to abstract this for other package managers.

nacozarina 10mo ago

successful things often have humble origins, it’s a feature not a bug for every project that managed to out-grow ext4/git there were a hundred that were well-served and never needed to over-invest in something else

PunchyHamster 10mo ago

The article conclusion is just... not good. There are many benefits to using Git as backend, you can point your project to every single commit as a version which makes testing any fixes or changes in libs super easy, it has built in integrity control and technically (sadly not in practice) you could just sign commits and use that to verify whether package is authentic. It being unoptimal bandwidth wise is frankly just a technical hurdle to get over it, with benefits well worth the drawback

0xbadcafebee 10mo ago

YOLO software engineering, the hallmark of the 21st century

cesarb 10mo ago

One of these is not like the others... > The problem was that go get needed to fetch each dependency’s source code just to read its go.mod file and resolve transitive dependencies. This article is mixing two separate issues. One is using git as the master database storing the index of packages and their versions. The other is fetching the code of each package through git. They are orthogonal; you can have a package index using git but the packages being zip/tar/etc archives, you can have a package index not using git but each package is cloned from a git repository, you can have both the index and the packages being git repositories, you can have neither using git, you can even not have a package index at all (AFAIK that's the case for Go).

bobpaw 10mo ago
I think the article takes issue not with fetching the code, but with fetching the go.mod file that contains index and dependency information. That’s why part of the solution was to host go.mod files separately.
jayd16 10mo ago
Even with git, it should be possible to grab the single file needed without the rest of the repo, but i'ts still trying to round a square peg.
- skywhopper 10mo ago
  Honestly I think the article is a bit ahistorical on this one. ‘go get’ pulls the source code into a local cache so it can build it, not just to fetch the go.mod file. If they were having slow CI builds because they didn’t or couldn’t maintain a filesystem cache, that’s annoying, but not really a fault in the design. Anyway, Go improved the design and added an easy way to do faster, local proxies. Not sure what the critique is here. The Go community hit a pain point and the Go team created an elegant solution for it.
  - everforward 10mo ago
    I was thinking this too. I think it might be talking about operations like “go mod tidy” or update operations where it updates your go.mod/sum but doesn’t actually build the code. I would guess enterprise tools do a lot of checking whether there are updates without actually doing any building.

sghiassy 10mo ago

Use the git clone —shallow option and you’ll only download the most recent commits. Yeesh

deleted 10mo ago
[deleted]
ZenoArrow 10mo ago
Did you read the article? It references the server-side overhead for shallow clones.
- sghiassy 10mo ago
  The “server’ cost you’re referencing is the CI system running git shallow then brew update on GitHubs CI servers.
  - cben 10mo ago
    That's not how I understood it. Full clones are big but simple — the server just sends all the packfiles. A first shallow clone needs some server work, but that's cachable, OK. But then on subsequent interactions between a git client that made a shallow clone various time ago and the git server, it's AFAIU actually expensive for the git server to compute the portion this particular client doesn't yet have. Intuitively, and very hand-wavingly, I suspect things could be improved by: (1) clients relaxing "exact depth" requests to "give me approximately N days of stuff, over-sending being OK", and server relaxing "minimal traffic" to roughly map time ranges to whole packfiles — CPU/traffic tradeoff. (2) allowing servers to under-send too (makes (1) tradeoffs easier), by client asking for missing parts right away and/or later — needs on-demand fetch ability to be transparent to user. With "promisor" mechanism in "partial clones" this sounds more realistic? (3) storing history/trees/blobs in entirely separate packfiles(?) I suspect recent years work on bitmaps & MIDX move in this direction, only less naively? I'm not saying Git can scale as well as a DB, but I do feel we sat on an effectively frozen Git format & protocol for a ~decade, and are now exploring more of the design space so hope future will be less clear-cut... And specifically, partial clones remove the hard "fully offline vs. centralized" dichotomy we long clinged to. Assuming you stay online (necessary anyway if you consider HTTP/DB), things that used to be up-front UX decisions can now be matters of perf tuning! * The most dramatic win is if you had to fetch info from every package's separate repo, like Go did. Then, a central DB/caching proxy can build global indexes, unlocking huge wins, no question. It's like "1+N" issues. However, most examples other than Go in the article talk of a single Git repo already storing a global view (still leaving opportunity for custom indexing and querying).

dleslie 10mo ago

GitHub is intoxicatingly free hosting, but Git itself is a terrible database. Why not maintain an _actual_ database on GitHub, with tagged releases? Sqlite data is paged and so you can get away with only fetching the pages you need to resolve your query. https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ https://phiresky.github.io/blog/2021/hosting-sqlite-database...

jarofgreen 10mo ago
This seems to be about hosting an Sqlite database on a static website like GitHub Pages - this can be a great plan, there is also Datasette in a browser now: https://github.com/simonw/datasette-lite https://github.com/simonw/datasette-lite But that's different from how you collect the data in a git repository in the first place - or are you suggesting just putting a Sqlite file in a git repository? If so I can think of one big reason against that.
- dleslie 10mo ago
  Yes, I'm suggesting hosting it on GitHub, leveraging their git lfs support. Just treat it like a binary blob and periodically update with a tagged release.
  - jarofgreen 10mo ago
    It's not clear if you are suggesting accepting contributions to the SQLite file via PR from people (but accepting contributions is generally the point of why people put these on projects on GitHub). But if you are I wouldn't recommend it. PR's won't be able to show diff's. Worse, as soon as multiple people send a PR at once you'll have a really painful merge to resolve, and GitHub's tools won't help you at all. And you can't edit the files in GitHub's web UI. I recommend one file per record, JSON, YAML, whatever non-binary format you want. But then you get: * PR's with diff's that show you what's being changed * Files that technical people can edit directly in GitHub's web editor * If 2 people make PR's on different records at once it's an easy merge with no conflicts * If 2 people make PR's on the same record at once ... ok, you might now have a merge conflict to resolve but it's in an easy text file and GitHub UI will let you see what it is. You can of course then compile these data files into a SQLite file that can be served in a static website nicely - in fact if you see my other comments on this post I have a tool that does this. And on that note, sorry, I've done a few projects in this space so I have views :-)

xpressvideoz 10mo ago

The article lists Git-based wiki engines as a bad usage of Git. Can anybody recommend alternatives? I want something that can be self-hosted, is easily modified by text editors, and has individual page history, preferably with Markdown.

cbondurant 10mo ago

Admittedly, I try and stay away from database design whenever possible at work. (Everything database is legacy for us) But the way the term is being used here kinda makes me wonder, do modern sql databases have enough security features and permissions management systems in place that you could just directly expose your database to the world with a "guest" user that can only make incredibly specific queries? Cut out the middle man, directly serve the query response to the package manager client. (I do immediately see issues stemming from the fact that you cant leverage features like edge caching this way, but I'm not really asking if its a good solution, im more asking if its possible at all)

brendoncarroll 10mo ago
I personally think that this is the future, especially since such an architecture allows for E2E encryption of the entire database. The protocol should just be a transaction layer for coordinating changes of opaque blobs. All of the complexity lives on the client. That makes a lot of sense for a package manager because it's something lots of people want to run, but no one really wants to host.
bob1029 10mo ago
There are still no realistic ways to expose a hosted SQL solution to the public without really unhappy things occurring. It doesn't matter which vendor you pick. Anything where you are opening a TCP connection to a hosted SQL server is a non-starter. You could hypothetically have so many read replicas that no one could blow anyone else up, but this would get to be very expensive at scale. Something involving SQLite is probably the most viable option.
- deleted 10mo ago
  [deleted]
- IshKebab 10mo ago
  Feels like there's an opening in the market there. Why can't you expose an SQL server to the public? Also Stackoverflow exposes a SQL interface so it isn't totally impossible.

dromologist 10mo ago

We wanted to pull updated code in our undockerized instances when they were instantiated, so we decided to pull the code from GitHub. Worked out pretty well though after a thousand trials we got a 502 and now we're one step closer to being forced into a CD pipeline.

keithgroves 10mo ago

When building https:/enact.tools we considered this. I'm glad we didn't go this route.

jarofgreen 10mo ago

It's not just package manager who do this - a lot of smaller projects crowd source data in git repositories. Most of these don't reach the scale where the technical limitations become a problem. Personally my view is that the main problem when they do this is that it gets much harder for non-technical people to contribute. At least that doesn't apply to package managers, where it's all technical people contributing. There are a few other small problems - but it's interesting to see that so many other projects do this. I ended up working on an open source software library to help in these cases: https://www.datatig.com/ https://www.datatig.com/ Here's a write up of an introduction talk about it: https://www.datatig.com/2024/12/24/talk.html https://www.datatig.com/2024/12/24/talk.html I'll add the scale point to future versions of this talk with a link to this post.

Hasnep 10mo ago
Oh, this would have been great for a project I was working on a while ago! I'll have to keep it in mind for the future. Thanks for sharing

pizlonator 10mo ago

What is the alternative? "Use a database" isn't actionable advice because it's not specific enough

yawaramin 10mo ago
Use an SQLite database file, stream out delta updates to clients using zipped plaintext or Litestream or something.

holyknight 10mo ago

It’s basically the same thing that always happens when you choose a technology because it’s convenient rather than a great fit for your problem. Sooner or later, you’ll hit a wall. Just because you can cook a salmon in your dishwasher doesn’t mean you should.

BlueTemplar 10mo ago

Wait, isn't fossil based on sqlite ? Or does fossil itself still have the same issues ?

dwardu 10mo ago

Worst thing is when you’re in a an office and your pc along with other pcs pulls from git unauthenticated, then you get hit with api limits

mukundesh 10mo ago

Though not Github, worth mentioning Huggingface, which is also using git, but managing large files with their(?) xet protocol. https://huggingface.co/docs/hub/en/xet/index https://huggingface.co/docs/hub/en/xet/index

Ericson2314 10mo ago

The Nixpkgs example is not like the others, because it is source code. I don't get what is so bad about shallow clones either. Why should they be so performance sensative?

ajb 10mo ago
In a compressed format, later commits would be added as a delta of some kind, to avoid increasing the size by the whole tree size each time. To make shallow clones efficient you'd need to rewrite the compressed form such that earlier commits are instead deltas on later ones, or something equivalent.
deleted 10mo ago
[deleted]
__MatrixMan__ 10mo ago
It also seems like it's not git that's emitting scary creaks and groans, but rather GitHub. As much as it would be a bummer to forgo some of GitHub's nice-to-have features, I expect we could survive without some of it.
- mindslight 10mo ago
  Furthermore, the issues given for nixpkgs are actually demonstrating the success of using git as the database! Those 20k forks are all people maintaining their own version of nixpkgs on Github, right? Each their own independent tree that users can just go ahead and modify for their own whims and purposes, without having to overcome the activation energy of creating their own package repository. If 83GB (4MB/fork) is "too big" then responsibility for that rests solely on the elective centralization encouraged by Github. I suspect if you could go and total up the cumulative storage used by the nixpkgs source tree distributed on computers spread throughout the world, that is many orders of magnitude larger.
  - __MatrixMan__ 10mo ago
    Agreed, nix really makes it easy to go from solving the problem for yourself to solving it for everybody. Not much else is easy, but when it comes to building an open source community, that criterion is a pretty powerful one.

teiferer 10mo ago

And this my friends is the reason why (only) focusing on CPU cycles and memory hierarchies is insufficient when thinking of the performance of a system. Yes they are important. But no level of low-level optimization will get you out of the hole that a wrong choice of algorithm and/or data structure may have dug you into.

iamwil 10mo ago

This sounds like a missing piece of software in the OSS world. If you have the inclination, you should write it.

yawaramin 10mo ago
People have: https://0install.net/ https://0install.net/

weiwenhao 10mo ago

For package management software that is rarely used, free is the biggest motivation.

drzaiusx11 10mo ago

One of the first things I did at my current place of employment was to detangle the mess of gemfile git dependencies and get them to adopt semver and an actual package repo. There were so many footguns with git dependencies in ruby we were getting taken down by friendly fire on the daily...

drzaiusx11 10mo ago

I'd add git gemfile dependencies to the list of languages called out here as well. It supports git repos, but in general it's a bad idea unless you are diligent with git tag use and disallow git tag mutability, which also assumes you have complete control of your git dependencies...

newswangerd 10mo ago

It’s always humbling when you go on the front page of HN and see an article titled “the thing you’re doing right now is a bad idea and here’s why” This has happened to me a few times now. The last one was a fantastic article about how PG Notify locks the whole database. In this particular case it just doesn’t make a ton of sense to change course. Im a solo dev building a thing that may never take off, so using git for plug-in distribution is just a no brainer right now. That said, I’ll hold on to this article in case I’m lucky enough to be in a position where scale becomes an issue for me.

baobun 10mo ago
The good news is you can easier avoid some of the pitfalls now even as you stick with it. Some good points in comments. I don't know if you rely on github.com but IMO vendor lock-in there might be a bigger issue which you can avoid.
- newswangerd 10mo ago
  Yeah, I'm implementing a couple of things to make my life easier in the future. I don't use any github APIs and I'm setting up my clients to load the plugin repo URLs from my server so I can change them later if I need to. I want all of the resources my clients need to come from my domain name so I can move it around if I need to.

ekjhgkejhgk 10mo ago

Uncertain if this is OT, but given that the CCC is politically inspired organization, I hope not: One thing that still seems absent is awareness of the complete takeover of "gadgets" in schools. Schools these days, as early as primary school, shove screens in front of children. They're expected to look at them, and "use" them for various activities, including practicing handwriting. I wish I was joking [1]. I see two problems with this. First is that these devices are engineered to be addictive by way of constant notifications/distractions, and learning is something that requires long sustained focus. There's a lot of data showing that under certain common circumstances, you do worse learning from a screen than from paper. Second is implicitly it trains children to expect that anything has to be done through a screen connected to a closed point-and-click platform. (Uninformed) people will say "people who work with computers make money, so I want my child to have an ipad". But interacting with a closed platform like an ipad is removing the possibilities and putting the interaction "on rails". You don't learn to think, explore and learn from mistakes, instead you learn to use the app that's put in front of you. This in turn reinforces the "computer says no" [2] approach to understanding the world. I think this is a matter of civil rights and freedom, but sadly I don't often see "civil rights" organizations talk about this. I think I heard Stallman say something along these lines once, but other than that I don't see campaigns anywhere. [1] https://www.letterjoin.co.uk/ https://www.letterjoin.co.uk/ [2] https://youtu.be/eE9vO-DTNZc https://youtu.be/eE9vO-DTNZc

AceJohnny2 10mo ago
It looks like you commented on the wrong post, although I don't immediately see a front-page post about the ongoing Chaos Computer Congress.
- kzrdude 10mo ago
  it's here https://news.ycombinator.com/item?id=46386211 https://news.ycombinator.com/item?id=46386211 (and it was last on the front page at the moment)
  - ekjhgkejhgk 10mo ago

grumbel 10mo ago

Do we have distributed databases that regular users can clone, modify and merge?

stephenlf 10mo ago

Omarchy

jama211 10mo ago

“It never works out” - hmm, seems like it worked out just fine, worked great to get the operation of the ground and when scale became an issue it was solvable by moving to something else. It served its purpose, sounds like it worked out to me.

swiftcoder 10mo ago
You appear to have glossed over the two projects in the list that are stuck due to architectural decisions, and don't have any route to migrate off of git-as-database?
- hombre_fatal 10mo ago
  Be more specific because I just see a list of workarounds deployed once they had the scale to warrant them, supporting the OP’s claim.
  - swiftcoder 10mo ago
    Read the vcpkg section, it explicitly states that they have no horizontal on a solution. The nix section also doesn’t explain any potential solution.
    - deleted 10mo ago
      [deleted]
- baobun 10mo ago
  The issues with nixpkgs stem from that it is a monorepo for all packages and doubling as an index. The issues are only fundamental with that architecture. Using a separate repo for each package, like the Arch User Repos, does not have the same problems. Nixpkgs certainly could be architected like that and submodules would be a graceful migration path. I'm not aware of discussion of this but guess that what's preventing it might be that github.com tooling makes it very painful to manage thousands of repos for a single project. So I think it can be a lesson not to that using git as a database is bad but that using github.com as a database is. PRs as database transactions is clunky and GitHub Actions isn't really ACID.

mikepurvis 10mo ago

The nix cli almost exclusively pulls GitHub as zipballs. Not perfect but certainly far faster than a real git clone.

pxc 10mo ago
That it supports fetching via Git as well as various via forge-specific tarballs, even for flakes, is pretty nice. It means that if your org uses Nix, you can fall back to distribution via Git as a solution that doesn't require you to stand up any new infra or tie you to any particular vendor, but once you get rolling it's an easy optimization to switch to downloading snapshots. The most pain probably just becomes from the hugeness of Nixpkgs, but I remain an advocate for the huge monorepo of build recipes.
- mikepurvis 10mo ago
  Yes agreed. It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates, but I suspect the folks who would have to build and maintain that are all on gigabit internet connections and don’t feel the complexity is worth it.
  - pxc 10mo ago
    > It’s possible to imagine some kind of cached-deltas scheme to get faster/smaller updates I think the snix¹ folks are working on something like this for the binary caches— the greater granularity of the content-addressing offers morally the same kind of optimization as delta RPMs: you can download less of what you don't need to re-download. But I'm not aware of any current efforts to let people download the Nixpkgs tree itself more efficiently. Somehow caching Git deltas would be cool. But I'd expect that kind of optimization to come from a company that runs a Git forge, if it's generally viable, and to benefit many projects other than Nix and Nixpkgs. -- 1: https://snix.dev/ https://snix.dev/
    - mikepurvis 10mo ago
      Yes indeed. That said nix typically throws away the .git dir so it would require some work to adapt a solution to nix that operates at the git repo level. The ideal for nix would be “I have all content at commit X and need the deltas for content at commit Y” and i suspect nix would be fairly unique in being able to benefit from that. To the point that it might actually make sense to just implement the fact git repo syncs and have a local client serving those tarballs to the nix daemon.

didip 10mo ago

So… What we need is a globally distributed git seeders of all open source github content, then? Seems possible if every git client is also a torrent client.

the__alchemist 10mo ago

The Cargo example at the top is striking. Whenever I publish a crate, and it blocks me until I write `--allow-dirty`, I am reminded that there is a conflation between Cargo/crates.io and Git that should not exist. I will write `--allow-dirty` because I think these are two separate functionalities that should not be coupled. Crates.io should not know about or care about my project's Git usage or lack thereof.

cesarb 10mo ago
> The Cargo example at the top is striking. Whenever I publish a crate, and it blocks me until I write `--allow-dirty`, I am reminded that there is a conflation between Cargo/crates.io and Git that should not exist. I will write `--allow-dirty` because I think these are two separate functionalities that should not be coupled. That's completely unrelated. The --allow-dirty flag is to bypass a local safety check which prevents you from accidentally publishing a crate with changes which haven't been committed to your local git repository. It has no relation at all to the use of git for the index of packages. > Crates.io should not know about or care about my project's Git usage or lack thereof. There are good reasons to know or care. The first one, is to provide a link from the crates.io page to your canonical version control repository. The second one, is to add a file containing the original commit identifier (commit hash in case of git) which was used to generate the package, to simplify auditing that the contents of the package match what's on the version control repository (to help defend against supply chain attacks). Both are optional.
- the__alchemist 10mo ago
  Those are great points, and reinforce the concept that there is conflation between Cargo and Git/commits. Commits and Cargo IMO should be separate concepts. Cargo should not be checking my Git history prior to publishing.

deleted 10mo ago

[deleted]

aidenn0 10mo ago

As far as I know, Nixpkgs doesn't use git as a package database. The packages definitions are stored and developed in git, but the channels certainly are not.

mcny 10mo ago

I want to take a quick detour here if anyone is knowledgeable about this topic. > The hosting problems are symptoms. The underlying issue is that git inherits filesystem limitations, and filesystems make terrible databases. Does this mean mbox is inherently superior to maildir? I really like the idea of maildir because there is nothing to compact but if we assume we never delete emails (on the local machine anyways), does that mean mbox or similar is preferable over maildir?

juped 10mo ago
No, of course not.

notorandit 10mo ago

Repsy

pxc 10mo ago

Loved this article. Just enough detail to make the broad scope compatible with a reasonable length, and well-argued. I feel sometimes like package management is a relatively second-class topic in computer science (or at least among many working programmers). But a package manager's behavior can be the difference between a grotesque, repulsive experience and a delightful, beautiful one. And there aren't quite yet any package managers that do well everything that we collectively have learned how to do well, which makes it an interesting space imo. Re: Nixpkgs, interestingly, pre-flakes Nix distributes all of the needed Nix expressions as tarballs, which does play nice with CDNs. It also distributes an index of the tree as a SQLite database to obviate some of the "too many files/directories" problem with enumerating files. (In the meantime, Nixpkgs has also started bucketing package directories by name prefix, too.) So maybe there was a lesson learned here that would be useful to re-learn. On the other hand, IIRC if you use the GitHub fetcher rather than the Git one, including for fetching flakes, Nix will download tarballs from GitHub instead of doing clones. Regardless, downloading and unpacking Nixpkgs has become kinda slow. :-\

themk 10mo ago

I think git is overkill, and probably a database is as well. I quite like the hackage index, which is an append-only tar file. Incremental updates are trivial using HTTP range requests making hosting it trivial as well.

deleted 10mo ago

[deleted]

leoh 10mo ago

The conclusion reached in this essay is 100% wrong. See " The reftable backend What it is, where it's headed, and why should you care?" >With release 2.45, Git has gained support for the “reftable” backend to read and write references in a Git repository. While this was a significant milestone for Git, it wasn‘t the end of GitLab’s journey to improve scalability in repositories with many references. In this talk you will learn what the reftable backend is, what work we did to improve it even further and why you should care. https://www.youtube.com/watch?v=0UkonBcLeAo https://www.youtube.com/watch?v=0UkonBcLeAo Also see Scalar, which Microsoft used to scale their 300GiB Windows repository, https://github.com/microsoft/scalar https://github.com/microsoft/scalar.

deleted 10mo ago

[deleted]

skywhopper 10mo ago

Not sure I can agree with the takeaway. It works well at first, but doesn’t scale, so folks found workarounds. That’s how literally every working system grows. There are always bottlenecks eventually. And you address them when they become an issue, not five years earlier.

juped 10mo ago

These are actually all problems with using Github as an ersatz CDN.

bandrami 10mo ago

Maybe I'm misreading the article but isn't every example about the downside of using github as a database host, not the downside of using git as a database? Like, yes, you should host your own database. This doesn't seem like an argument against that database being git.

khc 10mo ago

seems like the issue isn't with using git as a database, but using github as a distribution mechanism?

zzo38computer 10mo ago

Git commits will have a hash and each file will have a hash, which means that locking is unnecessary for read access. (This is also true of fossil, although fossil does have locking since it uses SQLite.) The other stuff mentioned in the article seems to be valid criticisms.

wg0 10mo ago

Why not use SQLite then as database for package managers? A local copy could be replicated easily with delta fetch.

kuahyeow 10mo ago

GitLab employee here. We have completed the move away from Gollum years ago (see https://gitlab.com/groups/gitlab-org/-/epics/2381 https://gitlab.com/groups/gitlab-org/-/epics/2381). It looks like that doc https://docs.gitlab.com/development/wikis/ https://docs.gitlab.com/development/wikis/ was outdated - since fixed to no longer mention Gollum.

venturecruelty 10mo ago

This is why I don't use programming languages that do that.

unit149 10mo ago

[dead]

skylurk 10mo ago

Sounds like it worked pretty well several times? But yeah it does not scale forever.

krbaccord941x 10mo ago

I understand article is concerning RFC2789, in cloning whole indexes for lang indexes, but /cargo/src shallow-clones need another layer, where tertiary compilation or decompression takes place in mutex libraries, whether its SSL certificate is dependent on HTTP fetch.

shellkr 10mo ago

I am not sure this is necessarily a git issue as it is mostly a GitHub issue. just look at the Aur of Arch Linux which works perfectly.

nottorp 10mo ago

> Auto-updates now run every 24 hours instead of every 5 minutes What the... why would you run an autoupdate every 5 minutes?

tarun_anand 10mo ago

why dont we have a P2P transfer platform for this (modulo security)

ferfumarma 10mo ago

I love this write-up. As a non-expert user of package managers I can quickly understand a set of patterns that have been deeply considered and carefully articulated. Thanks for taking the time to write up your observations!

whytevuhuni 10mo ago

No mention of Guix. Has its situation improved? I remember waiting almost an hour on “guix pull” to catch up with its git repo on a fresh install.

rudolph9 10mo ago

It’s worth considering if these package managers would have taken off if they didn’t use git. You get a bunch for free, why not use it while you’re small?

rldjbpin 10mo ago

the title and core argument do not seem to align much. subject is git, but most discourse is around github. the role discussed is for serving packages, while the title refers to it as "database". regardless of the semantics, git is not ideal for serving files. this has been more apparent in the ai world, where extensions such as git lfs has allowed larger file size. but as seen elsewhere, network effects trump over any design issues. we can always introduce an "lfs" for better shallow fetching (cached? compressed?) and this would resolve a majority of the op's grievences.

rurban 10mo ago

You can add Debian to the list of pain. They'll find out soon enough

schnatterer 10mo ago

> Even GitOps tools that embrace git as a source of truth have to work around its limitations I'd say this only applies to huge scale (or monorepos, as mentioned in the article). Another workaround might be gitless gitops via OCI.

cben 10mo ago

Obligatory link to the gold intro to So. Many. Aspects. of pkg manager design: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527 https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... Even if its section on "Central Package Registry" isn't very deep.

account42 10mo ago

> Windows restricts paths to 260 characters, a constraint dating back to DOS. It doesn't if you do it properly.