13 ms·
We put half a million files in one Git repository (2022)
- Smaug123 3y agoBold move to enable the "ours" merge strategy by default! I presume this is a typo for the "-Xours" merge option to `ort` or `recursive`, but that still seems pretty brave.
- deleted 3y ago[deleted]
- Waterluvian 3y agoA very useful flag. But as a default? That is scary…
- avidiax 3y agoIt is still only "ours" per hunk. But yes, it could obliterate changes. On the other hand, the default merge strategy is a huge waste of developer time. There is rarely a genuine conflict. It's usually just that we want to keep both sides.
- Smaug123 3y agoI've found `ort` (which I believe is now the default) to be better than `recursive` by leaps and bounds. Have you tried `ort`?
- mrAssHat 3y agoThe site is not opening. Thanks, CloudFlare.
- time4tea 3y agoWe learned they were 70% autogenerated so probably shouldn't have been in git at all, but our build process relied on that, and didnt want to fix it, so we bodged it.
- maccard 3y ago> probably shouldn't have been in git at all Something being autogenerated, or binary, doesn't mean it shouldn't be in version control. If step one of your instructions to build something from version control involve downloading a specific version of something else, then your VCS isn't doing it's job, and you're likely skirting around it to avoid limitations in the tool itself. People still use tools like P4 because they want versioned binary content that belongs in version control, or because they want to handle half a million files, and git chokes. In my last org, we vendored our entire toolchain, including SDKs. The project setup instructions were: - Install p4 - Sync, get coffee - Run build, get more coffee. A disruptive thing like a compiler upgrade just works out of the box in this scenario. It's a shame that the mantra of "do one thing well" devolves into "only support a few hundred text files on linux" with git.
- PMunch 3y agoWouldn't Git LFS be the tool for this job? Have the automated tool build a .zip file for example of the translations (possibly with compression level set to 0), then have your build toolchain unzip the archive before it runs. Then check that big .zip file into GitLFS, et voila you now have this large file versioned in Git.
- maccard 3y agoGit LFS isn't the same as git, though. It's better than putting everything in a separate store, but for one it disables offline work, and breaks the concept of D in the DVCS of git. > then have your build toolchain unzip the archive before it runs My build toolchain shouldn't have to work around the shortcomings of my environment, IMO. > et voila you now have this large file versioned in Git. No, it's on a separate http server that is fetched via git lfs. Subtle, but important difference.
- aidenn0 3y ago> it disables offline work, This is a non-issue for images and autogenerated files, since you shouldn't ever be doing a merge on them. > breaks the concept of D in the DVCS of git. git-annex is distributed and works well for files that will never be merged (such as images, or autogenerated files)
- hjgraca 3y agoOr as they call it, a simple "Hello World" Javascript project
- jiggawatts 3y agoSomething I learned about writing robust code is that scalability needs to be tested up-front. Test with 0, 1, and many where the latter is tens of millions, not just ten. I've seen production databases that had 40,000 tables for valid reasons. I've personally deployed an app that needed 80,000 security groups in a single LDAP domain, just for it. I can't remember what the total group number of groups across everything was, but it was a decent chunk of a million. Making something like Git, or a file system, or a package manager? Test what happens with millions of objects! Try billions and see where your app breaks. Fix the issues even if you never think anyone will trigger them. It's not about scaling to some arbitrary number, it's about scaling, period.
- vincnetas 3y agocould you elaborate on 40,000 tables DB? I want to learn what could be valid reasons for that?
- avidiax 3y agoHow does a "monorepo" differ from, say, using a master project containing many git submodules[1], perhaps recursively? You would probably need a bit of tooling. But the gain is that git commands in the submodules are speedy, and there is only O(logN) commit multiplication to commit the updated commit SHAs up the chain. Think Merkle tree, not single head commit SHA. Eventually, you may get a monstrosity like Android Repo [2] though. And an Android checkout and build is pushing 1TB these days. But there, perhaps, the submodule idea wins again. Replace most of the submodules with prebuilt variants, and have full source + building only for the module of interest. [1] https://git-scm.com/book/en/v2/Git-Tools-Submodules https://git-scm.com/book/en/v2/Git-Tools-Submodules [2] https://source.android.com/docs/setup/download#repo https://source.android.com/docs/setup/download#repo
- tazjin 3y agoMost large monorepos simply are not on git. Google has Piper, Yandex has arc, Facebook has eden (which is actually semi-open-source, btw!), some companies use Perforce and so on.
- avidiax 3y agoI think it's orthogonal. "Monorepo" is a culture around having a single branch with a single lineage, and not developing anything in any isolation greater than a single developer's workstation. I agree that Git is not very adequate for large monorepos, but I'd say that most open source projects are on Git, and most of them are trivial monorepos.
- hgsgm 3y agoNo. Branching is orthogonal to modules/monorepo.
- 0xcoffee 3y agoWindows is coming out with their own 'Dev Drive': https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/ I'm very curious how it performs compared to EdenFS: https://github.com/facebook/sapling/blob/main/eden/fs/docs/Overview.md https://github.com/facebook/sapling/blob/main/eden/fs/docs/O...
- steffres 3y agoAnyone know, what's the advantage of this over a big composite repo with several git submdolues? I think that submodules are better suited for separation of concerns and performance, even while achieving the same composite structure as an equivalent monorepo?
- tantalor 3y agoMonorepo allows a single commit to update across components, eg an API change
- steffres 3y agofor each submodule affected by some change you would need an additional commits, yes. But those commits are bundled together in the commit of the parent repo where they act as one. So, atomicity of changes can be guaranteed, but you need to write a few more commits. However this effort of small increases of commits is far outweighed by the modularity imo.
- marksomnian 3y agoIs it? I'm slightly struggling to understand what benefit you gain from having the "parent" repo but also having individual submodules. Sure, working in each individual project's module makes cloning faster, until you need to work on a module that references another module (at which point you need to check out the parent repo or risk using the wrong version), and now every change you make needs two commits (one to the sub-repo, and one to the base to bump the submodule reference),
- steffres 3y agoIn our case, we have a codebase that involves two submodules: one for persistence and one for python based management of internal git repos. Both of these are standalone applications and can run on their own. They are then used in a parent repo which represents the overarching architecture, which calls into the submodules. The advantage of this is, that work can be done by devs on the individual modules without much knowledge of the overarching architecture, nor strong code ties into it. Right now our persistence is done with SQL, but we could swap it with anything else, e.g. mongo, and the parent codebase wouldn't notice a thing since the submodule only returns well defined python objects. Of course, this comes at the cost of higher number of commits as you mentioned. But in my opinion these are still cheap because they only affect trivial quantity and not brain-demanding quality.
- baz00 3y agoProbably learned how enterprise software developers suffer.
- issafram 3y agoSince when was a "monorepo" ever considered a good idea?
- yashap 3y agoThey have some nice advantages: - Makes it easy to develop applications and libraries together in a single branch - Similarly, makes it easy to make a breaking change to a library, then change all clients of said library, in a single branch - And because of the above, makes it easy to keep all dependencies on internal libs at the latest version, which can greatly reduce all sorts of “dependency hell” issues - Generally makes integration testing a bit easier The downside is you have to invest a lot more time in tooling, keeping both local and CI builds fast. And even with that tooling, builds won’t be as fast as they trivially are with multi-repo. But if you do invest that time in tooling, you can generally get them fast enough, and then reap the other benefits for a very productive dev experience. Have done both monorepo and multi-repo at different, decent sized companies. Both have their pros/cons.
- deleted 3y ago[deleted]
- Cthulhu_ 3y agoA couple of years now, but whether it's a good idea depends on your use case and organization. Seems to work for some. It works for my current assignment too - two and possibly more React Native that reuse a lot of components, translations, have the same APIs, etc.
- Alacart 3y agoAh yes, I too have accidentally committed node_modules. Jokes aside, and coming from a place of ignorance, it's interesting to me that a file count that size is still a real performance issue for git. I'd have expected something that's so ubiquitous and core to most of the software world hasn't seen improvements there. Genuine, non snarky question: Are there some fundamental aspects of git that would make it either very difficult to improve that, or that would sacrifice some important benefits if they were made? Or is this a case of it being a large effort and no one has particularly cared enough yet to take it on?
- klodolph 3y ago> Are there some fundamental aspects of git that would make it either very difficult to improve that, or that would sacrifice some important benefits if they were made? It’s hard to look at a million files on disk and figure out which ones have changed. Git, by default, examines the filesystem metadata. It takes a long time to examine the metadata for a million files. The main alternative approaches are: - Locking: Git makes all the files read-only, so you have to unlock them first before editing. This way, you only have to look at the unlocked files. - Watching: Keep a process running in the background and listen to notifications that the files have changed. - Virtual filesystem: Present a virtual filesystem to the user, so all file modifications go through some kind of Git daemon running in the background. All three approaches have been used by various version control systems. They’re not easy approaches by any means, and they all have major impacts on the way you have to set up your Git repository. People also want e.g. sparse checkouts, when you’re working with such large repos.
- HALtheWise 3y agoIt's notable that git does support "watching", but it requires some setup on Linux to install and integrate with Watchman. On Windows and Mac, core.fsmonitor has been built in since version 2.37. https://www.infoq.com/news/2022/06/git-2-37-released/ https://www.infoq.com/news/2022/06/git-2-37-released/
- 10000truths 3y agoAre there any solutions that use libgit2's ability to define a custom ODB backend? There are even example backends already written [1] that use RDBMSs as the underlying data store. [1] https://github.com/libgit2/libgit2-backends https://github.com/libgit2/libgit2-backends
- psydvl 3y agoThere is VFS for git from Microsoft, that can solve problem more elegant way, I think: https://github.com/microsoft/scalar https://github.com/microsoft/scalar
- TiredOfLife 3y agoThat was discontinued (like multiple times under different names). And is moved into a git fork. https://github.com/microsoft/git https://github.com/microsoft/git
- ComputerGuru 3y agoDo you know what replaced it?
- MikusR 3y agoTheir fork https://github.com/microsoft/git https://github.com/microsoft/git
- WorldMaker 3y agogit They upstreamed almost everything. The last version of "scalar" was mostly just a configuration tool for sparse checkout "cones" which needed a bit of hand-holding, and that is easier to configure in git itself now, or so I hear.
- zellyn 3y agoAre they still trying to upstream everything? For a while they were being good about that…
- paulirish 3y ago> A git fetch trace that was captured Anyone know observability software are they using to visualize the GIT_TRACE details? (Or is the assumption that the UI is Olly as well?)
- Maxious 3y agoJaeger There's a couple of other tools like https://github.com/derrickstolee/trace2-flamechart https://github.com/derrickstolee/trace2-flamechart
- rsp1984 3y agoI am not sure what I'm looking at here. Surely those half million files are for dozens if not hundreds of different apps, libraries and tools and surely those do not all depend on each other, no? Because if so, why not just use one repo per app/library/tool? Sure, if you have a cluster of things that all depend on each other, or a cluster of things that typically is needed in bulk, by all means, put those in a single repo. But putting literally all your code in a single repo is not a very sane technical choice, is it?
- jayd16 3y agoDepends on the test tooling. If you want a single commit to pass integration tests then they need to be in a single commit. Otherwise you're tracking every version of every tool. But I like to look at the problem from another perspective. Why _not_ use a single repo. The only real reason would be to work around technical challenges with your source control of choice, not because having everything tracked together is inherently bad.
- compiler-guy 3y agoGoogle runs a single monorepo for 95% of its projects across the company. Google isn't perfect, but it's hard to argue that it isn't technically very good. One of the biggest advantages is that there is no version chasing and dependency questions. At commit X, everything works consistently. No debating about whether this or that dependency is out of sync.
- nottorp 3y ago> Our engineers generally work in small teams and interact with an even smaller subset of the monorepo. The article says. But if Google does it it must be good.
- Borg3 3y agoHmm, I've read this one: "These .xlf files are generated and contain translated strings for each locale." So why to store them under VCS at first place? I think they are doing it wrong.
- urbandw311er 3y agoI think that’s a pretty sharp observation. Analogous to something like node_modules in web dev, for example. The files could be gitignored and then generated both on local machines but also on the CI/build system.
- haizzz 3y agoAuthor here, the phrase "generated" was the wrong choice of word here. These translations are manually translated by humans in another systems and consolidated into these .xlf files so they're dependent on what the original strings are in code at a specific commit. They cannot be generated on the fly
- Borg3 3y agoAhh, ok, then indeed bad wording. Fix up the orginal article :) Im always strict what is going into VCS. Anything that is generated or preprocessed should not belong there.
- eigenvalue 3y agoSince 70% of the files were xlf files used for translation/localization, couldn't they instead just store all of those in a single SQLite file and solve their problem much more easily? Any of the nuances of the directory structure could be captured in SQLite tables and relationships, and it would be easy to access them for edits by non-coders using a tool like DB Browser. I feel like often people make problems much harder than they need to be by imposing arbitrary constraints on themselves that could be avoided if they approached the problem differently.
- Cthulhu_ 3y agoThat just sounds like adding another problem though. A filesystem (and git) already is a database, and plain files can be read and managed more easily than a possibly corruptible binary file. Plus, you'd lose history, unless you add more complexity to add history. I mean I don't know if they ever needed history but, just saying. You get certain things for free by using a filesystem / git.
- melx 3y agoYou commit the sqlite dump file(?) to git and have the history... I dunno but there are folks who would put anything in git. I work with someone who manages to exceed disk space of company's Gitlab instance by git adding everything. The disk is full again once a month.
- haizzz 3y agoThe translations are dependent on the original strings that is in code. For example if we change "design anything" to "design everything", the translations also needs to be updated to reflect that and by keeping it within vcs, we have an atomic change including both code, copy and translation. Moving it to a database would make updates easier but would now be a separate process to "sync" between copy change in code and translation changes in the database
- Groxx 3y agoDon't bother with watchman, it has consistently been so flaky that I simply live with the normal latency. Thankfully, nowadays git has one built in for some OSes, and it's much, MUCH better than watchman ever was.
- haizzz 3y agoYup, we switched over to using the built in monitoring daemon and it's much less hassle!
- fsckboy 3y agothis is one of those multipurpose PR articles (not all bad) to generate awareness of the company, their product, use case, and developers. >At Canva, we made the conscious decision to adopt the monorepo pattern with its benefits and drawbacks. Since the first commit in 2012, the repository has rapidly grown alongside the product in both size and traffic while reading it i was having trouble keeping track of where I was in the recursion, it's sort of "Xzibit A" for "yo dawg, we know you use source repositories, so check out our source repository (we keep it in our source repository) while you check out your source repository!"
- ufjfjjfjfj 3y agoI can't be the only thinking this is a small amount of files unless you keep them all in the same directory
- bob1029 3y agoOur monorepo is at ~500 megs right now. This is 7 years worth of changes. No signs of distress anywhere, other than a periodic git gc operation that now takes long enough to barely notice. I can't imagine using anything else for my current project. In fact, the only domain within which I would even consider something different would be game development. Even then, only if the total asset set is ever expected to exceed a gigabyte or so. Git is awful with large blobs. LFS is an option, but I've always felt like it was a bandaid and not a fundamental solve.
- SerCe 3y agoI'd like to add that the previous discussion was here, https://news.ycombinator.com/item?id=31762245 https://news.ycombinator.com/item?id=31762245. However, since then we've migrated our engineering blog from medium to a self-hosted stack, so HN doesn't link it to the previous discussion automatically.