24 ms·

We Put Half a Million Files in One Git Repository, Here’s What We Learned

222 points by Maxious 4y ago

Beltiras 4y ago
Every time I read a monorepo story my takeaway is that it's a really bad idea. Segregation of responsibility is a good thing.
sandgiant 4y ago
I've had millions of files in git using git-annex. It kind of works if you're very patient, it's definitely not something I'd recommend. I wish there was a solution, such as git-annex, that worked well with millions of files.
deleted 4y ago
[deleted]
Gigachad 4y ago
>just under 60 million lines of code in 2022. I'm always surprised at the fact that almost every product has more lines of code than the entire Linux repo. The scale of these products is astounding.
- mherrmann 4y ago
  From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.
  - b3n 4y ago
    Note that these files are not statically generated; they are translation files, generated by translators.
  - sdflhasjd 4y ago
    Yes, the article glosses over that a little bit, and it's an unusual decision.
  - snicker7 4y ago
    As a counter-point, package managers generate “lock files” that are designed to be tracked in VC. For these translation files, I’d imagine there may be occasional work to modify them even after they are initially generated.
  - onion2k 4y ago
    They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.
    - fomine3 4y ago
      I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.
      - kadoban 4y ago
        The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.
        Gigachad 4y ago
        Depends how much generated stuff is there. We have our graphql schema in git even though its auto generated via a library. But its useful in PRs to see exactly how the schema changed as a result of the root change.
        kadoban 4y ago
        Yeah if there's not a lot of it, and if it's easy to regenerate, it can be fine.
        _ikke_ 4y ago
        You may want to set the '-diff' attribute for these files so that git will not show diffs for these, instead it will show 'Binary files differ'. There is also '-merge', which will cause git to not attempt to merge the contents, but just ask you to pick a side. The challenge however is then verifying the contents of these files in things like merge requests. See https://gitirc.eu/gitattributes.html https://gitirc.eu/gitattributes.html
  - wichert 4y ago
    Reading the article they are not generated files, but files that are never touched by developers. Translators will work with those files. I expect that for translators they have a different sparse checkout that only fetches .xlf files for their target languages.
    - onion2k 4y ago
      xlf files are usually generated. They're XML. No one wants to write that by hand.
      - Xylakant 4y ago
        .docx files are archives of xml-files. No one wants to write that by hand. Or, in more words: The format of the files is just the representation on disk - it’s not directly connected to how the files are generated or edited. XML files can be written by hand with suitable editor support.
        onion2k 4y ago
        I think we have different ideas of what "written by hand" means. Someone making a Word doc is not writing XML by hand.
      - lultimouomo 4y ago
        Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand. I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in. Translations files, even if they are not created with a plain text editor but with some other tool that handles the XML layer, are clearly not generated, as long as the translation is done by a human.
        onion2k 4y ago
        For the app I work at the moment we use https://lokalise.com/ https://lokalise.com/. We add translation strings to a SaaS app, and then the translation team translate them. I've written a build tool that downloads the translation JSON files from the API using the CLI, or as part of our CI process. Other teams have tools that download their language packs for different iOS and Android apps. The translations are versioned in Lokalise and we using a branching strategy to manage the work. Lokalise has an option to generate xlf files (and JSON, xliff, arb, etc). This is a very typical workflow. Most people are not out there modifying xlf files by opening them in a text editor. For a start, translations usually aren't done by developers. (Huge shoutout to Lokalise btw. I can highly recommend it. It makes building a multi-lingual app across different platforms so much easier.)
  - epage 4y ago
    I've been switching a lot of generated files to being checked in (with CI verifying they haven't drifted from the source). The primary motivation has been performance. For example, in Rust code, it means I don't need to foist the code-gen process and all the dependencies needed for it on dependent crates. I've seen this play out similar in other build systems and circumstances. The key is the data needs to be independent of other factors (like the system doing the generation) and the rate of change of the code-gen source and generator has to be relatively low.
  - AlfeG 4y ago
    We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.
    - jokethrowaway 4y ago
      If they're generated you can just re-generate from source every time you need to track changes. You're using git as a cache. You don't need to version a cache.
      - gustavohenke 4y ago
        Regenerating certain things might be fast, but some might not be. Hundreds of engineers pushing code and having to wait for these to be regenerated both locally and on CI means that caching is quite cheap after all.
- shric 4y ago
  As well as generated files the sibling mentioned I wonder how much is due to vendoring.
- alexdowad 4y ago
  The natural tendency in almost any software is to keep adding and adding, while rarely throwing anything out. More features, more code, more supported platforms, more supported languages, more this, more that, more, more, more. If it was a physical product, you couldn't keep making it bigger and more complex ad infinitum, because making a physical thing bigger takes more material, and bounded physical resources would be consumed. With software, it's all just bits, and computers can hold a lot of bits. This leads to bigger problems than just git running slowly.
  - Gigachad 4y ago
    Canva is absolutely crushing the established design market with their product right now so I'd say having all those features is pretty important for a good tool these days.
37 4y ago
Pretty interesting.... I don't know that much about git, but still fun to read. I guess the main takeaway is don't put all your eggs in one basket? Although it kinda seems like they are going to stick with the monorepo, ("Here’s how we solve them at Canva") Also, I looked up .xlf files and I still don't understand. It's xml, that part makes sense, but it's basically a config file? To tell what process to read which files? Also, I've heard of Canva, but had no idea they were this big/ubiquitous/whatever.... and learning about pseudo localization is interesting too. And the graph for lines of code looks pretty exponential, maybe it's common up to a point, but if it continues at that rate, it will be infinite by about 2026 (okay, I just made that number up, but you get the idea)
- Bluecobra 4y ago
  Putting my rusty sysadmin hat on, I would wager a guess that if it keeps on growing exponentially something will break. For example, hitting some hard limit like maximum number of files possible in a file system. Seems like you’re playing with fire to me.
  - 37 4y ago
    I would think the same... and from reading the comments in this thread, it seems like a pretty spicy topic.
- auscompgeek 4y ago
  .xlf isn't a config file format here. It refers to XLIFF, a standard for translation files. https://en.wikipedia.org/wiki/XLIFF https://en.wikipedia.org/wiki/XLIFF
  - 37 4y ago
    I looked at the Wikipedia article and a couple other pages, I guess the key word is "localization", which basically seems to mean language. I'm pretty sure I knew and forgot that. The comments in this thread from the devs have been informative as well.
nindalf 4y ago
git status scales with the number of files in the repo. Ask HN - Are there any common git operations that scale with the number of commits?
- dymk 4y ago
  git log ;)
  - iqkznnft 4y ago
    I believe git log is limited by the number of lines on the screen...
    - kadoban 4y ago
      It may operate in a streaming fashion if it's going to a pager, but naw it'll output however much you want. Technically it has modes which need to scan all commits in the current branch as well, like the ones that grep for certain changes.
- duskwuff 4y ago
  Repository maintenance operations like "git gc" scale with the total number of objects in the repo.
  - milosmns 4y ago
    I used to work at a company where our git repository would get so polluted with change-related junk that we had to call (many times literally call) BitBucket and ask them to run git clean up on the remote. Otherwise, fetch/pull and many operations on remote would take forever or often even fail completely due to timeout or lack of RAM. It's Thursday and you wanna release a hotfix - but nope - all builds are failing because git can't cope with junk and your CI can't proceed. You then have to call BB instead and ask them to clean up, hoping they'll do it within the same day. Ah, exciting times.
rsanheim 4y ago
It took me a while to find out what Canva actually does, but from https://www.canva.com https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure, we can throw all these generated files into git!". Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP.
- arinlen 4y ago
  > Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.
  - rtpg 4y ago
    One reason for doing this is that someone might be using a very small set of the monorepo, so having everything prebuilt means that in general things are very fast, without constantly having to rebuild chunks of your system. This is, I think, very common with stuff like browsers, where you have artifact checkouts that basically include built stuff since otherwise you're sitting there compiling forever all the time. Stuff like Bazel in theory helps with this, but tools that help with this are either super idiosyncratic about how they work (meaning hard to adopt) or outright don't work. I mean personally I would find that pretty annoying and unclean but I like DAGs.
  - whynaut 4y ago
    They’re not assuming anything, but stating that the post itself does not cover this well/at all.
  - 4y ago
sydthrowaway 4y ago
barbs 4y ago
I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file. Does anyone have any good resources for why and how best to implement a monorepo?
- qbasic_forever 4y ago
  You kind of have to experience the worst of boths worlds to understand where and how each method works and breaks down. With multiple repos it's harder for teams to share code and collaborate. Each team has a repo that becomes a little fiefdom where they are oblivious to who is using their code and how they're using it. Suddenly they'll push out what they think is an innocuous refactor and inadvertently break core functionality other teams took a dependency on for better or worse. So what happens is the team with a dependency now copies the old code into their repo and take on all the extra burden of maintaining this old version, trying to backport fixes, etc. It becomes an enormous mess and time sink. No one ever has time to go back and fix things, and when they eventually are forced to do so it costs more time and effort than it would have taken to do it right from the start. You'll also run into horrible versioning problems where you're stuck on old version X but depend on widget foo which needs current version Y of that dependency. You might say well bad on that team they should have engaged the product managers, made sure their dependencies and usage were well tracked with them, been looped in the process of changes, etc... but in the real world when your boss says X feature needs to be shipped in a few days all of that process goes out the window.
  - milosmns 4y ago
    This is valid. But I think it would make sense to challenge this and even (try to) leave if product is dictating tech what to do with their repos... especially if there's a boss figure pushing imaginary deadlines without talking to the team beforehand. There's a job crisis now, I know, but millions of files were not checked in during 2021. We should do better when it comes to organizing how we work
- xmprt 4y ago
  Here are a few reasons that I've heard and experienced 1. Standardized tooling 2. Fewer issues related to dependencies 3. Hermetic tests 4. Reduced code duplication and easier code sharing
zoomablemind 4y ago
> "...we found that .xlf files made up almost 70% of the total number of files. These .xlf files are generated... they are never manually edited by engineers..." First thought is why not to zip/tar away all of these "convenience" files per generation and add a line into build/install script to unpack them after checkout? Additionally, add the .xlf into .gitignore to exclude them from untracked. Noone cares to diff them as long as their contents is consistent with the checkout. Text compresses quite efficiently so this should not introduce any unreasonable build/install delays.
- haizzz 4y ago
  I made another comment as well though tldr is these xlf files are translations tied to texts in code so we can't simply ignore them from the repository. The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc...
  - GeneralMaximus 4y ago
    Could these translations be moved to another repository? Maybe they could be published as a separate NPM package that devs could install if they needed to look at the translations?
    - klysm 4y ago
      Then you get the annoying problem of having to push an update to that repo and wait for the new version before you can merge changes into the new repo which use the new version. It’s tightly coupled, so they should be co-located.
  - zoomablemind 4y ago
    >...The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc... The mentioned .zip file is to be kept in the repo. Instead of a whatever number of individual .xlf files per generation, these would get zipped together (say, 'assets/xlf.zip') before the commit and the resulting .zip added to the commit. Similarly, when reverting or on a checkout, it's the .zip that gets checked out and then the .xlf files are unpacked. The packing/unpacking could be done by the same process that handles the .xlf generation (??build). Also this may be automated by git-hooks, though it's more natural to handle the packing of assets during the build stage.
syspec 4y ago
The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.
- deleted 4y ago
  [deleted]
- outsomnia 4y ago
  Yeah... also once such a team exists, they are naturally going to resist ideas like not having a monorepo but just using git submodules, not keeping the history of every generated file forever etc. Because then they won't need a 5-man team to handle the fallout any more.
- brutusborn 4y ago
  That bit surprised me also. I don't have much experience with companies of this size so I have no real idea; how many people would you expect to work on source control at a company of this size? I had a look at Linkedin, they have 4,345 employees.
  - kanu666 4y ago
    Usually it is considered good proportion to start with to be 10% to work on a Developer Productivity and it should increase to 15% as company grow. But regarding source control it really depends on the approach: if you are on polyrepo - then 0 is enough. If you decide to write you own source control system like Facebook or Google, then maybe couple of hundred engineers full time until the project is finished :)
- milosmns 4y ago
  Really weird. But maybe this team is in a country where developers cost 100x less? (: Still weird though.
tehlike 4y ago
Did anyone use scalar from Microsoft? https://github.com/microsoft/scalar https://github.com/microsoft/scalar
- amir734jj 4y ago
  We use scalar at Microsoft. Working with massive scale of some repos would be near impossible with "vanilla" Git.
- Maxious 4y ago
  As mentioned in the blog "we are moving towards providing a known version of git", probably Microsoft Git which includes the scalar command but also upstreams many of the optimizations to git core https://github.com/microsoft/scalar#why-did-scalar-move https://github.com/microsoft/scalar#why-did-scalar-move disclaimer: canva staff working on source control
ur-whale 4y ago
> These .xlf files are generated and contain translated strings for each locale ... and they make up 70% of their repo Why would you include generated file in a repo? Do they take to long to remake? [EDIT]: especially given the fact that they're using bazel which is supposed to be the bee's knees of build system?
- SerCe 4y ago
  The article uses the term generated, but a more precise term could be "produced by translators". The files can't be simply regenerated. We've recently published [1] a separate post that explains the localisation process in more detail. [1]: https://news.ycombinator.com/item?id=28931601 https://news.ycombinator.com/item?id=28931601
urban_winter 4y ago
What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project boundaries, but that seems like an easier job than making git scale while maintaining response times.
- nsoonhui 4y ago
  One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.
  - Galanwe 4y ago
    Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.
    - rtpg 4y ago
      Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".
      - doctor_eval 4y ago
        What I took away from TFA is that monorepo management at this scale is “extremely frustrating” too. ISTM that the complexity of managing any repo will be bounded by the size of that repo; a monorepo, being unbounded in size, will, in time, become arbitrarily complex to manage. While a multirepo might occasionally require developers to apply changes to more than one repo at a time, I’ve never found this to be much more than a minor inconvenience; one that could be solved readily with simple tooling, if we had ever felt that the “problem” was even worth solving.
deleted 4y ago
[deleted]
haizzz 4y ago
Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply removing them we would lose all the changes made to translations each time texts are changed. Sorry about that, hope this clears it up! Edit: for more information on translation and xlf files, we have another blog post all about them https://canvatechblog.com/how-to-design-in-every-language-at-once-f2dd66a2780f https://canvatechblog.com/how-to-design-in-every-language-at...
- fomine3 4y ago
  Thanks for additional info. I'm happy to hear that localization is on your priority. > English source strings used in our frontend code live in Typescript files (.messages.ts). Source strings used in our backend code live in Java files (Message.java). Our internationalization (i18n) pipeline converts this into a series of XLIFF files (ending in .xlf), with one file for each locale. All these files live in the repository, but the translated .xlf files should never be modified by hand since they are updated automatically when strings get translated. Isn't it cache?
  - haizzz 4y ago
    Not quite, the .messages.ts and Message.java files are only in English which are then converted into XLIFF files for other languages so they're the source of truth for what the string should be in different locales so they're not cache
- prmoustache 4y ago
  Can't those xlf files be stored in a separate repo or in an object storage and let the build system fetch them from there?
  - necovek 4y ago
    That wouldn't be called a "monorepo" then :D Obviously, the problems they are solving (and admitting to solving) are due to their dedication to the monorepo. With all the effort spent on working around the drawbacks, I really wonder what advantages they are seeing that make it worth their while?
nomilk 4y ago
> git status takes 10 seconds on average > running these commands multiple times a day reduces the total productive time engineers have every day I love the attention paid to this. Often opportunities to prioritise seemingly small efficiency gains are neglected. At 10 seconds per command, an engineer that uses git status 50 times per day spends ~10 minutes per day waiting; an entire work week per year!! Well above the threshold warranting optimisation, and that doesn't even factor in distractions and context switching.
- ngshiheng 4y ago
  I'm glad I'm not the only one that thinks that way. Even ~3s every git status is driving me insane now.
- codeyperson 4y ago
  It’s actually even worse than that I think. If something takes over a certain amount of time, then I’m more likely to go do something else while I wait, like check Hackernews. And there goes 20 minutes.
  - Sohcahtoa82 4y ago
    I created a `beep` alias that just does `echo "\x07\x07\x07"` which triggers the system notification sound three times. Then if I have a command that will take a while, like a stupidly long `git status`, I do `git status && beep`.
  - AshamedCaptain 4y ago
    I'm hoping at least you acknowledge this is _your_ problem, rather than a tooling problem or the like. You just can't expect everything to give you immediate feedback after a couple seconds. This is something that's really a degradation of the newer generations of engineers, since I clearly remember the time where these "somethings" would never take less than a couple minutes, and people did not immediately flee to their nearest distraction, but actually planned their time around it. In fact, if you go further back, these "somethings" would have taken hours, and the older generations still got work done.
bjarneh 4y ago
> to just under 60 million lines of code in 2022 I'd be somewhat intimidated on my first day of work if I cloned a repo with that much sloc...
the_biot 4y ago
They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's job and/or ego is heavily invested in keeping things as they are, even if it demonstrably does not scale to their needs, and the solution is blindingly obvious to even a casual observer. That is institutional insanity.
- andi999 4y ago
  What would be a better structure?
  - roflulz 4y ago
    monorepo until it becomes too big -> then, splitting it into 2-3 repos, until each one also gets too big to manage...
    - masklinn 4y ago
      The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing them would really only be useful if we rewrote the entire repository, which breaks all working copies. There’s been a task in a wishlist for years now, but the business incentive just isn’t there. Exit: actually the translation files are 80% of the working copy, they’re closer to 90% of the repo, and on the far side.
      - andi999 4y ago
        But isnt it sort of correct to have translation data, which is text data, and there might be bug fixes, in the repo?
EdSchouten 4y ago
Maybe at their scale it makes more sense to switch to a VCS like Eden? https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden Eden's equivalent of 'git status' should run almost instantaneous, as checkouts are hosted by a virtual file system (FUSE) that tracks changes.
Thorentis 4y ago
I don't understand the logic of combining microservices with a monorepo. The whole point of microservices is that you don't care what is beyond the external contract of the service. Who cares how each individual team decides to name their stuff. Why do microservice teams need to care about having every single service checked out? Who or what would be bulk applying changes to all services? This is madness.
- hamandcheese 4y ago
  Suppose you want to deprecate an API you wrote in favor of something else for $valid_reasons. In a monorepo with the right tooling, I can make a branch where I delete the API and get pretty immediate feedback as to every module I have broken. From there I can update all the call sites and I also know which teams/engineers I should give a heads-up to. In a multi-repo world, this is much more difficult. Even learning what all the reverse dependencies are could be a challenge. Most likely, other teams have to pull in my changes on their own schedule, and other teams have very different incentives than my own. The cost of mistakes is therefore higher because they are more difficult to undo. Monorepos are very important if you want to empower engineers to achieve broad changes across the org. Some people fundamentally disagree with this methodology. And a fair number of people just don’t care enough - marking the old thing deprecated, calling it a day, and letting it rot for eternity is good enough for them. These are the same people who you’ll find saying “not my job” a lot, in my experience.
  - wewtyflakes 4y ago
    The perceived ease of that operation in a monorepo is what makes it dangerous. Unless these changes all map to a single monolith service, then even though you have updated all call sites, these callers will not be deployed all at the same time, meaning as the change rolls out you may see random breakage. By using polyrepo, the deployment boundary can align with the code boundary, making the rollout problem obvious.
    - hamandcheese 4y ago
      People point this out often, but in practice I have never seen this cause issues. Remove callers first, then remove endpoint... its pretty obvious the order in which things need to be done. What I have seen as a real problem, time and time again, is having trouble locating all usages of an API in a multi-repo scenario. Anyone who fucks this up in a monoreppo will probably fuck it up worse with multiple repos.
anonymous_1234 4y ago
I wonder why these generated files are not part of a separate distributed, HA based, cloud managed document store. Seems like a perfect use case for it. This looks like git abuse to me.
compressedgas 4y ago
This reminded me of https://github.com/twosigma/git-meta https://github.com/twosigma/git-meta
- cryptonector 4y ago
  Yeah, git-meta is a reasonable alternative to megamonorepos. It's basically a repo-mega-forest. A few things: - your build system will need more code to deal with a repo-mega-forest than a megamonorepo - code indexers may not be able to see cross-repo dependencies etc. You'll have pain no matter what. My preference would be for a megamonorepo approach to scale properly. That means partial/sparse cloning, as well as shallow cloning, and also all the hacks that are supposed to make git-status and git-log (and git-blame, and...) fast in partial clones.
hathym 4y ago
the easy solution is to put node_modules in your .gitignore :D
maccard 4y ago
I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expensive and streams still feel like a band aid even years and years later). PlasticSCM (which we use at work) is fine, but closed source, mildly expensive, and has a terrible UX
- Ygg2 4y ago
  There was pijul. Which allowed partial pulls iirc. I haven't used it, so I can't really recommended it.
  - maccard 4y ago
    I tried pijul [0] last time this conversation came up and despite its claims it is not fast. [0] https://news.ycombinator.com/item?id=29992875 https://news.ycombinator.com/item?id=29992875
    - pmeunier 4y ago
      Just seeing this now. If you look at the conversation, this was fixed within a few hours.
- kanu666 4y ago
  But that's what those responses are all about: git doesn't scale, so don't use it if you need scale. If there are no tools that scale available, then change the approach so you don't need face this scaling problem at all.
- jeffbee 4y ago
  Just saying that Perforce "is Perforce" is pretty lazy. Perforce is great in my experience. It scales up to repo sizes you are unlikely to ever reach and has a UX that makes actual sense.
abrookewood 4y ago
I don't think I've ever seen "because your mono-repo is too big for git" used as an argument for micro-services, but maybe at this point it's valid.
- charcircuit 4y ago
  You can put microservices in a monorepo. You just put them in their own folder.
  - abrookewood 4y ago
    Of course, but can you split a monolith across multiple repos?
    - isaacremuant 4y ago
      Yes. With git submodules.
    - scarmig 4y ago
      Sure. It'd drastically complicate the build process and CI for little benefit compared to other approaches (e.g. sparse checkout), but you could even in principle create a repo for each code directory and stitch them all together.
      - charcircuit 4y ago
        >It'd drastically complicate the build process and CI This is a part of most monorepos. I figure having only a single build artifact is rare.
- jaypeedevlin 4y ago
  Canva use microservices FWIW
rep_movsd 4y ago
The Linux kernel has about 30 million lines of code. Windows 10 has about 60 million. I doubt that any other software project has as many. How would you even end up with 500K source files in the repo?
alephnan 4y ago
These engineering blogs are intended to promote the engineering culture of the company for recruitment process, and as a correlate an opportunity for the engineers to self-promote and boost their resume/promotion-worthiness internally. The sentiment on this thread, if it is indicative of the greater talent pool, suggests this blog post is having the complete opposite effect. I remember when Uber was proud of their thousands of repos. Here it’s the 60 million lines of code. It’s not just red flags, but seems like stuff that might get leaked to Programming Horrors / WTFs.
- hamandcheese 4y ago
  I checked out their careers page after reading this.
- deleted 4y ago
  [deleted]
mariusmg 4y ago
>across half a million files >we found that .xlf files made up almost 70% of the total number of files If one xlf file is used to keep the translation for 1 language, how the heck do you have so many then ? Makes no sense. If you have many xlf files for 1 language, may God save your soul.
- BenjiWiebe 4y ago
  One xlf per language _per component_. 5 components, 5 languages, would be 25 xlf files.
kanu666 4y ago
If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO. Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes every time they need to simply push their code changes.
- danielheath 4y ago
  I mean, if you have hundreds of engineers, that might be sound advice; if you have 20, you’ll probably be fine.
  - kanu666 4y ago
    Of course, you shouldn't start a project from 10 repos. But the downside is while you live in a single repo all the tooling get tuned to work with a single repo. And then there is no way to go multi-repo when you need to. So advice would be: start small but think big :)
- bregma 4y ago
  There is no point in making any argument with someone who has the conviction of their religion behind them. Followers of the monorepo religion are no different.
- jeffbee 4y ago
  Google did not need to write their own SCM to reach the scale mentioned in the article. Off-the-shelf Perforce served that company well enough when it had tens of thousands of engineers and millions of files. They had 11 million files in a monorepo with one 256GB machine which most of you would consider a bare bones desktop computer at this point. https://www.perforce.com/sites/default/files/still-all-one-server-perforce-scale-google-wp.pdf https://www.perforce.com/sites/default/files/still-all-one-s...
rdelpret 4y ago
This is funny, I stopped reading when it started talking about how engineer x will never touch section y of the code. Ok then put them in separate repos
- CodeShmode 4y ago
  How is that any better than telling git to ignore them? Esp when they're tightly coupled translations
stevenjgarner 4y ago
I get a headache imagining everything Google in a monorepo. So how do you reconcile this single GIT repository approach with Infoworld's "The case against monorepos"? [1]: Reason #1. Monorepos go against single-team ownership principles Reason #2. Monorepos encourage bad practices involving massive refactoring Reason #3: Small repositories are better than large ones [1] https://www.infoworld.com/article/3638860/the-case-against-monorepos.html https://www.infoworld.com/article/3638860/the-case-against-m...
- pphysch 4y ago
  One person's STOP is another's silo One person's massive refactoring is another's tech debt reduction One person's multirepo is another's inability to find that broken bit of upstream code
- scarmig 4y ago
  With regard to 3: > In Google’s case, more than 45,000 changes are made to its monorepo every day. This code management becomes an exponential problem in overhead as the number of developers of an application grows, and the number of components within the application expands. Speaking from experience, the overhead of code management is far less at Google than any other place I've worked at, even on projects with just a couple hundred lines of code. If this author is imagining a monorepo means each engineer having to constantly check out terabytes of code to make any change and race with other developers to merge to HEAD, they don't really understand the landscape well enough to write an article criticizing monorepos (which certainly have real cons, particularly with the source control tools that are available to most companies). Of course, you need actual tooling and infrastructure for that beyond what git offers.
- wtetzner 4y ago
  I think the biggest problem here is that git (out of the box) is not well suited to a monorepo. But in terms of the reasons in "The case against monorepos": > Reason #1. Monorepos go against single-team ownership principles I think it's up for debate about whether or not single-team ownership is desirable. But even if it is, I don't see the difference. Just have teams own their directories within the monorepo. If you want to enforce that a team owns a particular part of the repo, just put some rules into the code review tool to ensure that a change to that component can't be merged without someone from that team reviewing/approving it. > Reason #2. Monorepos encourage bad practices involving massive refactoring Again, this just appears to be the author's opinion that massive refactoring is somehow problematic, without providing much of an argument against it. If you're changing service APIs, then sure, you need to be careful about the order in which you roll out changes. But if you're changing the APIs of shared libraries, then being able to do a single large refactoring is absolutely valuable. > Reason #3: Small repositories are better than large ones This is just a tooling issue. You can still separate projects by directories, and if your tools allow you to just check out a subdirectory, or if you have some sort of virtual filesystem on top of your source control, then you don't have to pay the penalty of pulling down the entire repository.
TedShiller 4y ago
Just don’t run git gc Runs forever even on small repos
c8g 4y ago
Did you consider VFS For Git and Scalar from Microsoft? What was the result? https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit https://github.com/microsoft/scalar https://github.com/microsoft/scalar
- scarmig 4y ago
  VFS for Git is deprecated, although something incorporating its approach is likely to best way to implement a real monorepo. Which AFAIK doesn't exist, outside of proprietary implementations used inside certain big tech companies.
deleted 4y ago
[deleted]
pronik 4y ago
Somehow, it seems so backwards. We make our repositories more accessible for automated tools and less accessible for people and instead of investing coding time into better tools, we invest it into local workarounds at every single developer's machine. This is probably not a general conclusion on monorepos, but the notion of "let's put auto-generated stuff into version control" sounds like the point in time where it all went awry. Yes, there is ClearCase where this is the way to do it, Perforce probably does this too and there are areas where you are supposed to be archiving your generated binaries with your code and scripts for later audits, but I'd argue most people shouldn't be doing this, since there are ways to separate source and release artifacts once and for all in a way to make both bots and people happy without locally-brewed scripts for checking out code.

222 points by Maxious 4y ago

Every time I read a monorepo story my takeaway is that it's a really bad idea. Segregation of responsibility is a good thing.

sandgiant 4y ago

I've had millions of files in git using git-annex. It kind of works if you're very patient, it's definitely not something I'd recommend. I wish there was a solution, such as git-annex, that worked well with millions of files.

deleted 4y ago

[deleted]

Gigachad 4y ago

>just under 60 million lines of code in 2022. I'm always surprised at the fact that almost every product has more lines of code than the entire Linux repo. The scale of these products is astounding.

mherrmann 4y ago
From the rest of the article, it sounds like a big chunk of these lines are from generated files. What I don't understand is why they're checking in generated files into Git.
- b3n 4y ago
  Note that these files are not statically generated; they are translation files, generated by translators.
- sdflhasjd 4y ago
  Yes, the article glosses over that a little bit, and it's an unusual decision.
- snicker7 4y ago
  As a counter-point, package managers generate “lock files” that are designed to be tracked in VC. For these translation files, I’d imagine there may be occasional work to modify them even after they are initially generated.
- onion2k 4y ago
  They're using git as a cache. Having generated files stored there means they're available if they're needed (eg in CI) without needing further access controls, they're versioned, and it's a simple and understandable strategy. As the article states, most devs are set up to ignore those files so they're not much of a source of the slowness. It's a common pattern for apps that have to serve lots of different locales.
  - fomine3 4y ago
    I don't think it's good idea to store cache in Git. Any file remains forever in the repository after once committed. Local/remote repository become unnecessary big.
    - kadoban 4y ago
      The bigger issues for me are it makes history impossible to read (every change is hidden in an avalanche of crap), merges are a mess (you definitely want to spend forever merging autogened files, right?), PR reviews are annoying, etc.
      - Gigachad 4y ago
        Depends how much generated stuff is there. We have our graphql schema in git even though its auto generated via a library. But its useful in PRs to see exactly how the schema changed as a result of the root change.
        kadoban 4y ago
        Yeah if there's not a lot of it, and if it's easy to regenerate, it can be fine.
      - _ikke_ 4y ago
        You may want to set the '-diff' attribute for these files so that git will not show diffs for these, instead it will show 'Binary files differ'. There is also '-merge', which will cause git to not attempt to merge the contents, but just ask you to pick a side. The challenge however is then verifying the contents of these files in things like merge requests. See https://gitirc.eu/gitattributes.html https://gitirc.eu/gitattributes.html
- wichert 4y ago
  Reading the article they are not generated files, but files that are never touched by developers. Translators will work with those files. I expect that for translators they have a different sparse checkout that only fetches .xlf files for their target languages.
  - onion2k 4y ago
    xlf files are usually generated. They're XML. No one wants to write that by hand.
    - Xylakant 4y ago
      .docx files are archives of xml-files. No one wants to write that by hand. Or, in more words: The format of the files is just the representation on disk - it’s not directly connected to how the files are generated or edited. XML files can be written by hand with suitable editor support.
      - onion2k 4y ago
        I think we have different ideas of what "written by hand" means. Someone making a Word doc is not writing XML by hand.
    - lultimouomo 4y ago
      Also plain text files are usually generated. They are arrays of 1s and 0s. No one wants to write that by hand. I think that this is not a sensible definition of a generated file. A more sensible definition is that a generated file is created automatically from some source, which is not user input (i.e. an other file). This means generated files do not need to be kept under git, as long as their source is checked in. Translations files, even if they are not created with a plain text editor but with some other tool that handles the XML layer, are clearly not generated, as long as the translation is done by a human.
      - onion2k 4y ago
        For the app I work at the moment we use https://lokalise.com/ https://lokalise.com/. We add translation strings to a SaaS app, and then the translation team translate them. I've written a build tool that downloads the translation JSON files from the API using the CLI, or as part of our CI process. Other teams have tools that download their language packs for different iOS and Android apps. The translations are versioned in Lokalise and we using a branching strategy to manage the work. Lokalise has an option to generate xlf files (and JSON, xliff, arb, etc). This is a very typical workflow. Most people are not out there modifying xlf files by opening them in a text editor. For a start, translations usually aren't done by developers. (Huge shoutout to Lokalise btw. I can highly recommend it. It makes building a multi-lingual app across different platforms so much easier.)
- epage 4y ago
  I've been switching a lot of generated files to being checked in (with CI verifying they haven't drifted from the source). The primary motivation has been performance. For example, in Rust code, it means I don't need to foist the code-gen process and all the dependencies needed for it on dependent crates. I've seen this play out similar in other build systems and circumstances. The key is the data needs to be independent of other factors (like the system doing the generation) and the rate of change of the code-gen source and generator has to be relatively low.
- AlfeG 4y ago
  We check-in some generated CSS files. That are generated by external theme cli. Just to be sure, that after version update we can track all changes in the generated CSS files.
  - jokethrowaway 4y ago
    If they're generated you can just re-generate from source every time you need to track changes. You're using git as a cache. You don't need to version a cache.
    - gustavohenke 4y ago
      Regenerating certain things might be fast, but some might not be. Hundreds of engineers pushing code and having to wait for these to be regenerated both locally and on CI means that caching is quite cheap after all.
shric 4y ago
As well as generated files the sibling mentioned I wonder how much is due to vendoring.
alexdowad 4y ago
The natural tendency in almost any software is to keep adding and adding, while rarely throwing anything out. More features, more code, more supported platforms, more supported languages, more this, more that, more, more, more. If it was a physical product, you couldn't keep making it bigger and more complex ad infinitum, because making a physical thing bigger takes more material, and bounded physical resources would be consumed. With software, it's all just bits, and computers can hold a lot of bits. This leads to bigger problems than just git running slowly.
- Gigachad 4y ago
  Canva is absolutely crushing the established design market with their product right now so I'd say having all those features is pretty important for a good tool these days.

37 4y ago

Pretty interesting.... I don't know that much about git, but still fun to read. I guess the main takeaway is don't put all your eggs in one basket? Although it kinda seems like they are going to stick with the monorepo, ("Here’s how we solve them at Canva") Also, I looked up .xlf files and I still don't understand. It's xml, that part makes sense, but it's basically a config file? To tell what process to read which files? Also, I've heard of Canva, but had no idea they were this big/ubiquitous/whatever.... and learning about pseudo localization is interesting too. And the graph for lines of code looks pretty exponential, maybe it's common up to a point, but if it continues at that rate, it will be infinite by about 2026 (okay, I just made that number up, but you get the idea)

Bluecobra 4y ago
Putting my rusty sysadmin hat on, I would wager a guess that if it keeps on growing exponentially something will break. For example, hitting some hard limit like maximum number of files possible in a file system. Seems like you’re playing with fire to me.
- 37 4y ago
  I would think the same... and from reading the comments in this thread, it seems like a pretty spicy topic.
auscompgeek 4y ago
.xlf isn't a config file format here. It refers to XLIFF, a standard for translation files. https://en.wikipedia.org/wiki/XLIFF https://en.wikipedia.org/wiki/XLIFF
- 37 4y ago
  I looked at the Wikipedia article and a couple other pages, I guess the key word is "localization", which basically seems to mean language. I'm pretty sure I knew and forgot that. The comments in this thread from the devs have been informative as well.

nindalf 4y ago

git status scales with the number of files in the repo. Ask HN - Are there any common git operations that scale with the number of commits?

dymk 4y ago
git log ;)
- iqkznnft 4y ago
  I believe git log is limited by the number of lines on the screen...
  - kadoban 4y ago
    It may operate in a streaming fashion if it's going to a pager, but naw it'll output however much you want. Technically it has modes which need to scan all commits in the current branch as well, like the ones that grep for certain changes.
duskwuff 4y ago
Repository maintenance operations like "git gc" scale with the total number of objects in the repo.
- milosmns 4y ago
  I used to work at a company where our git repository would get so polluted with change-related junk that we had to call (many times literally call) BitBucket and ask them to run git clean up on the remote. Otherwise, fetch/pull and many operations on remote would take forever or often even fail completely due to timeout or lack of RAM. It's Thursday and you wanna release a hotfix - but nope - all builds are failing because git can't cope with junk and your CI can't proceed. You then have to call BB instead and ask them to clean up, hoping they'll do it within the same day. Ah, exciting times.

rsanheim 4y ago

It took me a while to find out what Canva actually does, but from https://www.canva.com https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure, we can throw all these generated files into git!". Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP.

arinlen 4y ago
> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.
- rtpg 4y ago
  One reason for doing this is that someone might be using a very small set of the monorepo, so having everything prebuilt means that in general things are very fast, without constantly having to rebuild chunks of your system. This is, I think, very common with stuff like browsers, where you have artifact checkouts that basically include built stuff since otherwise you're sitting there compiling forever all the time. Stuff like Bazel in theory helps with this, but tools that help with this are either super idiosyncratic about how they work (meaning hard to adopt) or outright don't work. I mean personally I would find that pretty annoying and unclean but I like DAGs.
- whynaut 4y ago
  They’re not assuming anything, but stating that the post itself does not cover this well/at all.
- 4y ago

sydthrowaway 4y ago

barbs 4y ago

I feel like I've read about several big companies using monorepos, but I've never understood why. It feels like the source-control equivalent of writing your code in one big file. Does anyone have any good resources for why and how best to implement a monorepo?

qbasic_forever 4y ago
You kind of have to experience the worst of boths worlds to understand where and how each method works and breaks down. With multiple repos it's harder for teams to share code and collaborate. Each team has a repo that becomes a little fiefdom where they are oblivious to who is using their code and how they're using it. Suddenly they'll push out what they think is an innocuous refactor and inadvertently break core functionality other teams took a dependency on for better or worse. So what happens is the team with a dependency now copies the old code into their repo and take on all the extra burden of maintaining this old version, trying to backport fixes, etc. It becomes an enormous mess and time sink. No one ever has time to go back and fix things, and when they eventually are forced to do so it costs more time and effort than it would have taken to do it right from the start. You'll also run into horrible versioning problems where you're stuck on old version X but depend on widget foo which needs current version Y of that dependency. You might say well bad on that team they should have engaged the product managers, made sure their dependencies and usage were well tracked with them, been looped in the process of changes, etc... but in the real world when your boss says X feature needs to be shipped in a few days all of that process goes out the window.
- milosmns 4y ago
  This is valid. But I think it would make sense to challenge this and even (try to) leave if product is dictating tech what to do with their repos... especially if there's a boss figure pushing imaginary deadlines without talking to the team beforehand. There's a job crisis now, I know, but millions of files were not checked in during 2021. We should do better when it comes to organizing how we work
xmprt 4y ago
Here are a few reasons that I've heard and experienced 1. Standardized tooling 2. Fewer issues related to dependencies 3. Hermetic tests 4. Reduced code duplication and easier code sharing

zoomablemind 4y ago

> "...we found that .xlf files made up almost 70% of the total number of files. These .xlf files are generated... they are never manually edited by engineers..." First thought is why not to zip/tar away all of these "convenience" files per generation and add a line into build/install script to unpack them after checkout? Additionally, add the .xlf into .gitignore to exclude them from untracked. Noone cares to diff them as long as their contents is consistent with the checkout. Text compresses quite efficiently so this should not introduce any unreasonable build/install delays.

haizzz 4y ago
I made another comment as well though tldr is these xlf files are translations tied to texts in code so we can't simply ignore them from the repository. The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc...
- GeneralMaximus 4y ago
  Could these translations be moved to another repository? Maybe they could be published as a separate NPM package that devs could install if they needed to look at the translations?
  - klysm 4y ago
    Then you get the annoying problem of having to push an update to that repo and wait for the new version before you can merge changes into the new repo which use the new version. It’s tightly coupled, so they should be co-located.
- zoomablemind 4y ago
  >...The changes have to be kept so that if we say revert to a certain commit, all the translations match with the texts of headers, buttons, etc... The mentioned .zip file is to be kept in the repo. Instead of a whatever number of individual .xlf files per generation, these would get zipped together (say, 'assets/xlf.zip') before the commit and the resulting .zip added to the commit. Similarly, when reverting or on a checkout, it's the .zip that gets checked out and then the .xlf files are unpacked. The packing/unpacking could be done by the same process that handles the .xlf generation (??build). Also this may be automated by git-hooks, though it's more natural to handle the packing of assets during the build stage.

syspec 4y ago

The fact that Canva has a `Source Control team` with at least 5 people on it (going by the thanks at the bottom of the article), means they should probably try a different approach. I think it's a cool company, with a good product, but they're WAY too small to be having a "source control team" on staff. That's at least 1.2MM a year salary / benefits cost.

deleted 4y ago
[deleted]
outsomnia 4y ago
Yeah... also once such a team exists, they are naturally going to resist ideas like not having a monorepo but just using git submodules, not keeping the history of every generated file forever etc. Because then they won't need a 5-man team to handle the fallout any more.
brutusborn 4y ago
That bit surprised me also. I don't have much experience with companies of this size so I have no real idea; how many people would you expect to work on source control at a company of this size? I had a look at Linkedin, they have 4,345 employees.
- kanu666 4y ago
  Usually it is considered good proportion to start with to be 10% to work on a Developer Productivity and it should increase to 15% as company grow. But regarding source control it really depends on the approach: if you are on polyrepo - then 0 is enough. If you decide to write you own source control system like Facebook or Google, then maybe couple of hundred engineers full time until the project is finished :)
milosmns 4y ago
Really weird. But maybe this team is in a country where developers cost 100x less? (: Still weird though.

tehlike 4y ago

Did anyone use scalar from Microsoft? https://github.com/microsoft/scalar https://github.com/microsoft/scalar

amir734jj 4y ago
We use scalar at Microsoft. Working with massive scale of some repos would be near impossible with "vanilla" Git.
Maxious 4y ago
As mentioned in the blog "we are moving towards providing a known version of git", probably Microsoft Git which includes the scalar command but also upstreams many of the optimizations to git core https://github.com/microsoft/scalar#why-did-scalar-move https://github.com/microsoft/scalar#why-did-scalar-move disclaimer: canva staff working on source control

ur-whale 4y ago

> These .xlf files are generated and contain translated strings for each locale ... and they make up 70% of their repo Why would you include generated file in a repo? Do they take to long to remake? [EDIT]: especially given the fact that they're using bazel which is supposed to be the bee's knees of build system?

SerCe 4y ago
The article uses the term generated, but a more precise term could be "produced by translators". The files can't be simply regenerated. We've recently published [1] a separate post that explains the localisation process in more detail. [1]: https://news.ycombinator.com/item?id=28931601 https://news.ycombinator.com/item?id=28931601

urban_winter 4y ago

What's the value of a monorepo if developers only ever check out a small subset of it? Wouldn't multiple repos allow greater scale without any practical reduction in utility? For example, all the localisation files could live in a separate project (if we accept the need to commit them at all). Some tools would be needed to deal with the inevitable problem that developer working sets would not align with project boundaries, but that seems like an easier job than making git scale while maintaining response times.

nsoonhui 4y ago
One of the benefits of monorepo is refactoring. You can just apply a renaming command across all of the files in the solution and all the related names are properly updated. Not that easy to get this to work on multiple repos.
- Galanwe 4y ago
  Sure, how did we ever manage to rename something without monorepos. Oh wait, maybe that's what this "versionning" thing is all about.
  - rtpg 4y ago
    Right, it's "I have to send 5 PRs to 5 different repos, get them all cross merged, and then at the end it's wrong anyways so I have to start all over". Multirepo management is extremely frustrating compared to "it's all in the same folder".
    - doctor_eval 4y ago
      What I took away from TFA is that monorepo management at this scale is “extremely frustrating” too. ISTM that the complexity of managing any repo will be bounded by the size of that repo; a monorepo, being unbounded in size, will, in time, become arbitrarily complex to manage. While a multirepo might occasionally require developers to apply changes to more than one repo at a time, I’ve never found this to be much more than a minor inconvenience; one that could be solved readily with simple tooling, if we had ever felt that the “problem” was even worth solving.

deleted 4y ago

[deleted]

haizzz 4y ago

Hey everyone, author here, the article is a bit misleading in that .xlf files aren't really generated files, they're created through our translation pipeline by real humans. I considered them generated in the sense that they're not directly worked on by engineers who have to deal with them in the repository. The content of these translation files are snapshot in time aligned with the text in our product so simply removing them we would lose all the changes made to translations each time texts are changed. Sorry about that, hope this clears it up! Edit: for more information on translation and xlf files, we have another blog post all about them https://canvatechblog.com/how-to-design-in-every-language-at-once-f2dd66a2780f https://canvatechblog.com/how-to-design-in-every-language-at...

fomine3 4y ago
Thanks for additional info. I'm happy to hear that localization is on your priority. > English source strings used in our frontend code live in Typescript files (.messages.ts). Source strings used in our backend code live in Java files (Message.java). Our internationalization (i18n) pipeline converts this into a series of XLIFF files (ending in .xlf), with one file for each locale. All these files live in the repository, but the translated .xlf files should never be modified by hand since they are updated automatically when strings get translated. Isn't it cache?
- haizzz 4y ago
  Not quite, the .messages.ts and Message.java files are only in English which are then converted into XLIFF files for other languages so they're the source of truth for what the string should be in different locales so they're not cache
prmoustache 4y ago
Can't those xlf files be stored in a separate repo or in an object storage and let the build system fetch them from there?
- necovek 4y ago
  That wouldn't be called a "monorepo" then :D Obviously, the problems they are solving (and admitting to solving) are due to their dedication to the monorepo. With all the effort spent on working around the drawbacks, I really wonder what advantages they are seeing that make it worth their while?

nomilk 4y ago

> git status takes 10 seconds on average > running these commands multiple times a day reduces the total productive time engineers have every day I love the attention paid to this. Often opportunities to prioritise seemingly small efficiency gains are neglected. At 10 seconds per command, an engineer that uses git status 50 times per day spends ~10 minutes per day waiting; an entire work week per year!! Well above the threshold warranting optimisation, and that doesn't even factor in distractions and context switching.

ngshiheng 4y ago
I'm glad I'm not the only one that thinks that way. Even ~3s every git status is driving me insane now.
codeyperson 4y ago
It’s actually even worse than that I think. If something takes over a certain amount of time, then I’m more likely to go do something else while I wait, like check Hackernews. And there goes 20 minutes.
- Sohcahtoa82 4y ago
  I created a `beep` alias that just does `echo "\x07\x07\x07"` which triggers the system notification sound three times. Then if I have a command that will take a while, like a stupidly long `git status`, I do `git status && beep`.
- AshamedCaptain 4y ago
  I'm hoping at least you acknowledge this is _your_ problem, rather than a tooling problem or the like. You just can't expect everything to give you immediate feedback after a couple seconds. This is something that's really a degradation of the newer generations of engineers, since I clearly remember the time where these "somethings" would never take less than a couple minutes, and people did not immediately flee to their nearest distraction, but actually planned their time around it. In fact, if you go further back, these "somethings" would have taken hours, and the older generations still got work done.

bjarneh 4y ago

> to just under 60 million lines of code in 2022 I'd be somewhat intimidated on my first day of work if I cloned a repo with that much sloc...

the_biot 4y ago

They made a bad design decision 10 years ago, have been fighting the fallout for years, and will be doing so forever and ever because things will only ever grow. They wrote a blog post on how clever they think all their workarounds are, at least one of which involves sparse-checkout -- which is perilously close to chopping up your monorepo into several, while still pretending monorepo is fine. I feel like somebody's job and/or ego is heavily invested in keeping things as they are, even if it demonstrably does not scale to their needs, and the solution is blindingly obvious to even a casual observer. That is institutional insanity.

andi999 4y ago
What would be a better structure?
- roflulz 4y ago
  monorepo until it becomes too big -> then, splitting it into 2-3 repos, until each one also gets too big to manage...
  - masklinn 4y ago
    The problem is not the repository, it’s having the translation data in the repo. We have the same issue at $dayjob, the repo is quite large and 80% of it is translation data. Even if they compress ridiculously well (99% last I checked) the number of translation files and the number of exports makes them the vast majority of the cost. However that structure remains a convenient nuisance, and more importantly removing them would really only be useful if we rewrote the entire repository, which breaks all working copies. There’s been a task in a wishlist for years now, but the business incentive just isn’t there. Exit: actually the translation files are 80% of the working copy, they’re closer to 90% of the repo, and on the far side.
    - andi999 4y ago
      But isnt it sort of correct to have translation data, which is text data, and there might be bug fixes, in the repo?

EdSchouten 4y ago

Maybe at their scale it makes more sense to switch to a VCS like Eden? https://github.com/facebookexperimental/eden https://github.com/facebookexperimental/eden Eden's equivalent of 'git status' should run almost instantaneous, as checkouts are hosted by a virtual file system (FUSE) that tracks changes.

Thorentis 4y ago

I don't understand the logic of combining microservices with a monorepo. The whole point of microservices is that you don't care what is beyond the external contract of the service. Who cares how each individual team decides to name their stuff. Why do microservice teams need to care about having every single service checked out? Who or what would be bulk applying changes to all services? This is madness.

hamandcheese 4y ago
Suppose you want to deprecate an API you wrote in favor of something else for $valid_reasons. In a monorepo with the right tooling, I can make a branch where I delete the API and get pretty immediate feedback as to every module I have broken. From there I can update all the call sites and I also know which teams/engineers I should give a heads-up to. In a multi-repo world, this is much more difficult. Even learning what all the reverse dependencies are could be a challenge. Most likely, other teams have to pull in my changes on their own schedule, and other teams have very different incentives than my own. The cost of mistakes is therefore higher because they are more difficult to undo. Monorepos are very important if you want to empower engineers to achieve broad changes across the org. Some people fundamentally disagree with this methodology. And a fair number of people just don’t care enough - marking the old thing deprecated, calling it a day, and letting it rot for eternity is good enough for them. These are the same people who you’ll find saying “not my job” a lot, in my experience.
- wewtyflakes 4y ago
  The perceived ease of that operation in a monorepo is what makes it dangerous. Unless these changes all map to a single monolith service, then even though you have updated all call sites, these callers will not be deployed all at the same time, meaning as the change rolls out you may see random breakage. By using polyrepo, the deployment boundary can align with the code boundary, making the rollout problem obvious.
  - hamandcheese 4y ago
    People point this out often, but in practice I have never seen this cause issues. Remove callers first, then remove endpoint... its pretty obvious the order in which things need to be done. What I have seen as a real problem, time and time again, is having trouble locating all usages of an API in a multi-repo scenario. Anyone who fucks this up in a monoreppo will probably fuck it up worse with multiple repos.

anonymous_1234 4y ago

I wonder why these generated files are not part of a separate distributed, HA based, cloud managed document store. Seems like a perfect use case for it. This looks like git abuse to me.

compressedgas 4y ago

This reminded me of https://github.com/twosigma/git-meta https://github.com/twosigma/git-meta

cryptonector 4y ago
Yeah, git-meta is a reasonable alternative to megamonorepos. It's basically a repo-mega-forest. A few things: - your build system will need more code to deal with a repo-mega-forest than a megamonorepo - code indexers may not be able to see cross-repo dependencies etc. You'll have pain no matter what. My preference would be for a megamonorepo approach to scale properly. That means partial/sparse cloning, as well as shallow cloning, and also all the hacks that are supposed to make git-status and git-log (and git-blame, and...) fast in partial clones.

hathym 4y ago

the easy solution is to put node_modules in your .gitignore :D

maccard 4y ago

I appreciate this post. It's nice to see that there are other teams that feel some of the pain points of git, (and unsurprising that most of the responses are "you're holding it wrong"). The fact is that git doesn't scale to _very large_ repos, We've seen it time and time again, but there isn't really a great alternative. Perforce is.... Perforce (centralized, very expensive to license, branches are incredibly expensive and streams still feel like a band aid even years and years later). PlasticSCM (which we use at work) is fine, but closed source, mildly expensive, and has a terrible UX

Ygg2 4y ago
There was pijul. Which allowed partial pulls iirc. I haven't used it, so I can't really recommended it.
- maccard 4y ago
  I tried pijul [0] last time this conversation came up and despite its claims it is not fast. [0] https://news.ycombinator.com/item?id=29992875 https://news.ycombinator.com/item?id=29992875
  - pmeunier 4y ago
    Just seeing this now. If you look at the conversation, this was fixed within a few hours.
kanu666 4y ago
But that's what those responses are all about: git doesn't scale, so don't use it if you need scale. If there are no tools that scale available, then change the approach so you don't need face this scaling problem at all.
jeffbee 4y ago
Just saying that Perforce "is Perforce" is pretty lazy. Perforce is great in my experience. It scales up to repo sizes you are unlikely to ever reach and has a UX that makes actual sense.

abrookewood 4y ago

I don't think I've ever seen "because your mono-repo is too big for git" used as an argument for micro-services, but maybe at this point it's valid.

charcircuit 4y ago
You can put microservices in a monorepo. You just put them in their own folder.
- abrookewood 4y ago
  Of course, but can you split a monolith across multiple repos?
  - isaacremuant 4y ago
    Yes. With git submodules.
  - scarmig 4y ago
    Sure. It'd drastically complicate the build process and CI for little benefit compared to other approaches (e.g. sparse checkout), but you could even in principle create a repo for each code directory and stitch them all together.
    - charcircuit 4y ago
      >It'd drastically complicate the build process and CI This is a part of most monorepos. I figure having only a single build artifact is rare.
jaypeedevlin 4y ago
Canva use microservices FWIW

rep_movsd 4y ago

The Linux kernel has about 30 million lines of code. Windows 10 has about 60 million. I doubt that any other software project has as many. How would you even end up with 500K source files in the repo?

alephnan 4y ago

These engineering blogs are intended to promote the engineering culture of the company for recruitment process, and as a correlate an opportunity for the engineers to self-promote and boost their resume/promotion-worthiness internally. The sentiment on this thread, if it is indicative of the greater talent pool, suggests this blog post is having the complete opposite effect. I remember when Uber was proud of their thousands of repos. Here it’s the 60 million lines of code. It’s not just red flags, but seems like stuff that might get leaked to Programming Horrors / WTFs.

hamandcheese 4y ago
I checked out their careers page after reading this.
deleted 4y ago
[deleted]

mariusmg 4y ago

>across half a million files >we found that .xlf files made up almost 70% of the total number of files If one xlf file is used to keep the translation for 1 language, how the heck do you have so many then ? Makes no sense. If you have many xlf files for 1 language, may God save your soul.

BenjiWiebe 4y ago
One xlf per language _per component_. 5 components, 5 languages, would be 25 xlf files.

kanu666 4y ago

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO. Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes every time they need to simply push their code changes.

danielheath 4y ago
I mean, if you have hundreds of engineers, that might be sound advice; if you have 20, you’ll probably be fine.
- kanu666 4y ago
  Of course, you shouldn't start a project from 10 repos. But the downside is while you live in a single repo all the tooling get tuned to work with a single repo. And then there is no way to go multi-repo when you need to. So advice would be: start small but think big :)
bregma 4y ago
There is no point in making any argument with someone who has the conviction of their religion behind them. Followers of the monorepo religion are no different.
jeffbee 4y ago
Google did not need to write their own SCM to reach the scale mentioned in the article. Off-the-shelf Perforce served that company well enough when it had tens of thousands of engineers and millions of files. They had 11 million files in a monorepo with one 256GB machine which most of you would consider a bare bones desktop computer at this point. https://www.perforce.com/sites/default/files/still-all-one-server-perforce-scale-google-wp.pdf https://www.perforce.com/sites/default/files/still-all-one-s...

rdelpret 4y ago

This is funny, I stopped reading when it started talking about how engineer x will never touch section y of the code. Ok then put them in separate repos

CodeShmode 4y ago
How is that any better than telling git to ignore them? Esp when they're tightly coupled translations

stevenjgarner 4y ago

I get a headache imagining everything Google in a monorepo. So how do you reconcile this single GIT repository approach with Infoworld's "The case against monorepos"? [1]: Reason #1. Monorepos go against single-team ownership principles Reason #2. Monorepos encourage bad practices involving massive refactoring Reason #3: Small repositories are better than large ones [1] https://www.infoworld.com/article/3638860/the-case-against-monorepos.html https://www.infoworld.com/article/3638860/the-case-against-m...

pphysch 4y ago
One person's STOP is another's silo One person's massive refactoring is another's tech debt reduction One person's multirepo is another's inability to find that broken bit of upstream code
scarmig 4y ago
With regard to 3: > In Google’s case, more than 45,000 changes are made to its monorepo every day. This code management becomes an exponential problem in overhead as the number of developers of an application grows, and the number of components within the application expands. Speaking from experience, the overhead of code management is far less at Google than any other place I've worked at, even on projects with just a couple hundred lines of code. If this author is imagining a monorepo means each engineer having to constantly check out terabytes of code to make any change and race with other developers to merge to HEAD, they don't really understand the landscape well enough to write an article criticizing monorepos (which certainly have real cons, particularly with the source control tools that are available to most companies). Of course, you need actual tooling and infrastructure for that beyond what git offers.
wtetzner 4y ago
I think the biggest problem here is that git (out of the box) is not well suited to a monorepo. But in terms of the reasons in "The case against monorepos": > Reason #1. Monorepos go against single-team ownership principles I think it's up for debate about whether or not single-team ownership is desirable. But even if it is, I don't see the difference. Just have teams own their directories within the monorepo. If you want to enforce that a team owns a particular part of the repo, just put some rules into the code review tool to ensure that a change to that component can't be merged without someone from that team reviewing/approving it. > Reason #2. Monorepos encourage bad practices involving massive refactoring Again, this just appears to be the author's opinion that massive refactoring is somehow problematic, without providing much of an argument against it. If you're changing service APIs, then sure, you need to be careful about the order in which you roll out changes. But if you're changing the APIs of shared libraries, then being able to do a single large refactoring is absolutely valuable. > Reason #3: Small repositories are better than large ones This is just a tooling issue. You can still separate projects by directories, and if your tools allow you to just check out a subdirectory, or if you have some sort of virtual filesystem on top of your source control, then you don't have to pay the penalty of pulling down the entire repository.

TedShiller 4y ago

Just don’t run git gc Runs forever even on small repos

c8g 4y ago

Did you consider VFS For Git and Scalar from Microsoft? What was the result? https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit https://github.com/microsoft/scalar https://github.com/microsoft/scalar

scarmig 4y ago
VFS for Git is deprecated, although something incorporating its approach is likely to best way to implement a real monorepo. Which AFAIK doesn't exist, outside of proprietary implementations used inside certain big tech companies.

deleted 4y ago

[deleted]

pronik 4y ago

Somehow, it seems so backwards. We make our repositories more accessible for automated tools and less accessible for people and instead of investing coding time into better tools, we invest it into local workarounds at every single developer's machine. This is probably not a general conclusion on monorepos, but the notion of "let's put auto-generated stuff into version control" sounds like the point in time where it all went awry. Yes, there is ClearCase where this is the way to do it, Perforce probably does this too and there are areas where you are supposed to be archiving your generated binaries with your code and scripts for later audits, but I'd argue most people shouldn't be doing this, since there are ways to separate source and release artifacts once and for all in a way to make both bots and people happy without locally-brewed scripts for checking out code.