7 ms·

It took me a while to find out what Canva actually does, but from https://www.canva.com https://www.canva.com it appears they are an online design / collab tool

It took me a while to find out what Canva actually does, but from https://www.canva.com https://www.canva.com it appears they are an online design / collab tool. To be fair, that could mean a _lot_ of functionality and code providing a rich, SPA, JS heavy experience. Modern JS frameworks aren't exactly known for being concise. But still, 60 million lines of code and half a million files is a sure sign that someone said at one point "sure, we can throw all these generated files into git!". Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP.

arinlen 4y ago
> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.
- rtpg 4y ago
  One reason for doing this is that someone might be using a very small set of the monorepo, so having everything prebuilt means that in general things are very fast, without constantly having to rebuild chunks of your system. This is, I think, very common with stuff like browsers, where you have artifact checkouts that basically include built stuff since otherwise you're sitting there compiling forever all the time. Stuff like Bazel in theory helps with this, but tools that help with this are either super idiosyncratic about how they work (meaning hard to adopt) or outright don't work. I mean personally I would find that pretty annoying and unclean but I like DAGs.
- whynaut 4y ago
  They’re not assuming anything, but stating that the post itself does not cover this well/at all.
- heyparkerj 4y ago
  The post itself covers in the intro that they made the conscious decision to go with a monorepo, accepting the downsides of it. Much more than this article though, I'd like to read one discussing that decision and why they went that direction.
SenHeng 4y ago
Remember back when people recommended commiting node_modules into git?
- jamesfinlayson 4y ago
  Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within. It is gone now.
  - SenHeng 4y ago
    The days before lock files were a thing and 'it works on my machine!' was rampant.
    - anitil 4y ago
      ... are we no longer doing 'works on my machine' ?
      - coding123 4y ago
        At companies that don't check in node_modules, build folders, and are using standard packaging tooling like maven or yarn or npm or what-have-you. Yes, I haven't experienced that in like 15 years.
      - dotancohen 4y ago
        Ostensibly if it works in Alice's Docker instance, it will run in Bob's Docker instance too.
        Beltiras 4y ago
        Except for weird Docker edge cases (extremely rare, but does happen).
        necovek 4y ago
ElemenoPicuares 4y ago
Canva gives users without design tools expertise the ability to make fairly polished looking graphics with a super easy and intuitive interface. (As a designer, I can assure you that polished looking is not the same thing as designed.) It’s a very popular service, so they’re dealing with huge scale. Intuitive interfaces often come with complex mechanisms and lots of assets, and they have clients on every major mobile and desktop platform, and the web. They also do a ton of heavy graphics processing that is likely done in lower-level languages than the interface. The likelihood of a 2000 employee software company simply not considering that they could streamline their build process is pretty slim.
- _the_inflator 4y ago
  Don't rely on this anecdotal heuristic. Have a look at some enterprises. My experience: "How to solve scaling issues?" - "Automation? No, another team." ;)
  - ElemenoPicuares 4y ago
    This is not a 15k employee enterprise riveting features onto a codebase from the 90s— it’s a <10yo company that makes one product. Big difference in approach.
- necovek 4y ago
  They have obviously invested a lot over time into streamlining their build process: so much so that they're putting an article about it. All of the problems they are having are basically due to their use of a monorepo: they do explain that they made the decision early, but I wonder what are the advantages over multiple repos they are seeing that it was worth it all this trouble?
  - ralphb 4y ago
    We can talk about the advantages of monorepos, but your questions is phrased in a way that makes me think that you don't see any "trouble" in multiple repositories. I would encourage you to do some research and keep an open mind.
haizzz 4y ago
hey hey author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them
- necovek 4y ago
  How do you split your XLIFF files? Does each project get one big one and the proliferation is simply due to number of languages, or do you have a more granular split (eg. if you've got one component, it will have dozens of XLIFF files for every language, instead of one per language)? By the numbers you mention, 70% of the files make a ratio of code files to translation files 1-3, so unless you only support 3-5 languages, it's definitely not one XLIFF file per source file, so I wonder at what granularity it is? (My experience is mostly with localizations using GNU gettext tools, and you usually do a small-finite-number of PO files per project per language, where that small-finite number is exactly one for like 99% of projects)
  - auscompgeek 4y ago
    It's one XLIFF file per locale per component, not including the source en_US. We currently support 104 locales. More info: https://news.ycombinator.com/item?id=28931601 https://news.ycombinator.com/item?id=28931601
    - necovek 4y ago
      Thanks: that's still a lot of components (3000+) if you've got at least 300k xlf files! Good job managing all that regardless of the approach :)
dotancohen 4y ago
> it takes 10 seconds to run git status People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.
- trasz 4y ago
  From my experience SVN isn’t significantly slower than git.
  - dotancohen 4y ago
    My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework. I do not remember if "stat" was particularly slow, but SVN in general is slow.
    - trasz 4y ago
      Yeah, branches are slow. Otoh blame is fast compared to git.
      - capitainenemo 4y ago
        These days mercurial has a cached blame/annotate called "fast annotate" which I love because of one particular awesome feature --deleted which must be seen to be appreciated I think. I have this alias in my .hgrc file fad=fastannotate -u -n -wbB --deleted It's by the Facebook engineer Jun Wu who also made the even more awesome "absorb"
    - capitainenemo 4y ago
      Huh. 10 gigabyte svn repo at work spanning about 40 projects.. Creating branches is virtually instantaneous. It's just a copy which is a free operation (just a link). Curious as to why it would be slow for you. svn cp https:/ /server/svn/trunk/project/ https:/ /server/svn/branches/project/ticket -m "making a branch here" svn status, even for an entire repo checkout (which is not common) is also fast. And yeah, it has virtue of simplicity as well doing very well at narrow and shallow even though I'd love to have mercurial's feature set. It's also rather good in the "wiki" situation since people can operate on their single files without needing to update, sync and merge. https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-return-to-sanity/ https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs... A fun rant, even though git has gotten better-ish at large files.
tasn 4y ago
One of my biggest pet peeves: tracking generated files in version control. The only exception is our generated OpenAPI spec, because we want people to be explicit about modifying the API, and have a CI task that verifies that the API and OpenAPI spec match.
- cnity 4y ago
  One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations. Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"
  - tasn 4y ago
    Hm... An interesting thought! What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.
    - cmcconomy 4y ago
      you could put it in the pre-commit githook
      - tasn 4y ago
        That's not enforced centrally (or visible by the reviewer) so hard to trust it that everyone will remember having it. Though even without that, I'm not sure how it'll even mechanically work.
    - cnity 4y ago
      Yeah either you'd have to maintain the generated spec as a versioned artifact hosted wherever is most convenient for you, or the CI could actually generate the before and after specs based on the PR diff. If the calculation of the spec is computationally expensive (it shouldn't be) then the latter approach could be a problem.
deleted 4y ago
[deleted]
withinboredom 4y ago
Not everyone has the skills to build the toolset and use it. My brother called last night to help him change some SASS variables in a bootstrap theme. He’s a data scientist and had no idea how to build bootstrap’s js and apply the new variables. If bootstrap came from npm fully built, over half of his problems he called me about (15 times!) would have been avoided.
hzhou321 4y ago
I think it is a crowd dumbing effect. Since hundreds of engineers sharing the mono-repo, no one can or care to make the decision or is able to push the decision for alternate. Even when every one is complaining, it is still far from every one agreeing on the alternate. Crowd settle at the lowest denominator.
jefurii 4y ago
> Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP. It's perfectly fine to use Git to track things other than sourcecode. In fact, right on the manpage, Git calls itself "the stupid content tracker". I've been using Git with git-annex to track archival files with their associated metadata. We keep our data separate from our sourcecode, and segment our data into individual Git repositories for each collection. Git gives us many features that we would have had to build into our app in other ways (data integrity, fixity, etc), though this came with costs. To my eye it probably would have been better for Canva to use multiple separate repositories instead of a monorepo, but I'm not them and their use-case is not mine.

arinlen 4y ago

> Didn't someone on the team say "hey, it takes 10 seconds to run git status, can we move this junk out and do this another way??" Why do you assume they didn't? Just because they arrived at a different conclusion than you that doesn't mean they didn't thought about it. I might very well mean you did not considered the tradeoffs they had to take into account, mainly because you're out of the loop.

rtpg 4y ago
One reason for doing this is that someone might be using a very small set of the monorepo, so having everything prebuilt means that in general things are very fast, without constantly having to rebuild chunks of your system. This is, I think, very common with stuff like browsers, where you have artifact checkouts that basically include built stuff since otherwise you're sitting there compiling forever all the time. Stuff like Bazel in theory helps with this, but tools that help with this are either super idiosyncratic about how they work (meaning hard to adopt) or outright don't work. I mean personally I would find that pretty annoying and unclean but I like DAGs.
whynaut 4y ago
They’re not assuming anything, but stating that the post itself does not cover this well/at all.
heyparkerj 4y ago
The post itself covers in the intro that they made the conscious decision to go with a monorepo, accepting the downsides of it. Much more than this article though, I'd like to read one discussing that decision and why they went that direction.

SenHeng 4y ago

Remember back when people recommended commiting node_modules into git?

jamesfinlayson 4y ago
Ah - that would explain why at my current job there was a node_modules directory in git with nearly 2 million lines of Javascript within. It is gone now.
- SenHeng 4y ago
  The days before lock files were a thing and 'it works on my machine!' was rampant.
  - anitil 4y ago
    ... are we no longer doing 'works on my machine' ?
    - coding123 4y ago
      At companies that don't check in node_modules, build folders, and are using standard packaging tooling like maven or yarn or npm or what-have-you. Yes, I haven't experienced that in like 15 years.
    - dotancohen 4y ago
      Ostensibly if it works in Alice's Docker instance, it will run in Bob's Docker instance too.
      - Beltiras 4y ago
        Except for weird Docker edge cases (extremely rare, but does happen).
        necovek 4y ago

ElemenoPicuares 4y ago

Canva gives users without design tools expertise the ability to make fairly polished looking graphics with a super easy and intuitive interface. (As a designer, I can assure you that polished looking is not the same thing as designed.) It’s a very popular service, so they’re dealing with huge scale. Intuitive interfaces often come with complex mechanisms and lots of assets, and they have clients on every major mobile and desktop platform, and the web. They also do a ton of heavy graphics processing that is likely done in lower-level languages than the interface. The likelihood of a 2000 employee software company simply not considering that they could streamline their build process is pretty slim.

_the_inflator 4y ago
Don't rely on this anecdotal heuristic. Have a look at some enterprises. My experience: "How to solve scaling issues?" - "Automation? No, another team." ;)
- ElemenoPicuares 4y ago
  This is not a 15k employee enterprise riveting features onto a codebase from the 90s— it’s a <10yo company that makes one product. Big difference in approach.
necovek 4y ago
They have obviously invested a lot over time into streamlining their build process: so much so that they're putting an article about it. All of the problems they are having are basically due to their use of a monorepo: they do explain that they made the decision early, but I wonder what are the advantages over multiple repos they are seeing that it was worth it all this trouble?
- ralphb 4y ago
  We can talk about the advantages of monorepos, but your questions is phrased in a way that makes me think that you don't see any "trouble" in multiple repositories. I would encourage you to do some research and keep an open mind.

haizzz 4y ago

hey hey author here, xlf files are translations that are coupled with the texts we set in the code so they're not really generated I admit that was misleading. What I wanted to get across is they're not touched directly by engineers but they're still created through our translation pipeline where real humans translate them

necovek 4y ago
How do you split your XLIFF files? Does each project get one big one and the proliferation is simply due to number of languages, or do you have a more granular split (eg. if you've got one component, it will have dozens of XLIFF files for every language, instead of one per language)? By the numbers you mention, 70% of the files make a ratio of code files to translation files 1-3, so unless you only support 3-5 languages, it's definitely not one XLIFF file per source file, so I wonder at what granularity it is? (My experience is mostly with localizations using GNU gettext tools, and you usually do a small-finite-number of PO files per project per language, where that small-finite number is exactly one for like 99% of projects)
- auscompgeek 4y ago
  It's one XLIFF file per locale per component, not including the source en_US. We currently support 104 locales. More info: https://news.ycombinator.com/item?id=28931601 https://news.ycombinator.com/item?id=28931601
  - necovek 4y ago
    Thanks: that's still a lot of components (3000+) if you've got at least 300k xlf files! Good job managing all that regardless of the approach :)

dotancohen 4y ago

> it takes 10 seconds to run git status People coming from the SVN world do not think that this is unusual or problematic. And unfortunately even recently I've seen SVN still in use at large legacy companies.

trasz 4y ago
From my experience SVN isn’t significantly slower than git.
- dotancohen 4y ago
  My experience is that anything dealing with a branch, especially but not exclusively creating branches, is very slow in SVN for a repo of any real size, basically anything with a framework. I do not remember if "stat" was particularly slow, but SVN in general is slow.
  - trasz 4y ago
    Yeah, branches are slow. Otoh blame is fast compared to git.
    - capitainenemo 4y ago
      These days mercurial has a cached blame/annotate called "fast annotate" which I love because of one particular awesome feature --deleted which must be seen to be appreciated I think. I have this alias in my .hgrc file fad=fastannotate -u -n -wbB --deleted It's by the Facebook engineer Jun Wu who also made the even more awesome "absorb"
  - capitainenemo 4y ago
    Huh. 10 gigabyte svn repo at work spanning about 40 projects.. Creating branches is virtually instantaneous. It's just a copy which is a free operation (just a link). Curious as to why it would be slow for you. svn cp https:/ /server/svn/trunk/project/ https:/ /server/svn/branches/project/ticket -m "making a branch here" svn status, even for an entire repo checkout (which is not common) is also fast. And yeah, it has virtue of simplicity as well doing very well at narrow and shallow even though I'd love to have mercurial's feature set. It's also rather good in the "wiki" situation since people can operate on their single files without needing to update, sync and merge. https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-return-to-sanity/ https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs... A fun rant, even though git has gotten better-ish at large files.

tasn 4y ago

One of my biggest pet peeves: tracking generated files in version control. The only exception is our generated OpenAPI spec, because we want people to be explicit about modifying the API, and have a CI task that verifies that the API and OpenAPI spec match.

cnity 4y ago
One alternative that enables you to keep generated files out but still feel like there's an explicit human check in place is to add a gated confirmation step in CI to confirm that the changes to the generated spec match expectations. Something like: "This change will result in the following new API endpoints: ... do you wish to continue?"
- tasn 4y ago
  Hm... An interesting thought! What does it compare against though? Need to add more state to the CI? We kinda like the interface be part of the version control and having an audit chain that's part of the code.
  - cmcconomy 4y ago
    you could put it in the pre-commit githook
    - tasn 4y ago
      That's not enforced centrally (or visible by the reviewer) so hard to trust it that everyone will remember having it. Though even without that, I'm not sure how it'll even mechanically work.
  - cnity 4y ago
    Yeah either you'd have to maintain the generated spec as a versioned artifact hosted wherever is most convenient for you, or the CI could actually generate the before and after specs based on the PR diff. If the calculation of the spec is computationally expensive (it shouldn't be) then the latter approach could be a problem.

deleted 4y ago

[deleted]

withinboredom 4y ago

Not everyone has the skills to build the toolset and use it. My brother called last night to help him change some SASS variables in a bootstrap theme. He’s a data scientist and had no idea how to build bootstrap’s js and apply the new variables. If bootstrap came from npm fully built, over half of his problems he called me about (15 times!) would have been avoided.

hzhou321 4y ago

I think it is a crowd dumbing effect. Since hundreds of engineers sharing the mono-repo, no one can or care to make the decision or is able to push the decision for alternate. Even when every one is complaining, it is still far from every one agreeing on the alternate. Crowd settle at the lowest denominator.

jefurii 4y ago

> Given that 70% of their repo is generated files, that discussion and the tradeoffs involved don't get nearly enough attention from OP. It's perfectly fine to use Git to track things other than sourcecode. In fact, right on the manpage, Git calls itself "the stupid content tracker". I've been using Git with git-annex to track archival files with their associated metadata. We keep our data separate from our sourcecode, and segment our data into individual Git repositories for each collection. Git gives us many features that we would have had to build into our app in other ways (data integrity, fixity, etc), though this came with costs. To my eye it probably would have been better for Canva to use multiple separate repositories instead of a monorepo, but I'm not them and their use-case is not mine.