5 ms·

I've worked at both FB (monorepo) and Amazon (massively federated repo where you only check out tiny slices at a time; a huge package dependency system pulls th

by newobj 10y ago

I've worked at both FB (monorepo) and Amazon (massively federated repo where you only check out tiny slices at a time; a huge package dependency system pulls things together), and I can testify that monorepo is great, and federated repo with dependencies-via-packaging is awful. It's slower to work with, harder to coordinate wide scale changes, slower to work with, the cognitive load is much higher, and it's slower to work with. Did I mention it's higher friction and slower to work with?

olalonde 10y ago
What do people mean by monorepo exactly (e.g. at Facebook)? For instance, FB's open source projects aren't all hosted in the same "open source monorepo", why not? And what goes into the monorepos exactly? All the closed source code? As a "one-man-team" who uses at least 15 different repos, it's hard for me to imagine how a massive company would manage things within a single repo and no package management. Also, aren't any of those monorepo companies concerned that a single rogue employee or stolen laptop could leak their entire source code? I can only guess I'm misunderstanding what is meant by monorepo and how they're used...
- encoderer 10y ago
  Friends tell me that very sensitive code (ranking algos, etc) is not in the repo. Personally, I also favor a single repo. You manage it the way you manage separate packages: with organization and some discipline. The magic is that command line tools like grep, sed & awk--and static analysis tools for refactoring--work really well. You can change a method signature and it just works. I've been part of monolith-breaking before (most recently at Trulia) and it definitely adds friction to the development process to work across such a rich graph of package dependencies.
  - honkhonkpants 10y ago
    Sensitive code _is_ in the repo, you just can't read it. This allows you to change library code, run the tests of the sensitive application code, and know that it worked, without being able to read the sensitive code.
    - lobster_johnson 10y ago
      What prevents you from reading it, if it's in the repo?
      - reitanqild 10y ago
        Access control if my sources are correct.
        elcapitan 10y ago
        So you can read the interface and compile against it, but not the source of the actual algorithm? Do you know how that works?
        reitanqild 10y ago
        No idea how it actually works but here are a couple of ideas about how it could work: * Compilation is done centrally, you code against a mock or only the interface and the submit the code for test and final build. * Or only libraries are supplied, possibly ofuscated.
- jtolmar 10y ago
  The majority of Google's code is in one gigantic repository. Sometimes there are some other repositories for very sensitive code, for example if you have a hardware vendor's closed-source driver code and they don't want many people to look at it. A few projects are in different repositories for their own reasons, like major open source projects. I don't think there's a single clear criteria you could come up with for what doesn't go in the main repository. There are dependency management tools that help enforce public/private code on a wider scale and that help the build tools make sense of it all. There are also ownership tools that say what people and teams are qualified to review code in certain directories. The config files for all these tools are checked into the repository. There's no versioning though. If you want to change an internal API you just update it and all the callers at once: patches in source control are already atomic. For truly massive changes (more code than most companies have) this gets too unwieldy and there are special tools and strategies people use. This means you can't have a project depending on an out of date (internal) library. Without that requirement, you don't have the situation where different libraries need to be synced to different versions. And without needing to sync different things differently, you can get away with just one repository. I worked at Amazon previously, which has world-class tools for dealing with versioned libraries in bulk, and Google's approach is vastly better. You spend less time worrying about breaking other people's dependencies, and you don't have someone spending a day fixing libraries every couple of weeks.
- jamesgpearce 10y ago
  We export each of the open source projects out of the monorepo and into separate GitHub projects using a tool called ShipIt (https://code.facebook.com/posts/1715560542066337/automatically-push-commits-to-github-with-fbshipit/ https://code.facebook.com/posts/1715560542066337/automatical...)
  - manojlds 10y ago
    FBShipIt has been primarily designed for branches with linear histories; in particular, it does not understand merge commits. That's sad.
    - serge2k 10y ago
      Sounds wonderful to me.
- Chyzwar 10y ago
  PHP dont have proper module system. It not like Facebook had any choice. Even with stolen FB code without dedicated infrastructure you still cannot do anything.

by newobj 10y ago

olalonde 10y ago

What do people mean by monorepo exactly (e.g. at Facebook)? For instance, FB's open source projects aren't all hosted in the same "open source monorepo", why not? And what goes into the monorepos exactly? All the closed source code? As a "one-man-team" who uses at least 15 different repos, it's hard for me to imagine how a massive company would manage things within a single repo and no package management. Also, aren't any of those monorepo companies concerned that a single rogue employee or stolen laptop could leak their entire source code? I can only guess I'm misunderstanding what is meant by monorepo and how they're used...

encoderer 10y ago
Friends tell me that very sensitive code (ranking algos, etc) is not in the repo. Personally, I also favor a single repo. You manage it the way you manage separate packages: with organization and some discipline. The magic is that command line tools like grep, sed & awk--and static analysis tools for refactoring--work really well. You can change a method signature and it just works. I've been part of monolith-breaking before (most recently at Trulia) and it definitely adds friction to the development process to work across such a rich graph of package dependencies.
- honkhonkpants 10y ago
  Sensitive code _is_ in the repo, you just can't read it. This allows you to change library code, run the tests of the sensitive application code, and know that it worked, without being able to read the sensitive code.
  - lobster_johnson 10y ago
    What prevents you from reading it, if it's in the repo?
    - reitanqild 10y ago
      Access control if my sources are correct.
      - elcapitan 10y ago
        So you can read the interface and compile against it, but not the source of the actual algorithm? Do you know how that works?
        reitanqild 10y ago
        No idea how it actually works but here are a couple of ideas about how it could work: * Compilation is done centrally, you code against a mock or only the interface and the submit the code for test and final build. * Or only libraries are supplied, possibly ofuscated.
jtolmar 10y ago
The majority of Google's code is in one gigantic repository. Sometimes there are some other repositories for very sensitive code, for example if you have a hardware vendor's closed-source driver code and they don't want many people to look at it. A few projects are in different repositories for their own reasons, like major open source projects. I don't think there's a single clear criteria you could come up with for what doesn't go in the main repository. There are dependency management tools that help enforce public/private code on a wider scale and that help the build tools make sense of it all. There are also ownership tools that say what people and teams are qualified to review code in certain directories. The config files for all these tools are checked into the repository. There's no versioning though. If you want to change an internal API you just update it and all the callers at once: patches in source control are already atomic. For truly massive changes (more code than most companies have) this gets too unwieldy and there are special tools and strategies people use. This means you can't have a project depending on an out of date (internal) library. Without that requirement, you don't have the situation where different libraries need to be synced to different versions. And without needing to sync different things differently, you can get away with just one repository. I worked at Amazon previously, which has world-class tools for dealing with versioned libraries in bulk, and Google's approach is vastly better. You spend less time worrying about breaking other people's dependencies, and you don't have someone spending a day fixing libraries every couple of weeks.
jamesgpearce 10y ago
We export each of the open source projects out of the monorepo and into separate GitHub projects using a tool called ShipIt (https://code.facebook.com/posts/1715560542066337/automatically-push-commits-to-github-with-fbshipit/ https://code.facebook.com/posts/1715560542066337/automatical...)
- manojlds 10y ago
  FBShipIt has been primarily designed for branches with linear histories; in particular, it does not understand merge commits. That's sad.
  - serge2k 10y ago
    Sounds wonderful to me.
Chyzwar 10y ago
PHP dont have proper module system. It not like Facebook had any choice. Even with stolen FB code without dedicated infrastructure you still cannot do anything.