5 ms·

A Mercurial compatible SCM (not sure if it is a fork) built for their workflow (monorepo) and scale (enormous, git is not usable at their scale, at least for a

A Mercurial compatible SCM (not sure if it is a fork) built for their workflow (monorepo) and scale (enormous, git is not usable at their scale, at least for a monorepo). Uses Python and Rust. Designed for efficient centralization rather than decentralization.

deanCommie 5y ago
So if you are at Facebook's or Google's scale, and also run a monorepo, this will be great for you. Which is to say, this is a product for one company - Facebook.
- pedrogpimenta 5y ago
  Or Google, as per your first sentence :)
  - onlyrealcuzzo 5y ago
    Google already has its own [=
    - dijit 5y ago
      Would kill to have piper open source though.
      - kevincox 5y ago
        I would be interesting but basically useless. - Requires a custom kernel to run. Although most of these patches are probably floating around the kernel mailing list in some form or another. - Requires Google's RPC and authentication system. (I guess GRPC is open source now, IDK if piper has switched, but you still need auth) - Requires Google's group membership seefvice. - Requires Google's storage engine. I don't remember if they have migrated to spanner yet but even then it would be using internal spanner APIs not the cloud ones. There are probably also more less obvious ones but the point is that when Google writes software to run in Google production it is based on top if a mountain of infrastructure. I'm not sure the design of Piper is in any way novel enough to be worth it. I mean piper works and scales well but I don't think it is a fantastic VCS.
- speedgoose 5y ago
  It’s interesting that they prefer to develop such a tool rather than giving up on the monorepo concept.
  - sulam 5y ago
    One very counterintuitive truth of large scale software development is that as you scale to multiple services, you are gravitationally pulled into a monorepo. There are different forces at work but the strongest at this scale are your data models and API definitions.
    - skippyboxedhero 5y ago
      The problems with data models can happen at small scale. I remember the first large-ish project that I built, it had a few different components but the problem started when I tried to introduce data models to the Java and Python parts (I had it in my head after reading a book that I needed domain objects or some other nonsense in each language)...mistake, the data was still changing, it took forever to make changes, wasn't critical but I learned my lesson very quickly. One perspective on this is that many ORM libraries don't take the DB as the source of truth (one very good ORM library that does, and which saved me in this case, was JOOQ). I think a lot of small-scale problems could be solved this way, monorepo is just another variation of this solution: moving the source of truth into the repo. It is surprising to me how often variations of this problem come up. Obviously, there are solutions from multiple directions: having cross-language definitions (ProtoBuf), Arrow (zero-copy abstractions suitable for high performance), maybe even Swagger which comes at the problem from documentation...but I think this problem still comes up anywhere (and the DB approach is, imo, a very strong approach with a decent ORM at smaller scale).
      - sulam 5y ago
        Does your schema for data models (inception!) have a revision associated with it? If not, deployments are going to be spicy. If so, you end up having to deal with version-rot. Part of why putting this in the repo with your source code is a winning solution is that when you're working off head, you will naturally pick up and test the latest thing, and in most cases your next deploy will also just naturally roll forward as well.
- voidfunc 5y ago
  Can't wait for the "We switched to Eden from Git" posts in 6 months.
  - zelphirkalt 5y ago
    Which will result in lots of people thinking "FB does it! It must be the future! We have to do this as well, or we will be left behind!" sigh.
- criddell 5y ago
  Why wouldn't it be great for other companies that run a monorepo? Would it be nuts to go from some other monorepo (like Subversion) to this?
  - madeofpalk 5y ago
    I wouldn’t think there would be many companies running repos as large as Facebook. If you use a tool like this you would basically be on your own. If you “stick to git” (or mercurial or whatever) at least you have all that momentum behind you, and you almost definitely won’t be the first people to encounter a problem.
    - dan-robertson 5y ago
      You don’t need to have millions of files to have hg performance that leaves you wanting. A few GB of repo should do the trick.
- kvathupo 5y ago
  To their credit, I don't think its use-case is limited to monorepos. I've personally had a multiple GB `.git` file due to storing many data files in a repo. In retrospect, data __shouldn't__ be version-controlled, but it's sometimes the simplest solution, e.g. having a unit-test suite intake a bunch of CSV data. Eden's "a file is checked out only if opened" and "scan only modified directories" would've allowed me to avoid decoupling data from code.
  - pronlover723 5y ago
    data shouldn't be version controlled? Are you saying all the AAA game studios that version control their assets in p4 are doing it wrong?
    - BryanLegend 5y ago
      This. I've got a 35gb repo for the game I'm working on mostly solo.
    - ai_ja_nai 5y ago
      Data can and should be versioned, but not by just `git add BLOAT`. Take a look at https://dvc.org/ https://dvc.org/: blobs are uploaded to a S3 compatible blob storage, metadata is versioned in a config file and this one gets versioned in git
  - beagle3 5y ago
    Data should be versioned if it’s one of your inputs, even if you can’t merge it. It’s just that the gut tooling for it (including git-lfs) is horrible. Perforce (and apparently Eden) make it usable.
markstos 5y ago
The docs still refer to the the tool as Mecurial/hg: https://github.com/facebookexperimental/eden/tree/main/eden/scm https://github.com/facebookexperimental/eden/tree/main/eden/...
- cheeze 5y ago
  This doesn't surprise me. It's a fork, nobody bothered to update the readme, and as much as folks wanted to update things like documentation, improving the software was a higher priority.
tux1968 5y ago
It says it was originally based on Mercurial, but is no longer a distributed source code control system. Are you sure it's still compatible?
TheRealDunkirk 5y ago
I thought Microsoft was working on a "mod" to git that made it work on huge repos, e.g., the Windows source. Did that ever come to fruition?
- ai_ja_nai 5y ago
  you mean git lfs? That is alive
- MikusR 5y ago
  First it was https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit then replaced with https://github.com/microsoft/scalar https://github.com/microsoft/scalar and then into their git fork https://github.com/microsoft/git https://github.com/microsoft/git

deanCommie 5y ago

So if you are at Facebook's or Google's scale, and also run a monorepo, this will be great for you. Which is to say, this is a product for one company - Facebook.

pedrogpimenta 5y ago
Or Google, as per your first sentence :)
- onlyrealcuzzo 5y ago
  Google already has its own [=
  - dijit 5y ago
    Would kill to have piper open source though.
    - kevincox 5y ago
      I would be interesting but basically useless. - Requires a custom kernel to run. Although most of these patches are probably floating around the kernel mailing list in some form or another. - Requires Google's RPC and authentication system. (I guess GRPC is open source now, IDK if piper has switched, but you still need auth) - Requires Google's group membership seefvice. - Requires Google's storage engine. I don't remember if they have migrated to spanner yet but even then it would be using internal spanner APIs not the cloud ones. There are probably also more less obvious ones but the point is that when Google writes software to run in Google production it is based on top if a mountain of infrastructure. I'm not sure the design of Piper is in any way novel enough to be worth it. I mean piper works and scales well but I don't think it is a fantastic VCS.
speedgoose 5y ago
It’s interesting that they prefer to develop such a tool rather than giving up on the monorepo concept.
- sulam 5y ago
  One very counterintuitive truth of large scale software development is that as you scale to multiple services, you are gravitationally pulled into a monorepo. There are different forces at work but the strongest at this scale are your data models and API definitions.
  - skippyboxedhero 5y ago
    The problems with data models can happen at small scale. I remember the first large-ish project that I built, it had a few different components but the problem started when I tried to introduce data models to the Java and Python parts (I had it in my head after reading a book that I needed domain objects or some other nonsense in each language)...mistake, the data was still changing, it took forever to make changes, wasn't critical but I learned my lesson very quickly. One perspective on this is that many ORM libraries don't take the DB as the source of truth (one very good ORM library that does, and which saved me in this case, was JOOQ). I think a lot of small-scale problems could be solved this way, monorepo is just another variation of this solution: moving the source of truth into the repo. It is surprising to me how often variations of this problem come up. Obviously, there are solutions from multiple directions: having cross-language definitions (ProtoBuf), Arrow (zero-copy abstractions suitable for high performance), maybe even Swagger which comes at the problem from documentation...but I think this problem still comes up anywhere (and the DB approach is, imo, a very strong approach with a decent ORM at smaller scale).
    - sulam 5y ago
      Does your schema for data models (inception!) have a revision associated with it? If not, deployments are going to be spicy. If so, you end up having to deal with version-rot. Part of why putting this in the repo with your source code is a winning solution is that when you're working off head, you will naturally pick up and test the latest thing, and in most cases your next deploy will also just naturally roll forward as well.
voidfunc 5y ago
Can't wait for the "We switched to Eden from Git" posts in 6 months.
- zelphirkalt 5y ago
  Which will result in lots of people thinking "FB does it! It must be the future! We have to do this as well, or we will be left behind!" sigh.
criddell 5y ago
Why wouldn't it be great for other companies that run a monorepo? Would it be nuts to go from some other monorepo (like Subversion) to this?
- madeofpalk 5y ago
  I wouldn’t think there would be many companies running repos as large as Facebook. If you use a tool like this you would basically be on your own. If you “stick to git” (or mercurial or whatever) at least you have all that momentum behind you, and you almost definitely won’t be the first people to encounter a problem.
  - dan-robertson 5y ago
    You don’t need to have millions of files to have hg performance that leaves you wanting. A few GB of repo should do the trick.
kvathupo 5y ago
To their credit, I don't think its use-case is limited to monorepos. I've personally had a multiple GB `.git` file due to storing many data files in a repo. In retrospect, data __shouldn't__ be version-controlled, but it's sometimes the simplest solution, e.g. having a unit-test suite intake a bunch of CSV data. Eden's "a file is checked out only if opened" and "scan only modified directories" would've allowed me to avoid decoupling data from code.
- pronlover723 5y ago
  data shouldn't be version controlled? Are you saying all the AAA game studios that version control their assets in p4 are doing it wrong?
  - BryanLegend 5y ago
    This. I've got a 35gb repo for the game I'm working on mostly solo.
  - ai_ja_nai 5y ago
    Data can and should be versioned, but not by just `git add BLOAT`. Take a look at https://dvc.org/ https://dvc.org/: blobs are uploaded to a S3 compatible blob storage, metadata is versioned in a config file and this one gets versioned in git
- beagle3 5y ago
  Data should be versioned if it’s one of your inputs, even if you can’t merge it. It’s just that the gut tooling for it (including git-lfs) is horrible. Perforce (and apparently Eden) make it usable.

markstos 5y ago

The docs still refer to the the tool as Mecurial/hg: https://github.com/facebookexperimental/eden/tree/main/eden/scm https://github.com/facebookexperimental/eden/tree/main/eden/...

cheeze 5y ago
This doesn't surprise me. It's a fork, nobody bothered to update the readme, and as much as folks wanted to update things like documentation, improving the software was a higher priority.

tux1968 5y ago

It says it was originally based on Mercurial, but is no longer a distributed source code control system. Are you sure it's still compatible?

TheRealDunkirk 5y ago

I thought Microsoft was working on a "mod" to git that made it work on huge repos, e.g., the Windows source. Did that ever come to fruition?

ai_ja_nai 5y ago
you mean git lfs? That is alive
MikusR 5y ago
First it was https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit then replaced with https://github.com/microsoft/scalar https://github.com/microsoft/scalar and then into their git fork https://github.com/microsoft/git https://github.com/microsoft/git