13 ms·
Eden
- gregwebs 5y agoA Mercurial compatible SCM (not sure if it is a fork) built for their workflow (monorepo) and scale (enormous, git is not usable at their scale, at least for a monorepo). Uses Python and Rust. Designed for efficient centralization rather than decentralization.
- deanCommie 5y agoSo if you are at Facebook's or Google's scale, and also run a monorepo, this will be great for you. Which is to say, this is a product for one company - Facebook.
- pedrogpimenta 5y agoOr Google, as per your first sentence :)
- onlyrealcuzzo 5y agoGoogle already has its own [=
- dijit 5y agoWould kill to have piper open source though.
- kevincox 5y agoI would be interesting but basically useless. - Requires a custom kernel to run. Although most of these patches are probably floating around the kernel mailing list in some form or another. - Requires Google's RPC and authentication system. (I guess GRPC is open source now, IDK if piper has switched, but you still need auth) - Requires Google's group membership seefvice. - Requires Google's storage engine. I don't remember if they have migrated to spanner yet but even then it would be using internal spanner APIs not the cloud ones. There are probably also more less obvious ones but the point is that when Google writes software to run in Google production it is based on top if a mountain of infrastructure. I'm not sure the design of Piper is in any way novel enough to be worth it. I mean piper works and scales well but I don't think it is a fantastic VCS.
- speedgoose 5y agoIt’s interesting that they prefer to develop such a tool rather than giving up on the monorepo concept.
- sulam 5y agoOne very counterintuitive truth of large scale software development is that as you scale to multiple services, you are gravitationally pulled into a monorepo. There are different forces at work but the strongest at this scale are your data models and API definitions.
- skippyboxedhero 5y agoThe problems with data models can happen at small scale. I remember the first large-ish project that I built, it had a few different components but the problem started when I tried to introduce data models to the Java and Python parts (I had it in my head after reading a book that I needed domain objects or some other nonsense in each language)...mistake, the data was still changing, it took forever to make changes, wasn't critical but I learned my lesson very quickly. One perspective on this is that many ORM libraries don't take the DB as the source of truth (one very good ORM library that does, and which saved me in this case, was JOOQ). I think a lot of small-scale problems could be solved this way, monorepo is just another variation of this solution: moving the source of truth into the repo. It is surprising to me how often variations of this problem come up. Obviously, there are solutions from multiple directions: having cross-language definitions (ProtoBuf), Arrow (zero-copy abstractions suitable for high performance), maybe even Swagger which comes at the problem from documentation...but I think this problem still comes up anywhere (and the DB approach is, imo, a very strong approach with a decent ORM at smaller scale).
- sulam 5y agoDoes your schema for data models (inception!) have a revision associated with it? If not, deployments are going to be spicy. If so, you end up having to deal with version-rot. Part of why putting this in the repo with your source code is a winning solution is that when you're working off head, you will naturally pick up and test the latest thing, and in most cases your next deploy will also just naturally roll forward as well.
- voidfunc 5y agoCan't wait for the "We switched to Eden from Git" posts in 6 months.
- zelphirkalt 5y agoWhich will result in lots of people thinking "FB does it! It must be the future! We have to do this as well, or we will be left behind!" sigh.
- criddell 5y agoWhy wouldn't it be great for other companies that run a monorepo? Would it be nuts to go from some other monorepo (like Subversion) to this?
- madeofpalk 5y agoI wouldn’t think there would be many companies running repos as large as Facebook. If you use a tool like this you would basically be on your own. If you “stick to git” (or mercurial or whatever) at least you have all that momentum behind you, and you almost definitely won’t be the first people to encounter a problem.
- dan-robertson 5y agoYou don’t need to have millions of files to have hg performance that leaves you wanting. A few GB of repo should do the trick.
- kvathupo 5y agoTo their credit, I don't think its use-case is limited to monorepos. I've personally had a multiple GB `.git` file due to storing many data files in a repo. In retrospect, data __shouldn't__ be version-controlled, but it's sometimes the simplest solution, e.g. having a unit-test suite intake a bunch of CSV data. Eden's "a file is checked out only if opened" and "scan only modified directories" would've allowed me to avoid decoupling data from code.
- pronlover723 5y agodata shouldn't be version controlled? Are you saying all the AAA game studios that version control their assets in p4 are doing it wrong?
- BryanLegend 5y agoThis. I've got a 35gb repo for the game I'm working on mostly solo.
- ai_ja_nai 5y agoData can and should be versioned, but not by just `git add BLOAT`. Take a look at https://dvc.org/ https://dvc.org/: blobs are uploaded to a S3 compatible blob storage, metadata is versioned in a config file and this one gets versioned in git
- beagle3 5y agoData should be versioned if it’s one of your inputs, even if you can’t merge it. It’s just that the gut tooling for it (including git-lfs) is horrible. Perforce (and apparently Eden) make it usable.
- markstos 5y agoThe docs still refer to the the tool as Mecurial/hg: https://github.com/facebookexperimental/eden/tree/main/eden/scm https://github.com/facebookexperimental/eden/tree/main/eden/...
- cheeze 5y agoThis doesn't surprise me. It's a fork, nobody bothered to update the readme, and as much as folks wanted to update things like documentation, improving the software was a higher priority.
- tux1968 5y agoIt says it was originally based on Mercurial, but is no longer a distributed source code control system. Are you sure it's still compatible?
- TheRealDunkirk 5y agoI thought Microsoft was working on a "mod" to git that made it work on huge repos, e.g., the Windows source. Did that ever come to fruition?
- ai_ja_nai 5y agoyou mean git lfs? That is alive
- MikusR 5y agoFirst it was https://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit then replaced with https://github.com/microsoft/scalar https://github.com/microsoft/scalar and then into their git fork https://github.com/microsoft/git https://github.com/microsoft/git
- tosh 5y agoI wonder if Eden is a reference to https://en.wikipedia.org/wiki/Eden_(2021_TV_series) https://en.wikipedia.org/wiki/Eden_(2021_TV_series)
- inanutshellus 5y agoThe project's name goes back to 2016. Granted, the README called the project a "filesystem" back then not an SCM, but... still, the name seems to predate the series by quite a bit.
- NobodyNada 5y agoGiven that your link says the show came out in 2021, and the GitHub repo is at least 3 years old, probably not.
- benatkin 5y agoI think it's probably from Facebook's culture of pretending it's a force for good when it's a mixed bag like every other Megacorp. Here's an example of what Zuck has said: > I believe the most important thing we can do is work to bring people closer together. It's so important that we're changing Facebook's whole mission to take this on. No wonder they have a major project named Yoga and now this...
- tekknolagi 5y agoPretty sure it's called yoga because it is a flexible layout engine. A joke.
- benatkin 5y agoHere's another one: https://facebook.github.io/prophet/ https://facebook.github.io/prophet/ It's used to predict things, but still strikes me as an odd name, just like Yoga.
- pc86 5y agoThe comment you're replying to is a perfectly good reason why it's named Yoga, not sure why it'd still seem "odd" with that context. Not particularly witty, but it definitely makes sense.
- Ancapistani 5y agoI think it's a tall order for another SCM to challenge git. I can't imagine how it could be any more entrenched in the industry. Further, I'm happy with git. I played with Mercurial years ago, long enough to work with it day-to-day, and just didn't find any relevant advantages versus git. I love that people are still out there trying to improve things, and certainly don't want that to stop, but it's difficult for me to imagine switching at this point.
- shashashasha___ 5y agoit solves scale issues that git can't solve at the moment. fb monorepos are huge. so for most people/companies this issue is not critical to solve and git is great.
- qbasic_forever 5y agoI don't think it's trying to compete with git, it's not decentralized or meant to support big distributed open source project development. This looks like a nice tool for Big Company to manage its internal, private code repositories.
- z3t4 5y agoThe decentralized part of Git and Mercurial is nice (eg. no need for Github et.al), but I think most software projects using Git or Mercurial do have a centralized server/hub...
- avgcorrection 5y agoThis looks like it’s supposed to be more appropriate for very, very big repos. Which current Git doesn’t support and isn’t fundamentally designed to support. So rather than use Git + Git Annex or something like that (maybe more), you’ll just use this alternative SCM. (I keep hearing about how Git will eventually support big repos, but it’s still not on the horizon. Big repos with Git still seems to be limited to big corps who have the resources to make something bespoke. Personally I don’t use big repos so I have no skin in this game.)
- gabrielrc 5y agoUnfortunately their open source support is very little to non existent. It's great, and it doesn't work outside Facebook.
- aliveli 5y agoStay tuned ;)
- epage 5y agoA bit skeptical when there are relatively recent commits like > Re-sync with internal repository
- joatmon-snoo 5y agoOpen sourcing an internal repository with extensive, ongoing work on it is always a difficult affair, because you're creating a second source of truth. (It isn't just how you manage external contributions, but also workflows like releases and CI.) I wouldn't consider this to be a problem.
- epage 5y agoIt means they haven't (yet?) transitioned to open-first and it'll require proving themselves that they'll do open-first development before trusting them. Not willing to bet my work on a product where the governance isn't open but everything is driven by and for a single company's needs.
- ripphab 5y agoFor what, another Phabricator, that I’ll inevitably have to migrate my company away from again? So if history serves the next announcement to watch for is your departure from Facebook and the launch of Edenity, which will be sunset and abandoned inside a decade once it fails to IPO. Am I close?
- wocram 5y agovaporware?
- xiphias2 5y agoIt looks like Google's Perforce/Piper, I think that would be a better comparision than Git, but I don't see any data on how the two compares. Anyways, Eden being open source is a great advantage already.
- zelphirkalt 5y agoNever forget though, that being open source doesn't mean that much, when there is only _one_ party deciding, what goes into the code base.
- josephg 5y agoBeing opensource does mean it’s free. And can dig in and read the code. And if you want to, fork it and add your own features.
- bb88 5y agoIt's GPL 2.0, on github, which makes it ridiculously easy to fork, tear apart, debug, etc, given your time and ability. That said, I would like to see it self hosted. I think then and only then will I give it a serious look.
- kccqzy 5y agoThat's just cathedral style free software.
- nemothekid 5y ago>that being open source doesn't mean that much, when there is only _one_ party deciding, what goes into the code base. Being open source doesn't mean I have to maintain your code contributions.
- tzury 5y agothey use git (and github) to maintain their own source control management system?
- ctur 5y agoThe SCM ecosystem at Facebook is tremendously powerful and the result of some of the best minds at Facebook working on those systems for many years. From the scaling of the monorepos to the code review workflows, nothing really matches it. The ergonomics of most of the tooling was simply top notch (which it needed to be... engineers, particularly at Meta, are an opinionated lot who don't tolerate poor tools). It's great to see this out in the wild now.
- peppertree 5y agoAfter doing a round of onsite with fb tools team, I got the impression there were lots of bright engineers that wanted fb pay without having to touch fb products.
- jfengel 5y agoA lot of programmers would rather write tools for other programmers than for non-programmer end users. It's the target market that they know best, and it's most prestigious. I wish more of those bright engineers would try to solve problems for other users rather than re-re-re-re-re-optimizing the life of their colleagues. But writing code for users involves, among other things, knowing something about users, which is more bogged down and less fun.
- mym1990 5y agoTheir colleagues ARE users, they are just a different type, and a type that engineers understand more. I would rather a good engineer put their skills towards helping others use their skills more effectively than put that same engineer in a place where they don't like the work and produce below par results.
- xiphias2 5y agoFor some of us it's really hard to understand / solve problems of people whose life is spent on browsing instagram most of the time - when it's not TikTok.
- bin_bash 5y agotl;dr: automatic sparse checkouts for massive monorepos
- tazjin 5y agoAlso check out josh: https://github.com/josh-project/josh https://github.com/josh-project/josh
- maxmcd 5y agoIs there a FUSE filesystem for Git that takes a similar strategy to edenFS? Only setting up files and folders as they are requested might be very helpful with various git monorepo access patterns. Maybe there is something inherit to Git's design that makes this less practical.
- zellyn 5y agohttps://github.com/microsoft/VFSForGit https://github.com/microsoft/VFSForGit Which seems to have been superseded by https://github.com/microsoft/scalar https://github.com/microsoft/scalar according to the README.
- 0xbadcafebee 5y agoHm. I can see why they'd build some of these features, but there's some significant downsides. The VFS in particular will end up a poor experience when a transitory network problem causes apps to hang when pulling code. What happens if you 'grep -r', or if 'mlocate' indexes it? On the build side.... holy jesus, are they really compiling 40 different dependencies from scratch every time they push code? This build has been running for 5 minutes and it's still just compiling dependencies: https://github.com/facebookexperimental/eden/runs/5997101905?check_suite_focus=true https://github.com/facebookexperimental/eden/runs/5997101905... Come on, ya'll. You're supposed to be the "advanced FAANG people". Cache some build deps, will ya?
- amarshall 5y agoI’d be surprised if it was the case that the build system there is just cobbled together for the OSS version, and likely quite different from what they actually use at FB.
- nbm 5y agoOne thing to keep in mind for how development at larger tech companies works is that you’re often not building on your own desktop, you’re usually building on a development server that’s on a well-connected (effectively production-quality, if not literally the same) network. You don’t see a ton of drops in those cases, so it works well. Not that there hasn’t been effort to recover from networking issues encountered in this and other build tooling - at scale, someone’s development server is going to have a bad day every day. You also need much better tools than grep and locate for a monorepo - or any sufficiently large repo probably. Just load the full repo into memory in a few places around the world, and use an API to find the text you’re looking for. If you already have expertise with search services in your company, this is not that challenging a step - and you can get fancy by using something like Tree-sitter to make those searches more advanced than text. Hitting disk (especially for whole directory trees for “grep -r”) is a losing approach in a large repo.
- hammycheesy 5y agoA similar effort was made by Microsoft a while back to scale up Git in order to use with the Windows monorepo: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
- 3a2d29 5y agoJunior engineer here, so got isn’t used for the windows repo? I assumed git was the holy tool or all version control.
- wincy 5y agoI thought this a few years ago when I was more junior as well. In open source circles git has been used for quite awhile but my company just recently moved one of our repos from TFVC to git. A Fortune 500 company I worked for was still using TFVC for some legacy products. Another product from a acquisition used Subversion and the migration to git (even with the company using GitHub Enterprise) still took almost three years. Getting the inertia amongst developers to migrate to a different SCM can be quite the challenge. Edit: initially said migrating to git was a challenge. But really it’s migrating from any SCM to another that’s challenging.
- actuallyalys 5y agoThere are a lot of advantages to using git, in part because it was developed for a massive project (the Linux kernel) so it's very thoroughly tested, in part because it's exploded in popularity so a lot of a tools, integrations, and documentation are available for it. However, it's not the alpha and omega of version control. Some people prefer the interface of Mercurial, some people use Fossil because it integrates issues and wikis, others have stuck to older tools like Subversion, and still others are experimenting with new approaches like Pijul.
- dharmaturtle 5y agogit was created in 2005. No idea what MS uses, though some history is here: https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe#Microsoft_in-house_use https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe#Mi...
- bogwog 5y agoDoes this do anything special for handling binary data, or is it mostly for text like git? I've heard that Perforce (another centralized SCM) does a good job with large binaries. I love git just as much as the next guy, but git-lfs sucks.
- martincmartin 5y agoI worked at Meta and used Eden. I remember it as a virtual, FUSE based file system for a huge monorepo. Basically, if you have GBs of data in your monorepo, and any individual user accesses < 1% of it, then there's no point in having each "hg update" update the other 99%. But we were explicitly dissuaded from having binary data. I worked on image processing, and wanted to store test images for unit tests in hg, but the consensus was, that was a bad idea. So we stored them in a separate data store, and had a make rule to fetch them as part of the build.
- kevincox 5y agoGit only really has two problems with binary files: 1. The take up a lot of space because the entire history needs to be downloaded with all past versions (andany types of binary files completely change when updates so delta compression doesn't help much) 2. The are slow to update on checkout (not much of an issue if they don't chance much). Basically any solution for large repos will also solve Git's "binary file problem" because in order to allow large repos you need to allow shallow and partial checkouts as well as efficiently updating the working copy (usually via a virtual filesystem). TL;DR Git doesn't have a binary file problem, it just has a big repo problem. Binary files are often mentioned because they are the quickest and easiest way to get a big repo.
- hestefisk 5y agoProceeds to book edenhub.com
- fs111 5y agoeden.garden is for sale :)
- psuedo_uh 5y ago
- deleted 5y ago[deleted]
- krashidov 5y agoFacebook/Meta’s dev tooling, libraries, and infrastructure has always impressed me. React, Hack, jest, and now this. I am very surprised they haven’t tried to monetize this aspect of their business like Microsoft and Amazon did. They could be a very strong contender in that space instead of being known as a terrible time suck for humanity.
- pavlov 5y agoFB did acquire Parse, a mobile-oriented PaaS (in a roughly similar market segment as Google’s Firebase). It was shut down when FB apparently realized that an open cloud platform business is completely incompatible with how they run their infra.
- disgruntledphd2 5y agoI think that this was more a margin issue, ads were higher margin so Parse got deprioritised, and eventually killed.
- deleted 5y ago[deleted]
- cosmiccatnap 5y agoSo, instead of breaking their repo up so git is more performance, they forked mercurial instead? Madness
- martinbooth 5y agoWhy do you think this is madness? FB doesn't use git for these monorepos, so that's not really relevant, but I don't understand why you think that it's better to break a repo up because the SCM can't handle the size vs fixing the SCM so it can handle the size. I work for meta (nothing to do with the teams working on this though) and I can assure you people have considered the tradeoffs of breaking repos up just to accommodate existing SCMs vs improving the SCMs.. I think if you believe improving the SCM instead of breaking up the code is madness you should probably provide a bit more of a justification
- worker767424 5y agoYou skipped a step. They actively contributed to mercurial before the full fork.
- Sophistifunk 5y agoHow is the URL still bad in this story after 9 hours?
- MichaelMoser123 5y ago> EdenFS speeds up operations in large repositories by only populating working directory files on demand, as they are accessed. but if i grep through the source code then it will download all of the stuff?
- kevincox 5y agoYes, generally for a large codebase you will have a separately code search tool like Google's Code Search or Sourcegraph which are super fast for large amounts of code.
- T3RMINATED 5y ago