18 ms·

Microsoft and GitHub team up to take Git virtual file system to macOS, Linux

796 points by dmmalam 9y ago

Nelkins 9y ago
I wonder if this is a precursor to open-sourcing Windows development...
- gmueckl 9y ago
  No. Open Source and walled gardens don't mix. MS wants to turn Windows into the later, obviously.
hasenj 9y ago
Moving Windows development to git sounds like a completely irrational decision from a technical standpoint that was driven mostly by marketing concerns.
- Jonnax 9y ago
  What did they use before and why is it better?
- akie 9y ago
  On what do you base that assumption?
- Tuna-Fish 9y ago
  Umm, why?
  - gmueckl 9y ago
    Because git is the latest fashion rage and by itself incapable of managing the size of the Windows code base. This is the ideal combination of reasons for corporate PHB to decide to use it. Exhibit A: GVFS was solely invented as a hack to make git usable by the Windows team according to the Windows team blog. (Yes, there is some sour grapes and sarcasm in this post)
    - HelloNurse 9y ago
      Does GVFS support fine-grained access control to parts of the repo, or all userss are guaranteed to access the whole repository if they want?
- chx 9y ago
  Marketing concerns by whom? The nonexisting company behind git? Github when MS doesn't use github? Or you think MS is going to be able to drum up some business just because they use git? Please elaborate!
  - enjo 9y ago
    While I don't agree with OP... you could make a case that switching to Git is a way for Microsoft to make itself more fashionable to the engineering community at large. The marketing here is: it's cool to work at Microsoft again!
    - Analemma_ 9y ago
      > you could make a case that switching to Git is a way for Microsoft to make itself more fashionable to the engineering community at large. The marketing here is: it's cool to work at Microsoft again! But that makes it a technical decision! If it's easier to attract talented engineers because you're using git, that improves the product. I was trying to make this point in my other comment too: OP is drawing a sharp line between "marketing decisions" and "engineering decisions" and sneering at the former, but actually the boundary can be pretty fuzzy.
- Analemma_ 9y ago
  I worked at Microsoft before and after my team (not Windows) did the Source Depot -> git transition, and I can say that it was mostly for the better. Source Depot was pretty good as non-distributed VCSs go, but it doesn't hold a candle to git. Some specifics: - Branching in SD sucked, the only way to work on multiple features at once was to either have multiple copies of the repo or constantly fiddle with "package files" that contained your changes, and the whole thing shit the bed if the server went down. - All the new hires both from college and the industry all know git and there were an increasing number of them who just found it bizarre that Microsoft was still on a proprietary, non-distributed system. - Higher-ups (correctly, IMO) decided that VSTS/TFS had to have git in order to remain competitive, and that we should be eating our own dogfood. That's partly a marketing concern but also a legitimate technical decision.
- sctb 9y ago
  Could you please try to comment more substantively? We've already asked you this. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - hasenj 9y ago
    > We've already asked you this. Am I on some kind of a black/gray list? Either that, or you specifically remember my name, which sounds odd.
scscsc 9y ago
The title is terrible. Here is an excerpt from the article that explains what's going on: "The company's solution was to develop Git Virtual File System (GVFS). With GVFS, a local replica of a Git repository is virtualized such that it contains metadata and only the source code files that have been explicitly retrieved. By eliminating the need to replicate every file (and, hence, check every file for modifications), both the disk footprint of the repository and the speed of working with it were greatly improved. Microsoft modified Git to handle this virtual file system. The client was altered so that it didn't needlessly try to access files that weren't available locally and a new transfer protocol was added for selectively retrieving individual files from a remote repository." This style of working is needed in large code bases, where not all files are checked out on developer workstations (for performance or privacy reasons).
- gmueckl 9y ago
  Wait... so they turned got into a centralized abomination and added access control to parts of the source tree (A feature that SVN had from the start but decentralized VCS cannot provide)? Is this correct?
  - coldtea 9y ago
    What do you think is more possible? (a) the result of the work of a major Git player like GitHub, and a major software company like Microsoft, with tens of excellent engineers devoted on it, and that solves a real pain point they have, is a centralized abomination that merely replicates a feature SVN already had (b) your description is a crude knee jerk reaction ?
    - syrrim 9y ago
      In a similar rhetorical style: If SVN is a terrible application with no merits to large developers like MS, why did it exist in the first place? If the features SVN brings over git are not considered detrimental by Torvalds, why did he create his own VCS with them removed, and why hasn't he added them back in?
tarruda 9y ago
Did I misunderstood the idea, or is GVFS essentially making git a non-distributed VCS?
- WorldMaker 9y ago
  It's more that it provides more control on the centralized<->distributed spectrum. Git by default, yes, is fully distributed with every copy of a repo supposed to maintain full history, etc. GVFS allows you the option to offload some of that storage effort to servers you specify. Those servers can be distributed themselves (similar to CDNs, etc), so there's still distribution flexibility there. You can think of it as giving you somewhat flexible control of the "torrenter/seeder ratio" in BitTorrent: how many complete copies of the repo are available/accessible at a given time.
- mikepurvis 9y ago
  I think once you're checked out, you're still distributed as far as being able to make and push commits which modify the files you have. But obviously someone can't clone files from you which you don't yourself have. See previous articles which discussions: https://blogs.msdn.microsoft.com/devops/2017/02/03/announcing-gvfs-git-virtual-file-system/ https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin... https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/ https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...
  - srcmap 9y ago
    From reading the first post "The largest Git repo on the planet" - "Windows code base" of "3.5M files", "repo of about 300GB", "4,000 engineers" - I assumed that "Windows code base" contains all the utilties, DLLs that included in windows release such as notepad, games, IE/Edge etc and not just the Windows KERNEL. If so, the comparable code base to check against is Android AOSP, Ubuntu, RedHat or FreeBSD. If that is true, I believe the source code base for Ubuntu/RedHat distribution with all the Apps likely be bigger compare to Windows in term of number files, source repo and number of engineers (open source developers for all the packages such as ff, chrome openoffice.) Microsoft folks feel free to correct me here. It seems that the existing git's process, dev model seems to work well for much bigger projects already by using different git repo for each apps. Still not sure what pain point does the new GVFS solve.....
jwilk 9y ago
For the avoidance of doubt, GitHub is not upstream of Git.
- tomxor 9y ago
  yeah this was really confusing in the article everywhere it mentions up-streaming it refers to "git" only then later mentions how GitHub is accepting it's modifications... so not entirely sure which it is. TL;DR pretty sure it's not touching git source at all. Looking though the GVFS readme: https://github.com/Microsoft/GVFS https://github.com/Microsoft/GVFS it appears that it actually wraps git, which would make sense because in that case they would have to explicitly add support to each git host and provide individual platform ports of GVFS (all of which would be unnecessary if it was actually upstreamed into git). So that's good... git is one of my favourite open source tools and I would really hate for M$ to start polluting it. I don't care if GVFS is a good idea or not I just don't trust the fuckers and they will always deserve that suspicion.
  - hdhzy 9y ago
    No, they needed to do some modifications to the git suite of tools. Generally git expects all objects to be on disk and Microsoft wanted to have sparse checkouts of files in a specific revision. Not really polluting but rather having some objects be fetched only on demand. Source: https://blogs.msdn.microsoft.com/devops/2017/02/03/announcing-gvfs-git-virtual-file-system/ https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin... > In addition to the GVFS sources, we’ve also made some changes to Git to allow it to work well on a GVFS-backed repo, and those sources are available at https://github.com/Microsoft/git https://github.com/Microsoft/git. For the record as far as I understand GVFS the article is correctly using git vs Github.
    - tomxor 9y ago
      Thanks for the clarification, sparse object fetch seems like a small change in concept at least.
crankylinuxuser 9y ago
Does this mean we have to fdisk/format if we forget the commands to correct a broken git repo? https://xkcd.com/1597/ https://xkcd.com/1597/ I seriously did that first learning Git. And there's plenty of niches and side-cases that I'm still not quite sure of. Going from client-server to distributed has a level of complexity that usually isn't discussed until you implement. EDIT: And further understanding is, this provides a GIT filesystem based connection so that one can work on a multi-TB repo without downloading everything locally. This seems to be the result of choosing to have all the software in one OMG-sized repo, rather than 1 project/repo. And evidently they need a "keep on server" for this. Makes me wonder more why they even went with Git or a distributed model at all. This seems more like they 'screwed up and now have tons of bandaids'.
- 0xfeba 9y ago
  Yep, I've done that.
- aeorgnoieang 9y ago
  Lookup the history of GVFS and why Microsoft chose Git. There are interesting reads about it. There's also lots of interesting reads about 'monorepos' and their pros and cons. Note tho that Google and Facebook, and lots of other companies, use mono-repos. It's not just Microsoft and, not surprisingly, they've all made (more or less) reasoned, thoughtful decisions taking into account lots of factors that almost no one else in the world would ever think to do.
chungy 9y ago
Anyone else get superconfused and think about the GNOME Virtual Filesystem?
- sam0x17 9y ago
  https://github.com/Microsoft/GVFS/issues/7 https://github.com/Microsoft/GVFS/issues/7
- awalton 9y ago
  Having worked on GVFS (the GNOME one) in the past... every single time. Maybe GNOME should start trademarking project names too :/...
sam0x17 9y ago
And they are still calling it GVFS -- a name that is already used by a prominent linux technology. https://github.com/Microsoft/GVFS/issues/7 https://github.com/Microsoft/GVFS/issues/7
- zellyn 9y ago
  Every single hacker news discussion these days seems to have a complaint about reusing a name. Every name is taken. There is nothing to be done about it. It's okay. I mean, it's sad, and terrible, but it's also fine.
  - camgunz 9y ago
    They are definitely not all taken
  - deleted 9y ago
    [deleted]
jcoffland 9y ago
I can't help thinking, there is more to this than is discussed in the article. This tech can also be used to give developers selective access to a git repo. It lets Microsoft and other commercial software vendors get on the git bandwagon with out actually sharing all the code with all the developers. Each git repo is a complete copy because git was designed to support Open-Source. In Open-Source there are no hidden parts. But that's a hard pill for corps like Microsoft to swallow. This looks more like a solution to meet developer demands to use git and also satisfy the demands of management to maintain control of their team's code based.
- Sevii 9y ago
  Got vfs is more motivated by Microsoft wanting to use git as a monorepo. I think they have around 2k windows devs on one git tree with this vfs.
  - philipov 9y ago
    Are there advantages to using a monorepo rather than having a proper package manager? Seems like the problems they're trying to solve would be better solved by not insisting on using a monorepo.
    - AaronFriel 9y ago
      While this debate keeps being hashed out, it's important to note that some of the largest and most successful tech companies have decided to use monorepos. Facebook, Google, Microsoft, and others use monorepos. I suspect one factor for Microsoft is that many patches will be cross-cutting, and tooling with multiple repositories or even submodules remains poor.
      - philipov 9y ago
        I think it's dangerous to assume a causal link between success and their use of monorepos. As aaronAgain points out, Microsoft's codebase existed since before monoliths were recognised to be a bad idea. They're already screwed, and need a solution for where they are today. That's a valid and important use-case, but doesn't suggest that someone starting from scratch should follow their lead.
kmicklas 9y ago
Is FUSE itself really not fast enough? Referencing Dropbox/Project Infinite seems to suggest this is a network latency issue.
paradite 9y ago
Uhm... the author seems to be confused about the relationship between Github and git. GitHub doesn't develop git, it's a service built on top of git. Any modifications to git has nothing to do with GitHub. GFVS is a GitHub project, not part of git.
- aeorgnoieang 9y ago
  > GitHub doesn't develop git I'd be surprised if none of their employees have contributed to Git. I didn't interpret anything in the article as saying that GitHub develops Git, in its entirety (or even largely). > GFVS is a GitHub project GFVS is a Microsoft project and GitHub seems to be contributing. And the GFVS developers have been submitting, successfully, their changes to Git itself upstream to the actual Git project. So it is becoming a part of Git.
  - jwilk 9y ago
    > I didn't interpret anything in the article as saying that GitHub develops Git "Microsoft [...] wanted to get these modifications accepted upstream and integrated into the standard Git client. That plan appears to be going well. Yesterday, the company announced that GitHub was adopting its modifications and that the two would be working together to bring suitable clients to macOS and Linux." This hints that Git upstream = GitHub. I mean, why mention GitHub at all if they aren't upstream? The rest of the article doesn't explain GitHub's role in this story either.
    - hdhzy 9y ago
      Maybe quickly changing topics is what causes the confusion. I guess they picked github because it has enterprise offering and enterprise customers are interested in large git repos. So github is like a test lab for GVFS. In the mean time they upstream the changes to real git adjusting GVFS to what git maintainers think is right.
    - aeorgnoieang 9y ago
      I still don't think your interpretation is warranted, but meh. Microsoft made some contributions and they're working to get them accepted upstream. They're maintaining a fork basically. GitHub is adopting their modifications, i.e. GitHub is running Microsoft's fork. Microsoft's modifications are necessary for them, particularly so they can use Git with the giant Windows repo. Presumably, GitHub also has a need or desire for those same or similar modifications. IIRC, some of GitHub's largest customers need or want a version of Git that can also handle large repos, or repos with large files or large numbers of files.
dfabulich 9y ago
> Microsoft and GitHub are also working to bring similar capabilities to other platforms, with macOS coming first, and later Linux. The obvious way to do this on both systems is to use FUSE, an infrastructure for building file systems that run in user mode rather than kernel mode (desirable because user-mode development is easier and safer than kernel mode). However, the companies have discovered that FUSE isn't fast enough for this—a lesson Dropbox also learned when developing a similar capability, Project Infinite. Currently, the companies believe that tapping into a macOS extensibility mechanism called Kauth (or KAuth) will be the best way forward. I'd love to read more technical details on this. How can Kauth support a virtual filesystem?
tankenmate 9y ago
"Git wasn't designed for such vast numbers of developers—more than 20,000 actively working on the codebase. Also, Git wasn't designed for a codebase that was so large, either in terms of the number of files and version history for each file, or in terms of sheer size, coming in at more than 300GB. When using standard Git, working with the source repository was unacceptably slow. Common operations (such as checking which files have been modified) would take multiple minutes." These sentences in some parts gloss over the details, and in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel), it was designed for huge numbers of files (but large numbers of files works fine on Linux due to the dentry cache, it sucks on Windows because they don't have a cache that has the same behaviour as the Linux dentry cache). Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size? That said, monorepos with several dozen software projects can be very slow due to O(n) functions such as git blame and the like. It's true that git wasn't designed for large numbers of projects in one repo that are only indirectly shared by library code and the like. Unfortunately that was not written in the article.
- godzillabrennus 9y ago
  Your comment reminds me that Microsoft should really double down on replacing NTFS. Even Apple has moved off HFS+ with their latest OS.
  - sekh60 9y ago
    They unfortunately removed the ability to create ReFS disks from Windows 10 Pro and lower. Hopefully they reverse that decision soon. That said I'm not sure how stable it was considered.
- alkonaut 9y ago
  I develop a desktop app with lots of test documents (test inputs, outputs) that are tied to specific revisions in order to work with the test suite that uses them. None are huge but there are many and some are frequently changed (2GB working copy, 15gb repo, 100k commits). They could be put in some (very) complex hierarchy on a network share, but having them in vcs is so convenient. If rev 9999 changes the code so that the expected output testfileA differs then you update the expected output in the same commit. If you don’t have the file and you want to be able to clone+build+test any revision then you still need the results for 9998 also on the share. This scenario is nicely handled in svn, mercurial, tfvc and similar. If this lets git do it then it looks like a great idea. Game developers with lots of churn in huge binary assets also have the same problem, but probably even worse.
jopsen 9y ago
Hmm, some part of me wouldn't mind, a FUSE setup where: /repository/github.com/<org>/<repo>/ is automatically cloned out from github when accessed the first time :) It would be pretty cool to use as a $GOPATH. Or just for making drive-by github contributions super easy.
- zimbatm 9y ago
  I built https://github.com/zimbatm/h https://github.com/zimbatm/h to satisfy the same need but it's a command-line tool, not a FUSE filesystem. In some way I prefer it because transparent network access doesn't work really well with failure modes. Usage: `h <owner>/<repo>` clones the repo, if it doesn't exist, and takes you there. This and hub are integral to my developer workflow: $ h some/repo $ git checkout -b some-branch [hack and commit] $ hub fork $ git push -u zimbatm $ hub pull-request Done
- hinkley 9y ago
  It's all fun and games until someone like me tries to run `find` on the parent directory...
  - silentOpen 9y ago
    Children would be lazily loaded and would not appear in directory listing of the parent directory. `find` would search your existing local checkouts only and the file system would still be POSIX.
    - hinkley 9y ago
      How do you trigger loading the projects if there isn’t a pointer to them showing up in the file system?
      - koffiezet 9y ago
        When you open or stat a directory that doesn't exist, try to checkout that path, if that fails, return an ENOENT error?
rokob 9y ago
Interesting to see them try to fix Git when Facebook considered it (due to similar constraints) and decided to go with Hg instead: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc...
- jng 9y ago
  Microsoft is used to implementing its own filesystems, so it probably feels more natural for them to do that than it does for Facebook.
- glandium 9y ago
  They decided to go with mercurial because it was possible to monkey patch its guts from extensions written in python. They're now talking about implementing parts in rust, which, ironically, would have prevented them from doing what they originally chose mercurial for.
  - masklinn 9y ago
    > They decided to go with mercurial because it was possible to monkey patch The ability to extend, swap and replace deep into the stack was always a core component of Mercurial, hence having to enable extensions for things many users would consider core features e.g. terminal colors or graph log Facebook didn't hack around like monkeys, they built extensions. And when they could not do it as extensions, they upstreamed improvements to the core. The alternative would have been to fork the codebase entirely. Monkey patching the internals would have been significantly less maintainable: having used it as a library, I can tell you that none of the internal stuff is considered stable and pretty major components will change between point releases (I was using the diff-parsing and patch-application routines for something else, the API changed basically every minor release, forking would at least give you a heads-up conflict when upstream changed, monkey-patching would either blow up at runtime or not go through the patch anymore) > They're now talking about implementing parts in rust, which, ironically, would have prevented them from doing what they originally chose mercurial for. No.
- ethomson 9y ago
Koshkin 9y ago
To me it looked like one of the points of Git was that one could work on anything from the entire repository locally, i.e., for example, without having an internet connection, e.g. when on an airplane. (I think having one huge repository is the problem here.)
- jaccarmac 9y ago
  Well, Git was indeed created to serve Linux kernel development. So decentralized access for the people writing patches all over the world was useful. However, when companies started using source control for their monorepos which developers access through a work network or VPN decentralization takes a backseat to business concerns. Different use cases and all that.
  - secstate 9y ago
    I think the problem there, is that you then basically wind up with Git on the backend and an SVN-like layer over the front. Which seems a bit silly. SVN is a very robust project that works really well. Why not add git-like porcelain to SVN if you're gonna force devs to have an network connection so they don't have to download a giant git working directory. Smart people are working on all this, so I'm sure there are reasons, but in the all the instances where I've had to interact with a monorepo, it was because the tech debt was too high to pay off to break it apart, not because it was better. And if you're indebted to the point where you have no point in paying it off, you damn well better have leveraged assets against it (i.e. a cash cow of a business, like MS Word or Facebook)
    - jaccarmac 9y ago
      That's interesting. I've never been an insider and it was several years ago, but Google spoke in pretty glowing terms of their monorepo when I heard them talk about it. That's neither here nor their though, not my personal preference either to be fair. As far as Git's killer feature I think decentralization is only one part of it. Being able to deal with the change graph directly is sometimes handy. And knowing the basis of the model, I find Git pretty easy to understand even when history gets fuzzy. Not sure but I think Darcs might be even better for that, just without the mindshare.
jeanmichelx 9y ago
This is only tangentially related, but is there any standard advice to get git on a remote file system (ceph) to run fast with a long history?
gumby 9y ago
A couple of thoughts as the implementation downloads mainly placeholders: that mechanism can also be combined with permissions on the server side so contractor X who is fixing a driver need only have access to the driver and some supporting code while still checking into the monorepo. It also means that every developer’s laptop won’t have a full copy of the repo (so if said laptop is lost, the risk is more contained) And it’s similar to Part of LFS, bigfiles and other “blob” add-o s to git...could see in future a way to put .doc files and the like into repos.
drdrey 9y ago
Sounds a bit similar to Google's Piper (https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...) that lets you check out a portion of the monorepo, no?
Myrmornis 9y ago
> The open source, free GitHub hosting doesn't need the scaling work Microsoft has done They might want to fix that sentence. It reads as saying that Github is open source. Perhaps it should read > The free GitHub hosting that is typically used by open source projects doesn't need the scaling work Microsoft has done
exabrial 9y ago
This is kinda funny, it undoes one of the original pillars of why git was made
nerdponx 9y ago
It's a little surprising to me that these companies use Git at all. What's the appeal of Git really? Is it just the "familiarity factor", and is that really a reason to pour so many resources into it?
- dingo_bat 9y ago
  I think almost all the people they hire nowadays would be familiar with git and not something else. So it would be a lot of time savings in training if you switch to git instead of training every newbie to your own peculiar source control mechanism.
  - nerdponx 9y ago
    Sure, but the CLI is the CLI. I can't imagine there's no way to replicate the Git CLI without rewriting the internals. Surely the basics of branch-and-merge will be the same, right? I doubt anyone would miss the dangerously overloaded `checkout` subcommand, or wrestling with rebase conflicts.
- OOPMan 9y ago
  Personally, I imagine low-cost in-place branching is useful to most people doing serious work...?
mg74 9y ago
"The biggest complexity is that Git has a very conservative approach to compatibility, requiring that repositories remain compatible across versions." This, above all else. That I never, or anyone else on our team, ever has to worry about wether they are running an "older version" of git is such a win.
- gear54rus 9y ago
  Big problem. Just update your machines. Am I supposed to miss great features because of your admin laziness? It has to fail gracefully on version mismatch, other than that it's just a bad decision.
- xyzzy_plugh 9y ago
  It's not really a surprise that monorepo advocates struggle here. Compatibility through versioning and stable APIs is challenging and requires discipline. Making wide-spread changes to all clients of your code when you make changes that break things doesn't require discipline.
iceman2654 9y ago
Does anyone else think this is another "Embrace, extend, and extinguish" move from MS? I don't know enough about the changes make that judgement.
m12k 9y ago
It seems that lazy-downloading files isn't all that useful if you need to compile all the source code. Does anyone know which build tools can be used to avoid having to open most of the source files - maybe some centralized build cache?

796 points by dmmalam 9y ago

Nelkins 9y ago

I wonder if this is a precursor to open-sourcing Windows development...

gmueckl 9y ago
No. Open Source and walled gardens don't mix. MS wants to turn Windows into the later, obviously.

hasenj 9y ago

Moving Windows development to git sounds like a completely irrational decision from a technical standpoint that was driven mostly by marketing concerns.

Jonnax 9y ago
What did they use before and why is it better?
akie 9y ago
On what do you base that assumption?
Tuna-Fish 9y ago
Umm, why?
- gmueckl 9y ago
  Because git is the latest fashion rage and by itself incapable of managing the size of the Windows code base. This is the ideal combination of reasons for corporate PHB to decide to use it. Exhibit A: GVFS was solely invented as a hack to make git usable by the Windows team according to the Windows team blog. (Yes, there is some sour grapes and sarcasm in this post)
  - HelloNurse 9y ago
    Does GVFS support fine-grained access control to parts of the repo, or all userss are guaranteed to access the whole repository if they want?
chx 9y ago
Marketing concerns by whom? The nonexisting company behind git? Github when MS doesn't use github? Or you think MS is going to be able to drum up some business just because they use git? Please elaborate!
- enjo 9y ago
  While I don't agree with OP... you could make a case that switching to Git is a way for Microsoft to make itself more fashionable to the engineering community at large. The marketing here is: it's cool to work at Microsoft again!
  - Analemma_ 9y ago
    > you could make a case that switching to Git is a way for Microsoft to make itself more fashionable to the engineering community at large. The marketing here is: it's cool to work at Microsoft again! But that makes it a technical decision! If it's easier to attract talented engineers because you're using git, that improves the product. I was trying to make this point in my other comment too: OP is drawing a sharp line between "marketing decisions" and "engineering decisions" and sneering at the former, but actually the boundary can be pretty fuzzy.
Analemma_ 9y ago
I worked at Microsoft before and after my team (not Windows) did the Source Depot -> git transition, and I can say that it was mostly for the better. Source Depot was pretty good as non-distributed VCSs go, but it doesn't hold a candle to git. Some specifics: - Branching in SD sucked, the only way to work on multiple features at once was to either have multiple copies of the repo or constantly fiddle with "package files" that contained your changes, and the whole thing shit the bed if the server went down. - All the new hires both from college and the industry all know git and there were an increasing number of them who just found it bizarre that Microsoft was still on a proprietary, non-distributed system. - Higher-ups (correctly, IMO) decided that VSTS/TFS had to have git in order to remain competitive, and that we should be eating our own dogfood. That's partly a marketing concern but also a legitimate technical decision.
sctb 9y ago
Could you please try to comment more substantively? We've already asked you this. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- hasenj 9y ago
  > We've already asked you this. Am I on some kind of a black/gray list? Either that, or you specifically remember my name, which sounds odd.

scscsc 9y ago

The title is terrible. Here is an excerpt from the article that explains what's going on: "The company's solution was to develop Git Virtual File System (GVFS). With GVFS, a local replica of a Git repository is virtualized such that it contains metadata and only the source code files that have been explicitly retrieved. By eliminating the need to replicate every file (and, hence, check every file for modifications), both the disk footprint of the repository and the speed of working with it were greatly improved. Microsoft modified Git to handle this virtual file system. The client was altered so that it didn't needlessly try to access files that weren't available locally and a new transfer protocol was added for selectively retrieving individual files from a remote repository." This style of working is needed in large code bases, where not all files are checked out on developer workstations (for performance or privacy reasons).

gmueckl 9y ago
Wait... so they turned got into a centralized abomination and added access control to parts of the source tree (A feature that SVN had from the start but decentralized VCS cannot provide)? Is this correct?
- coldtea 9y ago
  What do you think is more possible? (a) the result of the work of a major Git player like GitHub, and a major software company like Microsoft, with tens of excellent engineers devoted on it, and that solves a real pain point they have, is a centralized abomination that merely replicates a feature SVN already had (b) your description is a crude knee jerk reaction ?
  - syrrim 9y ago
    In a similar rhetorical style: If SVN is a terrible application with no merits to large developers like MS, why did it exist in the first place? If the features SVN brings over git are not considered detrimental by Torvalds, why did he create his own VCS with them removed, and why hasn't he added them back in?

tarruda 9y ago

Did I misunderstood the idea, or is GVFS essentially making git a non-distributed VCS?

WorldMaker 9y ago
It's more that it provides more control on the centralized<->distributed spectrum. Git by default, yes, is fully distributed with every copy of a repo supposed to maintain full history, etc. GVFS allows you the option to offload some of that storage effort to servers you specify. Those servers can be distributed themselves (similar to CDNs, etc), so there's still distribution flexibility there. You can think of it as giving you somewhat flexible control of the "torrenter/seeder ratio" in BitTorrent: how many complete copies of the repo are available/accessible at a given time.
mikepurvis 9y ago
I think once you're checked out, you're still distributed as far as being able to make and push commits which modify the files you have. But obviously someone can't clone files from you which you don't yourself have. See previous articles which discussions: https://blogs.msdn.microsoft.com/devops/2017/02/03/announcing-gvfs-git-virtual-file-system/ https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin... https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/ https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...
- srcmap 9y ago
  From reading the first post "The largest Git repo on the planet" - "Windows code base" of "3.5M files", "repo of about 300GB", "4,000 engineers" - I assumed that "Windows code base" contains all the utilties, DLLs that included in windows release such as notepad, games, IE/Edge etc and not just the Windows KERNEL. If so, the comparable code base to check against is Android AOSP, Ubuntu, RedHat or FreeBSD. If that is true, I believe the source code base for Ubuntu/RedHat distribution with all the Apps likely be bigger compare to Windows in term of number files, source repo and number of engineers (open source developers for all the packages such as ff, chrome openoffice.) Microsoft folks feel free to correct me here. It seems that the existing git's process, dev model seems to work well for much bigger projects already by using different git repo for each apps. Still not sure what pain point does the new GVFS solve.....

jwilk 9y ago

For the avoidance of doubt, GitHub is not upstream of Git.

tomxor 9y ago
yeah this was really confusing in the article everywhere it mentions up-streaming it refers to "git" only then later mentions how GitHub is accepting it's modifications... so not entirely sure which it is. TL;DR pretty sure it's not touching git source at all. Looking though the GVFS readme: https://github.com/Microsoft/GVFS https://github.com/Microsoft/GVFS it appears that it actually wraps git, which would make sense because in that case they would have to explicitly add support to each git host and provide individual platform ports of GVFS (all of which would be unnecessary if it was actually upstreamed into git). So that's good... git is one of my favourite open source tools and I would really hate for M$ to start polluting it. I don't care if GVFS is a good idea or not I just don't trust the fuckers and they will always deserve that suspicion.
- hdhzy 9y ago
  No, they needed to do some modifications to the git suite of tools. Generally git expects all objects to be on disk and Microsoft wanted to have sparse checkouts of files in a specific revision. Not really polluting but rather having some objects be fetched only on demand. Source: https://blogs.msdn.microsoft.com/devops/2017/02/03/announcing-gvfs-git-virtual-file-system/ https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin... > In addition to the GVFS sources, we’ve also made some changes to Git to allow it to work well on a GVFS-backed repo, and those sources are available at https://github.com/Microsoft/git https://github.com/Microsoft/git. For the record as far as I understand GVFS the article is correctly using git vs Github.
  - tomxor 9y ago
    Thanks for the clarification, sparse object fetch seems like a small change in concept at least.

crankylinuxuser 9y ago

Does this mean we have to fdisk/format if we forget the commands to correct a broken git repo? https://xkcd.com/1597/ https://xkcd.com/1597/ I seriously did that first learning Git. And there's plenty of niches and side-cases that I'm still not quite sure of. Going from client-server to distributed has a level of complexity that usually isn't discussed until you implement. EDIT: And further understanding is, this provides a GIT filesystem based connection so that one can work on a multi-TB repo without downloading everything locally. This seems to be the result of choosing to have all the software in one OMG-sized repo, rather than 1 project/repo. And evidently they need a "keep on server" for this. Makes me wonder more why they even went with Git or a distributed model at all. This seems more like they 'screwed up and now have tons of bandaids'.

0xfeba 9y ago
Yep, I've done that.
aeorgnoieang 9y ago
Lookup the history of GVFS and why Microsoft chose Git. There are interesting reads about it. There's also lots of interesting reads about 'monorepos' and their pros and cons. Note tho that Google and Facebook, and lots of other companies, use mono-repos. It's not just Microsoft and, not surprisingly, they've all made (more or less) reasoned, thoughtful decisions taking into account lots of factors that almost no one else in the world would ever think to do.

chungy 9y ago

Anyone else get superconfused and think about the GNOME Virtual Filesystem?

sam0x17 9y ago
https://github.com/Microsoft/GVFS/issues/7 https://github.com/Microsoft/GVFS/issues/7
awalton 9y ago
Having worked on GVFS (the GNOME one) in the past... every single time. Maybe GNOME should start trademarking project names too :/...

sam0x17 9y ago

And they are still calling it GVFS -- a name that is already used by a prominent linux technology. https://github.com/Microsoft/GVFS/issues/7 https://github.com/Microsoft/GVFS/issues/7

zellyn 9y ago
Every single hacker news discussion these days seems to have a complaint about reusing a name. Every name is taken. There is nothing to be done about it. It's okay. I mean, it's sad, and terrible, but it's also fine.
- camgunz 9y ago
  They are definitely not all taken
- deleted 9y ago
  [deleted]

jcoffland 9y ago

I can't help thinking, there is more to this than is discussed in the article. This tech can also be used to give developers selective access to a git repo. It lets Microsoft and other commercial software vendors get on the git bandwagon with out actually sharing all the code with all the developers. Each git repo is a complete copy because git was designed to support Open-Source. In Open-Source there are no hidden parts. But that's a hard pill for corps like Microsoft to swallow. This looks more like a solution to meet developer demands to use git and also satisfy the demands of management to maintain control of their team's code based.

Sevii 9y ago
Got vfs is more motivated by Microsoft wanting to use git as a monorepo. I think they have around 2k windows devs on one git tree with this vfs.
- philipov 9y ago
  Are there advantages to using a monorepo rather than having a proper package manager? Seems like the problems they're trying to solve would be better solved by not insisting on using a monorepo.
  - AaronFriel 9y ago
    While this debate keeps being hashed out, it's important to note that some of the largest and most successful tech companies have decided to use monorepos. Facebook, Google, Microsoft, and others use monorepos. I suspect one factor for Microsoft is that many patches will be cross-cutting, and tooling with multiple repositories or even submodules remains poor.
    - philipov 9y ago
      I think it's dangerous to assume a causal link between success and their use of monorepos. As aaronAgain points out, Microsoft's codebase existed since before monoliths were recognised to be a bad idea. They're already screwed, and need a solution for where they are today. That's a valid and important use-case, but doesn't suggest that someone starting from scratch should follow their lead.

kmicklas 9y ago

Is FUSE itself really not fast enough? Referencing Dropbox/Project Infinite seems to suggest this is a network latency issue.

paradite 9y ago

Uhm... the author seems to be confused about the relationship between Github and git. GitHub doesn't develop git, it's a service built on top of git. Any modifications to git has nothing to do with GitHub. GFVS is a GitHub project, not part of git.

aeorgnoieang 9y ago
> GitHub doesn't develop git I'd be surprised if none of their employees have contributed to Git. I didn't interpret anything in the article as saying that GitHub develops Git, in its entirety (or even largely). > GFVS is a GitHub project GFVS is a Microsoft project and GitHub seems to be contributing. And the GFVS developers have been submitting, successfully, their changes to Git itself upstream to the actual Git project. So it is becoming a part of Git.
- jwilk 9y ago
  > I didn't interpret anything in the article as saying that GitHub develops Git "Microsoft [...] wanted to get these modifications accepted upstream and integrated into the standard Git client. That plan appears to be going well. Yesterday, the company announced that GitHub was adopting its modifications and that the two would be working together to bring suitable clients to macOS and Linux." This hints that Git upstream = GitHub. I mean, why mention GitHub at all if they aren't upstream? The rest of the article doesn't explain GitHub's role in this story either.
  - hdhzy 9y ago
    Maybe quickly changing topics is what causes the confusion. I guess they picked github because it has enterprise offering and enterprise customers are interested in large git repos. So github is like a test lab for GVFS. In the mean time they upstream the changes to real git adjusting GVFS to what git maintainers think is right.
  - aeorgnoieang 9y ago
    I still don't think your interpretation is warranted, but meh. Microsoft made some contributions and they're working to get them accepted upstream. They're maintaining a fork basically. GitHub is adopting their modifications, i.e. GitHub is running Microsoft's fork. Microsoft's modifications are necessary for them, particularly so they can use Git with the giant Windows repo. Presumably, GitHub also has a need or desire for those same or similar modifications. IIRC, some of GitHub's largest customers need or want a version of Git that can also handle large repos, or repos with large files or large numbers of files.

dfabulich 9y ago

> Microsoft and GitHub are also working to bring similar capabilities to other platforms, with macOS coming first, and later Linux. The obvious way to do this on both systems is to use FUSE, an infrastructure for building file systems that run in user mode rather than kernel mode (desirable because user-mode development is easier and safer than kernel mode). However, the companies have discovered that FUSE isn't fast enough for this—a lesson Dropbox also learned when developing a similar capability, Project Infinite. Currently, the companies believe that tapping into a macOS extensibility mechanism called Kauth (or KAuth) will be the best way forward. I'd love to read more technical details on this. How can Kauth support a virtual filesystem?

tankenmate 9y ago

"Git wasn't designed for such vast numbers of developers—more than 20,000 actively working on the codebase. Also, Git wasn't designed for a codebase that was so large, either in terms of the number of files and version history for each file, or in terms of sheer size, coming in at more than 300GB. When using standard Git, working with the source repository was unacceptably slow. Common operations (such as checking which files have been modified) would take multiple minutes." These sentences in some parts gloss over the details, and in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel), it was designed for huge numbers of files (but large numbers of files works fine on Linux due to the dentry cache, it sucks on Windows because they don't have a cache that has the same behaviour as the Linux dentry cache). Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size? That said, monorepos with several dozen software projects can be very slow due to O(n) functions such as git blame and the like. It's true that git wasn't designed for large numbers of projects in one repo that are only indirectly shared by library code and the like. Unfortunately that was not written in the article.

godzillabrennus 9y ago
Your comment reminds me that Microsoft should really double down on replacing NTFS. Even Apple has moved off HFS+ with their latest OS.
- sekh60 9y ago
  They unfortunately removed the ability to create ReFS disks from Windows 10 Pro and lower. Hopefully they reverse that decision soon. That said I'm not sure how stable it was considered.
alkonaut 9y ago
I develop a desktop app with lots of test documents (test inputs, outputs) that are tied to specific revisions in order to work with the test suite that uses them. None are huge but there are many and some are frequently changed (2GB working copy, 15gb repo, 100k commits). They could be put in some (very) complex hierarchy on a network share, but having them in vcs is so convenient. If rev 9999 changes the code so that the expected output testfileA differs then you update the expected output in the same commit. If you don’t have the file and you want to be able to clone+build+test any revision then you still need the results for 9998 also on the share. This scenario is nicely handled in svn, mercurial, tfvc and similar. If this lets git do it then it looks like a great idea. Game developers with lots of churn in huge binary assets also have the same problem, but probably even worse.

jopsen 9y ago

Hmm, some part of me wouldn't mind, a FUSE setup where: /repository/github.com/<org>/<repo>/ is automatically cloned out from github when accessed the first time :) It would be pretty cool to use as a $GOPATH. Or just for making drive-by github contributions super easy.

zimbatm 9y ago
I built https://github.com/zimbatm/h https://github.com/zimbatm/h to satisfy the same need but it's a command-line tool, not a FUSE filesystem. In some way I prefer it because transparent network access doesn't work really well with failure modes. Usage: `h <owner>/<repo>` clones the repo, if it doesn't exist, and takes you there. This and hub are integral to my developer workflow: $ h some/repo $ git checkout -b some-branch [hack and commit] $ hub fork $ git push -u zimbatm $ hub pull-request Done
hinkley 9y ago
It's all fun and games until someone like me tries to run `find` on the parent directory...
- silentOpen 9y ago
  Children would be lazily loaded and would not appear in directory listing of the parent directory. `find` would search your existing local checkouts only and the file system would still be POSIX.
  - hinkley 9y ago
    How do you trigger loading the projects if there isn’t a pointer to them showing up in the file system?
    - koffiezet 9y ago
      When you open or stat a directory that doesn't exist, try to checkout that path, if that fails, return an ENOENT error?

rokob 9y ago

Interesting to see them try to fix Git when Facebook considered it (due to similar constraints) and decided to go with Hg instead: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc...

jng 9y ago
Microsoft is used to implementing its own filesystems, so it probably feels more natural for them to do that than it does for Facebook.
glandium 9y ago
They decided to go with mercurial because it was possible to monkey patch its guts from extensions written in python. They're now talking about implementing parts in rust, which, ironically, would have prevented them from doing what they originally chose mercurial for.
- masklinn 9y ago
  > They decided to go with mercurial because it was possible to monkey patch The ability to extend, swap and replace deep into the stack was always a core component of Mercurial, hence having to enable extensions for things many users would consider core features e.g. terminal colors or graph log Facebook didn't hack around like monkeys, they built extensions. And when they could not do it as extensions, they upstreamed improvements to the core. The alternative would have been to fork the codebase entirely. Monkey patching the internals would have been significantly less maintainable: having used it as a library, I can tell you that none of the internal stuff is considered stable and pretty major components will change between point releases (I was using the diff-parsing and patch-application routines for something else, the API changed basically every minor release, forking would at least give you a heads-up conflict when upstream changed, monkey-patching would either blow up at runtime or not go through the patch anymore) > They're now talking about implementing parts in rust, which, ironically, would have prevented them from doing what they originally chose mercurial for. No.
ethomson 9y ago

Koshkin 9y ago

To me it looked like one of the points of Git was that one could work on anything from the entire repository locally, i.e., for example, without having an internet connection, e.g. when on an airplane. (I think having one huge repository is the problem here.)

jaccarmac 9y ago
Well, Git was indeed created to serve Linux kernel development. So decentralized access for the people writing patches all over the world was useful. However, when companies started using source control for their monorepos which developers access through a work network or VPN decentralization takes a backseat to business concerns. Different use cases and all that.
- secstate 9y ago
  I think the problem there, is that you then basically wind up with Git on the backend and an SVN-like layer over the front. Which seems a bit silly. SVN is a very robust project that works really well. Why not add git-like porcelain to SVN if you're gonna force devs to have an network connection so they don't have to download a giant git working directory. Smart people are working on all this, so I'm sure there are reasons, but in the all the instances where I've had to interact with a monorepo, it was because the tech debt was too high to pay off to break it apart, not because it was better. And if you're indebted to the point where you have no point in paying it off, you damn well better have leveraged assets against it (i.e. a cash cow of a business, like MS Word or Facebook)
  - jaccarmac 9y ago
    That's interesting. I've never been an insider and it was several years ago, but Google spoke in pretty glowing terms of their monorepo when I heard them talk about it. That's neither here nor their though, not my personal preference either to be fair. As far as Git's killer feature I think decentralization is only one part of it. Being able to deal with the change graph directly is sometimes handy. And knowing the basis of the model, I find Git pretty easy to understand even when history gets fuzzy. Not sure but I think Darcs might be even better for that, just without the mindshare.

jeanmichelx 9y ago

This is only tangentially related, but is there any standard advice to get git on a remote file system (ceph) to run fast with a long history?

gumby 9y ago

A couple of thoughts as the implementation downloads mainly placeholders: that mechanism can also be combined with permissions on the server side so contractor X who is fixing a driver need only have access to the driver and some supporting code while still checking into the monorepo. It also means that every developer’s laptop won’t have a full copy of the repo (so if said laptop is lost, the risk is more contained) And it’s similar to Part of LFS, bigfiles and other “blob” add-o s to git...could see in future a way to put .doc files and the like into repos.

drdrey 9y ago

Sounds a bit similar to Google's Piper (https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...) that lets you check out a portion of the monorepo, no?

Myrmornis 9y ago

> The open source, free GitHub hosting doesn't need the scaling work Microsoft has done They might want to fix that sentence. It reads as saying that Github is open source. Perhaps it should read > The free GitHub hosting that is typically used by open source projects doesn't need the scaling work Microsoft has done

exabrial 9y ago

This is kinda funny, it undoes one of the original pillars of why git was made

nerdponx 9y ago

It's a little surprising to me that these companies use Git at all. What's the appeal of Git really? Is it just the "familiarity factor", and is that really a reason to pour so many resources into it?

dingo_bat 9y ago
I think almost all the people they hire nowadays would be familiar with git and not something else. So it would be a lot of time savings in training if you switch to git instead of training every newbie to your own peculiar source control mechanism.
- nerdponx 9y ago
  Sure, but the CLI is the CLI. I can't imagine there's no way to replicate the Git CLI without rewriting the internals. Surely the basics of branch-and-merge will be the same, right? I doubt anyone would miss the dangerously overloaded `checkout` subcommand, or wrestling with rebase conflicts.
OOPMan 9y ago
Personally, I imagine low-cost in-place branching is useful to most people doing serious work...?

mg74 9y ago

"The biggest complexity is that Git has a very conservative approach to compatibility, requiring that repositories remain compatible across versions." This, above all else. That I never, or anyone else on our team, ever has to worry about wether they are running an "older version" of git is such a win.

gear54rus 9y ago
Big problem. Just update your machines. Am I supposed to miss great features because of your admin laziness? It has to fail gracefully on version mismatch, other than that it's just a bad decision.
xyzzy_plugh 9y ago
It's not really a surprise that monorepo advocates struggle here. Compatibility through versioning and stable APIs is challenging and requires discipline. Making wide-spread changes to all clients of your code when you make changes that break things doesn't require discipline.

iceman2654 9y ago

Does anyone else think this is another "Embrace, extend, and extinguish" move from MS? I don't know enough about the changes make that judgement.

m12k 9y ago

It seems that lazy-downloading files isn't all that useful if you need to compile all the source code. Does anyone know which build tools can be used to avoid having to open most of the source files - maybe some centralized build cache?