10 ms·
A handful of us from the product team are around for a few hours to discuss if you're interested.
by vtbassmatt 9y ago
A handful of us from the product team are around for a few hours to discuss if you're interested.
- CiPHPerCoder 9y agoSure, a couple of questions: 1. How do you measure "largeness" of a git repo? 2. How are you confident that you have the largest? 3. How much technical debt does that translate to?
- vtbassmatt 9y ago1. Saeed is writing a really nice series of articles starting here: https://www.visualstudio.com/learn/git-at-scale/ https://www.visualstudio.com/learn/git-at-scale/ In the first one, he lays out how we think about small/medium/large repos. Summary: size at tip, size of history, file count at tip, number of refs, and number of developers. 2. Fairly confident, at least as far as usable repos go. Given how unusable the Windows repo is without GVFS and the other things we've built, it seems pretty unlikely anyone's out there using a bigger one. If you know of something bigger, we'd love to hear about it and learn how they solved the same problems! 3. Windows is a 30 year old codebase. There's a lot of stuff in there supporting a lot of scenarios.
- deleted 9y ago[deleted]
- MikusR 9y agoIs it possible to checkout (if not build) something like Windows 3.11 or NT 4?
- hyperrail 9y agoAs far as I can recall, this is not possible using Windows source control, as its history only goes back to the lifecycle of Windows XP (when the source control tool prior to GVFS was adopted). Microsoft does have an internal Source Code Archive, which does the moral equivalent of storing source code and binary artifacts for released software in a underground bunker. I used to have a bit of fun searching the NT 3.5 sources as taken from the Source Code Archive...
- qznc 9y agoI recently heard a story that someone tried to push a 1TB repo to our university Gitlab which then ran out of disk space. Sure, that might have been not be a usable repo but only an experiment. Still, I would bet against the claim that 300GB is the largest one.
- tormeh 9y ago1 TB of code?
- BenjiWiebe 9y agoI'd sure like to run that as my operating system, browser, virtual assistant, car automation system, and overall do-everything-for-me system...
- ethomson 9y ago300 GB is not the size of _the repository_. It's the size of the code base - the checked out tree of source, tests, build tools, etc - without history. It's certainly possible that somebody created a 1 TB source tree in Git, but what we've never heard of is somebody actually _using_ such a source tree, with 4000 or more developers, for their daily work in producing a product. I say this with some certainty because if somebody had succeeded, they would have needed to make similar changes to Git to be successful, though of course they could have kept such changes secret.
- rajathagasthya 9y agoVery cool blog! As I understand, you dynamically fetch a file from the remote git server once for the first time I open the file. Do you do any sort of pre-fetching of files? For example, if a file has an import and uses a few symbols from that file, do you also fetch the imported file beforehand or just fetch it when you access it first time?
- deleted 9y ago[deleted]
- saeednoursalehi 9y agoWe don't currently do that sort of predictive prefetching, but it's a feature we've thought a lot about. For now, users can explicitly call "gvfs prefetch" if they want to, or just allow files to be downloaded on demand.
- vtbassmatt 9y agoFor now, we're not that smart and simply fetch what's opened by the filesystem. With the cache servers in place, it's plenty fast. We do also have an optional prefetch to grab all the contents (at tip) for a folder or set of folders.
- muglug 9y agoWhat's the PR review UI built in?
- vtbassmatt 9y agoCustom JQuery-based framework, transitioning to React.
- taylorlafrinere 9y agoActually, I think we finished the conversion to React :). So, React.
- vtbassmatt 9y agoTaylor is the dev manager for that area so I'm inclined to believe his correction :)
- kingbirdy 9y agoThe article mentions relying on a windows filesystem driver. Two questions about that: 1) Why include it in default windows? It seems that 99.99% of users would never even know it existed, let alone use it 2) Does that mean GVFS isn't useable on *nix systems? Any plans to make it useable, if so?
- hirsin 9y agoIt's not included in Windows, which is why they have a signed drop of the driver for you to install. Even internally we have to install the driver. E- oops, missed that line. Cool, wonder if it will show up in more than Git
- saeednoursalehi 9y ago1) The file system driver is called GvFlt. If it does get included in Windows by default, it'll be to make it easier for products like GVFS, but GvFlt on its own is not usable by end users directly. 2) GVFS is currently only available on Windows, but we are very interested in porting to other platforms.
- ATsch 9y agoWhat are your thoughts on implementing something more general like linux's FUSE[1] instead? A general Virtual Filesystem driver in Windows could be used for a wide range of things and means you don't just have a single-purpose driver sitting around. [1] https://en.m.wikipedia.org/wiki/Filesystem_in_Userspace https://en.m.wikipedia.org/wiki/Filesystem_in_Userspace
- bluejekyll 9y agoI'm currently investigating using GitLFS for a large repo that has many binary and other large artifacts. I'm curious, did you experiment with LFS for prior to building GitVFS? Also, I know that there is an (somewhat) active effort to port GitVFS to Linux, do you know if any of the Git vendors (GitLab and/or GitHub) are planning to support GitVFS in their enterprise products?
- saeednoursalehi 9y agoYes we did evaluate LFS. The thing about LFS is that while it does help reduce the clone size, it doesn't reduce the number of files in the repo at all. The biggest bottleneck when working with a repo of this size is that so many of your local git operations are linear on the number of files. One of the main values of GVFS is that it allows Git to only consider the files you're actually working with, not all 3M+ files in the repo.
- bluejekyll 9y ago> One of the main values of GVFS is that it allows Git to only consider the files you're actually working with, not all 3M+ files in the repo. That is an excellent point. Thanks!
- sytse 9y agoWe at GitLab are looking at GitVFS but have not made a decision yet https://gitlab.com/gitlab-org/gitlab-ce/issues/27895 https://gitlab.com/gitlab-org/gitlab-ce/issues/27895
- sigil 9y agoThis is amazing, congrats. I worked on Windows briefly in 2005 (the same year git was released!) and was surprised at how well Source Depot worked, especially given the sheer size of the codebase and the other SCM tools at the time. Is there anything people particularly miss about Source Depot? Something SD was good at, but git is not?
- vtbassmatt 9y agoI just got a request today for an API equivalent to `sd files`, which is not something Git is natively great at without a local copy of the repo.
- taylorlafrinere 9y agoAnother interesting complaint is one that we hear from a lot of people who move from CVCS to DVCS: there are too many steps to perform each action. For example, "why do I have to do so many steps to update my topic branch". While we find that people get better with these things over time, I do think it would be interesting to build a suite of wrapper commands that roll a bunch of these actions up.
- hart_russell 9y agoWhat was the impetus for switching to git?
- vtbassmatt 9y agoMore or less: - Availability of tools - Familiarity of developers (both current and potential)
- chinhodado 9y agoWhy do you name it "GVFS" instead of something more descriptive like "GitVFS"?
- saeednoursalehi 9y agoThis was discussed a bit in the comments in Brian Harry's last post on GVFS: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-git-and-some-back-story/ https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-g... We're building a VFS (Virtual File System) for Git (G) so GVFS was a very natural name and it just kind of stuck once we came up with it.
- Kostic 9y agoAre you aware that the name was already taken[0] for something which also has to do with file systems? [0] https://wiki.gnome.org/Projects/gvfs https://wiki.gnome.org/Projects/gvfs
- alkonaut 9y agoDoes the virtualization work equally well for lots of history as it does for large working copies? I have a 100k commit svn repo I have been trying to migrate but the result is just too large. Partly this is due to tons of revisions of binary files that must be in the repo. Does the virtualization also help provide a shallow set of recent commits locally but keep all history at the server (which is hundreds of gigs that is rarely used)?
- vtbassmatt 9y agoGVFS helps in both dimensions, working copy and lots of history. For the lots of history case, the win is simply not downloading all the old content. A GVFS clone will contain all of the commits and all of the trees but none of the blobs. This lets you operate on history as normal, so long as you don't need the file content. As soon as you touch file content, GVFS will download those blobs on demand.
- alkonaut 9y agoThanks - that sounds perfect for lots of binary history as you never view history on the binaries, only the source files.
- mschuster91 9y agoHow do you prevent data exfiltration? I mean, in theory you could restrict the visibility of repos to the user based on team membership/roles and so prevent a single person from unauditably exfiltrating the whole Windows source code tree. In contrast with a monorepo there likely won't be any alerts triggered if someone does do a full git clone, except for someone saturating his switch port...
- asdfgadsfgasfdg 9y agoWhat would someone do with the source code for Windows? No one in open source would want to touch it. No large company would want to touch it. Grey/black hats are probably happier with their decompilers. Surely it would be easier to pirate than build (assuming their build system scales with most build systems I've observed in the wild). No small company would want to touch it. Anyway MS share source with various third parties (governments at least and I believe large customs in general) so any of these are a potential leak source.
- vtbassmatt 9y agoThis is all correct. Also, we'd notice someone grabbing the whole 300GB whether it's in 40 SD depots or a single Git repo.
- e40 9y agoMind if I ask how you'd notice?
- DoofusOfDeath 9y ago"A handful of us from the product team are around for a few hours to discuss if you're interested." Thanks! This is a little off-topic, but why can't Windows 10 users conclusively disable all telemetry? (I consider the question only a little off-topic, because I have the impression that this story is part of an ongoing Microsoft charm-offensive.)
- gear54rus 9y agoHaha, no answer, as expected. HN got butthurt as well lol.
- DoofusOfDeath 9y agoI knew there was a risk of getting downvoted, but I was surprised that it went to "-4".
- colejohnson66 9y agoWhy not TFS?
- wilatmsft 9y agohttps://news.ycombinator.com/item?id=14411588 https://news.ycombinator.com/item?id=14411588
- e40 9y agoAny plans to port GVFS to Linux or macOS?