4 ms·

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do y

by kk1274 9y ago

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

mschuster91 9y ago
Hmm. I believe it's likely there's also lots of binary assets - the WAV sound files, BMP images, the "hello world" videos, for example - and possibly also the raw versions of the assets. And if it's really the whole history of Windows in there, that's a LOT of binary assets in LOTS of versions.
taylorlafrinere 9y ago
Most of that 300GB isn't text. There are test assets, images, videos, built binaries, vhd's, etc. Also, I should be clear that that 300GB is just at tip (no history). We can debate about whether or not those things should be checked into the repo but they are there now.
- mschuster91 9y ago
  Whoa. 300GB with a shallow clone?! What size does the whole repo use on the server side?
  - wilatmsft 9y ago
    The pack file size for a full clone is 187GB. The 300GB is the working directory. We did not import the history of the code base, so the current repo only has about 5 months of history. As others have called out, there are a lot of assets in the repo that don't compress.
    - bokchoi 9y ago
      Why only 5 months? Will more of the history be added to the git repository eventually?
      - vtbassmatt 9y ago
        No, we'll keep the SD servers around for a while for servicing older products. We also have a "breadcrumbing" system that lets an engineer follow a file's history back from Git to the old system.
        ysleepy 9y ago
        Was importing the complete history tried during the development? This is very interesting. The git history will grow at break-neck speed and will reach similar size soon enough. Is this to delay the inevitable tech wrangling for dealing with terabyte histories or were there issues with the import/sync? Or maybe it was just the initial repo setup used for alpha testing that got promoted to production :)
- alkonaut 9y ago
  How did you go about creating the central repo and how long did it take? A 2Gb at tip svn repo with 100k commits is taking me many days and each odd failure typically has me restart the process after filtering out some obscure part of the tree. Edit: read in another comment that you dropped the history. Understandable, but can appreciate how that would add to friction (devs having to look through two different histories).
  - vtbassmatt 9y ago
    The Windows team developed a tool called "GitTrain" that knew how to: - migrate the tip of a branch to Git (yes, the 300GB number is the tips of all the interesting branches, not the history) - keep a Git branch and a SD branch in sync for a while - be re-run over each of the 400+ branches they care about But they went through some of the same trial-and-error process that you're describing.
zanny 9y ago
A lot of that is probably static art assets. A lot of big corporate projects are guilty of just throwing everything in one tree like this where for some bizarre reason you are checking out jpgs and video files. It is part of the reason why the Chrome or Unreal Engine repos are preposterously large (hundreds of megs to gigabytes each).
- ansible 9y ago
  And there was an article a few weeks back about how certain content creation tools (Adobe) stuff in lots of metadata into these assets by default. Some of the MS teams were good about stripping those out for release, others, didn't seem to notice.
  - ygra 9y ago
    Well, to be fair, the metadata in those files amounted to 5 MiB over the whole of Windows, so while there is a little overhead, it won't make up a significant chunk.
    - ansible 9y ago
      Oh, I didn't see that part. That's not nearly as bad as I thought based on my reading of the original article.
- MaulingMonkey 9y ago
  > A lot of big corporate projects are guilty of just throwing everything in one tree like this where for some bizarre reason you are checking out jpgs and video files. Would you want to build chrome without it's icons for bookmark folders, the launcher, etc.? No? What about a game without it's intro videos? Even if you need to debug a crash during the initial intro, related to playing said video? Some people prefer this kind of content to be managed with a different, media focused version control system, or a separate tree, but now you have 2+ disparate distribution / version control systems to try and keep in sync. You do have options like git submodules now for some VCSes, but I haven't been impressed with their workflow, and they weren't always an option. I will gladly burn several gigs just to simplify my workflow - I've got the spare SSD and bandwidth. I've had fully built checkouts in the hundreds of gigabytes range. And don't get me wrong, "hundreds of gigabytes" is getting annoyingly large if the IT department skimped and bought me a 250GB SSD. I'll abuse directory junctions to offload some of that onto mechanical drives. This is relatively fire and forget though - multi-repository is not.

by kk1274 9y ago

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

mschuster91 9y ago

Hmm. I believe it's likely there's also lots of binary assets - the WAV sound files, BMP images, the "hello world" videos, for example - and possibly also the raw versions of the assets. And if it's really the whole history of Windows in there, that's a LOT of binary assets in LOTS of versions.

taylorlafrinere 9y ago

Most of that 300GB isn't text. There are test assets, images, videos, built binaries, vhd's, etc. Also, I should be clear that that 300GB is just at tip (no history). We can debate about whether or not those things should be checked into the repo but they are there now.

mschuster91 9y ago
Whoa. 300GB with a shallow clone?! What size does the whole repo use on the server side?
- wilatmsft 9y ago
  The pack file size for a full clone is 187GB. The 300GB is the working directory. We did not import the history of the code base, so the current repo only has about 5 months of history. As others have called out, there are a lot of assets in the repo that don't compress.
  - bokchoi 9y ago
    Why only 5 months? Will more of the history be added to the git repository eventually?
    - vtbassmatt 9y ago
      No, we'll keep the SD servers around for a while for servicing older products. We also have a "breadcrumbing" system that lets an engineer follow a file's history back from Git to the old system.
      - ysleepy 9y ago
        Was importing the complete history tried during the development? This is very interesting. The git history will grow at break-neck speed and will reach similar size soon enough. Is this to delay the inevitable tech wrangling for dealing with terabyte histories or were there issues with the import/sync? Or maybe it was just the initial repo setup used for alpha testing that got promoted to production :)
alkonaut 9y ago
How did you go about creating the central repo and how long did it take? A 2Gb at tip svn repo with 100k commits is taking me many days and each odd failure typically has me restart the process after filtering out some obscure part of the tree. Edit: read in another comment that you dropped the history. Understandable, but can appreciate how that would add to friction (devs having to look through two different histories).
- vtbassmatt 9y ago
  The Windows team developed a tool called "GitTrain" that knew how to: - migrate the tip of a branch to Git (yes, the 300GB number is the tips of all the interesting branches, not the history) - keep a Git branch and a SD branch in sync for a while - be re-run over each of the 400+ branches they care about But they went through some of the same trial-and-error process that you're describing.

zanny 9y ago

A lot of that is probably static art assets. A lot of big corporate projects are guilty of just throwing everything in one tree like this where for some bizarre reason you are checking out jpgs and video files. It is part of the reason why the Chrome or Unreal Engine repos are preposterously large (hundreds of megs to gigabytes each).

ansible 9y ago
And there was an article a few weeks back about how certain content creation tools (Adobe) stuff in lots of metadata into these assets by default. Some of the MS teams were good about stripping those out for release, others, didn't seem to notice.
- ygra 9y ago
  Well, to be fair, the metadata in those files amounted to 5 MiB over the whole of Windows, so while there is a little overhead, it won't make up a significant chunk.
  - ansible 9y ago
    Oh, I didn't see that part. That's not nearly as bad as I thought based on my reading of the original article.
MaulingMonkey 9y ago
> A lot of big corporate projects are guilty of just throwing everything in one tree like this where for some bizarre reason you are checking out jpgs and video files. Would you want to build chrome without it's icons for bookmark folders, the launcher, etc.? No? What about a game without it's intro videos? Even if you need to debug a crash during the initial intro, related to playing said video? Some people prefer this kind of content to be managed with a different, media focused version control system, or a separate tree, but now you have 2+ disparate distribution / version control systems to try and keep in sync. You do have options like git submodules now for some VCSes, but I haven't been impressed with their workflow, and they weren't always an option. I will gladly burn several gigs just to simplify my workflow - I've got the spare SSD and bandwidth. I've had fully built checkouts in the hundreds of gigabytes range. And don't get me wrong, "hundreds of gigabytes" is getting annoyingly large if the IT department skimped and bought me a 250GB SSD. I'll abuse directory junctions to offload some of that onto mechanical drives. This is relatively fire and forget though - multi-repository is not.