5 ms·

> On $dayjob's repository, the base checkout is several gigs. Why is it several gigs? Is that really necessary?

by tasuki 6y ago

chrisseaton 6y ago
> Why is it several gigs? A lot of code written by a lot of engineers over a lot of years. I'm not sure what other answer you're expecting? I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual. > Is that really necessary? What would you do? Delete history every year or so? I regularly annotate files and see useful history from ten years ago that I need to do my work.
- andrewstuart2 6y ago
  > I'm not sure what other answer you're expecting? I've seen some things. Binaries, large files, isos, generally just any large file with no lines that somebody just didn't know better and committed, changed a few bytes a few times, and generated huge deltas quickly. I'm far more surprised you don't know what he was expecting than I am that he asked, considering how many times I've run into this in the past.
  - chrisseaton 6y ago
    If people want a concrete example, here's a 6 GB repo that's 90% Java, 5% C, then some other languages. https://github.com/oracle/graal https://github.com/oracle/graal It's not even a mono-repo - this is just part of the project. Maybe someone's got some tools that let them dig around in the history and find large things or explain why it's so large? I don't think they've been checking ISOs in.
    - andrewstuart2 6y ago
      I'm totally not surprised that large repos exist, without having fallen prey to any antipatterns. Just saying that I know "have you committed any binaries, dependencies, generated code, etc" are the first questions I tend to ask when somebody tells me their repo is massive. In part selection bias, though, because usually the people working on super old projects know exactly why.
    - neolog 6y ago
      https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/
    - cmurphycode 6y ago
      I was really taken aback by this given my previous experience with the Linux kernel git repo size. So, I just checked out Graal. Receiving objects: 100% (981372/981372), 187.08 MiB | 6.39 MiB/s, done. % du -sh graal 328M graal % du -sh .git 221M .git And that includes many binary files that were added and removed over time, but not filter-branched out of existence, including jars, pngs, pdfs :) (https://stackoverflow.com/questions/10622179/how-to-find-identify-large-commits-in-git-history https://stackoverflow.com/questions/10622179/how-to-find-ide...)
      - chrisseaton 6y ago
        Huh... maybe I have a degenerate copy for having it so long and I should try running the compression commands myself.
        lathiat 6y ago
        See also “git prune” to remove references to branches and objects no longer on the remote server. Check the docs though I think in some cases I might delete local branches or objects so make a backup first.
    - cerved 6y ago
      git filter repo has tools for analyzing history
  - masklinn 6y ago
    Nope, the answer is exactly what’s above, this is a large and old project with a fair amount of churn. There are assets in there but they’re a small minority of the issue. The biggest issue is actually the l10n stuff. It’s all text with lines but there is a lot of it and historically the exports were not really ordered so every major update to the translation files scrambled everything.
- pas 6y ago
  That just seems an excessive amount of text! But the Linux kernel repo (.git + working copy) is over 7GB. It turns out folks can write an ungodly amount of code.
  - cerved 6y ago
    but you have to write ungodly amounts of code to get a repo that big, and it's not even that big that or check-in binaries, logs, database backups or other things that don't belong in VC. TBH it's pretty much always this
- tasuki 6y ago
  > I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual. That's a lot of code! I think that is at least a little unusual. The monorepos I see are usually large because people gratuitously commit large binary files (and sometimes change them, generating huge changesets).
  - stkdump 6y ago
    It is. Let's say we are talking 30 person-years, that is 60000 days. It means 100KB of text written per day and engineer.
    - stkdump 6y ago
      Can't edit anymore, but ofc the math is off by factor 10: 'tens of engineers over more than a decade' is more in the realm of 300 person-years. Still 10KB per day and engineer, which is maybe 200-500 lines of code. Far above average still, but approaching something realistic. This is if we are talking 6GB with full history and without any compression.
dbeley 6y ago
Just clone any large open source software you know, most of them are several gigs. I don't know if it's necessary, but that's just the way it is: large monorepos are everywhere.
amelius 6y ago
> Why is it several gigs? Is that really necessary? Just wait until you have to store deep learning training data.
- skj 6y ago
  Sounds like something to store in not-git.
  - mayank 6y ago
    Git has large file storage now: https://git-lfs.github.com/ https://git-lfs.github.com/
    - remram 6y ago
      That is not-git, storing files this way will not increase your repo size. Same with git-annex, same with commiting .txt files with a URL to a file on S3.
      - amelius 6y ago
        I'm using git-shell to serve my repo. Can those extensions run over git-shell? Or do I need to set up a new server?
        remram 6y ago
        They are separate. git-lfs needs a separate server, that will communicate with the Git client over HTTP (not SSH), and git-annex can use a variety of different backends for storing the data (which doesn't need to be related to the Git repository) such as S3, FTP, webdav, Google Drive, ... and even git-lfs. GitLab and Gitea come with a git-lfs server, but I don't know if there is any standalone server that you could use with git-shell.
        amelius 6y ago
        Ok, thanks. This is why I hope large file support will be integrated in Git. It's just too much of a hassle to figure out the security implications of yet another server, and administrating it, setting up the connection between Git and the plugin, etc.
yalok 6y ago
Just depends on the size of the company and amount of developers. With monorepo approach, all of the projects are in one repo, all kids of technologies/languages, forks of various open source projects, sources of tools/utilities... At one of my work places one of the monorepos was over 50GB... took an hour to clone and many minutes to pull/rebase if I was a few days behind... I don’t like monorepo approach because of such monstrosity, but the upside is it’s easier to commit/sync changes that span multiple products/components, and easier to merge many commits a minute from thousands of engineers...

> On $dayjob's repository, the base checkout is several gigs. Why is it several gigs? Is that really necessary?

by tasuki 6y ago

> On $dayjob's repository, the base checkout is several gigs. Why is it several gigs? Is that really necessary?

> Why is it several gigs? A lot of code written by a lot of engineers over a lot of years. I'm not sure what other answer you're expecting? I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual. > Is that really necessary? What would you do? Delete history every year or so? I regularly annotate files and see useful history from ten years ago that I need to do my work.

andrewstuart2 6y ago
> I'm not sure what other answer you're expecting? I've seen some things. Binaries, large files, isos, generally just any large file with no lines that somebody just didn't know better and committed, changed a few bytes a few times, and generated huge deltas quickly. I'm far more surprised you don't know what he was expecting than I am that he asked, considering how many times I've run into this in the past.
- chrisseaton 6y ago
  If people want a concrete example, here's a 6 GB repo that's 90% Java, 5% C, then some other languages. https://github.com/oracle/graal https://github.com/oracle/graal It's not even a mono-repo - this is just part of the project. Maybe someone's got some tools that let them dig around in the history and find large things or explain why it's so large? I don't think they've been checking ISOs in.
  - andrewstuart2 6y ago
    I'm totally not surprised that large repos exist, without having fallen prey to any antipatterns. Just saying that I know "have you committed any binaries, dependencies, generated code, etc" are the first questions I tend to ask when somebody tells me their repo is massive. In part selection bias, though, because usually the people working on super old projects know exactly why.
  - neolog 6y ago
    https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/
  - cmurphycode 6y ago
    I was really taken aback by this given my previous experience with the Linux kernel git repo size. So, I just checked out Graal. Receiving objects: 100% (981372/981372), 187.08 MiB | 6.39 MiB/s, done. % du -sh graal 328M graal % du -sh .git 221M .git And that includes many binary files that were added and removed over time, but not filter-branched out of existence, including jars, pngs, pdfs :) (https://stackoverflow.com/questions/10622179/how-to-find-identify-large-commits-in-git-history https://stackoverflow.com/questions/10622179/how-to-find-ide...)
    - chrisseaton 6y ago
      Huh... maybe I have a degenerate copy for having it so long and I should try running the compression commands myself.
      - lathiat 6y ago
        See also “git prune” to remove references to branches and objects no longer on the remote server. Check the docs though I think in some cases I might delete local branches or objects so make a backup first.
  - cerved 6y ago
    git filter repo has tools for analyzing history
- masklinn 6y ago
  Nope, the answer is exactly what’s above, this is a large and old project with a fair amount of churn. There are assets in there but they’re a small minority of the issue. The biggest issue is actually the l10n stuff. It’s all text with lines but there is a lot of it and historically the exports were not really ordered so every major update to the translation files scrambled everything.
pas 6y ago
That just seems an excessive amount of text! But the Linux kernel repo (.git + working copy) is over 7GB. It turns out folks can write an ungodly amount of code.
- cerved 6y ago
  but you have to write ungodly amounts of code to get a repo that big, and it's not even that big that or check-in binaries, logs, database backups or other things that don't belong in VC. TBH it's pretty much always this
tasuki 6y ago
> I work with a compiler that has just tens of engineers working on it over just a decade or so and even that's a 6 GB repository. No binary assets. Just source code and configuration I think. I really don't think it's that unusual. That's a lot of code! I think that is at least a little unusual. The monorepos I see are usually large because people gratuitously commit large binary files (and sometimes change them, generating huge changesets).
- stkdump 6y ago
  It is. Let's say we are talking 30 person-years, that is 60000 days. It means 100KB of text written per day and engineer.
  - stkdump 6y ago
    Can't edit anymore, but ofc the math is off by factor 10: 'tens of engineers over more than a decade' is more in the realm of 300 person-years. Still 10KB per day and engineer, which is maybe 200-500 lines of code. Far above average still, but approaching something realistic. This is if we are talking 6GB with full history and without any compression.

dbeley 6y ago

Just clone any large open source software you know, most of them are several gigs. I don't know if it's necessary, but that's just the way it is: large monorepos are everywhere.

amelius 6y ago

> Why is it several gigs? Is that really necessary? Just wait until you have to store deep learning training data.

skj 6y ago
Sounds like something to store in not-git.
- mayank 6y ago
  Git has large file storage now: https://git-lfs.github.com/ https://git-lfs.github.com/
  - remram 6y ago
    That is not-git, storing files this way will not increase your repo size. Same with git-annex, same with commiting .txt files with a URL to a file on S3.
    - amelius 6y ago
      I'm using git-shell to serve my repo. Can those extensions run over git-shell? Or do I need to set up a new server?
      - remram 6y ago
        They are separate. git-lfs needs a separate server, that will communicate with the Git client over HTTP (not SSH), and git-annex can use a variety of different backends for storing the data (which doesn't need to be related to the Git repository) such as S3, FTP, webdav, Google Drive, ... and even git-lfs. GitLab and Gitea come with a git-lfs server, but I don't know if there is any standalone server that you could use with git-shell.
        amelius 6y ago
        Ok, thanks. This is why I hope large file support will be integrated in Git. It's just too much of a hassle to figure out the security implications of yet another server, and administrating it, setting up the connection between Git and the plugin, etc.

yalok 6y ago

Just depends on the size of the company and amount of developers. With monorepo approach, all of the projects are in one repo, all kids of technologies/languages, forks of various open source projects, sources of tools/utilities... At one of my work places one of the monorepos was over 50GB... took an hour to clone and many minutes to pull/rebase if I was a few days behind... I don’t like monorepo approach because of such monstrosity, but the upside is it’s easier to commit/sync changes that span multiple products/components, and easier to merge many commits a minute from thousands of engineers...