10 ms·

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

by danbruc 13y ago

cliveowen 13y ago
Exactly, there's no way they wrote 8GB of code.
- justincormack 13y ago
  I think they did, have heard similar figures from them before, all deployed as one file. Macroservices...
  - SnakeDoc 13y ago
    unlikely. they probably have a very dirty repo with tons of binaries, images, blah blah blah. It's highly unlikely they actually wrote 8GB of code, and the 46GB .git directory will be littered with binary blob changes, etc. This is really just to "impress" two people: 1) People who love Facebook 2) People who don't know anything about version control and/or how to do proper version control (no binaries in the scm).
    - justincormack 13y ago
      In 2012 the Facebook binary was 1.5GB http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/ http://arstechnica.com/business/2012/04/exclusive-a-behind-t...
    - vmarsy 13y ago
      I cited this interesting paper from Facebook engineers(2013) in another comment : https://news.ycombinator.com/item?id=7648802 https://news.ycombinator.com/item?id=7648802 In 2011 they had 10 millions LoC, up to 500 commits a day, but if we asssume the plots keeps going up like this, now in 2014 it can be pretty big. Their binary was 1.5GB when the paper was written.
      - SnakeDoc 13y ago
        1.5GB binary that includes their dependencies (other binaries).
maaaats 13y ago
The big .git directory is probably binary revisions? Is there any good way around that in git?
- rickyc091 13y ago
  Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.
  - masklinn 13y ago
    > Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revisions count), broader (more files), more complex and probably full of binary stuff.
    - Dylan16807 13y ago
      Is that all actually code? Each commit would have to add an average of 5KB compressed so maybe 20KB of brand new code.
      - masklinn 13y ago
        > Is that all actually code? Of course not, it's an actual project. There is code, there are data files (text and binary) and there are assets.
- Tobu 13y ago
  Stripping the binaries with filter-branch, or normalising them using smudge filters (for example, making sure zip-based formats use no compression).
- 13y ago
rquirk 13y ago
I bet they check in (or have checked in at some point) 3rd party libraries, jar files, generated files, etc. I battle this every day at $dayjob and we have a multi GB subversion repository with a separate one for the 3rd party binaries. Svn handles this a bit better than the DVCSes, so just checking out the HEAD is smaller than the full history you get in git/hg, and you can clean up some crud a bit. It just lives on the central server, not in everyone's working copy.
- michaelmior 13y ago
  You can clone with a single revision in git. Still ends up being around 2X the size I believe since you have the objects stored for the single revision as well as the working tree.
  - Dylan16807 13y ago
    2X the size of what? The raw files? Assuming little shared data, sure. But svn will make two copies of every file too.
    - deleted 13y ago
      [deleted]
vijayr 13y ago
What about unit tests (and other testing code)? Those could occupy decent size too.
mixedbit 13y ago
8GB would be at least 100 million lines of code (upper bound with 80 characters per each line). For comparison Linux has 15+ million lines of code, PHP 2+M.
- smsm42 13y ago
  PHP's repo is around 500M. But I'd say it is probably tens to a hundred times smaller than Facebook should be, especially if you count non-public stuff they must have there. So comes out about right.
danbruc 13y ago
Facebook now has 7000 employees and is 10 years old. Each employee would have had to write 14 kiB of code (357 LOC with 40 (random guess of mine) characters per LOC) every day during this 10 years to produce 8 GiB of code. (Obviously assuming the repository contains only non-compressed code and only one version of everything and no metadata and...)
- patmcguire 13y ago
  If they've got memcached with their own patches, linux with their own patches, Hadoop with their own patches, etc. and tons of translations I can see 8 gigs of text.
  - zhemao 13y ago
    Why would they put that all in the same repository? I'm pretty sure this 8 GB repo is just their website code. A frontend dev working on a Timeline feature shouldn't have to check out the Linux kernel.
hnriot 13y ago
how is 8gb a lot, that's a small thumb drive's worth.
hiphopyo 13y ago
Could it be it contains a DB of all their users?

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

by danbruc 13y ago

8 GB is still a lot. Would be interesting to know how much of it is actual code and how much is just images and so on.

cliveowen 13y ago

Exactly, there's no way they wrote 8GB of code.

justincormack 13y ago
I think they did, have heard similar figures from them before, all deployed as one file. Macroservices...
- SnakeDoc 13y ago
  unlikely. they probably have a very dirty repo with tons of binaries, images, blah blah blah. It's highly unlikely they actually wrote 8GB of code, and the 46GB .git directory will be littered with binary blob changes, etc. This is really just to "impress" two people: 1) People who love Facebook 2) People who don't know anything about version control and/or how to do proper version control (no binaries in the scm).
  - justincormack 13y ago
    In 2012 the Facebook binary was 1.5GB http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/ http://arstechnica.com/business/2012/04/exclusive-a-behind-t...
  - vmarsy 13y ago
    I cited this interesting paper from Facebook engineers(2013) in another comment : https://news.ycombinator.com/item?id=7648802 https://news.ycombinator.com/item?id=7648802 In 2011 they had 10 millions LoC, up to 500 commits a day, but if we asssume the plots keeps going up like this, now in 2014 it can be pretty big. Their binary was 1.5GB when the paper was written.
    - SnakeDoc 13y ago
      1.5GB binary that includes their dependencies (other binaries).

maaaats 13y ago

The big .git directory is probably binary revisions? Is there any good way around that in git?

rickyc091 13y ago
Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. I guess another way would be to just archive all the history up to a certain point somewhere.
- masklinn 13y ago
  > Re-cloning a fresh repo should keep it small. There's also a git gc method which cleans up the repo. There's only so much git gc can do. We've got a 500MB repo (.git, excluding working copy) at work, for 100k revisions. That's with a fresh clone and having tried most combinations of gc and repack we could think of. Considering the size of facebook, I can only expect that their repo is deeper (longer history in revisions count), broader (more files), more complex and probably full of binary stuff.
  - Dylan16807 13y ago
    Is that all actually code? Each commit would have to add an average of 5KB compressed so maybe 20KB of brand new code.
    - masklinn 13y ago
      > Is that all actually code? Of course not, it's an actual project. There is code, there are data files (text and binary) and there are assets.
Tobu 13y ago
Stripping the binaries with filter-branch, or normalising them using smudge filters (for example, making sure zip-based formats use no compression).
13y ago

rquirk 13y ago

I bet they check in (or have checked in at some point) 3rd party libraries, jar files, generated files, etc. I battle this every day at $dayjob and we have a multi GB subversion repository with a separate one for the 3rd party binaries. Svn handles this a bit better than the DVCSes, so just checking out the HEAD is smaller than the full history you get in git/hg, and you can clean up some crud a bit. It just lives on the central server, not in everyone's working copy.

michaelmior 13y ago
You can clone with a single revision in git. Still ends up being around 2X the size I believe since you have the objects stored for the single revision as well as the working tree.
- Dylan16807 13y ago
  2X the size of what? The raw files? Assuming little shared data, sure. But svn will make two copies of every file too.
  - deleted 13y ago
    [deleted]

vijayr 13y ago

What about unit tests (and other testing code)? Those could occupy decent size too.

mixedbit 13y ago

8GB would be at least 100 million lines of code (upper bound with 80 characters per each line). For comparison Linux has 15+ million lines of code, PHP 2+M.

smsm42 13y ago
PHP's repo is around 500M. But I'd say it is probably tens to a hundred times smaller than Facebook should be, especially if you count non-public stuff they must have there. So comes out about right.

danbruc 13y ago

Facebook now has 7000 employees and is 10 years old. Each employee would have had to write 14 kiB of code (357 LOC with 40 (random guess of mine) characters per LOC) every day during this 10 years to produce 8 GiB of code. (Obviously assuming the repository contains only non-compressed code and only one version of everything and no metadata and...)

patmcguire 13y ago
If they've got memcached with their own patches, linux with their own patches, Hadoop with their own patches, etc. and tons of translations I can see 8 gigs of text.
- zhemao 13y ago
  Why would they put that all in the same repository? I'm pretty sure this 8 GB repo is just their website code. A frontend dev working on a Timeline feature shouldn't have to check out the Linux kernel.

hnriot 13y ago

how is 8gb a lot, that's a small thumb drive's worth.

hiphopyo 13y ago

Could it be it contains a DB of all their users?