5 ms·

> At GitHub, we store a lot of Git data: more than 18.6 petabytes of it, to be precise. That actually seems kinda small. Git’s lack of good support for large

> At GitHub, we store a lot of Git data: more than 18.6 petabytes of it, to be precise. That actually seems kinda small. Git’s lack of good support for large files means there’s probably an exabyte of data that, imho, should be source control but isn’t.

kortex 4y ago
It would be amazing if Github/lab provided a backing store for www.dvc.org . I've been using to great effect, but I have to rely on separate AWS integration for storing the large objects in s3.
- arjvik 4y ago
  Take a look at DagsHub - https://dagshub.com https://dagshub.com They do exactly what you're looking for! They provide both a DVC remote to push and pull DVC-tracked objects, as well as a UI showing them integrated alongside git-tracked files in the repository view. (Disclaimer: I work at DagsHub)
kccqzy 4y ago
That's indeed small. I'd guess that Google probably stores 4 orders of magnitude more data than GitHub. (I was in fact asked a long time ago in an interview to estimate how much disk was needed to store Google's search index.)
- sulam 4y ago
  Glad it was a long time ago. Those kinds of questions are awful.
  - isatty 4y ago
    Agreed that it isn’t ideal, but about “awful” specifically - I’m not too sure. I would never ask such a question but I would assume the intent is just to find out how you think and not to get you to spit out a number. Would it be fun if the interviewer worked together with you to approximate it?
    - kccqzy 4y ago
      Yup that was exactly the case. A fun discussion that involved estimating sizes of web pages, number of web pages, the effect of compression, the effect of deduplication, etc.
  - triceratops 4y ago
    All of these questions boil down to "pull some numbers out of your ass, then multiply (or divide) as appropriate".
    - saagarjha 4y ago
      This is coincidentally exactly how I guess my code will perform.
      - gregoryl 4y ago
        With the resources/time most of us have, thats pretty much the peak of accuracy on performance estimates anyway.
  - tharkun__ 4y ago
    Why? Unlike my sibling saying it's "pull out of thin air", I think this is an opportunity to actually do the opposite: Ask questions back! I have no idea how to guess that. I don't know what kind of information Google stores in it's "index". What exactly do they mean by "index" even? I need more input. I can ask the interviewer questions and start a discussion! Of course it's possible that the interviewer really just wanted to be a smart-ass and expected me to "just have an answer" and else fail me. But then I don't want to work there anyway. Having an actual discussion, as if we were just working together and had to 'solve' whatever problem at hand is great.
    - sulam 4y ago
      Because these kinds of questions do not tell you anything about the person’s ability as a developer, a leader or give you any insight into the ability to be innovative. As others have noted, it simply measures their ability to make basic assumptions and then do math, plus bullshit your way through a discussion. Might be perfect for hiring product managers tho. /s ;)
      - rpadovani 4y ago
        But you don't have to make assumptions! I think this is the important bit: you are supposed to ask questions back until you are confident enough about your answer. You should highlight that you are able to work with customers that want something without knowing what they want.
      - tharkun__ 4y ago
        Like my sibling reply says, I think you didn't read past my first word in the reply. The "why". That was rhetorical. My entire point was basically that it actually isn't about assumptions at all. If you just assume when I ask that question this tells me you're out as a developer I want to hire. Nobody can build proper software by just assuming the first thing that comes to mind and bullshitting. You have to show that you can take a totally ambiguous and open question and show me that you can systematically try to find out as much information as you can in a reasonable amount of time to verify your assumptions or at least make them less assuming. Of course (a d this can happen when solving a real problem too) it might be that you cannot accurately measure some input you need and you will have to make assumptions. But you will want to try and you will want to have divided your problem into lots of smaller parts such that the part where you still have to assume is only one small part and you want to be very explicit that this part is still an assumption only which you will have to verify or improve upon as you actually implement something. You do have to start at some point, otherwise it's analysis paralysis. And no this has nothing to do with a PM. They are notoriously bad at the above approach and instead will just assume the first best thing and bullshit their way through with that for as long as they don't get caught (no /s there but my experience with 80+% of PMs)
- latchkey 4y ago
  > estimate how much disk was needed to store Google's search index 1 googol!
- nixgeek 4y ago
  You might be off by an additional order or perhaps two!
ajb 4y ago
You can't actually put the Android source in GitHub because of the 4GB per repo size limit. Niche problem, but shows the scale of things.
- ajb 4y ago
  As people have pointed out, there's not a hard 4GB limit. They do recommend less than 5GB.
  - naikrovek 4y ago
    paying customers and popular open source repos can go well beyond 5GB. GHEC customers have a 5GB limit per file in git-lfs.
- saagarjha 4y ago
  You could if someone contacted GitHub to have it mirrored.
  - naikrovek 4y ago
    it's not a real limit.
- est31 4y ago
  WDYM by 4GB size limit? When I run du -h on my linux checkout, I get 6.4G (no build artifacts, just git tracked source code plus the .git directory). Even if I only look at the .git/objects directory, it's 5.2G large.
  - deleted 4y ago
    [deleted]
  - londons_explore 4y ago
    It seems GitHub have whitelisted git projects containing certain commits or files to go over the size limit. Chromium is also well over the size limit.

kortex 4y ago

It would be amazing if Github/lab provided a backing store for www.dvc.org . I've been using to great effect, but I have to rely on separate AWS integration for storing the large objects in s3.

arjvik 4y ago
Take a look at DagsHub - https://dagshub.com https://dagshub.com They do exactly what you're looking for! They provide both a DVC remote to push and pull DVC-tracked objects, as well as a UI showing them integrated alongside git-tracked files in the repository view. (Disclaimer: I work at DagsHub)

kccqzy 4y ago

That's indeed small. I'd guess that Google probably stores 4 orders of magnitude more data than GitHub. (I was in fact asked a long time ago in an interview to estimate how much disk was needed to store Google's search index.)

sulam 4y ago
Glad it was a long time ago. Those kinds of questions are awful.
- isatty 4y ago
  Agreed that it isn’t ideal, but about “awful” specifically - I’m not too sure. I would never ask such a question but I would assume the intent is just to find out how you think and not to get you to spit out a number. Would it be fun if the interviewer worked together with you to approximate it?
  - kccqzy 4y ago
    Yup that was exactly the case. A fun discussion that involved estimating sizes of web pages, number of web pages, the effect of compression, the effect of deduplication, etc.
- triceratops 4y ago
  All of these questions boil down to "pull some numbers out of your ass, then multiply (or divide) as appropriate".
  - saagarjha 4y ago
    This is coincidentally exactly how I guess my code will perform.
    - gregoryl 4y ago
      With the resources/time most of us have, thats pretty much the peak of accuracy on performance estimates anyway.
- tharkun__ 4y ago
  Why? Unlike my sibling saying it's "pull out of thin air", I think this is an opportunity to actually do the opposite: Ask questions back! I have no idea how to guess that. I don't know what kind of information Google stores in it's "index". What exactly do they mean by "index" even? I need more input. I can ask the interviewer questions and start a discussion! Of course it's possible that the interviewer really just wanted to be a smart-ass and expected me to "just have an answer" and else fail me. But then I don't want to work there anyway. Having an actual discussion, as if we were just working together and had to 'solve' whatever problem at hand is great.
  - sulam 4y ago
    Because these kinds of questions do not tell you anything about the person’s ability as a developer, a leader or give you any insight into the ability to be innovative. As others have noted, it simply measures their ability to make basic assumptions and then do math, plus bullshit your way through a discussion. Might be perfect for hiring product managers tho. /s ;)
    - rpadovani 4y ago
      But you don't have to make assumptions! I think this is the important bit: you are supposed to ask questions back until you are confident enough about your answer. You should highlight that you are able to work with customers that want something without knowing what they want.
    - tharkun__ 4y ago
      Like my sibling reply says, I think you didn't read past my first word in the reply. The "why". That was rhetorical. My entire point was basically that it actually isn't about assumptions at all. If you just assume when I ask that question this tells me you're out as a developer I want to hire. Nobody can build proper software by just assuming the first thing that comes to mind and bullshitting. You have to show that you can take a totally ambiguous and open question and show me that you can systematically try to find out as much information as you can in a reasonable amount of time to verify your assumptions or at least make them less assuming. Of course (a d this can happen when solving a real problem too) it might be that you cannot accurately measure some input you need and you will have to make assumptions. But you will want to try and you will want to have divided your problem into lots of smaller parts such that the part where you still have to assume is only one small part and you want to be very explicit that this part is still an assumption only which you will have to verify or improve upon as you actually implement something. You do have to start at some point, otherwise it's analysis paralysis. And no this has nothing to do with a PM. They are notoriously bad at the above approach and instead will just assume the first best thing and bullshit their way through with that for as long as they don't get caught (no /s there but my experience with 80+% of PMs)
latchkey 4y ago
> estimate how much disk was needed to store Google's search index 1 googol!
nixgeek 4y ago
You might be off by an additional order or perhaps two!

ajb 4y ago

You can't actually put the Android source in GitHub because of the 4GB per repo size limit. Niche problem, but shows the scale of things.

ajb 4y ago
As people have pointed out, there's not a hard 4GB limit. They do recommend less than 5GB.
- naikrovek 4y ago
  paying customers and popular open source repos can go well beyond 5GB. GHEC customers have a 5GB limit per file in git-lfs.
saagarjha 4y ago
You could if someone contacted GitHub to have it mirrored.
- naikrovek 4y ago
  it's not a real limit.
est31 4y ago
WDYM by 4GB size limit? When I run du -h on my linux checkout, I get 6.4G (no build artifacts, just git tracked source code plus the .git directory). Even if I only look at the .git/objects directory, it's 5.2G large.
- deleted 4y ago
  [deleted]
- londons_explore 4y ago
  It seems GitHub have whitelisted git projects containing certain commits or files to go over the size limit. Chromium is also well over the size limit.