4 ms·
Odd. It's surprising to me that this example runs out of memory. What would be a possible solution? Admittedly I don't know that much about the inner-workings
by warent 9y ago
Odd. It's surprising to me that this example runs out of memory. What would be a possible solution?
Admittedly I don't know that much about the inner-workings of git, but off the top of my head, perhaps something with traversing the tree depth-first and releasing resources as you hit the bottom?
- peff 9y agoGit assumes it can keep a small struct in memory for each file in the repository (not the file contents, but a fixed per-file size). This repository just has a very large number of files.
- glandium 9y agoLarge as in 10 billions. Even if git only needed 1 byte in memory per file, it would need 10GB.
- ericfrederich 9y agoYou need a problem to have a solution to it. What do you consider to be the problem here? This is essentially something that can be expressed in relatively few bytes that expands to something much larger. Imagine I had a compressed file format for blank files "0x00" the whole way. It is implemented by writing in ascii the size of the uncompressed file. So the contents of a file called terrabyte.blank is just ascii "1000000000000" ... or the contents of a file called petabyte.blank is "10000000000000" I cannot decompress these files... what is the solution?
- warent 9y agoI'm not following; why can't you decompress it? Of course you cant decompress it into memory, but if it's trying to that then there's a problem in the code (problem identified). Naive solution, just write to the end of the file and make sure you have enough disk. More sophisticated solution, shard the file across multiple disks.
- Piskvorrr 9y agoThat's not a solution, that's sweeping the problem under the rug: "just have the OS provide storage, therefore it's not my problem any more, solved. (Never mind that with a few more layers, the tree would decompress into a structure larger than all the storage ever available to mankind)"
- geezerjay 9y ago>You need a problem to have a solution to it. What do you consider to be the problem here? > >This is essentially something that can be expressed in relatively few bytes that expands to something much larger. That seems to be the problem. I mean, if an object expands to something much larger to the point that it crashes services just by the sheer volume of the resources it takes... That is pretty much the definition of an attack vector of a denial-of-service attack.
- TeMPOraL 9y agoThere is a problem here, but it's not with data. It's with the service. Being able to express trees efficiently in a data format is an useful feature, but it requires the code processing it not to be lazy and assume people will never create pathological tree structures.
- koenigdavidmj 9y agoOne option is to modify each of the utilities so that it doesn't have a full representation of the whole tree in memory. I doubt this is feasible in all cases, though for something like 'git status' it should be doable. If the tree object format was required to store its own path, then you wouldn't be able to repeat the tree a bunch of times. The in-memory representation would be the same size, but you would now need that same number of objects in the repository. No more exponential fanout. But that would kind of defeat the purpose of Git for real use cases (renaming a directory shouldn't make the size of your repo blow up).
- TeMPOraL 9y agoHave git (the client) monitor its own memory usage and abort if it gets above a set limit (say, default, 1GB), with a message that tells you how to change or disable the limit.