4 ms·
5M items sounds suspiciously like one machine has to keep in RAM the complete list of a users files for certain operations. User accounts with large numbers
by londons_explore 4y ago
5M items sounds suspiciously like one machine has to keep in RAM the complete list of a users files for certain operations. User accounts with large numbers of objects were probably causing those machines to OOM. 5M items, at about 1000 bytes per item (name, metadata, a few uuids, etc) is 5Gbytes. 5Gbytes is about the amount of RAM most tasks will be given.
I could imagine that some services within Google haven't been carefully designed - for example, perhaps the quota service reads a complete file listing into RAM, adds up the sizes, and then writes the available quota.
This is probably the easy fix, rather than redesigning every service to be able to stream objects correctly.
- londons_explore 4y agoIt could also be a sharding/checkpointing issue. Imagine you want to scan every file in every user account, for example for blank documents (maybe because you want to downrank blank 'untitled documents' in search results, because they are easily accidentally made). You write your scanner to divide up the list of user accounts into chunks, process all chunks in different worker machines, and combine the results. Simple mapreduce. However, if there is one huge user account, then you either have to wait for the entire process to take far longer, or you need to have multiple workers working on the single account (adding a lot of complexity to every operation you wish to run across all accounts).
- cabirum 4y agoMore like an arbitrary number from the old times "cause 5M items ought to be enough for everybody."
- charcircuit 4y ago>from the old times That doesn't make sense because this is a new restriction.
- 8organicbits 4y agoI think the parent is referring to the 42 year old meme quote "640K ought to be enough for anybody". https://www.computerworld.com/article/2534312/the--640k--quote-won-t-go-away----but-did-gates-really-say-it-.html https://www.computerworld.com/article/2534312/the--640k--quo...
- mike503 4y agoProbably a reference to Bill Gates saying "640K ought to be enough for anybody"
- genewitch 4y agoCouldn't one programmatically test this by hitting 5GB sooner with excessively long filenames? Say 500 byte names?