2 ms·
and these data structures are themselves able to span hundreds/thousands of disks? eg. what happens when a customer's bucket/object are too many such that the s
by arhyth 3y ago
and these data structures are themselves able to span hundreds/thousands of disks? eg. what happens when a customer's bucket/object are too many such that the skip list "indexing" them cannot fit in a single harddrive?
- gregjor 3y agodata structures that span physical media got solved decades ago. the disks or ssd get abstracted into virtual storage. See: https://en.wikipedia.org/wiki/Logical_disk https://en.wikipedia.org/wiki/Logical_disk This kind of logical media mapped to physical media goes back to magnetic tape, predating disks and solid state storage. Database systems solve this same problem. A database table may include millions of rows and indexes, too big to fit into RAM all at once. The index trees, for example, get loaded in pages. There’s no need to have the entire index in memory, only the relevant parts of it.