33 ms·
Whenever you have that much data stored how do you actually know the data is still there and can be retrieved? Even if you have absolutely insane connectivity t
by sidewndr46 2y ago
Whenever you have that much data stored how do you actually know the data is still there and can be retrieved? Even if you have absolutely insane connectivity to it at some point don't you run out of time to check it? Apparent 200 PiB at 1 GiB per second would take about 58254 hours to retrieve.
- danielheath 2y agoIt's not like it's all coming from one disk, or going to one single CPU. 20TB drives with 500mb/s sequential read are available today. Reading the whole disk takes about half a day. If your storage pod has 12 of those, even a $50 n100 CPU can run xxHash at 6gb / s (could probably even manage MurmurHash).
- tap-snap-or-nap 2y agoI won't even pretend to know how to begin with this type of project.
- InDubioProRubio 2y agoCrawlers with jobs, building searchable indexes? Similar to youtube. Down at the source its blobs, but above it all floats a layers of tags, metrics and searchable text. That is what the searches run against and the preferences algo builds its lineup against?
- deleted 2y ago[deleted]