3 ms·
> Huh. With the context of the rest of the comment, I realize (the very obvious comparison) that a database engine designed to shard to many thousands of small
by sam0x17 5y ago
> Huh. With the context of the rest of the comment, I realize (the very obvious comparison) that a database engine designed to shard to many thousands of small workers could potentially be a very attractive future development path.
Yes. I've been patiently waiting for the database community to realize this for the last 5 years now :)
- exikyut 5y agoI have no delusions I'd be able to viably make a dent in the area (at least anytime soon), but I do wonder how this would actually work. The optimal solution would of course be to shard both the compute I/O and the storage footprint, so each worker only needed to hold onto maybe 1-100MB of data. Perhaps some existing (simpler) designs could be modified to "hyper-shard" the compute angle, but would still likely require carrying around a large percentage of the database. In any case, you'd need an internal signalling fabric capable of (cost-effectively) handling very bursty many-to-many I/O across thousands of endpoints to make consensus work in realtime. It would honestly be really interesting to see how something like this would work in practice.