4 ms·
> 90% of our bill is database related currently. Huh. With the context of the rest of the comment, I realize (the very obvious comparison) that a database engi
by exikyut 5y ago
> 90% of our bill is database related currently.
Huh. With the context of the rest of the comment, I realize (the very obvious comparison) that a database engine designed to shard to many thousands of small workers could potentially be a very attractive future development path.
Iff the current trends in cloud computing (workers, lambda, etc) continues and some other fundamental doesn't come along and overtake.
Which is probably (part of) the reason why this doesn't exist, since I think I've basically just described the P=NP of storage engineering :)
- sam0x17 5y ago> Huh. With the context of the rest of the comment, I realize (the very obvious comparison) that a database engine designed to shard to many thousands of small workers could potentially be a very attractive future development path. Yes. I've been patiently waiting for the database community to realize this for the last 5 years now :)
- exikyut 5y agoI have no delusions I'd be able to viably make a dent in the area (at least anytime soon), but I do wonder how this would actually work. The optimal solution would of course be to shard both the compute I/O and the storage footprint, so each worker only needed to hold onto maybe 1-100MB of data. Perhaps some existing (simpler) designs could be modified to "hyper-shard" the compute angle, but would still likely require carrying around a large percentage of the database. In any case, you'd need an internal signalling fabric capable of (cost-effectively) handling very bursty many-to-many I/O across thousands of endpoints to make consensus work in realtime. It would honestly be really interesting to see how something like this would work in practice.