4 ms·
Distributed data structures is an interesting idea, but I’m not sure this particular implementation is the right abstraction level for many problems. These sor
by slaymaker1907 8y ago
Distributed data structures is an interesting idea, but I’m not sure this particular implementation is the right abstraction level for many problems.
These sorts of things form a spectrum from low level message passing all the way to a full fledged database. Some things in between are abstractions like MapReduce which still gives a lot of control, but also gives you fault tolerance and lower configuration of your cluster to run different applications.
Another abstraction in this space that is just a bit higher than ordinary message passing are systems like Kafka and RabbitMQ which give additional guarantees over plain message passing using sockets as well as less configuration to set up/remove machines.
One thing a lot of the more successful abstractions in this space seem to have is a pretty clear line between which parts of the system are local and which are distributed. It seems like this system doesn’t have as clear of a demarcation which would make understanding performance much more difficult. Databases can also have this problem since they do a lot of low level performance tuning automatically, but at least they provide a very high level of abstraction for applications to work with. This seems like it could end up being confusing to tune without providing a nice high-level interface like databases.
- xiii1408 8y agoThis post is really just about an abstraction for storing objects and doesn't explain how you'd make data structures that distribute objects across nodes using remote pointers. Remote pointers end up being pretty nice for locality, since you can explicitly see what process a remote pointer is pointing to. None of the backends that my remote pointer implementations depend on really offer fault tolerance, though. So you're right: if you're in a situation where you have nodes entering or leaving your cluster before your program ends, this model needs extending before it works for you. That's true for MPI-style programs, generally, though.