6 ms·

This paper is very light on details. It defines "RInK architecture" as something which uses stateless application servers w/ key-value store as backend (Redis/M

This paper is very light on details. It defines "RInK architecture" as something which uses stateless application servers w/ key-value store as backend (Redis/Memcached). Section 3 then shows that it's faster to use a stateful application server, but skips any details about scaling/durability which makes the comparison kinda strange. Are they really just comparing a single-threaded in-memory application server with a scalable system of stateless application servers? Their solution (LInK) is a key-value system "as a library": You link it into your application servers and you can then store data (they become stateful). The system will automatically handle sharding, replication, consistency and so on. From your perspective you're working directly on memory. You still need to implement marshalling, but this is only used for sharding/replication across nodes (which the system handles completely for you). They say nothing about how this system handles data consistency/replication/durability, so it's a bit tricky to assess it. They also have this nice note: "(since, for performance, persistence of state is not guaranteed)". Summary: This is a system which couples your application logic with your persistence layer (in the same OS process). This can give you great performance as they co-side with each other. Looks like a very cool project, and I'm actually surprised there not more "database as a library"-projects out there.

liveoneggs 7y ago
SQLite (:memory:), lmdb, berkeleydb, memcached-over-unix-socket, APCu (php), EHCache (java).. I'm sure there are others; I don't get what this paper presents that is new
- judofyr 7y ago
  Oh, sorry, I was imprecise: I meant "distributed database as a library". As in, you compile/bundle your application logic together with the persistence/replication/transaction logic and that is what you're running in a cluster.
  - liveoneggs 7y ago
    EHCache has tons of capabilities and is bundled directly into your app('s war file)
  - ryanworl 7y ago
    This is do-able with FoundationDB. You could distribute your application colocated with the database nodes and write a router to redirect requests to the machine which owns the data you're looking for. Then your application logic is computing against data on the same machine. FoundationDB exposes the key-range to node mapping to you to make this router possible.
  - s_kilk 7y ago
    The BeamVM languages (Erlang, Elixir) seem to have this in the form of Mnesia, and libraries that are built on top of it, such as riak-core.
    - StreamBright 7y ago
      Riak uses Mnesia?
      - ergl 7y ago
        Not that I know, but it uses ETS to handle state, and the global process registry to store peer information, as far as I know. Also, be aware that OP mentioned riak-core, which is a different thing from Riak. riak-core is a library to build dynamo-style distributed applications, which gives you virtual nodes, consistent hashing and data migration as building blocks.
      - elcritch 7y ago
        Riak KV uses leveldb, but as @ergl mentioned, riak-core is essentially a hash-ring and replication (aka dynamo) layer.
    - dillonmckay 7y ago
      Ha! I was just about to ask about that. ‘I bet Elixir already has something like this’.
  - stingraycharles 7y ago
    Datomic is a distributed database written in Clojure, and is best described as "cqrs-as-a-service". It allows you to run a "peer" inside your local Java VM, which essentially does exactly this: it caches the parts your local vm needs, and if not it looks elsewhere. https://docs.datomic.com/on-prem/peer-server.html https://docs.datomic.com/on-prem/peer-server.html
- mey 7y ago
  Apache Geode, Hazelcast, how some people are using Kafka. I am just listing a few others, please don't consider this endorsement.
maxmcd 7y ago
https://www.usenix.org/system/files/conference/osdi16/osdi16-adya.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16...
perlgeek 7y ago
> I'm actually surprised there not more "database as a library"-projects out there. For distributed systems, you tend to need quite a bit of configuration (how to find the peers, authentication, things like namespaces). And then if the communication is TCP-based, you tend to keep connections open to avoid the handshake latency when you need to retrieve a value. Which sounds like something you typically don't want running inside a library, but rather as a separate process. And then you've basically arrived at the memcached design.
continuations 7y ago
> Summary: This is a system which couples your application logic with your persistence layer (in the same OS process). I don't think the paper is talking about the persistence layer. Rather it's focusing on the cache layer. From the paper: "Modern internet-scale services often rely on remote, in-memory, key-value (RInK) stores such as Redis and Memcached. These stores serve at least two purposes. First, they may provide a cache over a storage system to enable faster retrieval of persistent state. Second, they may store short-lived data, such as per-session state, that does not warrant persistence." If you look at Fig.2 in the paper, the database (persistence) layer is always there. What the paper is advocating is to move from an out-of-process cache server like Redis to an in-process cache library like Java Caffeine
- zaroth 7y ago
  So for the canonical use case of session state in a load balanced server farm, is the session state being replicated across every single one of these in-process caches? Or, if a request is routed to a server without a copy of the session state, does the in-process cache have to go find a peer server with a copy of the session state? This seems like an easy “solution” to show how it speeds up the happy path while not fully addressing the real-world tradeoffs. Stateless application servers come and go without a care in the world. You can reason very simply about the cost and effects of bringing them up and down. Add in a KV store with sharding, replication, a discovery protocol, a heartbeat protocol, a sync and recovery protocol, etc... and put it all in-process on every application server? Have fun monitoring and debugging this. I much prefer the 3-tier system of a local RAM cache, a Redis cache, and a persistence layer. As much state as possible is idempotent so you can cache it locally as well as in Redis. The load balancer can best-effort route back to the same app server, and your KV lookups automatically check local RAM first. But the central KV store handles replication and is easy to monitor and scale out if needed.
  - lallysingh 7y ago
    How's in-process issuing the same library much different than using a separate process? What's complicated there in monitoring/debugging that's different that's not otherwise? I think that debugging with one less process is a good thing. And monitoring with one less process boundary to traverse is easier as well.
PopeDotNinja 7y ago
On the topic of pruning logic more complex than put & ge in the application, to me that is kind of like want to add iterable functions to a language that only has for loops. That was enough for me to appreciate the paper. I was recently working on adding some query logic to an app that talks to Redis, and it definitely felt like more work than I cared for. At a minimum I would have liked to see that logic go into a more feature rich Redis client
jandrewrogers 7y ago
Apparently I have designed a few production "LInK" systems over the years (now that someone else has named it), with relatively sophisticated algorithms for load balancing and replication. These architectures are great for fast-twitch mixed workload use cases, rapid mutation rates concurrent with relatively light, low-latency analytic queries. Think operational real-time data models. Conceptually, these are similar to the simple database kernels used as caches inside BI systems like Tableau but extended to support write workloads and scale-out. These may have a large amount of storage in them but only to increase the quantity of data each server can handle (i.e. swap space). Storage is not for durability, other systems handle that. The primary advantages of this architecture is it dramatically reduces data motion and wasteful computation, and you can often push complex data model logic to this layer since it implicitly contains more application context. All "databases as a library" (e.g. SQLite) are designed around a much simpler set of constraints. The combination of high write rates and continuous background resharding/rebalancing across servers requires low-level architectural support that you can't graft onto a database kernel that was not explicitly designed for it. To start with, any query execution has to have sophisticated topology awareness in a system where the topology is constantly changing due to resharding/rebalancing and you can't rely on global consensus for everything. The library you are linking into your application contains all the clever algorithms for navigating a data cache that behaves like this, you don't want the app developer to have to reason about it. The level of dynamic data model "fluidity" in the underlying organization and algorithms means that a mature design has little in common with traditional in-memory databases even though there are many superficial similarities. This isn't something you grind out over the weekend, there is a lot complex and non-obvious theory behind robust designs. (I agree that the paper is not very clear about what they are talking about. This is my interpretation of the paper based on my recognition of the claims and architecture.)
- hungnmsu 7y ago
  Do you have any reference for your load balancing and replication algorithms?
zbentley 7y ago
> I'm actually surprised there not more "database as a library"-projects out there Sqlite?
btown 7y ago
So the crux of this paper is "if you have perfect auto-sharding as a service, it's better to cut out a middleman and have it shard stateful app servers, rather than just sharding a KV store layer." In many ways, this paper is an extension of the 2016 paper it references, which introduces their internal sharding-as-a-service, Slicer: https://www.usenix.org/system/files/conference/osdi16/osdi16-adya.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16... Once you separate out Slicer, this result is pretty obvious in theory - in fact, in that Slicer paper they mention machine learning as a driving example, where it absolutely makes sense to colocate application logic with user-specific models cached in memory. Honestly, this paper, with its one-paragraph implementation section, reads almost like clickbait - "let's say something controversial, release an un-reproducable paper that adds very little to the literature besides 'this is something cool you can do if you have a tool only Google has,' btw do you want to work for Google?" The Slicer paper itself has much more detail and IMO is a much better read.

liveoneggs 7y ago

SQLite (:memory:), lmdb, berkeleydb, memcached-over-unix-socket, APCu (php), EHCache (java).. I'm sure there are others; I don't get what this paper presents that is new

judofyr 7y ago
Oh, sorry, I was imprecise: I meant "distributed database as a library". As in, you compile/bundle your application logic together with the persistence/replication/transaction logic and that is what you're running in a cluster.
- liveoneggs 7y ago
  EHCache has tons of capabilities and is bundled directly into your app('s war file)
- ryanworl 7y ago
  This is do-able with FoundationDB. You could distribute your application colocated with the database nodes and write a router to redirect requests to the machine which owns the data you're looking for. Then your application logic is computing against data on the same machine. FoundationDB exposes the key-range to node mapping to you to make this router possible.
- s_kilk 7y ago
  The BeamVM languages (Erlang, Elixir) seem to have this in the form of Mnesia, and libraries that are built on top of it, such as riak-core.
  - StreamBright 7y ago
    Riak uses Mnesia?
    - ergl 7y ago
      Not that I know, but it uses ETS to handle state, and the global process registry to store peer information, as far as I know. Also, be aware that OP mentioned riak-core, which is a different thing from Riak. riak-core is a library to build dynamo-style distributed applications, which gives you virtual nodes, consistent hashing and data migration as building blocks.
    - elcritch 7y ago
      Riak KV uses leveldb, but as @ergl mentioned, riak-core is essentially a hash-ring and replication (aka dynamo) layer.
  - dillonmckay 7y ago
    Ha! I was just about to ask about that. ‘I bet Elixir already has something like this’.
- stingraycharles 7y ago
  Datomic is a distributed database written in Clojure, and is best described as "cqrs-as-a-service". It allows you to run a "peer" inside your local Java VM, which essentially does exactly this: it caches the parts your local vm needs, and if not it looks elsewhere. https://docs.datomic.com/on-prem/peer-server.html https://docs.datomic.com/on-prem/peer-server.html
mey 7y ago
Apache Geode, Hazelcast, how some people are using Kafka. I am just listing a few others, please don't consider this endorsement.

maxmcd 7y ago

https://www.usenix.org/system/files/conference/osdi16/osdi16-adya.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16...

perlgeek 7y ago

> I'm actually surprised there not more "database as a library"-projects out there. For distributed systems, you tend to need quite a bit of configuration (how to find the peers, authentication, things like namespaces). And then if the communication is TCP-based, you tend to keep connections open to avoid the handshake latency when you need to retrieve a value. Which sounds like something you typically don't want running inside a library, but rather as a separate process. And then you've basically arrived at the memcached design.

continuations 7y ago

> Summary: This is a system which couples your application logic with your persistence layer (in the same OS process). I don't think the paper is talking about the persistence layer. Rather it's focusing on the cache layer. From the paper: "Modern internet-scale services often rely on remote, in-memory, key-value (RInK) stores such as Redis and Memcached. These stores serve at least two purposes. First, they may provide a cache over a storage system to enable faster retrieval of persistent state. Second, they may store short-lived data, such as per-session state, that does not warrant persistence." If you look at Fig.2 in the paper, the database (persistence) layer is always there. What the paper is advocating is to move from an out-of-process cache server like Redis to an in-process cache library like Java Caffeine

zaroth 7y ago
So for the canonical use case of session state in a load balanced server farm, is the session state being replicated across every single one of these in-process caches? Or, if a request is routed to a server without a copy of the session state, does the in-process cache have to go find a peer server with a copy of the session state? This seems like an easy “solution” to show how it speeds up the happy path while not fully addressing the real-world tradeoffs. Stateless application servers come and go without a care in the world. You can reason very simply about the cost and effects of bringing them up and down. Add in a KV store with sharding, replication, a discovery protocol, a heartbeat protocol, a sync and recovery protocol, etc... and put it all in-process on every application server? Have fun monitoring and debugging this. I much prefer the 3-tier system of a local RAM cache, a Redis cache, and a persistence layer. As much state as possible is idempotent so you can cache it locally as well as in Redis. The load balancer can best-effort route back to the same app server, and your KV lookups automatically check local RAM first. But the central KV store handles replication and is easy to monitor and scale out if needed.
- lallysingh 7y ago
  How's in-process issuing the same library much different than using a separate process? What's complicated there in monitoring/debugging that's different that's not otherwise? I think that debugging with one less process is a good thing. And monitoring with one less process boundary to traverse is easier as well.

PopeDotNinja 7y ago

On the topic of pruning logic more complex than put & ge in the application, to me that is kind of like want to add iterable functions to a language that only has for loops. That was enough for me to appreciate the paper. I was recently working on adding some query logic to an app that talks to Redis, and it definitely felt like more work than I cared for. At a minimum I would have liked to see that logic go into a more feature rich Redis client

jandrewrogers 7y ago

Apparently I have designed a few production "LInK" systems over the years (now that someone else has named it), with relatively sophisticated algorithms for load balancing and replication. These architectures are great for fast-twitch mixed workload use cases, rapid mutation rates concurrent with relatively light, low-latency analytic queries. Think operational real-time data models. Conceptually, these are similar to the simple database kernels used as caches inside BI systems like Tableau but extended to support write workloads and scale-out. These may have a large amount of storage in them but only to increase the quantity of data each server can handle (i.e. swap space). Storage is not for durability, other systems handle that. The primary advantages of this architecture is it dramatically reduces data motion and wasteful computation, and you can often push complex data model logic to this layer since it implicitly contains more application context. All "databases as a library" (e.g. SQLite) are designed around a much simpler set of constraints. The combination of high write rates and continuous background resharding/rebalancing across servers requires low-level architectural support that you can't graft onto a database kernel that was not explicitly designed for it. To start with, any query execution has to have sophisticated topology awareness in a system where the topology is constantly changing due to resharding/rebalancing and you can't rely on global consensus for everything. The library you are linking into your application contains all the clever algorithms for navigating a data cache that behaves like this, you don't want the app developer to have to reason about it. The level of dynamic data model "fluidity" in the underlying organization and algorithms means that a mature design has little in common with traditional in-memory databases even though there are many superficial similarities. This isn't something you grind out over the weekend, there is a lot complex and non-obvious theory behind robust designs. (I agree that the paper is not very clear about what they are talking about. This is my interpretation of the paper based on my recognition of the claims and architecture.)

hungnmsu 7y ago
Do you have any reference for your load balancing and replication algorithms?

zbentley 7y ago

> I'm actually surprised there not more "database as a library"-projects out there Sqlite?

btown 7y ago

So the crux of this paper is "if you have perfect auto-sharding as a service, it's better to cut out a middleman and have it shard stateful app servers, rather than just sharding a KV store layer." In many ways, this paper is an extension of the 2016 paper it references, which introduces their internal sharding-as-a-service, Slicer: https://www.usenix.org/system/files/conference/osdi16/osdi16-adya.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16... Once you separate out Slicer, this result is pretty obvious in theory - in fact, in that Slicer paper they mention machine learning as a driving example, where it absolutely makes sense to colocate application logic with user-specific models cached in memory. Honestly, this paper, with its one-paragraph implementation section, reads almost like clickbait - "let's say something controversial, release an un-reproducable paper that adds very little to the literature besides 'this is something cool you can do if you have a tool only Google has,' btw do you want to work for Google?" The Slicer paper itself has much more detail and IMO is a much better read.