3 ms·

At my company, we're actually doing that as well, but we took it further: no tiers at all. We do load-balancing at the client, every server has a public IP add

At my company, we're actually doing that as well, but we took it further: no tiers at all. We do load-balancing at the client, every server has a public IP address, we use DNS round-robin only to distribute the login phase; after that, it's location-based with re-connect logic in the app. We don't store the entire database on every node, but we do the Cassandra-style equivalent: each datacenter has all of the data. We can store all of the data in a single datacenter. We never have to worry if the "database" is down, because the app server is also down whenever it is. We even have global visibility on all machines in a Cassandra-like design -- something that took Google's Spanner team custom GPS and atomic timekeeping hardware to achieve. We're doing it with a lightweight consensus algorithm that achieves a reliable commit timestamp globally, and NTP. We can take a global snapshot, or return a globally-consistent query across all data at any time. I think we'll see a lot of architectures built like this in the (near) future. Collapsing the tiers is great for maintainability and performance. ---- FWIW I found http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html to be...lacking. He ignores all problems related to multi-update completely. Nathan's solution does nothing to solve consistency problems except in the trivial situation where every piece of data can be "updated" (really appended) completely in isolation from any other update. You actually do need multi-data consistency somewhere to write virtually all non-trivial apps, and both Spanner (and the architecture we use) deal with that in a sane way, that developers can actually use. Nathan is definitely on the path towards how to implement such a beast; hopefully his book will have a fuller implementation of where he's heading.

lifeisstillgood 14y ago
Would you mind taking a little time to explain the architecture again - this clearly is a new cut of dealing with the distributed problem and I am not clear how it fits together afaik you have 3+ data centers, and need to ensure consistency between them. They agree on an exact time (hmmm how - Ntp offsets where each data center acts as stratum two for the others, eventually you see each others drift???) anyway, exact time agreed, timestamps on every transaction, labelled uuids and then send them as updates to other centers. Is thatroughly it? Or am I way off how do you deal with collisions - if I sell the same room twice in two data centers who loses, how is the business prcess handled? Thx
- erichocean 14y ago
  Sure. afaik you have 3+ data centers, and need to ensure consistency between them. They agree on an exact time (hmmm how - Ntp offsets where each data center acts as stratum two for the others, eventually you see each others drift???) anyway, exact time agreed, timestamps on every transaction, labelled uuids and then send them as updates to other centers. Is thatroughly it? Or am I way off The servers do not agree on exact time, but they do agree on the timestamp for a particular update, using a lightweight consensus protocol. So, here's the properties that actually matter: 1. A client is connected to a node in a single datacenter. 2. Multiple clients are connected to multiple nodes across datacenters. 3. At timestamp T, all clients should be able to read the same data from every datacenter if it can read from any datacenter (this is what is meant by "global consistency"). 4. It should always be possible to read at time T even if it is actually some time T' in the future. (This is what's needed to support snapshots.) We label an arbitrary collection of records as an "entity". All records for an entity can be stored on a single machine, and are updated atomically (transactionally). On average, an entity should be < 1MB in size, but we currently allow them to get as large as 64MB if needed. (We serialize updates to the same entity in the same datacenter; in practice, we could use Paxos here (like Google), but our particular application doesn't need that complexity.) Entities are identified by a UUID, and a particular commit id refers to a particular version of an entity. (We do append updates.) The commit id is a git-style hash of the contents and the parent commit(s). This facilitates both auditing and merging (see below). I'll cover how that commit gets a timestamp and thus becomes globally visible as part of the next question. how do you deal with collisions - if I sell the same room twice in two data centers who loses, how is the business prcess handled? The same entity can be updated at multiple datacenters containing it at the "same" time. If these updates are identical, they result in the same commit id (remember, it's a hash); otherwise, the commits will be different. (Unlike git, we DO NOT hash a timestamp with the commit.) When committed, initially there is no timestamp. The commit is "in the system", but it has not yet become visible, except (conditionally) on the datacenter where it was just written to. It's not globally visible at this point, and we don't let any other client see it. Next, the datacenter "votes" for the commit by associating it with a timestamp, and also forwards the commit/vote/timestamp to another datacenter. That datacenter receives the commit, and if it has no other, conflicting commit, votes with its timestamp, and then forwards it on to the next data center, recursively, until it has reached every datacenter. A commit in our system becomes globally visible precisely when all datacenters have voted. The commit is made globally visible at the latest vote/timestamp. At this point, we are guaranteed two things: 1. All datacenter have the commit. 2. They all have voted there is no conflict, and also provided a timestamp. So what about the case of a conflict? When that occurs, by definition, one of the datacenters will not vote. Instead, what happens is our version of "read repair", which really, is just a merge between two commits (think: git). The merged commit and its parent commits are sent on to the next datacenter, and also given a vote/timestamp. (Our merge procedure is a pure function, and can be customized per entity.) The only remaining details are as follows: 1. How does each datacenter know when it has received all votes? 2. How do we handle partions, which by definition, prevent voting? We handle 1. with a global datacenter topology, that is distributed as an entity, so it too becomes globally visible atomically. We handle 2. by arranging our datacenters in a tree topology. When a partition occurs, we disable branches lower in a tree, and we empower the parent node to "vote" for the disabled node. In practice, thanks to this topology, we don't actually need to send around all of the votes; we can actually just forward the timestamp of the parent node to its parent, recursively. NTP works for all of this because they all agree on the timestamp, since it's part of the vote. Each datacenter has the invariant that each new vote timestamp is monotonically the same or increasing up the datacenter tree, so we'll never have a situation where an "old" commit becomes visible after a new commit. A datacenter can only vote (for a particular dataset) with a timestamp greater than its previous vote. At the business process level, we basically don't react to a commit until it's globally visible. That's when consensus is achieved and it's safe to move forward with a workflow, such as updating other entities. Anyway, hope that helps!
cwp 14y ago
We do this at instandomainsearch.com too. Our domain database is relatively small and changes slowly, so we do daily updates and distribute snapshots to all the servers. We have copies in 4 different AWS regions, and all the load-balancing and failover is done in client-side Javascript. One server in each region is plenty for our current traffic levels, but if we need to, scaling horizontally is trivial. More important for us is that the in-browser load-balancing is based on measured latency, which ensures that each user gets the snappiest interface that we can provide. All in all, it works really well.
- cvillecsteele 14y ago
  At Room Key we're using Amazon's DNS solution for load balancing (lowest latency).
  - cwp 14y ago
    When Amazon came out with DNS-based load-balancing, we thought about switching, but it'd be a downgrade from what we've got already. We direct traffic to the server that's showing the lowest latency right now and switch to another server if the measured latency changes. Amazon's latency data will be much slower to adapt to network conditions, and with DNS caching, it wouldn't provide fail-over if the lowest-latency server happened to go down or become unreachable. Amazon's scheme is pretty nifty though, and we'd definitely use it if we didn't already have a more dynamic solution already in place.
chubot 14y ago
I don't think your solution is comparable to Spanner, because you said somewhere you have to write custom merge functions for data. Spanner doesn't impose that on application developers. Your system sounds really interesting though, thanks for writing it up. For most people, the hardware requirement of a Spanner-like solution does seem impractical, so maybe that's a good tradeoff. I'm curious what industry this is in? Or what company? 17ms is really low! I wish more sites would put that much effort into their latency.

lifeisstillgood 14y ago

Would you mind taking a little time to explain the architecture again - this clearly is a new cut of dealing with the distributed problem and I am not clear how it fits together afaik you have 3+ data centers, and need to ensure consistency between them. They agree on an exact time (hmmm how - Ntp offsets where each data center acts as stratum two for the others, eventually you see each others drift???) anyway, exact time agreed, timestamps on every transaction, labelled uuids and then send them as updates to other centers. Is thatroughly it? Or am I way off how do you deal with collisions - if I sell the same room twice in two data centers who loses, how is the business prcess handled? Thx

erichocean 14y ago
Sure. afaik you have 3+ data centers, and need to ensure consistency between them. They agree on an exact time (hmmm how - Ntp offsets where each data center acts as stratum two for the others, eventually you see each others drift???) anyway, exact time agreed, timestamps on every transaction, labelled uuids and then send them as updates to other centers. Is thatroughly it? Or am I way off The servers do not agree on exact time, but they do agree on the timestamp for a particular update, using a lightweight consensus protocol. So, here's the properties that actually matter: 1. A client is connected to a node in a single datacenter. 2. Multiple clients are connected to multiple nodes across datacenters. 3. At timestamp T, all clients should be able to read the same data from every datacenter if it can read from any datacenter (this is what is meant by "global consistency"). 4. It should always be possible to read at time T even if it is actually some time T' in the future. (This is what's needed to support snapshots.) We label an arbitrary collection of records as an "entity". All records for an entity can be stored on a single machine, and are updated atomically (transactionally). On average, an entity should be < 1MB in size, but we currently allow them to get as large as 64MB if needed. (We serialize updates to the same entity in the same datacenter; in practice, we could use Paxos here (like Google), but our particular application doesn't need that complexity.) Entities are identified by a UUID, and a particular commit id refers to a particular version of an entity. (We do append updates.) The commit id is a git-style hash of the contents and the parent commit(s). This facilitates both auditing and merging (see below). I'll cover how that commit gets a timestamp and thus becomes globally visible as part of the next question. how do you deal with collisions - if I sell the same room twice in two data centers who loses, how is the business prcess handled? The same entity can be updated at multiple datacenters containing it at the "same" time. If these updates are identical, they result in the same commit id (remember, it's a hash); otherwise, the commits will be different. (Unlike git, we DO NOT hash a timestamp with the commit.) When committed, initially there is no timestamp. The commit is "in the system", but it has not yet become visible, except (conditionally) on the datacenter where it was just written to. It's not globally visible at this point, and we don't let any other client see it. Next, the datacenter "votes" for the commit by associating it with a timestamp, and also forwards the commit/vote/timestamp to another datacenter. That datacenter receives the commit, and if it has no other, conflicting commit, votes with its timestamp, and then forwards it on to the next data center, recursively, until it has reached every datacenter. A commit in our system becomes globally visible precisely when all datacenters have voted. The commit is made globally visible at the latest vote/timestamp. At this point, we are guaranteed two things: 1. All datacenter have the commit. 2. They all have voted there is no conflict, and also provided a timestamp. So what about the case of a conflict? When that occurs, by definition, one of the datacenters will not vote. Instead, what happens is our version of "read repair", which really, is just a merge between two commits (think: git). The merged commit and its parent commits are sent on to the next datacenter, and also given a vote/timestamp. (Our merge procedure is a pure function, and can be customized per entity.) The only remaining details are as follows: 1. How does each datacenter know when it has received all votes? 2. How do we handle partions, which by definition, prevent voting? We handle 1. with a global datacenter topology, that is distributed as an entity, so it too becomes globally visible atomically. We handle 2. by arranging our datacenters in a tree topology. When a partition occurs, we disable branches lower in a tree, and we empower the parent node to "vote" for the disabled node. In practice, thanks to this topology, we don't actually need to send around all of the votes; we can actually just forward the timestamp of the parent node to its parent, recursively. NTP works for all of this because they all agree on the timestamp, since it's part of the vote. Each datacenter has the invariant that each new vote timestamp is monotonically the same or increasing up the datacenter tree, so we'll never have a situation where an "old" commit becomes visible after a new commit. A datacenter can only vote (for a particular dataset) with a timestamp greater than its previous vote. At the business process level, we basically don't react to a commit until it's globally visible. That's when consensus is achieved and it's safe to move forward with a workflow, such as updating other entities. Anyway, hope that helps!

cwp 14y ago

We do this at instandomainsearch.com too. Our domain database is relatively small and changes slowly, so we do daily updates and distribute snapshots to all the servers. We have copies in 4 different AWS regions, and all the load-balancing and failover is done in client-side Javascript. One server in each region is plenty for our current traffic levels, but if we need to, scaling horizontally is trivial. More important for us is that the in-browser load-balancing is based on measured latency, which ensures that each user gets the snappiest interface that we can provide. All in all, it works really well.

cvillecsteele 14y ago
At Room Key we're using Amazon's DNS solution for load balancing (lowest latency).
- cwp 14y ago
  When Amazon came out with DNS-based load-balancing, we thought about switching, but it'd be a downgrade from what we've got already. We direct traffic to the server that's showing the lowest latency right now and switch to another server if the measured latency changes. Amazon's latency data will be much slower to adapt to network conditions, and with DNS caching, it wouldn't provide fail-over if the lowest-latency server happened to go down or become unreachable. Amazon's scheme is pretty nifty though, and we'd definitely use it if we didn't already have a more dynamic solution already in place.

chubot 14y ago

I don't think your solution is comparable to Spanner, because you said somewhere you have to write custom merge functions for data. Spanner doesn't impose that on application developers. Your system sounds really interesting though, thanks for writing it up. For most people, the hardware requirement of a Spanner-like solution does seem impractical, so maybe that's a good tradeoff. I'm curious what industry this is in? Or what company? 17ms is really low! I wish more sites would put that much effort into their latency.