5 ms·

For incredible scale, and heretic ideas, I think people should consider the decisions that Colin Steele made as CTO of RoomKey. He was thinking about how to bui

For incredible scale, and heretic ideas, I think people should consider the decisions that Colin Steele made as CTO of RoomKey. He was thinking about how to build a database that could offer nearly real time searching and yet handle any level of traffic. The idea he hit upon was compiling his code with a static snapshot of his database. Think about this carefully: instead of lots of webservers putting strain on a central database, every web server has its own snapshot of the database. If traffic doubles, you can double the number of instances you have on Amazon, and you are doubling both the number of web servers you have, and also the number of databases you have. The team at RoomKey has built this app with Clojure, and stores the data in an instance of Solr that actually gets compiled with the code. With an automated build system, you could probably roll out new snapshots of the database several times a day, so the data is never very stale. This is how Colin Steele wrote about it on his blog: "Put another way, users of this system have a high tolerance for inconsistent reads. Bob’s and Jane’s hotel universes need not be identical. (They can’t be completely divergent; eventual consistency is fine.) So: A-ha! The messy relational data could live in a secluded back-end “content sausage factory”, whose sole purpose in life would be to produce a crisp, non-relational version of the hotel universe as known best at that point in time. This “golden master”, non-relational database of hotels could then be shipped off to the live, operational system which faces users. Moreover, different users might be exposed to different versions of the “golden master” hotel database, allowing us to test and to do progressive and continuous rollouts. Decision One: I put relational data on one side and “static”, non-relational data on the other, with a big wall of verification process between them. This led to Decision Two. Because the data set is small, we can “bake in” the entire content database into a version of our software. Yep, you read that right. We build our software with an embedded instance of Solr and we take the normalized, cleansed, non-relational database of hotel inventory, and jam that in as well, when we package up the application for deployment. Egads, Colin! That’s wrong! Data is data and code is code! We earn several benefits from this unorthodox choice. First, we eliminate a significant point of failure - a mismatch between code and data. Any version of software is absolutely, positively known to work, even fetched off of disk years later, regardless of what godawful changes have been made to our content database in the meantime. Deployment and configuration management for differing environments becomes trivial. Second, we achieve horizontal shared-nothing scalabilty in our user-facing layer. That’s kinda huge. Really huge. One of our crack developers and now our Front-End Development Manager drove Decision Three. Our user-facing servers render JSON. A Javascript application, bootstrapped by a static HTML page, creates the entire UI, rendering elements into the brower’s DOM as it interacts with our API. This so-called “fat client” or “single page Javascript app” has been steadily coming into vogue for the last few years, but it was far from an obvious choice at the time. We reap several benefits from this decision. First, our UI gets faster without us having to pay for new servers that take advantage of Moore’s Law. Second, our UI gets faster because browser makers are dumping tons of time an energy into improved JS performance. Third, HTML is verbose and JSON isn’t, saving us both compute cycles and bandwidth on the server side. These decisions en toto yield a picture of a three-layer cake, with a messy relational tier on the bottom, an operational API and SOA tier in the middle, and a fat client presentation layer on top." http://www.colinsteele.org/post/27929539434/60-000-growth-in-7-months-using-clojure-and-aws http://www.colinsteele.org/post/27929539434/60-000-growth-in...

erichocean 14y ago
At my company, we're actually doing that as well, but we took it further: no tiers at all. We do load-balancing at the client, every server has a public IP address, we use DNS round-robin only to distribute the login phase; after that, it's location-based with re-connect logic in the app. We don't store the entire database on every node, but we do the Cassandra-style equivalent: each datacenter has all of the data. We can store all of the data in a single datacenter. We never have to worry if the "database" is down, because the app server is also down whenever it is. We even have global visibility on all machines in a Cassandra-like design -- something that took Google's Spanner team custom GPS and atomic timekeeping hardware to achieve. We're doing it with a lightweight consensus algorithm that achieves a reliable commit timestamp globally, and NTP. We can take a global snapshot, or return a globally-consistent query across all data at any time. I think we'll see a lot of architectures built like this in the (near) future. Collapsing the tiers is great for maintainability and performance. ---- FWIW I found http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html to be...lacking. He ignores all problems related to multi-update completely. Nathan's solution does nothing to solve consistency problems except in the trivial situation where every piece of data can be "updated" (really appended) completely in isolation from any other update. You actually do need multi-data consistency somewhere to write virtually all non-trivial apps, and both Spanner (and the architecture we use) deal with that in a sane way, that developers can actually use. Nathan is definitely on the path towards how to implement such a beast; hopefully his book will have a fuller implementation of where he's heading.
- lifeisstillgood 14y ago
  Would you mind taking a little time to explain the architecture again - this clearly is a new cut of dealing with the distributed problem and I am not clear how it fits together afaik you have 3+ data centers, and need to ensure consistency between them. They agree on an exact time (hmmm how - Ntp offsets where each data center acts as stratum two for the others, eventually you see each others drift???) anyway, exact time agreed, timestamps on every transaction, labelled uuids and then send them as updates to other centers. Is thatroughly it? Or am I way off how do you deal with collisions - if I sell the same room twice in two data centers who loses, how is the business prcess handled? Thx

erichocean 14y ago

At my company, we're actually doing that as well, but we took it further: no tiers at all. We do load-balancing at the client, every server has a public IP address, we use DNS round-robin only to distribute the login phase; after that, it's location-based with re-connect logic in the app. We don't store the entire database on every node, but we do the Cassandra-style equivalent: each datacenter has all of the data. We can store all of the data in a single datacenter. We never have to worry if the "database" is down, because the app server is also down whenever it is. We even have global visibility on all machines in a Cassandra-like design -- something that took Google's Spanner team custom GPS and atomic timekeeping hardware to achieve. We're doing it with a lightweight consensus algorithm that achieves a reliable commit timestamp globally, and NTP. We can take a global snapshot, or return a globally-consistent query across all data at any time. I think we'll see a lot of architectures built like this in the (near) future. Collapsing the tiers is great for maintainability and performance. ---- FWIW I found http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html to be...lacking. He ignores all problems related to multi-update completely. Nathan's solution does nothing to solve consistency problems except in the trivial situation where every piece of data can be "updated" (really appended) completely in isolation from any other update. You actually do need multi-data consistency somewhere to write virtually all non-trivial apps, and both Spanner (and the architecture we use) deal with that in a sane way, that developers can actually use. Nathan is definitely on the path towards how to implement such a beast; hopefully his book will have a fuller implementation of where he's heading.

lifeisstillgood 14y ago
Would you mind taking a little time to explain the architecture again - this clearly is a new cut of dealing with the distributed problem and I am not clear how it fits together afaik you have 3+ data centers, and need to ensure consistency between them. They agree on an exact time (hmmm how - Ntp offsets where each data center acts as stratum two for the others, eventually you see each others drift???) anyway, exact time agreed, timestamps on every transaction, labelled uuids and then send them as updates to other centers. Is thatroughly it? Or am I way off how do you deal with collisions - if I sell the same room twice in two data centers who loses, how is the business prcess handled? Thx