4 ms·

Interesting perspective. The article talks alot about big companies that make lots of money but as a user of databases I always saw the competition to ReThink

by hoodoof 10y ago

Interesting perspective. The article talks alot about big companies that make lots of money but as a user of databases I always saw the competition to ReThink DB being Postgres, MySQL and to a lesser extent ArangoDB and MongoDB. The core problem for Rethink being that, well, I can get as much database as I want for free. I explored many/most of the new/nosql databases and in the end always found that Postgres was the best solution - you just shouldn't bet against Postgres it seems. I really liked RethinkDB and thought it was well implemented but it just wasn't needed. I do wonder if ArangoDB is going to go the same way.

lobster_johnson 10y ago
Exactly this. RethinkDB's value proposition wasn't really that attractive compared to Postgres. Replication and a nice UI, sure, those are nice. But Postgres has fantastic single-host performance, and Rethink is not yet in a position to compete there, except in some horizontally scaled use cases. Rethink doesn't even have transactions, making it potentially useless for some things that require strict atomicity over a complicated data model. In general, Postgres has such a richness of features that it's hard to compete with it; PostGIS alone is a "killer app" if you do any kind of GIS stuff. For me, there weren't any such "killer app" features. The change streaming is nice, but it's not difficult to do that with Postgres either. That's it really: Every solution offered by Rethink could be countered with "that's nice, but trivial with Postgres". The one exception being horizontal replication. I have yet to launch a project where Postgres couldn't scale out well enough, and I suspect that if I got to that point, I would reach for bigger/different guns such as Cassandra. For our apps we typically use Elasticsearch in front as a kind of read-only mirror of Postgres; the latter being the slow, serializable path, and the former the fast, slightly-out-of-sync path. Users don't know the difference.
- zigzigzag 10y ago
  Firstly, to the OP, that is an excellent post-mortem. Introspective, goes deep, generally confidence inspiring even though a part of it is literally "we were incompetent and didn't know it", well, lessons learned. I'd like to add some colour, although I've never used RethinkDB so take it all with a large pile of salt. lobster_johnson says: > Rethink doesn't even have transactions The post-mortem dwells heavily on how RethinkDB focused on correctness and was beaten by MongoDB which is, to put it mildly, not concerned about correctness. But Rethink is a JSON DB that apparently doesn't have transactions and which on its web page says "Query JSON documents with Python, Ruby, Node.js or dozens of other languages". Here are three technologies I do not strongly associate with correctness: * JSON * Schemaless databases * Scripting languages Rethink's own FAQ says "RethinkDB is not a good choice if you need full ACID support or strong schema enforcement". So in the database market there are of course lots of users who do strongly value correctness, but they tend to use databases and technologies that are statically typed (Java/C#) and database engines that have are ACID and provide schemas. Very few of them want a database that doesn't provide ACID, doesn't provide schema enforcement and which appears to either not support standard industrial languages at all, or is at least distinctly uninterested in them. In other words Rethink targeted the hipster startup market of companies building social networks, ride-hailing apps and other things where correctness is not important and low-correctness tech dominates, but tried to sell a product that was oriented around correctness. This seems like a market/product mismatch.
  - lobster_johnson 10y ago
    That's a good point. Competing with MongoDB is difficult because "correctness" alone is not a selling point. It's the "worse is better" phenomenon as a marketing challenge. That said, I don't think lack of correctness itself is a design goal. I think it's accidental, or at least an intermediate state before we have something better. Case in point: I'm currently working on an open-source (it will be released soon) data store that uses JSON to express "documents" of data, but also has schemas, joins and transactions. It uses PostgreSQL internally, but completely hides the underlying storage. The developers who are currently using it for actual work are pleasantly surprised about how nice it is to work with: You get to work with rich, structured data in its natural shape without needing some kind of ORM to mediate between an app's data model and the database's, and at the same time it's very strictly validated without being pushy (it has gradual typing: you can start out schemaless and then tighten the schema). When we designed it, we wanted to build something that reflected how people actually use a database, and I think we're achieving that. (I'm looking forward to make it public and share it on HN.) So it's certainly possible to have the best of both worlds. It's just that the direction that the world moved in with "NoSQL" temporarily moved us off course for a little while. It started out as a natural reaction to scaling issues, and in that sense I think it might have been a necessary step while people figured out what they were doing. We're seeing the same evolution in the ways of people's thinking with regard to dynamically/statically typed languages, too.

by hoodoof 10y ago

lobster_johnson 10y ago

Exactly this. RethinkDB's value proposition wasn't really that attractive compared to Postgres. Replication and a nice UI, sure, those are nice. But Postgres has fantastic single-host performance, and Rethink is not yet in a position to compete there, except in some horizontally scaled use cases. Rethink doesn't even have transactions, making it potentially useless for some things that require strict atomicity over a complicated data model. In general, Postgres has such a richness of features that it's hard to compete with it; PostGIS alone is a "killer app" if you do any kind of GIS stuff. For me, there weren't any such "killer app" features. The change streaming is nice, but it's not difficult to do that with Postgres either. That's it really: Every solution offered by Rethink could be countered with "that's nice, but trivial with Postgres". The one exception being horizontal replication. I have yet to launch a project where Postgres couldn't scale out well enough, and I suspect that if I got to that point, I would reach for bigger/different guns such as Cassandra. For our apps we typically use Elasticsearch in front as a kind of read-only mirror of Postgres; the latter being the slow, serializable path, and the former the fast, slightly-out-of-sync path. Users don't know the difference.

zigzigzag 10y ago
Firstly, to the OP, that is an excellent post-mortem. Introspective, goes deep, generally confidence inspiring even though a part of it is literally "we were incompetent and didn't know it", well, lessons learned. I'd like to add some colour, although I've never used RethinkDB so take it all with a large pile of salt. lobster_johnson says: > Rethink doesn't even have transactions The post-mortem dwells heavily on how RethinkDB focused on correctness and was beaten by MongoDB which is, to put it mildly, not concerned about correctness. But Rethink is a JSON DB that apparently doesn't have transactions and which on its web page says "Query JSON documents with Python, Ruby, Node.js or dozens of other languages". Here are three technologies I do not strongly associate with correctness: * JSON * Schemaless databases * Scripting languages Rethink's own FAQ says "RethinkDB is not a good choice if you need full ACID support or strong schema enforcement". So in the database market there are of course lots of users who do strongly value correctness, but they tend to use databases and technologies that are statically typed (Java/C#) and database engines that have are ACID and provide schemas. Very few of them want a database that doesn't provide ACID, doesn't provide schema enforcement and which appears to either not support standard industrial languages at all, or is at least distinctly uninterested in them. In other words Rethink targeted the hipster startup market of companies building social networks, ride-hailing apps and other things where correctness is not important and low-correctness tech dominates, but tried to sell a product that was oriented around correctness. This seems like a market/product mismatch.
- lobster_johnson 10y ago
  That's a good point. Competing with MongoDB is difficult because "correctness" alone is not a selling point. It's the "worse is better" phenomenon as a marketing challenge. That said, I don't think lack of correctness itself is a design goal. I think it's accidental, or at least an intermediate state before we have something better. Case in point: I'm currently working on an open-source (it will be released soon) data store that uses JSON to express "documents" of data, but also has schemas, joins and transactions. It uses PostgreSQL internally, but completely hides the underlying storage. The developers who are currently using it for actual work are pleasantly surprised about how nice it is to work with: You get to work with rich, structured data in its natural shape without needing some kind of ORM to mediate between an app's data model and the database's, and at the same time it's very strictly validated without being pushy (it has gradual typing: you can start out schemaless and then tighten the schema). When we designed it, we wanted to build something that reflected how people actually use a database, and I think we're achieving that. (I'm looking forward to make it public and share it on HN.) So it's certainly possible to have the best of both worlds. It's just that the direction that the world moved in with "NoSQL" temporarily moved us off course for a little while. It started out as a natural reaction to scaling issues, and in that sense I think it might have been a necessary step while people figured out what they were doing. We're seeing the same evolution in the ways of people's thinking with regard to dynamically/statically typed languages, too.