16 ms·
RethinkDB: An open-source distributed database built with love over three years
- d0m 14y agoI absolutely love the website. Congrats on the public launch. In the FAQ, I would suggest a "How do you compare with Mongo?" I've read the intro, the faq and a couple of quick guides to find out what was different (read better). If I'm a happy mongo user, why would I switch to RethinkDB?
- alexpopescu 14y ago1. If you are a happy mongo user, we'd still be happy to hear your feedback about RethinkDB. 2. We're putting together some comparisons, hope to add them to the site soon. 3. There are already a couple of answers to this question on this thread: https://news.ycombinator.com/item?id=4764137 https://news.ycombinator.com/item?id=4764137 https://news.ycombinator.com/item?id=4763939 https://news.ycombinator.com/item?id=4763939 alex @ rethinkdb
- shykes 14y agoI am very excited about this. The RethinkDB team is rock-solid and the market is only going to get bigger. I particularly like the perspective of an easy onramp to get started, knowing that I will never have to leave because of scale or reliability. Please, please give me a SQL adapter! My marketing team needs SQL. My business app developers need SQL. Give them an adapter and I will get them to use RethinkDB - knowing that 1) my data is safe and I'm not 6 months away from a painful re-architecture and migration, and 2) as my developers hit the limits of SQL they can gradually (gradually!) peel the paint off and start using your more powerful query language.
- sandstrom 14y agoOn the contrary, I hope you [RethinkDb] don't spend time on an SQL adapter. I'd rather see time spent on improvements to the database itself.
- Heff 14y agoCongrats on the launch guys!
- sutro 14y agoHow does RethinkDB perform when compared to open-source distributed databases built with hate?
- bsg75 14y agoMaybe not built with hate, but used in anger?
- pc 14y agoThis took tenacity. Congrats on shipping.
- jbellis 14y agoI'll ask the obvious question not in the FAQ: How is this different from MongoDB?
- coffeemug 14y agoHey, this is Slava, founder of rethinkdb. There are some obvious high level differences: * A far more advanced query language -- distributed joins, subqueries, etc. -- almost anything you can do in SQL you can do in RethinkDB * MVCC -- which means you can run analytics on your realtime system without locking up * All queries are fully parallelized -- the compiler takes the query, breaks it up, distributes it, runs it in parallel, and gives you the results But beyond that, details matter. Database system differ on what they make easy, not what they make possible. We spent an enormous amount of time on building the low-level architecture and working on a seamless user experience. If you play with the product, I think you'll see these differences right away. Note: rethink is a new product, so it'll inevitably have quirks. We'll fix all the bugs as quickly as we can, but it'll take a few months to iron things out that didn't come up in testing.
- mej10 14y agoWhat do you see as the potential areas where RethinkDB will shine? Also, I am excited to try this out. I always enjoyed your writings and I am sure you + team have made something awesome.
- Goopplesoft 14y agoHah that question is almost word for word in the faq: http://www.rethinkdb.com/docs/faq/ http://www.rethinkdb.com/docs/faq/
- mej10 14y agoYep, you got me. I hadn't read the FAQ. Thanks for that.
- jdoliner 14y ago
- pspeter3 14y agoAre there any performance tips or information? This looks really cool
- masukomi 14y agoYeah, I'm curious too as to how it stacks up against MongoDB performance-wise and why I'd want to use one over the other.
- 3amOpsGuy 14y agoI'm sure they'll provide performance examples in due course but we should be conscious how difficult it is to do this in practice. Sure they can take the shortcut and just produce some single case and claim 'faster than a.n. other db' and that will probably satisfy the 'my db is faster than your db' brigade. However, based on the effort that appears to have gone into this product to get it to this stage, they probably wouldn't be happy with that. More power to them!
- coffeemug 14y agoPerformance analysis of complex systems is really tricky, and good benchmarks are even trickier. We'll be working through posting numbers, docs, tips, etc. over the next few months. Documenting this well is very hard work, so it'll take a little bit of time. In the meantime, you can always chat with us and we'll help you work through any issues you might run into.
- shawn-butler 14y agoCan I provide an intelligent sharding algorithm in place of a naive partition by primary key? I always get into trouble encoding things into primary keys eventually.
- coffeemug 14y agoNot at the moment, unfortunately.
- dested 14y agoThis certainly seems interesting enough to look into! One note, there's a typo in the code in the tutorial on top r.table('users).insert({'name': 'Slava', 'age': 29 }).run() users needs a closing quote.
- coffeemug 14y agoThanks -- fixed.
- wildmXranat 14y agoIf authors are lurking here, link from github to a quick start page is broken: http://www.rethinkdb.com/docs/guides/basic_quickstart.html http://www.rethinkdb.com/docs/guides/basic_quickstart.html GH issue https://github.com/rethinkdb/rethinkdb/issues/2 https://github.com/rethinkdb/rethinkdb/issues/2 edit: fixed already!
- coffeemug 14y agoThanks -- fixed.
- l0stman 14y agoI was just wondering if you've planned on open-sourcing the code since the very beginning or if the idea came much later. Anyway congrats on launching.
- coffeemug 14y agoWe wanted to open-source all along, but there were concerns raised by investors about IP, etc. It took us a bit of time to work through all the issues people raised, which is why it took so long.
- whyleyc 14y agoHow do you plan on monetizing the business ? Are you looking at some kind of consultancy play, ala Redhat / MySQL (pre-Oracle) ?
- coffeemug 14y agoWe want everyone who has a reason to use the product to be able to get it and use it for free (as in beer and speech). As companies grow and operations become more important, we'll be there to provide paid support. It's actually a great business model because you don't have to hire a really expensive enterprise sales team (you still need a sales team, but it mostly has to handle inbound requests). We're also looking into launching services, which is a great revenue stream for people who prefer to pay for convenience of not having to deal with operations at all.
- mmorearty 14y agoI don't know much about RethinkDB yet, but I will say that I have been a big fan (online) of one of its founders, Slava Akhmechet, for years. I've never met him, but he wrote some terrific articles on his website, http://www.defmacro.org/ http://www.defmacro.org/ , a few years ago. Start at the bottom of the list of articles, with "The Nature of Lisp." Slava is a deep thinker, which makes me very excited to take a look at RethinkDB.
- reinhardt 14y agoSo is RethinkDB written in Lisp?
- NathanKP 14y agoNo according the website it is written in C++.
- erichocean 14y agoAnd yet the Clojure guy did his distributed DB in Clojure (aka, Lisp). Kind of makes me wonder why C++ was chosen...
- alexpopescu 14y agoI've joined RethinkDB just a couple of months ago, so I might not have all the historical facts right, but here is what I know. In a previous incarnation RethinkDB was a highly optimized storage engine for SSDs implemented in C++ to be able to take full advantage of both low level SSD and kernel access. The current distributed engine was built on top on this storage engine and I think it only made sense to continue with C++.
- leif 14y agoOriginally, RethinkDB was to be a MySQL storage engine, which made C++ the natural choice. They pivoted away from MySQL after my short stint in the beginning so I can't speak to why the storage code was kept (though I can't imagine it's because my code was so great they couldn't bear to throw it away). Storage people tend to stick close to the metal, in general. This means C or C++ in most cases, for better or for worse.
- ch0wn 14y agoThis looks really interesting. I'm interested to see how their license choice works out. The server is AGPL-licensed while the drivers are under Apache 2.0. This should at least avoid the issues we all know from libmysqlclient.
- tbrock 14y agoSame licensing that 10gen uses for MongoDB (AGPL) and drivers (apache).
- jedberg 14y agoSuggestion: It would be great to have a page on your website that explains why RethinkDB is better than the other prevailing options. Right now I don't know why I'd want to invest time setting up yet another database.
- coffeemug 14y agoThanks -- will do in the next few days.
- eventdroid 14y agoyeh, that would be interesting to know.
- stock_toaster 14y agoAgreed. It would be very useful as a potential user. As an example, I like how Basho provides some comparisons[1] of Riak vs other popular options. http://docs.basho.com/riak/latest/references/appendices/comparisons/ http://docs.basho.com/riak/latest/references/appendices/comp...
- alexpopescu 14y agoWe're already working on something in this direction. alex @ rethinkdb
- xutopia 14y agoWhat's the elevator pitch? Maybe we can help you with those advantages if you can tell us right now.
- jdoliner 14y agoThe elevator pitch is: "Mongo's ease of use without the gotchas." We have a nice simple to use query language and quick setup process. But things like analytic queries like map reduce don't lock up the entire database. Our product aims to not be a ticking time bomb of technical debt.
- dkhenry 14y agoI don't see much on the documentation on indexes. Also this looks awesome, I would love to see a option to let it be eventually consistent and still keep the great querying ability.
- coffeemug 14y agoWe currently support primary key indexing, but no secondary indexes. This is definitely planned -- it will take a few months to get this out. EDIT: also, you can run queries with an out_of_date_ok flag, which will give you what you want. This only works for read queries though, the architecture is pretty much set up in a way where this would be very very difficult to do for write queries.
- aterreno 14y agoFrom the homepage it already looks more usable than mongo, liking the query syntax. Looking forward in trying the performances. Well done.
- codewright 14y agoI'm hoping this'll be a viable replacement for MongoDB. (Sparse/Schema-free is incredibly useful for me, as is JSON-centric modeling) jedberg already asked for a compare/contrast, but let me provide some specifics I care about that you might be able to answer. 1. Is it fair to say that thanks to MVCC, running an aggregation or map-reduce job isn't going to lock the whole damn thing up like it does on MongoDB? 2. You've got a distributed system that is seemingly CP, do the availability/consistency semantics compare with HBase? Master-slave? Replication? Sharding? 3. Latency is a big one for us and is a large part of why we use ElasticSearch. How does the read-latency on RethinkDB compare with Mongo/MySQL/Redis/et al ?
- coffeemug 14y ago1. Yes -- that was the main motivation for MVCC. We wanted to allow people to use rethinkdb for analytics and map/reduce on top of the realtime system without dealing with having to replicate data into something else. 2. Short answer: we favor consistency (via master/slave under the hood). It allows for much easier API, much fewer issues in production, etc. The user experience is just better. If you're ok with out of date results, you can do that too without paying the price of consistency guarantees. The downsite of our design is that you might lose write availability in case of netsplits (if the client is on the wrong side of the split). Longer answer: checkout the FAQ at http://www.rethinkdb.com/docs/advanced-faq/ http://www.rethinkdb.com/docs/advanced-faq/ 3. Read latency should be equivalent to other comparable master/slave systems. We don't do quorums, so latency will be much better than quorum/dynamo-based designs.
- rbranson 14y agoI want to preface my comment: this is impressive work, congratulations on shipping, and this is what MongoDB should have been from the start. In reality, most transactional database deployments are heavily skewed towards read workload, so reading from hot slaves is basically a requirement for master/slave databases. So, in most real world applications at scale, apps already deal with inconsistencies between slaves and the master and are making the "difficult" choice of dealing with CAP trade-offs. Asynchronous replication also creates a potential for difficult or impossible to recover from data loss in the sense that masters & slaves always have a continuous possibility for split-brain. RethinkDB does not provide multi-shard transaction atomicity and/or isolation, which in my experience is the biggest difficulty thrown up in front of developers coming from single-node databases. I feel like the difficulty of dealing with inconsistencies across multiple versions of a single object is far more familiar as most developers have at least dealt with cache invalidation in some form. It's really having to ensure and deal with potentially out of order operations (inconsistency in the ACID sense) across a "graph" of data that's more insidious.
- gruseom 14y agoThe most exciting news in quite a while! It will be interesting to hear what people think as they try it out and differences begin to emerge. In the meantime, congratulations on both the release and the open-sourcing.
- continuations 14y ago* In the previous incarnation of rethinkdb the focus was on maximizing performance on SSDs. Is this still the case - does rethinkDB perform better than other databases on SSDs? Do you have any benchmark numbers? * How does rethinkdb compare to MySQL Cluster? Both are distributed, replicated databases with a sql-like query language. * Any plan to offer a java client?
- coffeemug 14y ago* The SSD-optimized storage engine is running under the clustering engine. I'm wary of saying 'better' or 'worse' in case of benchmarks, because they're really tricky to do right. We'll be publishing well-researched benchmarks as soon as we can, but it will take time. * RethinkDB has flexible schemas and a query language that integrates straight into the host programming language and doesn't require string interpolation. As far as clustering goes, RethinkDB is a) really really really easy to use, and b) does a lot of query parallelization and distribution that MySQL cluster doesn't do. The product feels totally different, I think in a good way. The downside, of course, is that rethink is new and it will take some time to work out all the kinks. * I can't commit to a timeline yet, but yes, absolutely.
- Xorlev 14y agoI'm impressed that you're taking the time to do proper, well-researched benchmarks. They're really tough to get right. It really comes down to your own specific workload that matters anyways. This feels new and refreshing, I hope things turn out positively for you.
- coffeemug 14y agoThank you!
- hagope 14y agoGreat work, can't wait to try it out! :-)
- tjic 14y agoWhat the heck does "built with love" even mean? Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages? I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.
- jacquesm 14y agoReleasing a project like this means working very hard for a long time without anybody patting you on the back saying you're doing a good job. It's exhausting, a labour of love. Get it now?
- haberman 14y agoAs someone who's been doing that for several years on a side project that I believe in, this really hit home.
- whyleyc 14y agoI reckon $1.2 million in funding helped soften the pain :)
- skeletonjelly 14y agoTrue, but nothing beats positive reinforcement, for me at least. The whole happiness vs money thing.
- gruseom 14y agoYou could not have gotten these guys more wrong. They are serious technologists who have been working day and night for years to build something that they deeply believe in. Every hacker's heart should be warmed by the fact that they kept at it. When you have a vision of something great that ought to exist and set about bringing it into the world, you are in an isolated position: other people don't yet see what you see. This leads to a lot of doubt by others and by yourself too. The longer it takes, the more exposed you are. To make it through that you are going to need a deeper source of motivation – an underground spring. Love is a fine word for this, and it makes me happy that Slava put it in his title: it's a clue to this experience that rarely gets mentioned, especially in the land of pivots and MVPs and weekend hacks.
- eknkc 14y agoSo, no secondary indexes right? Will doing a query like "age > 25" perform something equivalent to a full table scan?
- coffeemug 14y agoYes, there are no secondary indexes right now (they're coming, but it will take a little time). Currently "age > 25" will do a scan, but if the db is sharded it will touch only the relevant shards and the query will be completely parallelized. We also do internal sharding for multicore performance, and the query will be parallelized across cores as well. All of that is only good for relatively small amounts of data though, so we'll be adding secondary indexes soon.
- spicyj 14y agoHow does it know which shards are relevant? I thought that data was sharded by primary key.
- jdoliner 14y agoCorrect currently no secondary indices. Ranges on fields other than the primary key will require a full table scan. -Joe Doliner, engineer at RethinkDB
- jamesli 14y agoGreat work! One question: is there any manual that explains the implementation details of the internals? Some manual similar to those Oracle, MySQL, Postgres, etc. provide? The only docs I found in the company website that goes deep into the internals are Advanced FAQ (http://www.rethinkdb.com/docs/advanced-faq/ http://www.rethinkdb.com/docs/advanced-faq/). It is more of an architecture view, though. The reason I ask is that with a good understanding on the internals, the engineers who understand database internals and distributed systems will have an "more" accurate idea on the capabilities and the limits of the features. Thus, if they decide to adopt RethinkDB, the understanding will help them design their applications to take advantages of the benefits and avoid the potential issues (or surprises!). MongoDB was not very good at documentation. It claims this or that feature works smoothly. Then, people found out many potential issues and limitations. That is one reason it leaves a bad tastes to many engineers.
- coffeemug 14y agoThere currently isn't, beyond the advanced FAQ. This isn't by design -- writing really good detailed architecture papers takes a lot of time, and we were 100% focused on getting the product out. We'll get much better at documenting the internals, but it will take some time.
- sidcool 14y agoCatchy title, I must say.
- wildmXranat 14y agoIs the package up on Ubuntu PPA already ? it seems that the installation instructions use the ppa, but apt-get doesn't find the package. edit: Indeed, my architecture i386 doesn't match the only available amd64 binaries. Thanks
- coffeemug 14y agoThe package is up. Which ubuntu version are you using? We support 11.04 and above. Anything less than that is missing some kernel features we use. EDIT: the main thing missing from earlier ubuntu versions is TCP_USER_TIMEOUT. We can work around it in the server, but we haven't done it yet.
- deleted 14y ago[deleted]
- arzvi 14y agoQuery language and ruby like function-chaining are what I feel the selling factors. I like the ease with which I added a node to the cluster. But naming the version as Rashomon scares me..
- lobster_johnson 14y agoA release named Rashomon should be the one where they introduce versioning support, in my opinion.
- joevandyk 14y agoWhy would you use this over PostgreSQL, especially with pg's new json support?
- tcwc 14y agoJSON support in Postgres is currently limited to a validated plain text field, it doesn't let you efficiently query inside the json object.
- shawn-butler 14y agoWhile technically correct I assume the parent was referring to hstore [0] which has both GIST and GIN indexes as well as btree and hash for equality. With some simple formatting functions (its just json after all) it's sixes to me. [0] http://www.postgresql.org/docs/9.2/static/hstore.html http://www.postgresql.org/docs/9.2/static/hstore.html
- dbuxton 14y agohstore is great, I won't hear a word against it, but it doesn't deal with complex nested objects like JSON can (I would love to be wrong about this) - it's just a key/value store with indexing of the objects inside.
- alphadogg 14y agohttp://momjian.us/main/blogs/pgblog/2012.html#October_2_2012 http://momjian.us/main/blogs/pgblog/2012.html#October_2_2012
- joevandyk 14y agoThere's nothing stopping you from writing functions that can index json documents. plv8 makes this easy.
- perfunctory 14y agoPlease stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.
- crazygringo 14y agoI have nothing to do with RethinkDB, but what are you talking about? Just represent them as strings. (What databases support colors as native types, anyways?) If you format dates YYYY-MM-DD, then you can do string comparisons for ranges. And JSON has the huge advantage of supporting hierarchical data -- arrays with objects inside, etc. It seems a like a huge step forward.
- kemiller 14y agoBut there's no standard way of representing the dates as strings, or indicating that this field here is a date and not a string that happens to look like one, so nothing can rely on what you do. You can represent literally anything with a string, but you lose type information when you do.
- cpeterso 14y agoISO 8601 is a standard way of representing dates as strings, e.g. "2012-11-09T08:08Z".
- inportb 14y agoHopefully, your application knows that this field here is a date encoded in a string.
- rdtsc 14y agoThere is a standard for representing string (see sibling comment) and your type information is the field name. So you know your document has a type if you give it a d_type field for example and then based on that you know your tstamp field is the date.
- 14y ago
- jedahan 14y agoLast I heard RethinkDB was a tail-append style engine for MySQL that was optimized for SSDs. Interesting to see a drastic pivot like this. Looks good, and good luck.
- szopa 14y agoNice work! It seems that you are well aware of the tradeoffs that you are taking and communicating it openly in your documentation (and your choices seem to be very reasonable). I really like the tone of your communication – it seems essentially BS/koolaid free. 1. How much data can you put in one instance before seeing performance degradation? I know that you still working on good benchmarks – but do you have any ballpark figures? 2. How does replication work? Is it closer to row/document or statement based (or something completely different)? How fast is the replication? 3. What is your envisioned used of the replication? Are replicas supposed to serve read traffic, or their goal is to keep the data safe in case of a catastrophe? 4. Can you tell me something more about cluster configuration propagation? The Advanced FAQ answer doesn't get into much detail. 5. Am I correct to assume that you are using protocol buffers? What motivated your choice?
- coffeemug 14y ago1. As long as active dataset everything fits in RAM, performance will be great. E.g. you can have terabytes of data but as long as the actively accessed dataset is < ~80GB the system will perform well. Once things get out of RAM, everything will work well if you have an SSD. On a rotational disk, performance will degrade very very quickly. This is a bottleneck on all modern databases, but clustering makes this problem go away because you can effectively increase the amount of RAM at linear cost by just adding more nodes (e.g. two nodes at 100GB of RAM each cost about four times less than one node with 200GB of RAM). 2. We do do block-level replication. On each node of the btree we store replication timestamps. When a node asks for new data, we can cull away parts of the tree the node has almost instantly. So replication is very very efficient for most OLTP workloads. We don't have statement-level replication yet, so if you do a range update on a large table, we'll have to replicate data block by block. It'll take a while to add statement-based replication - we'd have to do a pretty significant refactoring to make it happen. 3. Either. Replicas are great for failover -- if the master dies, you just failover and a replica picks up where the master left off. If you're ok with out-of-date reads, you can also hit replicas directly (e.g. for reports, etc.) and spread out the read load across the cluster. 4. This is a really complex question - we didn't document this because doing it properly would take a lot of time. I'll ask jdoliner to chime in -- he designed the architecture and wrote most of the code, perhaps he can describe it succinctly while we write deeper docs on this :) 5. We use protocol buffers between the client drivers and the server. We picked that because there were libraries for the initial three languages we picked (Ruby, JS, Python), they were really easy to use, and very efficient. We could also have a single spec for the client/server API. Internally we use our own serialization scheme which allows us to dump arbitrary C++ objects on the network. It doesn't support other languages (which we didn't need), but is much more versatile for writing complex cross-machine code.
- krob 14y agoI'd really like to know why they don't have a PHP library, considering it powers half the web. The key shouldn't be to promote someone elses own language when using infrastructure products but rather to support everyone in using their tool. Mongo supports everything under the sun, so should this.
- alexpopescu 14y ago... and it will. We had to start from somewhere and we also wanted to get a feel of what are the most requested libraries so we can focus our energy in those directions. alex @ rethinkdb
- sergius 14y agoNo C++ API???
- DaemonXI 14y agoWill there ever be a 32-bit version?
- coffeemug 14y agoIt should be easy to do a port, but it's one of the many things to do. I can't commit to a timeline, but it will probably happen.
- shin_lao 14y agoCongratulations! How do you plan on making money?
- harryh 14y agoIf I were you guys I'd strongly consider adding support for hashing of the shard key. There are many cases where you care about distributing your writes(1) a lot more than fast range queries on the PK. -harryh 1. Yes, I know there are other ways to do this besides hashing the shard key, but this is often the best way.
- coffeemug 14y agoWe actually support hash sharding underneath -- each range shard is further broken down into hash shards internally to support multicore scalability. This isn't exposed to the users currently, but it can be. Another option is to allow the user to provide a hash function on the PK (I think this is what you're suggesting). We'll be addressing this at some point, we have to sort through the list of feature requests first. It's a long list :)
- harryh 14y agoI don't feel the need to provide the hash function, as long as it's something reasonable. I understand that it's a long list! Just adding a vote to one particular item. Good luck!
- bjhoops1 14y agoThe querying capabilities here look amazing. Having to manually figure out how do joins and group by in something like CouchDB is really a pain, but this looks really slick. Very impressed and I will be trying this out!
- jdoliner 14y agoWe spent a long time trying to reimplement other people's protocols but with our engine underneath. Being able to have features like this is one of the things that eventually convinced us it would be worth it to control our own. Bear in mind that our joins are also distributed which we think is really cool.
- haberman 14y agoIs schemaless a win over an object schema like a JSON schema (or a Protocol Buffer .proto file)? Schemaless is clearly a convenience win over SQL because SQL's way of modeling nested/repeated data doesn't map as easily onto programming languages. But for all the people who are using JSON-based databases these days, I'm curious how many of them couldn't easily write a JSON schema or a .proto file that describes their de facto schema. I ask because a lot of things become easier to reason about (and optimize) if you know that a field won't be a string in one record and a number in another. And writing a .proto file (or equivalent JSON schema) would give you an authoritative place to document what all the fields actually mean. I don't have any actual experience with JSON-based databases, so I was interested to hear the opinions of people who do.
- embwbam 14y agothis. I don't like SQL columns. They make life hard. But I'm spending time learning TypeScript specifically so I can add some types/schemas to my JavaScript. That doesn't mean I want to deal with the implementation detail of columns, but I definitely wouldn't mind some type safety.
- ollysb 14y agoI hear a lot of talk about how hard it is to maintain sql schemas and columns. I've used mongodb on projects previously and while it was interesting I didn't find that the lack of schema made life any easier. Writing a migration in rails is so easy, I just can't understand how managing a schema really makes life more difficult. What is it about managing a schema that makes people so eager to jump to schema-less?
- coffeemug 14y agoThere is of course no fundamental reason why JSON-based db's has to be schemaless. This is one interesting direction that might be worth exploring.
- m0th87 14y agoI would love a system that is schema-less by design, but has guards that can be enforced at insert/update. That way, the underlying data structures don't have to be locked up from complex migrations (as needed w/ ALTER TABLE), but you still get type safety. A migration instead would simply involve a change in guards and an asynchronous update of existing entries. Plus you'd get all the wins of something resembling optional types: you only enforce guards if you want.
- banachtarski 14y agoWould love a comparison of this to couchbase which seems to have a similar sharded distributed setup. Congrats on shipping!
- m0th87 14y agoHow do filters work? They seem pretty difficult implementation-wise since you can write them in any of the language bindings. My first guess is that you pipe all the data in a table to the client, and the client itself does the filtration. But this would be extraordinarily inefficient.
- old-gregg 14y agoNope, they build an AST for filter expressions and compile it on the server IIRC. The client gets filtered data from the server.
- jdoliner 14y agoPiping all the data to the client would be extremely inefficient. Fortunately we don't do that. When a filter is written in the client language it gets compiled into a protocol buffer which is sent to the cluster. This gets compiled into a query which is sent to each of the relevant shards for the table. This query has the filter baked right into it. The shards then go through their local copy of the data and filter out the rows which do not meet the query predicate. This data gets returned to the coordinating node and eventually to the user. Thus only the data the will actually be returned is ever transferred over the network. Furthermore this process is done lazily. On the client side rather than getting back a huge array with the results of your filter you get back an iterator. This iterator stores a buffer of data which will be refilled as it is incremented.
- coffeemug 14y agoTo add to jdoliner's answer, the reason why you can write table('foo').filter(lambda x: x['bar'] > 5).run() is because we do some language trickery on the client side to compile the query to an AST. In this case, we overload greater than operator, call the lambda function once on the client with a special object, and return an AST. This AST is then sent to the server and executed there. It's rather difficult to integrate into a host language like that smoothly from the driver implementation perspective, but once the driver is written the user experience is amazing because you can write queries that look exactly like Python, but they're executed entirely on the server.
- jpettersson 14y agoWow, looks great! I'm excited to try it out. The Github graphs are really interesting too, that's a lot of love/work right there! https://github.com/rethinkdb/rethinkdb/graphs/impact https://github.com/rethinkdb/rethinkdb/graphs/impact
- dbro 14y agoI wonder what happens in the RethinkDB office on Monday evenings? https://github.com/rethinkdb/rethinkdb/graphs/punch-card https://github.com/rethinkdb/rethinkdb/graphs/punch-card
- mcartyem 14y agoAre there plans for an Arc driver?
- jdoliner 14y agoPG said he'd post facto reject us from YC if we don't. In all seriousness while we'd eventually like to have support for every language Arc is farther down our list than others such as PHP and Java. We will get to it eventually though. If Arc could be made to use protocol buffers it wouldn't be too hard for a contributor to write the driver themselves though.
- coffeemug 14y agoHey guys, Slava here. I've been up since yesterday, so I'm going to clock out (though some of the team members are still lurking here). I wanted to thank everyone for great feedback. We're working hard to improve Rethink over the next few months. FYI, you can always hop on IRC (#rethinkdb on freenode) or github tracker (https://github.com/rethinkdb/rethinkdb/issues https://github.com/rethinkdb/rethinkdb/issues) with questions and we'll help you out.
- biturd 14y agoThanks for this work, it looks really nice. I was looking at the github comments about a home brew recipe in which it was stated that aside from a recipe creating a VM, the Mac OS X port would take a bit longer. Is that a full port from one language to another? Or just an issue of the different flavors of *nix that need dealing with and probably some of the dependency tree issues that come with it? I'm curious what needs be done to get it building on Mac OS X — perhaps I could assist somehow. I see a few dependencies that don't immediately sound familiar. You may have better luck with MacPorts, which uses tcl as the language for their portfiles. Portfiles are just like homebrews recipes, but MacPorts always builds new, including the entire dependency tree ( and dependencies of dependencies etc., etc. ), for which they have thousands of working portfiles. Since those are completed and working, you wouldn't have to worry about those until you wanted to be able to make a binary outside of any package manager. MacPorts can build binaries now ( new feature ), so you could just as easily instruct it to create a standard Mac OS X installer .pkg which makes sure everything goes in the right place, on the right platform, for the right architecture. They are an exceedingly friendly and helpful group, I'm sure they would live to see this software in their package/portfiles list.
- i386 14y agoSeems to me that most of us who have used MacPorts have moved to Homebrew or that could just be the bubble I'm living in. Is there anyone still who still uses MacPorts who could chime in and say why they never made the switch?
- 14y ago
- dxbydt 14y agoThought I'll share this with you. A yc company hired me. I showed up at their mountain view office. The founder said "This is the former office of RethinkDB! I hope we are as successful as them." I didn't know who/what RethinkDB was, so I said ok, sure. 3 days later he asked me to clear my desk and leave. He said "You are the sort of person who should work in RethinkDB". So I asked "What does that mean ?" He said "RethinkDB is trying to solve very deep algorithm problems. They want somebody with CS knowledge to do deep research. That is what you are good at. But here we are just trying to run a business. You are not a good fit for that!". So I left.
- fragsworth 14y agoI know lots of engineers who have trouble talking to people who don't share their knowledge. This problem is extremely pervasive - I'd say a good 25% or more have this problem to some extent. It's not a good thing when this happens - you need to be able to speak to laymen or you're gonna have a bad time. I am going to go out on a limb here and suggest you try to work on being a bit more practical. Don't complicate things for the sake of solving difficult problems. Don't try to shower people with your engineering knowledge when it's not necessary, and don't expect everyone to know everything you do. And don't be an asshole about it either.
- borplk 14y agoI adjust the level of details and technical stuff pretty well according to who I am speaking to. But sometimes someone with very limited knowledge of something asks me a detailed question about X. What they ask is too difficult and complex to be described in a simple way. Either I have to overly simplify it which may insult them and will do no good or I have to go on and step by step give them digestible chunks of explanation which will inevitably be a bit technical even though I try to minimise that. I think it's a two-way thing. The person should also consider their own level of knowledge before asking for an explanation of something and adjust their question based on that. I don't ask my doctor to tell me why my heart does this and that because I simply don't have the knowledge to be able to understand his answer. I ask, my heart is doing ok? cool!
- erichocean 14y agoI find JSON-oriented databases to be a huge limitation for writing applications managing any kind of financial data, due to the lack of a decimal number type and a timestamp/date type, both of which SQL provides (and are used A LOT). Sure, you can put that stuff in strings, but then you'll run into limitation with queries where you want to, e.g., aggregate a total, or do timestamp arithmetic. I could do everything with strings, custom map-reduce, etc., if you're inclined to suggest that as a workaround. Still doesn't mean JSON's a good idea.
- cuu508 14y agoOr you could use fixed point math: 600 in database means $6.00 Then aggregates and comparison operators would work, but you would have to decide upfront how much precision you might ever need
- erichocean 14y agoThat sounds good, but with only 53 bits of integer precision in JSON (51 if you move the decimal point to account for cents), there's just not enough digits for finance these days. A similar problem exists if you use JSON numbers (aka doubles) for timestamps –- the numbers just aren't big enough to do it accurately.
- javajosh 14y agoWhat is the business model, if any? (This question is not addressed in the FAQ, and I believe has at least some relevance to the longevity/shape of the reDB community over time.) Also, have you talked to the Meteor folks about swapping Mongo out for this? Or would this be 'newness overload'?
- ww520 14y agoI like their DSL-style query API. It's fantastic! Disclosure: Ok. I'm biased. :) I've designed a similar DSL-style query API in another project. https://github.com/williamw520/jsoda https://github.com/williamw520/jsoda
- ww520 14y agoCongratulate on releasing. Well done! A few questions: 1. Will secondary indices be ever supported? Range scan with a different order than the primary key is very welcomed. E.g. date range query. 2. Do you support conditional update? Or any kind of optimistic locking or versioning to coordinate concurrent updates from different clients? 3. Related to 2. How can loosely-sequential Id be generated using a table? 4. Will some transaction support be added? Don't need full ACID, just grouping updates (intra-table and/or inter-tables) in one shot would be nice. Should be feasible with MVCC already in place. 5. Do all the clients hit a central server to initiate queries which then farms out the requests to different shards? Or the client library knows how to get to different shards directly? First case has a single-point-of-failure, and bottleneck in scaling. 6. Do you support automatically re-balancing of shard data (data migration) when new shards are added or old ones retired? 7. How are authentication and authorization done? Or any clients can come in? 8. Internal detail. For out-of-date distributed query on the slave replicas, is there a cost-based (or load-based) decision process to pick the most idle replica to do the sub-query? 9. Internal detail. Do you use Bloom Filter to optimize distributed joins?
- DanielRibeiro 14y agoDid they not launch a while ago: http://techcrunch.com/2011/06/06/rethinkdb-expands-beyond-ssds-launches-its-speedy-database-to-the-public/ http://techcrunch.com/2011/06/06/rethinkdb-expands-beyond-ss... ?
- ww520 14y agoWell any product release is a huge effort, especially database product. Things got done and pushed out of door. Congratulation well deserved.
- jdoliner 14y ago> 1. Will secondary indices be ever supported? Range scan with a different order than the primary key is very welcomed. E.g. date range query. Secondary indices are one of the most asked for features so they'll probably be added in the next release. No promises though secondary indices are tough to do right and we won't ship them if they're not great. > 2. Do you support conditional update? Or any kind of optimistic locking or versioning to coordinate concurrent updates from different clients? Updates can be done with conditions on the row. For example: table.filter(lambda x: x['age'] > 25).update(lambda x: {"salary" : x["salary"] + 25) > 3. Related to 2. How can loosely-sequential Id be generated using a table? Loosely-sequential IDs would have to be generated client side for now. > 4. Will some transaction support be added? Don't need full ACID, just grouping updates (intra-table and/or inter-tables) in one shot would be nice. Should be feasible with MVCC already in place. Eventually. No concrete timeline for this right now though. > 5. Do all the clients hit a central server to initiate queries which then farms out the requests to different shards? Or the client library knows how to get to different shards directly? First case has a single-point-of-failure, and bottleneck in scaling. A client makes a connection to a specific server and all queries go through that server. However every server can file this role so connections can be distributed and there's no single point of failure. An even better option is to run a proxy on the same machine as the client. For more info run: rethinkdb --help proxy > 6. Do you support automatically re-balancing of shard data (data migration) when new shards are added or old ones retired? Right now sharding is a manual process. You tell the server how many shards you want and it handles figuring out how to evenly split the data, picking machines to host them and getting the data where it needs to go. What it doesn't do is readjust the split points when the data distribution changes. This will be a feature in RethinkDB 1.3. > 7. How are authentication and authorization done? Or any clients can come in? RethinkDB has no authentication built in to it. You should not allow people you don't trust to have access to it. 8. Internal detail. For out-of-date distributed query on the slave replicas, is there a cost-based (or load-based) decision process to pick the most idle replica to do the sub-query? Right now we just select randomly. This is slated as a potential upgrade for 1.3. Especially if it proves to be a problem for people. Thus far it hasn't been for us in profiling runs but this is the type of problem that's more likely to show up in real world workloads. 9. Internal detail. Do you use Bloom Filter to optimize distributed joins? We do not currently use bloom filters to optimize this.
- Meai 14y agoCould you guys offer a C driver? (preferably not c++)
- lambda 14y agoTaking a look at their API, a C++ driver would actually fit much better with their style, which relies on operator overloading to make queries involving arithmetic easy to write in the host language. A C driver would definitely be possible, but would be a little bit clunky.
- WhatsACamelToe 14y agoMay I ask why is there a link to "Bob Marley Everything's Gonna Be Alright" (youtube) at the bottom right of your website? : )
- nnash 14y agojoe@alchemist~$ rethinkdb joe@clockwerk~$ rethinkdb -j alchemist:29015 Dota player?
- tehansen 14y agoam I the only one that noticed the lambda at the bottom right of the webpage? :)
- willytobler 14y agoThis looks like very interesting. And a nice interface to deal with. But ... I do not find anything about authentication. Whoever wants can fiddle with this 8080 Port. Did you rethinkAuth or am I just too stupid to RTFM?
- coffeemug 14y agoNo authentication support yet unfortunately - you'd have to do an ssh tunnel for the web admin. This is one of the things on the todo list.
- hcarvalhoalves 14y agoIt seems very interesting, and having to deal with ORMs daily makes me appreciate the clean API. I feel being based on JSON is a big con though. While it's popular, it was never meant to be a rich serialization format, just simple. How to implement more complex fields like dates, and query efficiently on RethinkDB?
- shrughes 14y agoA benefit to JSON is that people know immediately what it is. It certainly does have its deficiencies (binary data being one that outweighs dates in terms of immediate importance). Being limited to strict JSON is not a permanent decision (I'm saying this as a member of the RethinkDB engineering team), it's a conservative one in terms of API design, and in terms of limiting scope for the first release.
- juzfoo 14y agoCongrats guys! Looks great from the specs. Any thoughts on architecting for multitenancy?
- goranp 14y agoI am a bit late to the game but how does this compare to HyperDex in terms of scalability and sharding?