18 ms·
RethinkDB 1.5 released: secondary indexes, soft durability, perf improvements
- dkhenry 13y agoLooks like one by one the features are being checked off to make rethink a real contender. Keep up the good work guys.
- jdoliner 13y agoThank you!!
- jonthepirate 13y agoVery cool.. I'm going to try this.
- manacit 13y agoI'm really excited to see this project take off - I played with it a few months ago and it seemed like it was getting close to prime time. The two issues I see now are: * More packages for more distros - It's great that I can install it on my OS X machine for testing, but I can only roll it out on an Ubuntu machine, and last time I checked the packages were a bit iffy when I tested them. * More languages - It looks like you guys are working hard on this one, which is great. Hopefully it'll be production ready soon! Great work guys.
- mglukhovsky 13y agoMike @ RethinkDB here. Great feedback-- while we've been a bit behind in rolling out releases to other distros and languages, we've been delighted to see the community pick up our slack: - Our new build system has people successfully compiling from source and building community packages for CentOS, Slackware, Gentoo, Arch, and lots of other distros. - One intrepid soul is porting RethinkDB to FreeBSD, where it builds successfully (almost ready to merge in): https://github.com/rethinkdb/rethinkdb/pull/688 https://github.com/rethinkdb/rethinkdb/pull/688 - We've seen an enormous number of client drivers for different languages pop up in a short time, with some impressive collaboration: * rethink-net (C#/.NET): https://github.com/mfenniak/rethinkdb-net * php-rql (PHP): https://github.com/danielmewes/php-rql * rethinkgo (Go): https://github.com/christopherhesse/rethinkgo * cl-rethinkdb (Common Lisp): https://github.com/orthecreedence/cl-rethinkdb * haskell-rethinkdb (Haskell): https://github.com/atnnn/haskell-rethinkdb * lethink (Erlang): https://github.com/taybin/lethink * librethinkdb (C): https://github.com/unbit/librethinkdb * Lots of emerging efforts for Java, Perl, and other languages. It's great to see people passionate about bringing our project to other platforms / distros, and we're doing whatever we can to support them. If anyone's interested in a similar project, shoot me an email at mike [at] rethinkdb.com
- rdudekul 13y agoI love RethinkDB API. It makes joins possible without going through mapreduce. Though I am using MongoDB currently, I plan on switching to RethinkDB as soon as it is more stable and provides better performance. Excited to see the team making progress on all fronts.
- dude_abides 13y agoIt makes joins possible without going through mapreduce. This is pure gold https://twitter.com/shit_hn_says https://twitter.com/shit_hn_says material :)
- mintplant 13y ago...Compared to MongoDB, which is referred to in the very next sentence.
- RyanZAG 13y agoI'm not sure what you think is pure gold here - the statement is correct. When dealing with big data, joins are the first thing to go as you need to shard your data. Whether RethinkDB will really solve joins across sharded data is up in the air though - ultimately you still need to do filtering per shard and then reduce the result somewhere - and doing filtering across two attributes means you are going to be doing a lot of data transfer, but RethinkDB is about the only database I've seen really making progress in this area.
- wmf 13y agoClustered databases are actually older than mapreduce, hence the hilarity.
- hendzen 13y agoSeems a bit unconventional to manually specify which index you'd like to use when writing queries. This would normally be something handled by a query optimizer. Of course, many DBMS provide some facility for specifying indices manually through query hinting, but for non-trivial queries, the optimizer often can do a better job than a programmer at picking the most efficient query plan.
- jdoliner 13y agoHi, I implemented secondary indexes in RethinkDb. This was an intentional choice we made for a couple of reasons. The most pragmatic one was that we don't have an optimizer and we thought that secondary indexes could still be useful to a lot of people without one. And we've found it's always better to ship early and get feedback sooner rather than later. Another reason though is we're not totally sure we're sold on the state of the art of query optimizers. From what we've seen they can introduce a number of non intuitive behaviors and it can be hard to tell which operations will use which index. Query optimizers are nice when they work correctly but they don't always work correctly so for the first release of secondary indexes we opted to make something that had very clear precise semantics and then build on it. We will eventually have a query optimizer but it's a big problem and we don't just want to implement what others have because we think we can do better.
- antirez 13y ago+1, a good approach could be to still make users able to specify manually what index to use even when the optimizer will be available. Sometimes humans just know better.
- jdoliner 13y agoThat's something we'll make sure to keep around indefintely. I always hate things that are too magically, which optimizers frequently seem to me, but I also get why they're useful to people. The best compromise I think is to give people the magic and a way to opt out of the magic.
- hcarvalhoalves 13y agoRethinkDB is looking great. The feature gap is closing, now I'm itching to use it in a real project.
- agildehaus 13y agoBuild fails on a "brew upgrade" for me. http://pastebin.com/3GWetUTE http://pastebin.com/3GWetUTE
- mglukhovsky 13y agoThanks for the report, I've sent up the Bat-Signal for our @atnnn (our Homebrew expert) to sort this out.
- coffeemug 13y agoOk, we've got a fix for this. We'll release 1.5.1 shortly.
- agildehaus 13y agoLooked into it and discovered it was a simple lack of the pyyaml python module. Should probably put that in your build instructions and have the configure script throw an error.
- rb2k_ 13y agopip install pyyaml fixed it for me
- craigyk 13y agoIt's not always 100%, but who else think homebrew is awesome? I miss it when I go to my linux machines.
- Game_Ender 13y agoWhat do you miss about it?
- nickpresta 13y agoI like Homebrew as well and I don't want to detract from how awesome it is, but dpkg + aptitude is much better in my opinion and I find I actually miss aptitude on OSX.
- dscrd 13y agoDoes RethinkDB do multiple atomic updates?
- hendzen 13y agoNope, operations on a single document are atomic, but not on multiple documents. MongoDB works the same way. AFAIK, the only NoSQL DB with support for multi-doc transactions is HyperDex [0]. [0] - http://hyperdex.org/ http://hyperdex.org/
- nanodeath 13y agoOrientDB also has multi-document transactions, I believe.
- prodigal_erik 13y agoAlso http://www.rethinkdb.com/docs/advanced-faq/ http://www.rethinkdb.com/docs/advanced-faq/. RethinkDB started out as a MySQL driver that was lock-free and did schema updates non-stupidly, but it sounds like they've abandoned ACID and schema enforcement entirely like almost all NoSQL systems (as if what I wanted is faster wrong answers).
- wmf 13y agoIf you want schemas and transactions, just use NewSQL.
- tomjen3 13y agoI see this tude' among DB guys quite often -- unless the database guarantees unconditionally that it will always give the right answer and that no data is ever lost then it is worse than useless and most be nuked from orbit. Quite frankly that is pretty arrogant. ACID may be required for a banks transactions, but in far the majority of cases it isn't required (yes, even in your business) -- nothing of any consequence is going to happen if your A/B testing DB lose a couple houndred entries because those weren't flushed to the cache. As for non-schema enforcement, well you have too add a few if statements to your database model class -- and that is it.
- themgt 13y agoWe've updated our demo to 1.5 as well: http://rethinkdb-demo.pogoapp.com/ http://rethinkdb-demo.pogoapp.com/
- dennis82 13y agothis is a good update and not to rain on their parade, but doesn't this seem market redundant and reinvent-the-wheel?
- coffeemug 13y agoRethinkDB is definitely late to the party, so this is a great question. Here's how we think about it. 1. Technically, Rethink already does very useful things leading NoSQL contenders don't do. An extremely expressive query language, massive query parallelization, distributed joins, etc. You can't get that anywhere else, and we have lots of features in the pipeline to keep raising the bar. 2. I believe we have a unique take on usability and design that is very valuable to users but isn't present in other products. Rethink is pleasant to set up and (hopefully) pleasant to use. To us, design matters, and we put an inordinate amount of effort to make the product beautiful. It turns out users care about this a lot. 3. I feel like leading NoSQL contenders are somewhat stagnant. There is an enormous amount of innovation that could be done, needs to be done, and isn't being done. We'll continue releasing really useful features that nobody has, and (I think) nobody expects. I think we bring a unique philosophy to product design that's very valuable. 4. Details matter (which is why Rethink is late to the party). Once you outgrow the ten-minute blog stage, a lot of underlying architectural decisions start to really matter, and we took the time to do them right. TL;DR: Rethink is already really good for building apps on top of it and offers things nobody else does. It will continue getting better. We'd be honored if you took it for a spin!
- mintplant 13y agoI'd rather have had them take their time to get it right than rush it out the door to be first-to-market.
- papsosouid 13y agoWow, I thought this was a joke press release making fun of "web scale" fad craziness. They seriously gave a 1.0 release number to a "database" that you couldn't search?
- apendleton 13y agoNo, you've always been able to execute queries. They just weren't indexed before.
- jdoliner 13y agoData has always been indexed it just was only indexed by a single primary key. Before this release the indexing scheme wasn't more sophisticated than a key value store now it is. I feel like there are several other products on the market that just do key value indexing.
- papsosouid 13y agoI didn't say "execute queries", I said search. Unless you have a tiny amount of data, no indexes means searching is not feasible.
- JulianMorrison 13y agoThe way you search on a key-value store with no secondary indexes is create the secondary indexes manually as separate tables.
- papsosouid 13y agoOr just use a database instead of creating problems for no reason.
- LeafStorm 13y ago> Early in the development of RethinkDB, we made the design decision to be extremely conservative about durability and safety of users's data. I'm glad that the Rethink engineers are taking this approach to development instead of "let's win all the benchmarks!" Overall, it's apparent that they are really thinking through the consequences of their decisions and the tradeoffs in database systems, and that they want to build something that will last instead of something that will just become popular in the short term.
- jdoliner 13y agoThanks. If we can succeed in being Mongo without the gotchas we'll be very happy.
- Ecio78 13y agoCongrats for your release, RethinkDB seems very interesting and I really appreciate how coffeemug and all the other team members genuinely and clearly reply to questions here. Is there already a list of sites/services that use RethinkDB in production?
- coffeemug 13y agoHi :) There isn't a list yet we can share, but we're starting to work very closely with a small group of early customers -- shoot me an email to slava@rethinkdb.com if you'd like to see if we can work together!
- troyk 13y agoAm I the only one that hardly ever needs a case-sensitive query/index on string data? This is the one feature I miss from the MySQL glory days. With postgres, I use the citext type (case-insensitive text) but I guess Mongo and ReThinkDB expect you to either store the data as all lower or upper, or duplicate the field (for indexing).
- coffeemug 13y agoThe reason we (and probably Mongo folks) do it this way is because JSON is case-sensitive, so making indexes case-insensitive wouldn't work very well. EDIT: it looks like I misread your comment. I was talking about index/field names, while I think you were talking about field values. In this case Sam's comment below is correct -- once we add string.toLower() (which is very easy) you'll be able to do case-insensitive indexing if you like because the indexing system supports arbitrary reql expressions. If you're using rethink and this is important to you, let me (slava@rethinkdb.com) know, and we'll prioritize this.
- pixelcort 13y agoAre the indices based on reql expressions updated incrementally? How about making indices based on map-reduce? Would that work? Would they be updated incrementally?
- coffeemug 13y agoYes, indices are updated incrementally. You can currently only evaluate an index for a given row based only on that row, not on other rows. The reasons for this are pretty deep -- if you start evaluating indexes on a row based on other rows, the result becomes non-deterministic and can't easily be replicated across the cluster. Once we introduce value-based replication (as opposed to evaluation-based replication we have now) this will become possible.
- troyk 13y agoInstalled and played with (1.3 I think) and it looks very promising but the ability to query and order by case-insensitive text is a requirement for us. Our app deals with user provided names of people, places and things that are commonly searched and sorted. So not using RethinkDB today, but I am 100% in love with what you guys are trying to accomplish.
- psykotic 13y agoDid anyone notice the name of the example database in the video demo? The RethinkDB crew must be Dota fans.
- L0j1k 13y agoI can't wait to give Rethink a spin later this weekend. I've been reading everything and just pleased as a pig in shit about almost all of it. The conversations with Rethink crew here also help me tremendously to feel good about any decision to try it out in production. I definitely feel good about throwing it against some weird situations just to see how it does. :)
- hoodoof 13y agoI like it alot but even simple queries are hideously complex. It doesn't need to be this hard. The ReThink developers are exceptionally helpful but goodness. Try to do a search returning all items where an array contains a value. In simpledb it's just select * where field='value', even if field is an array. In RethinkDB I had to ask for developer help and ended up writing a deep, sophisticated query. Querying has to be dramatically more simple.
- coffeemug 13y agoWe're actually working on adding commands to ReQL to make queries a lot simpler. (It turns out that just a few commands that are very easy for us to implement can make enormous difference) If you can, would you mind giving examples of things you'd like to do that are hard? We'd love to make it easy, and we think we've got most common things covered in the next release, but if there's something specific you have in mind we'd really appreciate it.
- hoodoof 13y agoThe basic stuff needed to let a beginner developer write a web app should be super super simple. REALLY simple. No map no reduce nothing hard. Here are some of the basic examples of typical things people might need to do when writing a web app: How do I select records based on a value in an array? How would I find all records whose name begins with "star" i.e. 'Star Trek' 'Star Base' 'Star Wars' 'Star Anise' I have records which contain an array of email addresses. How would I get all documents which contain one or more of a list of email addresses? How would I add a where clause to my queries to ensure for example that the "status" field in my document equals "open"? How would I implement paging through a result set? How would I implement count to control the number of records returned during paging? What is the right way to store dates in ReThinkDB so that they can be queried? How would I select all documents in a date range? How do I do queries on fields that are within other fields in my JSON structure? How do I do wildcard searches? How do I define which fields are returned from my query, or does it always return the entire document? How do I make sure that I do not add documents to the database that are missing required fields, thereby breaking all queries that do not contain those required fields? It should be incredibly easy (and well documented) to do all the above. I wanted to work with ReThinkDB but chose a different database because I was so intimidated by the query writing.
- anatari 13y agoSo is this production ready? Excited about this, but I don't have time at the moment to dabble on things I can't use and the last comment regarding this is a post on Quora from the founder saying that it is not production ready yet. 1.0+ version numbers makes you think this isn't beta software anymore, but that doesn't seem to be the case.
- coffeemug 13y agoWe'll bump the version to 2.0 when the product is production ready (the 1.x versioning scheme is an accident that unfortunately we can't do much about).