4 ms·

ReQL: hassle-free JSON processing

40 points by coffeemug 13y ago

orthecreedence 13y ago
As a user of RethinkDB in my new apps and also the author of a RethinkDB driver, I have to say that the query language is the absolute best I've worked with. I've been working with databases and writing queries for over 8 years now and ReQL is my favorite. Being able to compose queries in a language-native way is really effective but also powerful in that you are essentially running programs on the database that it understands enough to optimize and distribute amongst its fellow servers. Not only that, you really can marry your code with the forms you send to the database, making it easy to compose queries that are built from conditionals from local variables. Also because of the expressiveness, there are almost no places where I ever run two queries when selecting data. For instance in Mongo, I always used to do "get X. If X is this, then get Y, otherwise get Z." Now all that logic can be shipped off to the database (without having to build some huge SQL string) and when I get an answer back, it's the one I want. Great query language, great database, great team.
- neumino 13y ago
  You forgot great community! And you are part of it!
- rogerbinns 13y ago
  What does a query look like when you print it (to help with diagnostics and debugging)? I don't mean the results but rather what you have before calling run.
  - orthecreedence 13y ago
    On my primary platform (common lisp) when printed it looks like a large set of nested structures made up of "terms" (ie "create_database", "table", "function", etc) and datums (array, float, kv pair, etc). It's easy to pretty-print and get a view of the entire query structure before it's shipped off the the protocol buffer serializer. That's lisp though, and the print methods for the protobuf objects are built into the protobuf library. It's really going to depend on your language, probably. The nice thing about the query language is that the mapping between what you send the driver and what the driver sends to the server is pretty much 1:1, so what you see in front of you is the printed query (save for bugs in the driver).
mpweiher 13y ago
This is pretty neat, I certainly like not having a separate query language embedded as strings and manipulated with string concatenation etc. (SQL injection ...). Similar: - Avi Bryant's Relational Object Expressions bring relational algebra into the programming language: courseTitlesForProfessorNamed: aString |courses| courses := (professors * courses select: [[:ea | (ea lastName = aString) & (ea id = ea professorID)]. ^ (courses project: #title) copyFrom: 1 to: 10. http://www.cincomsmalltalk.com/userblogs/avi/blogView?showComments=true&entry=3246121322 http://www.cincomsmalltalk.com/userblogs/avi/blogView?showCo...: and http://c2.com/cgi/wiki?RelationsAsFirstClassObjects http://c2.com/cgi/wiki?RelationsAsFirstClassObjects - The query system for the Gemstone object database evaluates query blocks against proxies that capture the messages and translate the procedural queries into logical queries: myEmployees select: { :emp | emp.age > emp.lengthOfService } http://community.gemstone.com/download/attachments/6816350/GS64-ProgGuide-3.0.pdf http://community.gemstone.com/download/attachments/6816350/G...
lazyloop 13y ago
All that lambda and operator overloading magic makes the development of new drivers a lot more complicated than it has to be. And i wouldn't be surprised if some languages end up feeling like second class citizens in the RethinkDB eco system because of it.
- coffeemug 13y ago
  That's definitely true and is a big tradeoff. I think it's a good one because people use drivers much more frequently than they write them, so in retrospect I think it was a good decision (though the jury's still out). Interestingly, the server protocol doesn't require the use of lambdas in the driver (and knows nothing about it). So in languages that don't support them, one could design a driver with a very different feel that work for that language, and it would still work well with RethinkDB. You could even write a completely string-based driver, and it would work well. (Though we didn't think through what a good C driver for Rethink would look like; we really wanted to focus on Ruby, Python, and JavaScript and offer a great experience in those languages).
  - lazyloop 13y ago
    Sure, but the next problem is documentation, suddenly you have very different and incompatible examples for different languages. Possibly even between different drivers for the same language. MongoDB gets that part very right, you can apply pretty much all documentation to all drivers, with minimal changes.
- deleted 13y ago
  [deleted]
- dkhenry 13y ago
  When you consider the lambdas are really just sugar around chaining together some of the method primitives you don't _really_ have a second class citizen. I mean different languages give you different features so you _choose_ to work in java you might be limited to using just the primitives and not the full lambda syntax. However given your working in java and not scala you already made that decision ( BTW scala looks like it will support the lambda syntax really really well )
kbenson 13y ago
I was going to say that it didn't look to offer much more than what DBIC does for perl (in how it composes queries), but the operator overloading to do conditionals is really pretty slick.
crazygringo 13y ago
I like the idea of building queries via functions with jQuery-like chaining. However, the whole idea of overloading operators to pass in Python functions that somehow get converted... makes me feel uneasy, like it's too-clever-by-half -- having to keep track of which operators are actually supported, which client languages (Python, PHP, etc.) support it, if something's being messed up along the way... Is there an alternative string syntax for such functions, written in a ReQL-specific language that can be used instead?
- coffeemug 13y ago
  This is definitely a big concern -- we thought about this a lot (and still do). In practice, it turns out that most people pick ReQL up very easily and don't get confused or annoyed by these issues after getting acquainted with the language. That being said, we've toyed with the idea of having a string-based implementation, so people who are uneasy about these features could use the model they're already familiar with. (We didn't do this yet, but we likely will if there's enough demand for it)
  - crazygringo 13y ago
    Thanks. But, for example, how do you even do it in PHP pre-5.3, when anonymous functions became available? (I see PHP-RQL doesn't support pre-5.3). Or I guess the answer just is, you don't. So for now, ReQL won't really work with languages without anonymous functions. [Never mind, saw your reply about that below.]
- rogerbinns 13y ago
  Appengine NDB does the operator overloading too https://developers.google.com/appengine/docs/python/ndb/queries#filter_by_prop https://developers.google.com/appengine/docs/python/ndb/quer... Guido contributed to NDB.
DenisM 13y ago
So, it's like LINQ for Python?
rogerbinns 13y ago
I far prefer the MongoDB query syntax which is a JSON object. For example to find all last name smith you use {"last_name": "smith"}. To also find year less than 2001 {"last_name": "smith", "year": {"$lt": 2001}}. Since dictionaries are first class objects in Python this is very easy to work with. Accepting extras from the command line is an example: query={"last_name": "smith", "year": {"$lt": 2001}} query.update(json.loads(argv)) They do use the chaining for the non-query values bit (sort order, limits etc).

40 points by coffeemug 13y ago

As a user of RethinkDB in my new apps and also the author of a RethinkDB driver, I have to say that the query language is the absolute best I've worked with. I've been working with databases and writing queries for over 8 years now and ReQL is my favorite. Being able to compose queries in a language-native way is really effective but also powerful in that you are essentially running programs on the database that it understands enough to optimize and distribute amongst its fellow servers. Not only that, you really can marry your code with the forms you send to the database, making it easy to compose queries that are built from conditionals from local variables. Also because of the expressiveness, there are almost no places where I ever run two queries when selecting data. For instance in Mongo, I always used to do "get X. If X is this, then get Y, otherwise get Z." Now all that logic can be shipped off to the database (without having to build some huge SQL string) and when I get an answer back, it's the one I want. Great query language, great database, great team.

neumino 13y ago
You forgot great community! And you are part of it!
rogerbinns 13y ago
What does a query look like when you print it (to help with diagnostics and debugging)? I don't mean the results but rather what you have before calling run.
- orthecreedence 13y ago
  On my primary platform (common lisp) when printed it looks like a large set of nested structures made up of "terms" (ie "create_database", "table", "function", etc) and datums (array, float, kv pair, etc). It's easy to pretty-print and get a view of the entire query structure before it's shipped off the the protocol buffer serializer. That's lisp though, and the print methods for the protobuf objects are built into the protobuf library. It's really going to depend on your language, probably. The nice thing about the query language is that the mapping between what you send the driver and what the driver sends to the server is pretty much 1:1, so what you see in front of you is the printed query (save for bugs in the driver).

mpweiher 13y ago

This is pretty neat, I certainly like not having a separate query language embedded as strings and manipulated with string concatenation etc. (SQL injection ...). Similar: - Avi Bryant's Relational Object Expressions bring relational algebra into the programming language: courseTitlesForProfessorNamed: aString |courses| courses := (professors * courses select: [[:ea | (ea lastName = aString) & (ea id = ea professorID)]. ^ (courses project: #title) copyFrom: 1 to: 10. http://www.cincomsmalltalk.com/userblogs/avi/blogView?showComments=true&entry=3246121322 http://www.cincomsmalltalk.com/userblogs/avi/blogView?showCo...: and http://c2.com/cgi/wiki?RelationsAsFirstClassObjects http://c2.com/cgi/wiki?RelationsAsFirstClassObjects - The query system for the Gemstone object database evaluates query blocks against proxies that capture the messages and translate the procedural queries into logical queries: myEmployees select: { :emp | emp.age > emp.lengthOfService } http://community.gemstone.com/download/attachments/6816350/GS64-ProgGuide-3.0.pdf http://community.gemstone.com/download/attachments/6816350/G...

lazyloop 13y ago

All that lambda and operator overloading magic makes the development of new drivers a lot more complicated than it has to be. And i wouldn't be surprised if some languages end up feeling like second class citizens in the RethinkDB eco system because of it.

coffeemug 13y ago
That's definitely true and is a big tradeoff. I think it's a good one because people use drivers much more frequently than they write them, so in retrospect I think it was a good decision (though the jury's still out). Interestingly, the server protocol doesn't require the use of lambdas in the driver (and knows nothing about it). So in languages that don't support them, one could design a driver with a very different feel that work for that language, and it would still work well with RethinkDB. You could even write a completely string-based driver, and it would work well. (Though we didn't think through what a good C driver for Rethink would look like; we really wanted to focus on Ruby, Python, and JavaScript and offer a great experience in those languages).
- lazyloop 13y ago
  Sure, but the next problem is documentation, suddenly you have very different and incompatible examples for different languages. Possibly even between different drivers for the same language. MongoDB gets that part very right, you can apply pretty much all documentation to all drivers, with minimal changes.
deleted 13y ago
[deleted]
dkhenry 13y ago
When you consider the lambdas are really just sugar around chaining together some of the method primitives you don't _really_ have a second class citizen. I mean different languages give you different features so you _choose_ to work in java you might be limited to using just the primitives and not the full lambda syntax. However given your working in java and not scala you already made that decision ( BTW scala looks like it will support the lambda syntax really really well )

kbenson 13y ago

I was going to say that it didn't look to offer much more than what DBIC does for perl (in how it composes queries), but the operator overloading to do conditionals is really pretty slick.

crazygringo 13y ago

I like the idea of building queries via functions with jQuery-like chaining. However, the whole idea of overloading operators to pass in Python functions that somehow get converted... makes me feel uneasy, like it's too-clever-by-half -- having to keep track of which operators are actually supported, which client languages (Python, PHP, etc.) support it, if something's being messed up along the way... Is there an alternative string syntax for such functions, written in a ReQL-specific language that can be used instead?

coffeemug 13y ago
This is definitely a big concern -- we thought about this a lot (and still do). In practice, it turns out that most people pick ReQL up very easily and don't get confused or annoyed by these issues after getting acquainted with the language. That being said, we've toyed with the idea of having a string-based implementation, so people who are uneasy about these features could use the model they're already familiar with. (We didn't do this yet, but we likely will if there's enough demand for it)
- crazygringo 13y ago
  Thanks. But, for example, how do you even do it in PHP pre-5.3, when anonymous functions became available? (I see PHP-RQL doesn't support pre-5.3). Or I guess the answer just is, you don't. So for now, ReQL won't really work with languages without anonymous functions. [Never mind, saw your reply about that below.]
rogerbinns 13y ago
Appengine NDB does the operator overloading too https://developers.google.com/appengine/docs/python/ndb/queries#filter_by_prop https://developers.google.com/appengine/docs/python/ndb/quer... Guido contributed to NDB.

DenisM 13y ago

So, it's like LINQ for Python?

rogerbinns 13y ago

I far prefer the MongoDB query syntax which is a JSON object. For example to find all last name smith you use {"last_name": "smith"}. To also find year less than 2001 {"last_name": "smith", "year": {"$lt": 2001}}. Since dictionaries are first class objects in Python this is very easy to work with. Accepting extras from the command line is an example: query={"last_name": "smith", "year": {"$lt": 2001}} query.update(json.loads(argv)) They do use the chaining for the non-query values bit (sort order, limits etc).