3 ms·

Is anyone using RethinkDB as a "lightweight" Business Intelligence / Analytics / Datawharehouse store? (Maybe for use cases like Amazon Redshift?) It seems lik

by pquerna 12y ago

Is anyone using RethinkDB as a "lightweight" Business Intelligence / Analytics / Datawharehouse store? (Maybe for use cases like Amazon Redshift?) It seems like there could be a sweet spot of their nice query language, schemaless, and easy scaling for moderate size data sets? I'm kinda tired of having to go all in on a big hadoop-ecosystem just to figure out average X to Y in a dataset....

coffeemug 12y ago
Slava @ RethinkDB here. Some of our biggest users (to be announced in a few weeks for 2.0) use RethinkDB this way. You can't really do deep analytics/machine learning as RethinkDB wasn't designed for that, but if you want to store a lot of data, and then run lightweight aggregation or map-reduce queries on that data, Rethink turns out to be a really good product for it. One issue I see with this path is that if your queries ever get a lot more complex, you'd have to migrate off of RethinkDB onto Hadoop (which is a pain). I think that if you know for certain you just want lightweight querying capabilities RethinkDB can be really wonderful, but if there is a good chance you might need something deeper, it might be worth the effort to set up Hadoop early on.
- pquerna 12y ago
  Have you thought about a "read at timestamp" construct in RethinkDB? It's not really an MVCC thing, and you can work around it in data model, but for lots of reports (say, running in a cronjob), I want to run a query "as" the database saw things from at midnight UTC, even if i start running it at 2am? It would also make reports a more reproducible... but maybe this is really a datamodel problem. I felt when I read the Google Spanner papers that it was a pretty potentially useful feature for Read-Only queries.
  - coffeemug 12y ago
    Currently you'd have to do it in the data model as you would in any other database. It's a pretty cool idea, though -- I'll think about what we can do (though admittedly, this is a bit removed from the current direction).
  - emidln 12y ago
    I frequently use Datomic for this and it's awesome.
- JulianMorrison 12y ago
  That sounds like a job for the changes feed, pre-digest data with a query and then pipe its changes feed into Hadoop's storage. (How fast can change feeds run? Would that end up being a bottleneck?)

by pquerna 12y ago

coffeemug 12y ago

Slava @ RethinkDB here. Some of our biggest users (to be announced in a few weeks for 2.0) use RethinkDB this way. You can't really do deep analytics/machine learning as RethinkDB wasn't designed for that, but if you want to store a lot of data, and then run lightweight aggregation or map-reduce queries on that data, Rethink turns out to be a really good product for it. One issue I see with this path is that if your queries ever get a lot more complex, you'd have to migrate off of RethinkDB onto Hadoop (which is a pain). I think that if you know for certain you just want lightweight querying capabilities RethinkDB can be really wonderful, but if there is a good chance you might need something deeper, it might be worth the effort to set up Hadoop early on.

pquerna 12y ago
Have you thought about a "read at timestamp" construct in RethinkDB? It's not really an MVCC thing, and you can work around it in data model, but for lots of reports (say, running in a cronjob), I want to run a query "as" the database saw things from at midnight UTC, even if i start running it at 2am? It would also make reports a more reproducible... but maybe this is really a datamodel problem. I felt when I read the Google Spanner papers that it was a pretty potentially useful feature for Read-Only queries.
- coffeemug 12y ago
  Currently you'd have to do it in the data model as you would in any other database. It's a pretty cool idea, though -- I'll think about what we can do (though admittedly, this is a bit removed from the current direction).
- emidln 12y ago
  I frequently use Datomic for this and it's awesome.
JulianMorrison 12y ago
That sounds like a job for the changes feed, pre-digest data with a query and then pipe its changes feed into Hadoop's storage. (How fast can change feeds run? Would that end up being a bottleneck?)