4 ms·

It's great to see an amazing product like RethinkDB move so quickly and improve so much The counter-argument is that they can move so quickly and improve so mu

It's great to see an amazing product like RethinkDB move so quickly and improve so much The counter-argument is that they can move so quickly and improve so much because so much is so poorly implemented. While it may seem cynical, I have formed a habit of eliminating projects that still see magnitudes performance improvements from such rudimentary activities. Once they get to the point where a yearly release are a couple of syntax improvements and some minor speedups, it is more likely to be production ready.

gruseom 13y ago
You have a point in general, because that is how these things usually go, but these guys spent like three years working on the foundation for what they're doing now. This level of rapid improvement may be the visible payoff of a lot of careful design and engineering.
- andrewflnr 13y ago
  Speaking of which, for the rethink people on the thread, how much of old "key-value store for SSDs" code is still active? Is the document store at all a layer on top of that?
  - SamReidHughes 13y ago
    Most of that code still exists, extensively modified. Here are some things that don't: - Talking to files with libaio -- we now only support use of blocking I/O calls in a thread pool. - Use of block devices instead of files, and code optimized for older SSDs that spread writes to different parts of the disk. - A fsck utility and a utility for extracting data from a corrupted file. Most other stuff has been rewritten and refactored, though -- RethinkDB was once implemented entirely using callbacks on an epoll loop. Then we introduced coroutines (cooperative green threads that sit atop an epoll loop) into the codebase. Nowadays almost everything is implemented in terms of coroutines -- there aren't many APIs left that take a callback and say it'll get called sometime later. There's even still a secret memcache interface. The storage engine itself does a bit more now than it used to. Support for secondary indexes, MVCC, efficiently bringing back out-of-date replicas, and better on-disk storage are the main ones I can think of at this hour.
coffeemug 13y ago
Overall, you're right, but I'd point out that this is by design. In the past two years our top two goals (hopefully like any high-tech startup) were to build a product that electrifies people and has a solid architecture. The reasoning was that if people don't want the product nothing else matters, and if people do want the product but the architecture doesn't allow rapid changes, we'll lose. Really good implementations of specific features was priority number three, and while we did a lot of that too, it sometimes took a backseat in light of the first two. That's ok though -- now that we have a rapidly growing user base and a solid technical architecture we can go back and replace placeholder implementations with better ones according to user demand. We're not entirely happy with all our choices, but no one ever is. I think that the overall direction worked out really really well for RethinkDB and its users. EDIT: gruseom and mrkurt beat me to it.
mrkurt 13y ago
Rethink is more deliberately scoped small than it is poorly implemented. Their roadmap has all sorts of known performance improvements on it, and I greatly respect their decision to keep things as simple as possible and deliberately defer some areas of complexity.

gruseom 13y ago

You have a point in general, because that is how these things usually go, but these guys spent like three years working on the foundation for what they're doing now. This level of rapid improvement may be the visible payoff of a lot of careful design and engineering.

andrewflnr 13y ago
Speaking of which, for the rethink people on the thread, how much of old "key-value store for SSDs" code is still active? Is the document store at all a layer on top of that?
- SamReidHughes 13y ago
  Most of that code still exists, extensively modified. Here are some things that don't: - Talking to files with libaio -- we now only support use of blocking I/O calls in a thread pool. - Use of block devices instead of files, and code optimized for older SSDs that spread writes to different parts of the disk. - A fsck utility and a utility for extracting data from a corrupted file. Most other stuff has been rewritten and refactored, though -- RethinkDB was once implemented entirely using callbacks on an epoll loop. Then we introduced coroutines (cooperative green threads that sit atop an epoll loop) into the codebase. Nowadays almost everything is implemented in terms of coroutines -- there aren't many APIs left that take a callback and say it'll get called sometime later. There's even still a secret memcache interface. The storage engine itself does a bit more now than it used to. Support for secondary indexes, MVCC, efficiently bringing back out-of-date replicas, and better on-disk storage are the main ones I can think of at this hour.

coffeemug 13y ago

Overall, you're right, but I'd point out that this is by design. In the past two years our top two goals (hopefully like any high-tech startup) were to build a product that electrifies people and has a solid architecture. The reasoning was that if people don't want the product nothing else matters, and if people do want the product but the architecture doesn't allow rapid changes, we'll lose. Really good implementations of specific features was priority number three, and while we did a lot of that too, it sometimes took a backseat in light of the first two. That's ok though -- now that we have a rapidly growing user base and a solid technical architecture we can go back and replace placeholder implementations with better ones according to user demand. We're not entirely happy with all our choices, but no one ever is. I think that the overall direction worked out really really well for RethinkDB and its users. EDIT: gruseom and mrkurt beat me to it.

mrkurt 13y ago

Rethink is more deliberately scoped small than it is poorly implemented. Their roadmap has all sorts of known performance improvements on it, and I greatly respect their decision to keep things as simple as possible and deliberately defer some areas of complexity.