4 ms·

I think the problem being described here is he is using a database where he doesn't need to. This is the problem things like cassandra were made to solve. Effec

by dkhenry 15y ago

I think the problem being described here is he is using a database where he doesn't need to. This is the problem things like cassandra were made to solve. Effectivally he is not storing things to the DB he is logging them to the DB in case of a program crash. To speed things up he stores his entire data set in RAM client side. Writes are buffered and written to the DB, after they have modified the client side data. I assume the point is when the program crashes or dies it just rebuilds the client side data from the DB and then proceed to continue to log to the DB. He can get rid of a lot of his code and add in features but using something like MongoDB, Casandra , or Riak

willvarfar 15y ago
I might trust Riak's reputation, but it also has a reputation for being slow. MongoDB and Cassandra both seem immature and unstable at the moment; all you hear is horror stories.
- rkalla 15y ago
  I don't think "all you hear" is accurate. Mongo and Cassandra have some truly huge deployments and the "horror stories" are a function of the number of deployments. Search "MySQL sucks" or "Oracle sucks" and you'll find much the same thing. It is a result of being (successfully) deployed so many places. As an aside, if you want confidence in your NoSQL solution, reading either the Mongo or Cassandra mailing lists is an excellent way to realize how focused, efficient and powerful those companies and communities are. Of all the NoSQL db lists I read, Cassandra is one of the lowest incident levels and when there is, no matter how esoteric, it seems like one of the Datastax guys or someone from the community is able to help them out. As for Mongo, the fact that Eliot and team still replies 100s of times a day to all the questions, even complex reports, is beyond impressive to me. I am just trying to frame your comment a bit; these projects have left the "cool, large-scale toy" category a long time ago and have many companies better their existence on them. Netflix currently has one of the largest global Cassandra deployments (US east-west and UK) and continue to move out of SimpleDB and move more and more into their own custom deployment.
  - willvarfar 15y ago
    Fair enough, they may have come along way. I worked on that project with a chap who happend to also be a cassandra contributor, but we still didn't dare go with cassandra back then. And mongodb feels more squid than varnish; its just an uneasy feeling I get when I follow their crazy technical journeys ;)
    - rkalla 15y ago
      Just to clarify; I am not saying you are wrong and trying to talk you into it, just didn't want other readers to come across that comment and throw them out the window especially if they are a good fit. That being said, I am a big believer in gut feelings and if yours is telling you the answer is elsewhere then it is probably right.
jamii 15y ago
> Effectivally he is not storing things to the DB he is logging them to the DB in case of a program crash. To speed things up he stores his entire data set in RAM client side. At this point its not worth even using a database. I'm currently working on an experimental framework at Smarkets where we just write events directly to disk and do everything else in memory (with the occasional snapshot to speed recovery). If all goes well it will be open-sourced in a few weeks. LMAX have taken this idea to extremes and have some mind-blowing benchmarks - http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html
- willvarfar 15y ago
  I think the merging and reordering stands alone as a solution; the extension where, in my particular project, I cache to avoid some common SELECTs is a distinct other subject. They complement each other, and are in the same post, is all.

by dkhenry 15y ago

willvarfar 15y ago

I might trust Riak's reputation, but it also has a reputation for being slow. MongoDB and Cassandra both seem immature and unstable at the moment; all you hear is horror stories.

rkalla 15y ago
I don't think "all you hear" is accurate. Mongo and Cassandra have some truly huge deployments and the "horror stories" are a function of the number of deployments. Search "MySQL sucks" or "Oracle sucks" and you'll find much the same thing. It is a result of being (successfully) deployed so many places. As an aside, if you want confidence in your NoSQL solution, reading either the Mongo or Cassandra mailing lists is an excellent way to realize how focused, efficient and powerful those companies and communities are. Of all the NoSQL db lists I read, Cassandra is one of the lowest incident levels and when there is, no matter how esoteric, it seems like one of the Datastax guys or someone from the community is able to help them out. As for Mongo, the fact that Eliot and team still replies 100s of times a day to all the questions, even complex reports, is beyond impressive to me. I am just trying to frame your comment a bit; these projects have left the "cool, large-scale toy" category a long time ago and have many companies better their existence on them. Netflix currently has one of the largest global Cassandra deployments (US east-west and UK) and continue to move out of SimpleDB and move more and more into their own custom deployment.
- willvarfar 15y ago
  Fair enough, they may have come along way. I worked on that project with a chap who happend to also be a cassandra contributor, but we still didn't dare go with cassandra back then. And mongodb feels more squid than varnish; its just an uneasy feeling I get when I follow their crazy technical journeys ;)
  - rkalla 15y ago
    Just to clarify; I am not saying you are wrong and trying to talk you into it, just didn't want other readers to come across that comment and throw them out the window especially if they are a good fit. That being said, I am a big believer in gut feelings and if yours is telling you the answer is elsewhere then it is probably right.

jamii 15y ago

> Effectivally he is not storing things to the DB he is logging them to the DB in case of a program crash. To speed things up he stores his entire data set in RAM client side. At this point its not worth even using a database. I'm currently working on an experimental framework at Smarkets where we just write events directly to disk and do everything else in memory (with the occasional snapshot to speed recovery). If all goes well it will be open-sourced in a few weeks. LMAX have taken this idea to extremes and have some mind-blowing benchmarks - http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html

willvarfar 15y ago
I think the merging and reordering stands alone as a solution; the extension where, in my particular project, I cache to avoid some common SELECTs is a distinct other subject. They complement each other, and are in the same post, is all.