4 ms·

Why Flowdock migrated from Cassandra to MongoDB

43 points by mutru 16y ago

jbellis 16y ago
Note that flowdock was on the obsolete Cassandra 0.5; the little they described of their problems suggested that upgrading to the 0.6 series would solve the worst of them.
- mutru 16y ago
  Actually, we were running Cassandra 0.6.1. I'm a bit sorry that we didn't try to upgrade to 0.6.3, because it could have fixed some of the problems.
- ergo98 16y ago
  I have noticed a disturbing trend used in the defense of many NoSQL products --> When faced with criticism point out that there is version n+1 now available and they should have used that, it would have fixed everything, etc. We've seen this, ironically, in reference to many MongoDB horror stories, so it's interesting that in this case it's the case where someone is moving to MongoDB.
  - akmiller 16y ago
    To be fair, many of these products are in their infancy so yes in general the fix may be in version n+1. I don't think that's a defense of NoSQL products it's a common thing with any young product. Anyone who chooses an early product would hopefully understand that they will be experiencing issues and will be upgrading the product many times as issues are found and fixed.
  - jbellis 16y ago
    Cassandra has a lot more installations pushing hard on it in 0.6 than it did in 0.5, many of whom have worked with me and other committers to resolve their issues, so it's not just paint-by-numbers handwaving on my part. Here are some of the changelog entries from 0.6.2 that are relevant to high-volume sites: - improve default JVM GC options (CASSANDRA-1014) - better defaults for flush sorter + writer executor queue sizes (CASSANDRA-1100) - added Collections.shuffle(splits) before returning the splits in ColumnFamilyInputFormat (CASSANDRA-1096) - StreamingService.StreamDestinations wouldn't empty itself after streaming finished (CASSANDRA-1076) - Add option to turn off Hinted Handoff (CASSANDRA-894) - split CommitLogExecutorService into BatchCommitLogExecutorService and PeriodicCommitLogExecutorService (CASSANDRA-1014) from 0.6.3: - add ability to lower compaction priority (CASSANDRA-1181) ... and 0.6.4: - extend option to lower compaction priority to hinted handoff as well (CASSANDRA-1260) - pre-emptively drop requests that cannot be processed within RPCTimeout (CASSANDRA-685) If that's so much greek to you, I'll be doing explanations of these over on the Riptano blog in the next few days, like the one I did for 0.6.1 here: http://riptano.com/blog/annotated-changelog-cassandra-061 http://riptano.com/blog/annotated-changelog-cassandra-061 (sorry for excessive spacing. how do you get HN to format bullets?)
tszming 16y ago
In you article, you mentioned 'Dots are not allowed in BSON document keys. Typically it might not be a problem, but we had to work around it in our data migration.' Do you have more information?
- mutru 16y ago
  In MongoDB this is not possible: > db.test.insert({"foo.bar": 1}); Mon Jul 26 14:24:39 JS Error: uncaught exception: can't have . in field names [foo.bar] After all that's only natural, because the dot notation is supported in queries. For example, consider storing an e-mail in MongoDB: { content: "<p>HTML mail. <img src="cid:1234.0" alt="from attachment"/></p>", attachments: { "1234.0": { some details about the attachment } } } This is not legal, but in this case it's of course easy to work around (have attachments in an array and have the CID as a value instead of a key).
  - tszming 16y ago
    Thanks, this is interesting.
oldgregg 16y ago
Clearly articles like this are designed to promote the service rather have any substantive "hacker" news. That said, the front page looks compelling except for one thing -- NO PRICING. It doesn't matter if it's still in beta. You are asking me to commit to a completely new workflow -- it's not worth my time unless I have some idea how much you are going to hit me for once it comes out of beta. $9/mo? $99/mo? I mean c'mon, give me something...
superchink 16y ago
Quick question: is the service meant to work in the latest DEV release of Chrome? I just get a blank screen: http://cl.ly/92a3f95975927920868b http://cl.ly/92a3f95975927920868b
- mutru 16y ago
  Indeed, latest development version of Chrome doesn't work. We found and fixed the bug earlier today, the fix will be deployed later this week.

43 points by mutru 16y ago

Note that flowdock was on the obsolete Cassandra 0.5; the little they described of their problems suggested that upgrading to the 0.6 series would solve the worst of them.

mutru 16y ago
Actually, we were running Cassandra 0.6.1. I'm a bit sorry that we didn't try to upgrade to 0.6.3, because it could have fixed some of the problems.
ergo98 16y ago
I have noticed a disturbing trend used in the defense of many NoSQL products --> When faced with criticism point out that there is version n+1 now available and they should have used that, it would have fixed everything, etc. We've seen this, ironically, in reference to many MongoDB horror stories, so it's interesting that in this case it's the case where someone is moving to MongoDB.
- akmiller 16y ago
  To be fair, many of these products are in their infancy so yes in general the fix may be in version n+1. I don't think that's a defense of NoSQL products it's a common thing with any young product. Anyone who chooses an early product would hopefully understand that they will be experiencing issues and will be upgrading the product many times as issues are found and fixed.
- jbellis 16y ago
  Cassandra has a lot more installations pushing hard on it in 0.6 than it did in 0.5, many of whom have worked with me and other committers to resolve their issues, so it's not just paint-by-numbers handwaving on my part. Here are some of the changelog entries from 0.6.2 that are relevant to high-volume sites: - improve default JVM GC options (CASSANDRA-1014) - better defaults for flush sorter + writer executor queue sizes (CASSANDRA-1100) - added Collections.shuffle(splits) before returning the splits in ColumnFamilyInputFormat (CASSANDRA-1096) - StreamingService.StreamDestinations wouldn't empty itself after streaming finished (CASSANDRA-1076) - Add option to turn off Hinted Handoff (CASSANDRA-894) - split CommitLogExecutorService into BatchCommitLogExecutorService and PeriodicCommitLogExecutorService (CASSANDRA-1014) from 0.6.3: - add ability to lower compaction priority (CASSANDRA-1181) ... and 0.6.4: - extend option to lower compaction priority to hinted handoff as well (CASSANDRA-1260) - pre-emptively drop requests that cannot be processed within RPCTimeout (CASSANDRA-685) If that's so much greek to you, I'll be doing explanations of these over on the Riptano blog in the next few days, like the one I did for 0.6.1 here: http://riptano.com/blog/annotated-changelog-cassandra-061 http://riptano.com/blog/annotated-changelog-cassandra-061 (sorry for excessive spacing. how do you get HN to format bullets?)

tszming 16y ago

In you article, you mentioned 'Dots are not allowed in BSON document keys. Typically it might not be a problem, but we had to work around it in our data migration.' Do you have more information?

mutru 16y ago
In MongoDB this is not possible: > db.test.insert({"foo.bar": 1}); Mon Jul 26 14:24:39 JS Error: uncaught exception: can't have . in field names [foo.bar] After all that's only natural, because the dot notation is supported in queries. For example, consider storing an e-mail in MongoDB: { content: "<p>HTML mail. <img src="cid:1234.0" alt="from attachment"/></p>", attachments: { "1234.0": { some details about the attachment } } } This is not legal, but in this case it's of course easy to work around (have attachments in an array and have the CID as a value instead of a key).
- tszming 16y ago
  Thanks, this is interesting.

oldgregg 16y ago

Clearly articles like this are designed to promote the service rather have any substantive "hacker" news. That said, the front page looks compelling except for one thing -- NO PRICING. It doesn't matter if it's still in beta. You are asking me to commit to a completely new workflow -- it's not worth my time unless I have some idea how much you are going to hit me for once it comes out of beta. $9/mo? $99/mo? I mean c'mon, give me something...

superchink 16y ago

Quick question: is the service meant to work in the latest DEV release of Chrome? I just get a blank screen: http://cl.ly/92a3f95975927920868b http://cl.ly/92a3f95975927920868b

mutru 16y ago
Indeed, latest development version of Chrome doesn't work. We found and fixed the bug earlier today, the fix will be deployed later this week.