7 ms·

Bye, MongoDB. Hello, Cloudant

150 points by chinmoy 14y ago

d0m 14y ago
Cloudant seems interesting, never heard of them before. The pricing is intriguing. (https://cloudant.com/pricing/ https://cloudant.com/pricing/) From what I understand, it's free for most small projects but could get very pricey very fast, but then, handling millions of customers is far from being trivial to handle so I'm sure it's worth the price. Thoughts about the Cloudant pricing versus in-house solution (or vps)?
- ahoff 14y ago
  Our pricing as stated on the page can be somewhat confusing. We're working on making it both less expensive and more transparent to the end user.
- pooriaazimi 14y ago
  IrisCouch's pricing is also interesting: http://www.iriscouch.com/service http://www.iriscouch.com/service
- amaris1113 14y ago
  I might also note, thats its worth contact Cloudant directly if you are working on a larger scale production app. We have the metered pricing listed on the website and then dedicated which is an "all you can eat" style service and you pay a flat monthly rate based on how many nodes are in your cluster. That is the system that Postmark is using along with most of our other bigger customers.
josh2600 14y ago
We run BigCouch from Cloudant and it's amazing. They are awesome folks and spoke at our conference last year. https://github.com/cloudant/bigcouch https://github.com/cloudant/bigcouch Disclaimer: I'm the community manager for http://www.2600hz.com http://www.2600hz.com
- rdtsc 14y ago
  I love CouchDB. There was talk on the mailing list and IRC channel about eventually merging in BigCouch with CouchDB code base. That would be awesome.
  - mlmilleratmit 14y ago
    That's well underway and planned for the 2.0 release of CouchDB.
    - clark-kent 14y ago
      Can't wait!
      - fidanov 14y ago
        WoW! Really good news.
    - obiterdictum 14y ago
      Does it include chained map-reduce?
  - mlmilleratmit 14y ago
    That's well underway and planned for the 2.0 release of CouchDB.
chrissharkey 14y ago
We were running our own Bigcouch cluster and having all sorts of issues scaling to large numbers of databases (50k+), but after moving to Cloudant it handled it just fine. It does cost more, but we don't worry about our database now and can focus on what we're actually good at. Their search feature seems to be pretty good too. It is based on Lucene and is very flexible! Chris Sharkey, Bislr
gingerlime 14y ago
Interesting to find out what was going on behind the scenes at postmark. As a (albeit small) customer, we were noticing something was going on, but not exactly sure what. It's great to see the postmark guys sharing the info and being transparent. Overall I really like the service despite the recent issues. As far as cloudant, I also never heard of it. Trying to go through the website, is this effectively a managed CouchDB in the cloud? Also, just wondering what's the migration path out of it (or who provides alternative service) just in case something doesn't work out.
- mlmilleratmit 14y ago
  Cloudant sits under the Apache CouchDB API, so the easiest path for migration in/out of Cloudant is to use replication to/from another CouchDB-speaking service like Cloudant, Apache CouchDB, TouchDB (mobile), PouchDB (browser). For most databases, you can also get a JSON dump of the document bodies by doing: curl 'https://username.cloudant.com/dbname/_changes https://username.cloudant.com/dbname/_changes | gzip > dbname.json.gz. The tricker part of course is understanding how to best model data as you move between different database systems. There are some nice tools emerging in the no/new-sql ecosystem to help with that. Here's one that some of our mysql/postgres converted customers seem to like: http://www.foxweave.com/moving-on-premise-data-to-the-cloud-postgressql-to-cloudant/ http://www.foxweave.com/moving-on-premise-data-to-the-cloud-.... My understanding is that it works in both directions.
- cnagele 14y ago
  Chris from Postmark here. We definitely had some trouble over the last few weeks. I am hoping we are beyond that in terms of architecture. So far everything looks great. Regarding lock-in, this was pretty important to us when we decided to migrate. We prefer to control everything in our data center, so if it came down to it, we wanted that option. Since the Cloudant API is compatible with CouchDB, the migration path is not that hard. Although, considering our impression so far and their expertise, I don't expect it will come to that point.
  - gingerlime 14y ago
    Thanks for the info Chris. I certainly hope you won't have to make another big move, but vendor lock-in is always a consideration. Also, I'm curious about network/other latency and bandwidth limits when it comes to accessing cloud-based database services. Did you do any measurements on that before or after switching? ...and regards to Natalie, Dana and the rest of the postmark team! EDIT: noticed info about latency from cloudant on another comment.
thesis 14y ago
Can anyone share how latency is overcome using a cloud based database?
- jameswyse 14y ago
  It's not a problem if they're both in the same data center, not sure who cloudant are using though.
  - mlmilleratmit 14y ago
    (Mike Miller, Cloudant). Exactly, we run cloudant clusters in nearly 20 data centers globally on many different providers (ec2, softlayer, rackspace, joyent, azure, hostway, ...) so we make sure that the data and application tiers are co-located. Dealing with direct connections from mobile is more fun.
    - dmix 14y ago
      You should address this more clearly on your sales website. It's the first question people will have regarding using a database-as-a-service. So mentioning the data centers you are in would be helpful.
      - mlmilleratmit 14y ago
        You're right, we should address that more clearly. Thanks for the feedback.
        taligent 14y ago
        Seriously do. I spent a few minutes looking for the list of data centres. And when I couldn't find it I left with the assumption that it was going to be a slow, pricey mess. Nothing is more important than transparency when you are providing core infrastructure.
      - taligent 14y ago
        Actually you know what would be even better. I provide an IP address and you tell me the best, worst and median response times for a simple request.
benmccann 14y ago
I really wish they had been more specific about the problems they faced. It sounds like part of the headache was that they had to write their own MongoDB to ElasticSearch connector. However, 10gen has now written one which they could use to avoid writing and maintaining their own: http://blog.mongodb.org/post/29127828146/introducing-mongo-connector http://blog.mongodb.org/post/29127828146/introducing-mongo-c...
- cnagele 14y ago
  Hi Ben, the ES connector was part of it. At a certain point we had our custom oplog follower working well. After so many years working with MongoDB, we just lost trust in it, to be honest. We ran into so many bugs, replication issues (two secondaries and no primary) and the only way to get what we needed out of it was to use all SSD storage (Even flashcache did not work). I have to admit that this was for our use case, and I know that MongoDB works extremely well for some people. We still use it in Beanstalk for our deployment logs and have no complaints. It's a much smaller dataset though. Really nice to see that 10gen made an ES connector. ES is still young, but it's amazing. One thing that I always respected about 10gen is how they maintain all of the drivers and code.
neya 14y ago
Is this going to become a trend here? Some random start-up writing "Bye-bye <insert database name>" or "Why we moved from <db1 name> to <db2 name>"??
- georgemcbay 14y ago
  Going to become a trend? Posts like this have been fairly common for years now.
  - neya 14y ago
    No, I meant the frequency of such posts have increased in the past few months..
    - taligent 14y ago
      NoSQL databases are becoming very common place now and there are a lot more companies and hosted providers than there used to be. So naturally you are going to see more stories of people switching away from traditional SQL databases. I work for a large enterprise who previously was an Oracle shop and even they are starting to using MongoDB and CouchDB internally.
      - neya 14y ago
        Yeah, maybe you're right.. >I work for a large enterprise who previously was an Oracle shop and even they are starting to using MongoDB and CouchDB internally. Whoa..that's a pretty serious claim..seriously? If yes, then I think it really is the start of a new revolution...
        lwat 14y ago
        They're not shutting down their Oracle DBs though. Or at least I hope not.
        taligent 14y ago
pbbakkum 14y ago
On Mongo: "we were not big fans of the master/slave architecture" I don't quite understand this, that architecture was basically deprecated a while back, can you elaborate?
- cnagele 14y ago
  Sorry, I need to correct that. I meant that shards are basically still primary / secondary with an arbiter for failover. We prefer horizontal nodes instead, like Cloudant, Elastic Search or Cassandra offers.
  - pbbakkum 14y ago
    Ahh, I see. Why do you guys prefer that configuration? Space/resources, reliability?
    - kapilvt 14y ago
      probably large amounts of writes.
    - mlmilleratmit 14y ago
      Space/resources was certainly a concern in this case. A disk-based solution with predictable latencies enables storage of far more data than something that requires holding the working set in RAM.
- philfreo 14y ago
  Can you elaborate on what you mean by it was deprecated a while back?
nsainaney 14y ago
We looked at MongoDB for our back end at http://moj.io http://moj.io and preferred CouchDB. It's currently a toss up between Iriscouch and Cloudant. Both seem like great offerings. We are particularly interested in the ability to sync to CouchDB on iOS/Android/Windows Phone in the future. Does anyone prefer one service over the other?
- deleted 14y ago
  [deleted]
pshirishreddy 14y ago
the whole blog post has a single line which explains why they moved away from mongodb "At that point we still had some issues and bugs that would appear in MongoDB and we were not big fans of the master/slave architecture, so we decided to move on." Simply not being a fan of architecture cannot be a reason. Only wish they had elaborated on problems more
zapfmann 14y ago
Grr.. I can't figure if you can deploy cloudant in an on ec2 avail zone that you have your app in. Can you you even deploy it on ec2? I don't see anything stating answers to these questions clearly on their site. And they want my email to let me see tech docs! Are they run by former IBM staff? Finally, if they are 'hosted and managed all day everyday' some place far from me. I dunno if I want to use it.
- zapfmann 14y ago
  Ah. Spoke too soon. The comments here lead me to believe cloudant is to be used as a couchdb replica provider. Maybe cloudant should say that on their pages! :-)

150 points by chinmoy 14y ago

d0m 14y ago

Cloudant seems interesting, never heard of them before. The pricing is intriguing. (https://cloudant.com/pricing/ https://cloudant.com/pricing/) From what I understand, it's free for most small projects but could get very pricey very fast, but then, handling millions of customers is far from being trivial to handle so I'm sure it's worth the price. Thoughts about the Cloudant pricing versus in-house solution (or vps)?

ahoff 14y ago
Our pricing as stated on the page can be somewhat confusing. We're working on making it both less expensive and more transparent to the end user.
pooriaazimi 14y ago
IrisCouch's pricing is also interesting: http://www.iriscouch.com/service http://www.iriscouch.com/service
amaris1113 14y ago
I might also note, thats its worth contact Cloudant directly if you are working on a larger scale production app. We have the metered pricing listed on the website and then dedicated which is an "all you can eat" style service and you pay a flat monthly rate based on how many nodes are in your cluster. That is the system that Postmark is using along with most of our other bigger customers.

josh2600 14y ago

We run BigCouch from Cloudant and it's amazing. They are awesome folks and spoke at our conference last year. https://github.com/cloudant/bigcouch https://github.com/cloudant/bigcouch Disclaimer: I'm the community manager for http://www.2600hz.com http://www.2600hz.com

rdtsc 14y ago
I love CouchDB. There was talk on the mailing list and IRC channel about eventually merging in BigCouch with CouchDB code base. That would be awesome.
- mlmilleratmit 14y ago
  That's well underway and planned for the 2.0 release of CouchDB.
  - clark-kent 14y ago
    Can't wait!
    - fidanov 14y ago
      WoW! Really good news.
  - obiterdictum 14y ago
    Does it include chained map-reduce?
- mlmilleratmit 14y ago
  That's well underway and planned for the 2.0 release of CouchDB.

chrissharkey 14y ago

We were running our own Bigcouch cluster and having all sorts of issues scaling to large numbers of databases (50k+), but after moving to Cloudant it handled it just fine. It does cost more, but we don't worry about our database now and can focus on what we're actually good at. Their search feature seems to be pretty good too. It is based on Lucene and is very flexible! Chris Sharkey, Bislr

gingerlime 14y ago

Interesting to find out what was going on behind the scenes at postmark. As a (albeit small) customer, we were noticing something was going on, but not exactly sure what. It's great to see the postmark guys sharing the info and being transparent. Overall I really like the service despite the recent issues. As far as cloudant, I also never heard of it. Trying to go through the website, is this effectively a managed CouchDB in the cloud? Also, just wondering what's the migration path out of it (or who provides alternative service) just in case something doesn't work out.

mlmilleratmit 14y ago
Cloudant sits under the Apache CouchDB API, so the easiest path for migration in/out of Cloudant is to use replication to/from another CouchDB-speaking service like Cloudant, Apache CouchDB, TouchDB (mobile), PouchDB (browser). For most databases, you can also get a JSON dump of the document bodies by doing: curl 'https://username.cloudant.com/dbname/_changes https://username.cloudant.com/dbname/_changes | gzip > dbname.json.gz. The tricker part of course is understanding how to best model data as you move between different database systems. There are some nice tools emerging in the no/new-sql ecosystem to help with that. Here's one that some of our mysql/postgres converted customers seem to like: http://www.foxweave.com/moving-on-premise-data-to-the-cloud-postgressql-to-cloudant/ http://www.foxweave.com/moving-on-premise-data-to-the-cloud-.... My understanding is that it works in both directions.
cnagele 14y ago
Chris from Postmark here. We definitely had some trouble over the last few weeks. I am hoping we are beyond that in terms of architecture. So far everything looks great. Regarding lock-in, this was pretty important to us when we decided to migrate. We prefer to control everything in our data center, so if it came down to it, we wanted that option. Since the Cloudant API is compatible with CouchDB, the migration path is not that hard. Although, considering our impression so far and their expertise, I don't expect it will come to that point.
- gingerlime 14y ago
  Thanks for the info Chris. I certainly hope you won't have to make another big move, but vendor lock-in is always a consideration. Also, I'm curious about network/other latency and bandwidth limits when it comes to accessing cloud-based database services. Did you do any measurements on that before or after switching? ...and regards to Natalie, Dana and the rest of the postmark team! EDIT: noticed info about latency from cloudant on another comment.

thesis 14y ago

Can anyone share how latency is overcome using a cloud based database?

jameswyse 14y ago
It's not a problem if they're both in the same data center, not sure who cloudant are using though.
- mlmilleratmit 14y ago
  (Mike Miller, Cloudant). Exactly, we run cloudant clusters in nearly 20 data centers globally on many different providers (ec2, softlayer, rackspace, joyent, azure, hostway, ...) so we make sure that the data and application tiers are co-located. Dealing with direct connections from mobile is more fun.
  - dmix 14y ago
    You should address this more clearly on your sales website. It's the first question people will have regarding using a database-as-a-service. So mentioning the data centers you are in would be helpful.
    - mlmilleratmit 14y ago
      You're right, we should address that more clearly. Thanks for the feedback.
      - taligent 14y ago
        Seriously do. I spent a few minutes looking for the list of data centres. And when I couldn't find it I left with the assumption that it was going to be a slow, pricey mess. Nothing is more important than transparency when you are providing core infrastructure.
    - taligent 14y ago
      Actually you know what would be even better. I provide an IP address and you tell me the best, worst and median response times for a simple request.

benmccann 14y ago

I really wish they had been more specific about the problems they faced. It sounds like part of the headache was that they had to write their own MongoDB to ElasticSearch connector. However, 10gen has now written one which they could use to avoid writing and maintaining their own: http://blog.mongodb.org/post/29127828146/introducing-mongo-connector http://blog.mongodb.org/post/29127828146/introducing-mongo-c...

cnagele 14y ago
Hi Ben, the ES connector was part of it. At a certain point we had our custom oplog follower working well. After so many years working with MongoDB, we just lost trust in it, to be honest. We ran into so many bugs, replication issues (two secondaries and no primary) and the only way to get what we needed out of it was to use all SSD storage (Even flashcache did not work). I have to admit that this was for our use case, and I know that MongoDB works extremely well for some people. We still use it in Beanstalk for our deployment logs and have no complaints. It's a much smaller dataset though. Really nice to see that 10gen made an ES connector. ES is still young, but it's amazing. One thing that I always respected about 10gen is how they maintain all of the drivers and code.

neya 14y ago

Is this going to become a trend here? Some random start-up writing "Bye-bye <insert database name>" or "Why we moved from <db1 name> to <db2 name>"??

georgemcbay 14y ago
Going to become a trend? Posts like this have been fairly common for years now.
- neya 14y ago
  No, I meant the frequency of such posts have increased in the past few months..
  - taligent 14y ago
    NoSQL databases are becoming very common place now and there are a lot more companies and hosted providers than there used to be. So naturally you are going to see more stories of people switching away from traditional SQL databases. I work for a large enterprise who previously was an Oracle shop and even they are starting to using MongoDB and CouchDB internally.
    - neya 14y ago
      Yeah, maybe you're right.. >I work for a large enterprise who previously was an Oracle shop and even they are starting to using MongoDB and CouchDB internally. Whoa..that's a pretty serious claim..seriously? If yes, then I think it really is the start of a new revolution...
      - lwat 14y ago
        They're not shutting down their Oracle DBs though. Or at least I hope not.
        taligent 14y ago

pbbakkum 14y ago

On Mongo: "we were not big fans of the master/slave architecture" I don't quite understand this, that architecture was basically deprecated a while back, can you elaborate?

cnagele 14y ago
Sorry, I need to correct that. I meant that shards are basically still primary / secondary with an arbiter for failover. We prefer horizontal nodes instead, like Cloudant, Elastic Search or Cassandra offers.
- pbbakkum 14y ago
  Ahh, I see. Why do you guys prefer that configuration? Space/resources, reliability?
  - kapilvt 14y ago
    probably large amounts of writes.
  - mlmilleratmit 14y ago
    Space/resources was certainly a concern in this case. A disk-based solution with predictable latencies enables storage of far more data than something that requires holding the working set in RAM.
philfreo 14y ago
Can you elaborate on what you mean by it was deprecated a while back?

nsainaney 14y ago

We looked at MongoDB for our back end at http://moj.io http://moj.io and preferred CouchDB. It's currently a toss up between Iriscouch and Cloudant. Both seem like great offerings. We are particularly interested in the ability to sync to CouchDB on iOS/Android/Windows Phone in the future. Does anyone prefer one service over the other?

deleted 14y ago
[deleted]

pshirishreddy 14y ago

the whole blog post has a single line which explains why they moved away from mongodb "At that point we still had some issues and bugs that would appear in MongoDB and we were not big fans of the master/slave architecture, so we decided to move on." Simply not being a fan of architecture cannot be a reason. Only wish they had elaborated on problems more

zapfmann 14y ago

Grr.. I can't figure if you can deploy cloudant in an on ec2 avail zone that you have your app in. Can you you even deploy it on ec2? I don't see anything stating answers to these questions clearly on their site. And they want my email to let me see tech docs! Are they run by former IBM staff? Finally, if they are 'hosted and managed all day everyday' some place far from me. I dunno if I want to use it.

zapfmann 14y ago
Ah. Spoke too soon. The comments here lead me to believe cloudant is to be used as a couchdb replica provider. Maybe cloudant should say that on their pages! :-)