5 ms·

I really want an excuse to build something with CouchDB and PouchDB (https://pouchdb.com/ https://pouchdb.com/). Can you expand on your experiences with it?

by skeoh 10y ago

mikekchar 10y ago
I'm not the original poster, but I can give you some of my limited experience with CouchDB from an application I inherited. The original idea for the project still seems like a good idea to me. Basically they wanted to record events that came into the system and store them in a write only ledger. Then they wanted to version every change so that you have an audit trail. Finally they wanted to be able to create views of that ledger to create the kind of data that they would work with on a day to day basis. For this, CouchDB seems like a perfect fit. Unfortunately, it didn't work out as well as one might hope because the people who implemented the idea didn't seem to be able to resist using the DB the way they would use a relational db. Instead of maintaining the concept of a write only ledger, they started to use it as a data store for things that were ephemeral. Also, instead of replicating the db, using a view to create a new db that was optimal for certain queries, they wrote a huge number of views in the main db. Finally they organised the views by relation rather than by use, so you would have 60-80 views in the same design document that would have to be reindexed if one of them changed. The result was something with very poor performance and where the storage for the indexes was more than an order of magnitude more than the storage for the documents themselves. CouchDB is also not super speedy at the best of times. There is a lot of latency involved in serializing the documents and farming them out to view servers, etc. So it takes a good 10 minutes to process a million documents, but you will find that your CPU is chugging along at 30-40% utilisation. Having said all that, one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). Then instead of building something like a rails application to farm out the data, make "couch applications" where you serve the HTML and JS directly from attachments on documents in the DB. In fact, I've written a React application to allow users to interact with portions of the data and it was quite simple. Then you can write a really small coordinating application to allow users to navigate to the parts of the system (really single page apps) that they want to use. Again, the nice thing about this is that you have a write only data store with versioning and the ability to audit history. You have views that allow you to interact with a small subset of the overall data. You can easily write single page applications where deployment is as easy as pushing a document to the DB. Replication is relatively cheap and you can move expensive view creation to restricted versions of the DB. You can stick the whole thing behind a load balancer and scale it as cheaply as setting up a new replication (again just another document in your DB). But, I will warn you. Don't use it like you would a relational DB, or else you will be in for a world of hurt. Especially you will see comments in this thread about migrations. If you are migrating your data, by definition you do not have a write-only-with-versioning application. Your application will have to deal with multiple versions of data or else you will not have the ability to audit history. If you do not care about this, then possibly there are better solutions than this.
- skeoh 10y ago
  Thanks for the detailed response. I'll keep this all in mind next time I need to choose a DB for even a small project.
- fiatjaf 10y ago
  one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). How is that different from the current concept of CouchDB views? You meant to replicate the DB to various different places and use the same CouchDB views from there? Or you meant something like a replication-view, in which some calculations are done with the documents in the source database and the target database receives the result of those calculations as their primary documents?
  - mikekchar 10y ago
    Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys. An example of this might be that you have a large number of daily reports. They all need different aspects of the data, but you end up writing views that sort by date and then collate the result in the server. So you end up maintaining an index for data that you will never query again and you are doing lots of extra processing merging the data after the query. Much better to replicate one day's worth of data to a new db every evening (possibly setting up a continuous replication to keep it up to date) and then add views on that db to do what you want. Like I said a full replication of a million documents takes about 10 minutes, so it's a reasonable thing to do.
    - fiatjaf 10y ago
      I like the concept of views. I think it is the best part of CouchDB. Saving data and later defining views (that should be very fast). Once I wrote an app that stored massively enormous documents with lots of data, the views were used to turn that data into queriable information later. I like what you suggested very much, because what CouchDB can currently do with views is very limited to what a powerful views implementation would, and the one you've suggested is a good suggestion of how to do it better. Thank you. I'll give this a lot of thinking when I restart writing my https://github.com/fiatjaf/summadb/ https://github.com/fiatjaf/summadb/
SCdF 10y ago
A bit late to the party, but we use Couch + Pouch at work. It is in some ways magic, and I literally don't know how we'd achieve what we do without it. But. The big pain for us is filtered replication: we have a bunch of cell phones (Pouch) that should only see some of the documents from the server (Couch). No mater how you slice it pretty much, you need filtered replication in there somewhere. And it's sllllooooooowwwwwwwww. It basically canes a CPU core while it's running per user, which means you can have CORE user's replicating at any given time, which is terrible. There isn't really much a solution for it, except "don't use filtered replication". I recently posted to the mailing list about it, you can read more here: https://mail-archives.apache.org/mod_mbox/couchdb-user/201605.mbox/%3C78765234-9DCE-4D77-8267-18342D494051%40medicmobile.org%3E https://mail-archives.apache.org/mod_mbox/couchdb-user/20160... (I'm the OP in that thread).

I really want an excuse to build something with CouchDB and PouchDB (https://pouchdb.com/ https://pouchdb.com/). Can you expand on your experiences with it?

by skeoh 10y ago

I really want an excuse to build something with CouchDB and PouchDB (https://pouchdb.com/ https://pouchdb.com/). Can you expand on your experiences with it?

mikekchar 10y ago

I'm not the original poster, but I can give you some of my limited experience with CouchDB from an application I inherited. The original idea for the project still seems like a good idea to me. Basically they wanted to record events that came into the system and store them in a write only ledger. Then they wanted to version every change so that you have an audit trail. Finally they wanted to be able to create views of that ledger to create the kind of data that they would work with on a day to day basis. For this, CouchDB seems like a perfect fit. Unfortunately, it didn't work out as well as one might hope because the people who implemented the idea didn't seem to be able to resist using the DB the way they would use a relational db. Instead of maintaining the concept of a write only ledger, they started to use it as a data store for things that were ephemeral. Also, instead of replicating the db, using a view to create a new db that was optimal for certain queries, they wrote a huge number of views in the main db. Finally they organised the views by relation rather than by use, so you would have 60-80 views in the same design document that would have to be reindexed if one of them changed. The result was something with very poor performance and where the storage for the indexes was more than an order of magnitude more than the storage for the documents themselves. CouchDB is also not super speedy at the best of times. There is a lot of latency involved in serializing the documents and farming them out to view servers, etc. So it takes a good 10 minutes to process a million documents, but you will find that your CPU is chugging along at 30-40% utilisation. Having said all that, one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). Then instead of building something like a rails application to farm out the data, make "couch applications" where you serve the HTML and JS directly from attachments on documents in the DB. In fact, I've written a React application to allow users to interact with portions of the data and it was quite simple. Then you can write a really small coordinating application to allow users to navigate to the parts of the system (really single page apps) that they want to use. Again, the nice thing about this is that you have a write only data store with versioning and the ability to audit history. You have views that allow you to interact with a small subset of the overall data. You can easily write single page applications where deployment is as easy as pushing a document to the DB. Replication is relatively cheap and you can move expensive view creation to restricted versions of the DB. You can stick the whole thing behind a load balancer and scale it as cheaply as setting up a new replication (again just another document in your DB). But, I will warn you. Don't use it like you would a relational DB, or else you will be in for a world of hurt. Especially you will see comments in this thread about migrations. If you are migrating your data, by definition you do not have a write-only-with-versioning application. Your application will have to deal with multiple versions of data or else you will not have the ability to audit history. If you do not care about this, then possibly there are better solutions than this.

skeoh 10y ago
Thanks for the detailed response. I'll keep this all in mind next time I need to choose a DB for even a small project.
fiatjaf 10y ago
one of the things I want to try (but have only done some preliminary trials with) is to keep the concept of the write only ledger, but to replicate the db into several views of the data (some with severely restricted content). How is that different from the current concept of CouchDB views? You meant to replicate the DB to various different places and use the same CouchDB views from there? Or you meant something like a replication-view, in which some calculations are done with the documents in the source database and the target database receives the result of those calculations as their primary documents?
- mikekchar 10y ago
  Yes, the latter. One of the main problems I've seen is that indexing things that you will never query is both expensive in time and space. Also it's amazing how many views tend to have exactly the same data, only sorted differently. And the reason to sort it differently is because you only want to work on a subset of the data, but you can only restrict the query in a contiguous section of keys. An example of this might be that you have a large number of daily reports. They all need different aspects of the data, but you end up writing views that sort by date and then collate the result in the server. So you end up maintaining an index for data that you will never query again and you are doing lots of extra processing merging the data after the query. Much better to replicate one day's worth of data to a new db every evening (possibly setting up a continuous replication to keep it up to date) and then add views on that db to do what you want. Like I said a full replication of a million documents takes about 10 minutes, so it's a reasonable thing to do.
  - fiatjaf 10y ago
    I like the concept of views. I think it is the best part of CouchDB. Saving data and later defining views (that should be very fast). Once I wrote an app that stored massively enormous documents with lots of data, the views were used to turn that data into queriable information later. I like what you suggested very much, because what CouchDB can currently do with views is very limited to what a powerful views implementation would, and the one you've suggested is a good suggestion of how to do it better. Thank you. I'll give this a lot of thinking when I restart writing my https://github.com/fiatjaf/summadb/ https://github.com/fiatjaf/summadb/

SCdF 10y ago

A bit late to the party, but we use Couch + Pouch at work. It is in some ways magic, and I literally don't know how we'd achieve what we do without it. But. The big pain for us is filtered replication: we have a bunch of cell phones (Pouch) that should only see some of the documents from the server (Couch). No mater how you slice it pretty much, you need filtered replication in there somewhere. And it's sllllooooooowwwwwwwww. It basically canes a CPU core while it's running per user, which means you can have CORE user's replicating at any given time, which is terrible. There isn't really much a solution for it, except "don't use filtered replication". I recently posted to the mailing list about it, you can read more here: https://mail-archives.apache.org/mod_mbox/couchdb-user/201605.mbox/%3C78765234-9DCE-4D77-8267-18342D494051%40medicmobile.org%3E https://mail-archives.apache.org/mod_mbox/couchdb-user/20160... (I'm the OP in that thread).