5 ms·

Why I'm Walking Away From CouchDB

39 points by aaronpk 14y ago

deleted 14y ago
[deleted]
mandreko 14y ago
Unfortunately it's getting hammered enough that the content is offline. Maybe that's why he's walking away from CouchDB?
- tjgillies 14y ago
  I was thinking the same thing
- pwf 14y ago
  I just tried googling for the cached version, hoping that they were able to grab it, but interestingly they have only a screenshot of the article. It's even clear enough to read all the text, but there's no cached version. Surely the page itself would have been easier to store than an image?
  - sp332 14y ago
    https://webcache.googleusercontent.com/search?strip=1&q=cache:http%3A%2F%2Fdonpark.org%2Fblog%2F2012%2F06%2F01%2Fwhy-i-m-walking-away-from-couchdb%23commentform https://webcache.googleusercontent.com/search?strip=1&q=...
- donpdonp 14y ago
  Sorry for the slow performance, its a very tiny fanless ITX server on a not-too-performant rails blogging app (typo). The traffic seems to have died down now.
AdrianRossouw 14y ago
I really like using ElasticSearch to index the couchdb database[1]. It allows you to scale out the data store and the search index separately, since they actually have separate concerns. You can also do a lot of really complex queries with the elasticsearch query-dsl[2]. [1] http://daemon.co.za/2012/05/elasticsearch-5-minutes/ http://daemon.co.za/2012/05/elasticsearch-5-minutes/ [2] http://www.elasticsearch.org/guide/reference/query-dsl http://www.elasticsearch.org/guide/reference/query-dsl
gabrtv 14y ago
While the blog is mostly down, here is a summary of the problems pointed out by the author: 1. View index updates - "Every write sets up a trap for the first reader to come along after the write" 2. Append-only database file - "Create(s) a huge operational headache", disk utilization, compaction, etc CouchDB has always been a poor choice for write-heavy applications. Sounds to me like a case of selecting the wrong tool for the job. Though I'm a big CouchDB fan myself, I wonder why the author didn't go with a traditional RDBMS for this application.
- MSM 14y ago
  In the next article, titled "What I look for", he mentions that he'd never use a stored proc to access data through SQL. To me it sounds like he is almost refusing to use an RDBMS to its fullest.
  - donpdonp 14y ago
    Using map/reduce has taught me where RDBMS systems excel and what I've taken for granted. Its the simple master/master replication feature that is very attractive. RDMBS emphasises perfect consistency when eventual consistency is a better tradeoff for most webapps (IMHO).
- donpdonp 14y ago
  The ratio of write/read does not seem to be the problem. After a certain amount of write traffic the view indexing and storage problems appear - regardless of the amount of read traffic. More writing means more of 1. and 2. and the more likely the database enters the dreaded more-than-half full phase. I have cron job compact the database every 6 hours. (I'm the blog post author.)
einhverfr 14y ago
Given that it's only 20k new documents a day my immediate thoughts were of PostgreSQL. In particular thinking of use of things like the XML storage capabilities (XQuery is not yet supported but that's something folks are working on, but you can do a lot of processing with XML parsing/SQL if you like) or the new JSON type in 9.2 beta, particularly with Javascript as a stored procedure language....
mark242 14y ago
After reading the article, I'm convinced that the author has something really, really, REALLY wrong with their data format, or their view code. 20,000 new documents per _day_ (from the article) is absolutely nothing for CouchDB. Our dataset in Couch is 28 million records, takes up 13GB of data, and we can create a brand new view against those records and have it run the first time in less than ten minutes. Adding 20k records to this dataset might cause the view to take a couple of seconds to update. One update to the view, which is what the article complains about, takes one or two milliseconds to be added. But let's hypothetically say that the author has the most complex data format in history, and the most complex view code in history, and ten writes to the database actually cause minutes worth of view index updating. Adding the "&stale=update_after" request parameter to your view query returns instantly, with the last-generated view. After the query is issued, the view is automatically updated. No cron jobs necessary-- it is (and should be) a drop-in addition for all of your view queries. Now, about the append-only database file. We've been playing with autocompaction from CouchDB 1.2, and it seems to be working quite well, but again, at the scale the author is talking about, 20k records per day is a very predictable delta in disk space. Even running CouchDB off a 32GB compact flash card (and it is possible!) will give ample room for the size of his dataset. We run off a 120GB fibre channel mount and haven't needed to increase the partition size yet. From the article: "The system in question does a lot of writes of temporary data that is followed up by deletes a few days later. There is also a lot of permanent storage that hardly gets used." That, to me, speaks to two separate databases, one for the temporary data, one for long-term storage. The temporary database can be compacted once per day, and the permanent storage could be compacted once per week/month/never.
- donpdonp 14y ago
  The documents are simple and small. Maybe 15 keys and short string values like datetime and latitude/longitude. 20,000 documents should be nothing but adding that much then doing a read triggers a view reindexing that was taking minutes. There are 2 million records taking about 1.6GB of disk. Doing on order of 100,000 deletes was bring disk usage up to 1.8GB or higher. stale=update_after is not useful in my case. the client reads the database for the latest geographical position. it must return the lastest data in the database. update_after means its ready 'next time' but there wont be a next time, the client will simply have stale data. What the site needs is view updating after every write. If couchdb were to add that, I'd be excited. I was excited about auto-compaction in 1.2 but the core problem remains that rewriting every single record to a new spot on disk at every compaction is a tremendous waste of I/O and write cycles on a (SSD) disk. Splitting the data in two sounds untenable for even simple queries (every search done twice, merging answer sets, etc).
escoz 14y ago
I'm just walking into CouchDB, specially with the couchsync mobile part of it, and about to submit my first iOS app with CouchMobile to the AppStore. It's good to read these kinds of informed posts and comments. Thanks guys.
mark_l_watson 14y ago
Don: I don't doubt that you had problems with you particular project, but I have issues with your article because the tone is that you want to warn everyone off of CouchDB. Sweeping generalizations usually are wrong. Also, 20K writes a day are nothing.

39 points by aaronpk 14y ago

deleted 14y ago

[deleted]

Unfortunately it's getting hammered enough that the content is offline. Maybe that's why he's walking away from CouchDB?

tjgillies 14y ago
I was thinking the same thing
pwf 14y ago
I just tried googling for the cached version, hoping that they were able to grab it, but interestingly they have only a screenshot of the article. It's even clear enough to read all the text, but there's no cached version. Surely the page itself would have been easier to store than an image?
- sp332 14y ago
  https://webcache.googleusercontent.com/search?strip=1&q=cache:http%3A%2F%2Fdonpark.org%2Fblog%2F2012%2F06%2F01%2Fwhy-i-m-walking-away-from-couchdb%23commentform https://webcache.googleusercontent.com/search?strip=1&q=...
donpdonp 14y ago
Sorry for the slow performance, its a very tiny fanless ITX server on a not-too-performant rails blogging app (typo). The traffic seems to have died down now.

AdrianRossouw 14y ago

I really like using ElasticSearch to index the couchdb database[1]. It allows you to scale out the data store and the search index separately, since they actually have separate concerns. You can also do a lot of really complex queries with the elasticsearch query-dsl[2]. [1] http://daemon.co.za/2012/05/elasticsearch-5-minutes/ http://daemon.co.za/2012/05/elasticsearch-5-minutes/ [2] http://www.elasticsearch.org/guide/reference/query-dsl http://www.elasticsearch.org/guide/reference/query-dsl

gabrtv 14y ago

While the blog is mostly down, here is a summary of the problems pointed out by the author: 1. View index updates - "Every write sets up a trap for the first reader to come along after the write" 2. Append-only database file - "Create(s) a huge operational headache", disk utilization, compaction, etc CouchDB has always been a poor choice for write-heavy applications. Sounds to me like a case of selecting the wrong tool for the job. Though I'm a big CouchDB fan myself, I wonder why the author didn't go with a traditional RDBMS for this application.

MSM 14y ago
In the next article, titled "What I look for", he mentions that he'd never use a stored proc to access data through SQL. To me it sounds like he is almost refusing to use an RDBMS to its fullest.
- donpdonp 14y ago
  Using map/reduce has taught me where RDBMS systems excel and what I've taken for granted. Its the simple master/master replication feature that is very attractive. RDMBS emphasises perfect consistency when eventual consistency is a better tradeoff for most webapps (IMHO).
donpdonp 14y ago
The ratio of write/read does not seem to be the problem. After a certain amount of write traffic the view indexing and storage problems appear - regardless of the amount of read traffic. More writing means more of 1. and 2. and the more likely the database enters the dreaded more-than-half full phase. I have cron job compact the database every 6 hours. (I'm the blog post author.)

einhverfr 14y ago

Given that it's only 20k new documents a day my immediate thoughts were of PostgreSQL. In particular thinking of use of things like the XML storage capabilities (XQuery is not yet supported but that's something folks are working on, but you can do a lot of processing with XML parsing/SQL if you like) or the new JSON type in 9.2 beta, particularly with Javascript as a stored procedure language....

mark242 14y ago

After reading the article, I'm convinced that the author has something really, really, REALLY wrong with their data format, or their view code. 20,000 new documents per _day_ (from the article) is absolutely nothing for CouchDB. Our dataset in Couch is 28 million records, takes up 13GB of data, and we can create a brand new view against those records and have it run the first time in less than ten minutes. Adding 20k records to this dataset might cause the view to take a couple of seconds to update. One update to the view, which is what the article complains about, takes one or two milliseconds to be added. But let's hypothetically say that the author has the most complex data format in history, and the most complex view code in history, and ten writes to the database actually cause minutes worth of view index updating. Adding the "&stale=update_after" request parameter to your view query returns instantly, with the last-generated view. After the query is issued, the view is automatically updated. No cron jobs necessary-- it is (and should be) a drop-in addition for all of your view queries. Now, about the append-only database file. We've been playing with autocompaction from CouchDB 1.2, and it seems to be working quite well, but again, at the scale the author is talking about, 20k records per day is a very predictable delta in disk space. Even running CouchDB off a 32GB compact flash card (and it is possible!) will give ample room for the size of his dataset. We run off a 120GB fibre channel mount and haven't needed to increase the partition size yet. From the article: "The system in question does a lot of writes of temporary data that is followed up by deletes a few days later. There is also a lot of permanent storage that hardly gets used." That, to me, speaks to two separate databases, one for the temporary data, one for long-term storage. The temporary database can be compacted once per day, and the permanent storage could be compacted once per week/month/never.

donpdonp 14y ago
The documents are simple and small. Maybe 15 keys and short string values like datetime and latitude/longitude. 20,000 documents should be nothing but adding that much then doing a read triggers a view reindexing that was taking minutes. There are 2 million records taking about 1.6GB of disk. Doing on order of 100,000 deletes was bring disk usage up to 1.8GB or higher. stale=update_after is not useful in my case. the client reads the database for the latest geographical position. it must return the lastest data in the database. update_after means its ready 'next time' but there wont be a next time, the client will simply have stale data. What the site needs is view updating after every write. If couchdb were to add that, I'd be excited. I was excited about auto-compaction in 1.2 but the core problem remains that rewriting every single record to a new spot on disk at every compaction is a tremendous waste of I/O and write cycles on a (SSD) disk. Splitting the data in two sounds untenable for even simple queries (every search done twice, merging answer sets, etc).

escoz 14y ago

I'm just walking into CouchDB, specially with the couchsync mobile part of it, and about to submit my first iOS app with CouchMobile to the AppStore. It's good to read these kinds of informed posts and comments. Thanks guys.

mark_l_watson 14y ago

Don: I don't doubt that you had problems with you particular project, but I have issues with your article because the tone is that you want to warn everyone off of CouchDB. Sweeping generalizations usually are wrong. Also, 20K writes a day are nothing.