4 ms·

The documents are simple and small. Maybe 15 keys and short string values like datetime and latitude/longitude. 20,000 documents should be nothing but adding t

The documents are simple and small. Maybe 15 keys and short string values like datetime and latitude/longitude. 20,000 documents should be nothing but adding that much then doing a read triggers a view reindexing that was taking minutes. There are 2 million records taking about 1.6GB of disk. Doing on order of 100,000 deletes was bring disk usage up to 1.8GB or higher. stale=update_after is not useful in my case. the client reads the database for the latest geographical position. it must return the lastest data in the database. update_after means its ready 'next time' but there wont be a next time, the client will simply have stale data. What the site needs is view updating after every write. If couchdb were to add that, I'd be excited. I was excited about auto-compaction in 1.2 but the core problem remains that rewriting every single record to a new spot on disk at every compaction is a tremendous waste of I/O and write cycles on a (SSD) disk. Splitting the data in two sounds untenable for even simple queries (every search done twice, merging answer sets, etc).

mark242 14y ago
You have something extremely wrong. Maybe it's at the hardware level. Maybe you have a crap SSD. Something. I just created a test database on my workstation, with a spinning disk, with the same type of data, and a temporary view against 50k documents takes 10 seconds to establish. Saving the view, re-running, then adding 10k documents and re-running the view takes 4 seconds to update. The 60k records take up 110MB. "What the site needs is view updating after every write." So why not build that into your application? For every write that you want to be consistently shown in the very next client view, ensure that you spawn a thread to immediately issue a query to the view after the write. Remember, CouchDB does not try to be an ACID-level database, substituting the C for raw speed. In fact, you can't be guaranteed that a client won't get the view while a new record is being added, and with no blocking on write, your client now has out of date data. With such a small amount of database traffic, you don't really notice this side-effect of Couch. "I was excited about auto-compaction in 1.2 but the core problem remains that rewriting every single record to a new spot on disk at every compaction is a tremendous waste of I/O and write cycles on a (SSD) disk." With the number of IOPS that we're talking about here, your SSD will last for years and years and years, even if you ran compaction once every minute. A 256GB SSD costs $200 at Amazon, and with your Couch instance taking 1.8GB, I'd be more worried about the heat death of the Universe before worrying about running out of disk space or SSD write operations.
- donpdonp 14y ago
  Thanks for doing a quick benchmark. In my frustration I may have misremembered the amount of time the indexing takes, but the number of seconds really a distraction from the post. Its the principle that the next read operation time is so wildly unpredictable. Slowing down based on the size of the database makes some sense but slowing down based on the number of inserts since the last read is downright irrational to someone used to rdbms. The number of write cycles on an SSD is also a distraction. Its the principle that gigs and gigs of records that are essentially read only are physically moved to another spot on the disk every six hours (thats how often a cron job kicks off compaction) is still a tremendous waste of effort/bandwidth/whathave-you.

mark242 14y ago

You have something extremely wrong. Maybe it's at the hardware level. Maybe you have a crap SSD. Something. I just created a test database on my workstation, with a spinning disk, with the same type of data, and a temporary view against 50k documents takes 10 seconds to establish. Saving the view, re-running, then adding 10k documents and re-running the view takes 4 seconds to update. The 60k records take up 110MB. "What the site needs is view updating after every write." So why not build that into your application? For every write that you want to be consistently shown in the very next client view, ensure that you spawn a thread to immediately issue a query to the view after the write. Remember, CouchDB does not try to be an ACID-level database, substituting the C for raw speed. In fact, you can't be guaranteed that a client won't get the view while a new record is being added, and with no blocking on write, your client now has out of date data. With such a small amount of database traffic, you don't really notice this side-effect of Couch. "I was excited about auto-compaction in 1.2 but the core problem remains that rewriting every single record to a new spot on disk at every compaction is a tremendous waste of I/O and write cycles on a (SSD) disk." With the number of IOPS that we're talking about here, your SSD will last for years and years and years, even if you ran compaction once every minute. A 256GB SSD costs $200 at Amazon, and with your Couch instance taking 1.8GB, I'd be more worried about the heat death of the Universe before worrying about running out of disk space or SSD write operations.

donpdonp 14y ago
Thanks for doing a quick benchmark. In my frustration I may have misremembered the amount of time the indexing takes, but the number of seconds really a distraction from the post. Its the principle that the next read operation time is so wildly unpredictable. Slowing down based on the size of the database makes some sense but slowing down based on the number of inserts since the last read is downright irrational to someone used to rdbms. The number of write cycles on an SSD is also a distraction. Its the principle that gigs and gigs of records that are essentially read only are physically moved to another spot on the disk every six hours (thats how often a cron job kicks off compaction) is still a tremendous waste of effort/bandwidth/whathave-you.