3 ms·

...except it's not durable. If your server (clojure) crashes, you lose the data in memory.

damienkatz 16y ago
It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.
- saurik 16y ago
  The point of this article is queuing requestd at the webtier until you have enough to swamp the HTTP transport overhead to CouchDB. If the website crashes you are going to lose that data.
  - damienkatz 16y ago
    I'm not sure I understand. CouchDB won't lose any committed (fsynced) data when it crashes. And when configured, it won't return a success until it's committed. It's completely durable. edit: Oh I see, the stuff written in the clojure side won't be committed. If he wanted to write each one synchronously to CouchDB, that would work and CouchDB would still batch up the concurrent updates while writing.
    - technoweenie 16y ago
      "As requests come in, instead of connecting immediately to the database, why not queue them up until we have an optimal number and then do a bulk insert?" Judging by the code, it looks like he stores about 50 documents before sending them to CouchDB. I believe the original poster was asking about the durability of Clojure STM, not Couch DB.
- epochwolf 16y ago
  If Clojure crashes, not the database.
apage43 16y ago
Except that's always the case. You mean if Clojure crashes before it queues up 50 docs and sends them, the docs in the queue are lost. Durability is a concern when it means the integrity of your data is unknown. You'd probably be doing something like this in a bulk load, in which case you'd know where it crashed. If this were queuing and batching inserts from many users it might be more of an issue (a user thinks something happened and it didn't), but realize that the at a queue size of 50 this performs at about ~5500 inserts/second. (And you can reduce this queue size if you don't need to insert that fast) This would affect users who happen to make their request within a roughly 0.18 millisecond window. If confirmation is still absolutely important, the post also links a version at the end that -does- wait for and return the IDs of the inserted documents from couchdb, guaranteeing the inserts have happened, and it still performs quite well.
swannodette 16y ago
But not much more data than a webserver crash in a fully concurrent situation. One enhancement that I didn't bother to implement because it would muddy up the example was a timed flush so that data never just sits around in memory because the queue isn't full.

It's quite durable. CouchDB has pure tail append storage and will fsync every update and batch concurrent updates into a single commit. It can get fairly close to a drives max sustained write speed.

saurik 16y ago
The point of this article is queuing requestd at the webtier until you have enough to swamp the HTTP transport overhead to CouchDB. If the website crashes you are going to lose that data.
- damienkatz 16y ago
  I'm not sure I understand. CouchDB won't lose any committed (fsynced) data when it crashes. And when configured, it won't return a success until it's committed. It's completely durable. edit: Oh I see, the stuff written in the clojure side won't be committed. If he wanted to write each one synchronously to CouchDB, that would work and CouchDB would still batch up the concurrent updates while writing.
  - technoweenie 16y ago
    "As requests come in, instead of connecting immediately to the database, why not queue them up until we have an optimal number and then do a bulk insert?" Judging by the code, it looks like he stores about 50 documents before sending them to CouchDB. I believe the original poster was asking about the durability of Clojure STM, not Couch DB.
epochwolf 16y ago
If Clojure crashes, not the database.

apage43 16y ago

Except that's always the case. You mean if Clojure crashes before it queues up 50 docs and sends them, the docs in the queue are lost. Durability is a concern when it means the integrity of your data is unknown. You'd probably be doing something like this in a bulk load, in which case you'd know where it crashed. If this were queuing and batching inserts from many users it might be more of an issue (a user thinks something happened and it didn't), but realize that the at a queue size of 50 this performs at about ~5500 inserts/second. (And you can reduce this queue size if you don't need to insert that fast) This would affect users who happen to make their request within a roughly 0.18 millisecond window. If confirmation is still absolutely important, the post also links a version at the end that -does- wait for and return the IDs of the inserted documents from couchdb, guaranteeing the inserts have happened, and it still performs quite well.

swannodette 16y ago

But not much more data than a webserver crash in a fully concurrent situation. One enhancement that I didn't bother to implement because it would muddy up the example was a timed flush so that data never just sits around in memory because the queue isn't full.