5 ms·

I'm new to both (and noSQL in general), first heard about CouchDB a week or two ago. Would Couch be a good fit for building a logging system, or is there someth

I'm new to both (and noSQL in general), first heard about CouchDB a week or two ago. Would Couch be a good fit for building a logging system, or is there something else I should look into?

rdtsc 15y ago
I wouldn't use it for a write-heavy logging system ( and I am a big fan of it). Well actually it depends on what type of logging. If you plan on pouring megabytes of data/sec second into it continuously you might want to find something else. I don't think CouchDB would be a good candidate for that. It might be if you plan on archiving your post-processed logged data for a long term storage but you have to be smart about its format. Yeah 'logging' can mean a lot of things that it is hard to give a better advice without more information. Other DBs/tools you might want to look at: * rrdtool http://oss.oetiker.ch/rrdtool/ http://oss.oetiker.ch/rrdtool/ * MongoDB http://www.mongodb.org/ http://www.mongodb.org/ * Redis http://redis.io/ http://redis.io/ * Custom format + plain text files
- andrewvc 15y ago
  So, both MongoDB and Redis are DB's optimized for keeping stuff heavily in memory. I wouldn't use either for heavy logging. RRDTool is nice, but only for data where you don't mind losing records. For straight logging, SQL is actually a better solution than a lot of those options. If you need weird queries, something like Elastic Search or Solr on top of your SQL DB can buy you quite a bit of freedom. As can denormalizing your SQL. One other thing to consider of course, is something like syslog.
  - tilgovi 15y ago
    HBase might be a good solution, too, but I don't know a whole lot about it. CouchDB would work fine but you'd want to add some operational automation and a proxy to roll over between temporary databases since deleting old logs from a single database won't fully reduce its size and, as stated above, compaction isn't terribly fun or fast but you can avoid it by just flat out deleting old log databases.
mark242 15y ago
Absolutely it would be great for logging. Assuming that you don't need atomicity for your logs (eg, I want to see that log line reflect in my reports _right_ _now_) Couch is a perfect way to dump a huge amount of data really quickly. For reporting, the map/reduce system of Couch is a great fit. There's an initial learning curve -- as with anything, really -- but you'll find that Couch would be a great way to get logged data (which can be pretty unstructured) and report on it easily and quickly.
- buster 15y ago
  no, it wouldn't. It would be a horrible fit, imo. You don't need versioning info for logging, and the extra compacting of the database is another aspect, since with logging heavy applications you will write all the time... I love CouchDB but i definitely wouldn't use it for logging.
  - mark242 15y ago
    You don't need to worry about the versioning for a document that you never update. You can just blast the logfiles into the database and not worry about conflicts. Similarly, database compaction is really the biggest concern when you're updating documents, not just dumping new ones into the database. I would expect a very linear growth for a logging app.
    - buster 15y ago
      ok, my initial thought was one document per logfile, containing an increasing amount of loglines, but that's probably the worst approach possible. One document per logline which is appended to the database should be much better.
    - tilgovi 15y ago
      The problem doesn't come with the inserts, it comes with the deletes, where CouchDB will keep some metadata and stubs around to support its replication use cases. In a pure logging setup this is perhaps less than ideal.
- 15y ago
mey 15y ago
I'm not an expert on CouchDB (and I would love someone to correct where my approach is wrong). I could see having a CouchDB cluster per co-location, recording a document per logging event. Then back haul through replication to a central service for analysis/dashboard use. The only tricky part I see is to delete old logging data requires a compaction period and according to http://wiki.apache.org/couchdb/Compaction http://wiki.apache.org/couchdb/Compaction it's entirely possible for the compaction process to be slower then write data and never actually finish. Edit: Also the back end server doing analysis may need to be a BigCouch cluster to handle the load (depending on size) https://github.com/cloudant/bigcouch https://github.com/cloudant/bigcouch
schumihan 15y ago
Cassandra is a good fit for write heavy workload, as it is optimized for write...
wisty 15y ago
No. CouchDB is good for storing important data, slow at writing, and bad at ad-hoc querying. MongoDB is lightning fast at writing (provided you haven't got indexes, or your indexes fit in RAM), good at ad-hoc stuff, and a little more durable than /dev/null. Just remember to back up occasionally.

I'm new to both (and noSQL in general), first heard about CouchDB a week or two ago. Would Couch be a good fit for building a logging system, or is there something else I should look into?

rdtsc 15y ago

I wouldn't use it for a write-heavy logging system ( and I am a big fan of it). Well actually it depends on what type of logging. If you plan on pouring megabytes of data/sec second into it continuously you might want to find something else. I don't think CouchDB would be a good candidate for that. It might be if you plan on archiving your post-processed logged data for a long term storage but you have to be smart about its format. Yeah 'logging' can mean a lot of things that it is hard to give a better advice without more information. Other DBs/tools you might want to look at: * rrdtool http://oss.oetiker.ch/rrdtool/ http://oss.oetiker.ch/rrdtool/ * MongoDB http://www.mongodb.org/ http://www.mongodb.org/ * Redis http://redis.io/ http://redis.io/ * Custom format + plain text files

andrewvc 15y ago
So, both MongoDB and Redis are DB's optimized for keeping stuff heavily in memory. I wouldn't use either for heavy logging. RRDTool is nice, but only for data where you don't mind losing records. For straight logging, SQL is actually a better solution than a lot of those options. If you need weird queries, something like Elastic Search or Solr on top of your SQL DB can buy you quite a bit of freedom. As can denormalizing your SQL. One other thing to consider of course, is something like syslog.
- tilgovi 15y ago
  HBase might be a good solution, too, but I don't know a whole lot about it. CouchDB would work fine but you'd want to add some operational automation and a proxy to roll over between temporary databases since deleting old logs from a single database won't fully reduce its size and, as stated above, compaction isn't terribly fun or fast but you can avoid it by just flat out deleting old log databases.

mark242 15y ago

Absolutely it would be great for logging. Assuming that you don't need atomicity for your logs (eg, I want to see that log line reflect in my reports _right_ _now_) Couch is a perfect way to dump a huge amount of data really quickly. For reporting, the map/reduce system of Couch is a great fit. There's an initial learning curve -- as with anything, really -- but you'll find that Couch would be a great way to get logged data (which can be pretty unstructured) and report on it easily and quickly.

buster 15y ago
no, it wouldn't. It would be a horrible fit, imo. You don't need versioning info for logging, and the extra compacting of the database is another aspect, since with logging heavy applications you will write all the time... I love CouchDB but i definitely wouldn't use it for logging.
- mark242 15y ago
  You don't need to worry about the versioning for a document that you never update. You can just blast the logfiles into the database and not worry about conflicts. Similarly, database compaction is really the biggest concern when you're updating documents, not just dumping new ones into the database. I would expect a very linear growth for a logging app.
  - buster 15y ago
    ok, my initial thought was one document per logfile, containing an increasing amount of loglines, but that's probably the worst approach possible. One document per logline which is appended to the database should be much better.
  - tilgovi 15y ago
    The problem doesn't come with the inserts, it comes with the deletes, where CouchDB will keep some metadata and stubs around to support its replication use cases. In a pure logging setup this is perhaps less than ideal.
15y ago

mey 15y ago

I'm not an expert on CouchDB (and I would love someone to correct where my approach is wrong). I could see having a CouchDB cluster per co-location, recording a document per logging event. Then back haul through replication to a central service for analysis/dashboard use. The only tricky part I see is to delete old logging data requires a compaction period and according to http://wiki.apache.org/couchdb/Compaction http://wiki.apache.org/couchdb/Compaction it's entirely possible for the compaction process to be slower then write data and never actually finish. Edit: Also the back end server doing analysis may need to be a BigCouch cluster to handle the load (depending on size) https://github.com/cloudant/bigcouch https://github.com/cloudant/bigcouch

schumihan 15y ago

Cassandra is a good fit for write heavy workload, as it is optimized for write...

wisty 15y ago

No. CouchDB is good for storing important data, slow at writing, and bad at ad-hoc querying. MongoDB is lightning fast at writing (provided you haven't got indexes, or your indexes fit in RAM), good at ad-hoc stuff, and a little more durable than /dev/null. Just remember to back up occasionally.