3 ms·

This varies by the amount of RAM, if the total RAM being talked about is <= 256 GB (or in some cases 128) this statement will usually be false. But for RAM tota

This varies by the amount of RAM, if the total RAM being talked about is <= 256 GB (or in some cases 128) this statement will usually be false. But for RAM totals above 256 GB (and in some cases 512) it is almost always true. But these watermarks will change over time so this assumption must but re-evaluated from time to time.

msiebuhr 12y ago
$WORK anecdote: We had a quartet of legacy-ish databases with sharded customer data (had some interconnection, but we sharded in the application-layer, so the database didn't have to move any data internally). It was a bitch to maintain, but it performed OK, by throwing fairly beefy machines at it. We began a quest to replace the database with MongoDB/HDFS/Riak/CouchBase/Cassandra/whatever. We wrote a script that converted our data to JSON for easier ingestion and began hammering away. Until we discovered that by converting it to JSON (and simplifying some parts of the data-model), our dataset had actually become small enough to fit on a single server in memory (still ~100GB). So we ditched the cluster-thingie and moved everything to CouchDB. Pricy hardware, but the saved man-hours recouped that within a month or two. (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)
- ddorian43 12y ago
  But why throw it in couchdb when you could use a rdbms ?
  - collyw 12y ago
    It must be web scale.
  - msiebuhr 12y ago
    We don't need a RDBMS; we get along quite well with key/value + keyscans, making it a lot easier to switch to a database with roughly similar features later (CouchBase, Cassandra, HBase, possibly others)
    - collyw 12y ago
      OK, slightly ignorant question here, but SQL is kind of a standard for RDBMS's. As far as I am aware no similar starts exists in the NoSQL world, so I would have assumed that it would make switching harder. If you are just using it for basic stuff, then "SELECT * FROM table" is pretty standard across RDBMS's.
    - ddorian43 12y ago
      Reading this: (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...) You need a rdbms. Field names won't be repeated for each query and you will push later the time to 'start over'.
- wisty 12y ago
  That's odd. When I last checked, CouchDB took a lot more memory. Unless you are doing something very odd with how you model your data ("select (key, value) from attributes where recored_id = ?" ... why are you even using SQL then?). I expect that slimming down the data model was the main factor, or you simply didn't want to use SQL at all (or your data wasn't really structured enough).

msiebuhr 12y ago

$WORK anecdote: We had a quartet of legacy-ish databases with sharded customer data (had some interconnection, but we sharded in the application-layer, so the database didn't have to move any data internally). It was a bitch to maintain, but it performed OK, by throwing fairly beefy machines at it. We began a quest to replace the database with MongoDB/HDFS/Riak/CouchBase/Cassandra/whatever. We wrote a script that converted our data to JSON for easier ingestion and began hammering away. Until we discovered that by converting it to JSON (and simplifying some parts of the data-model), our dataset had actually become small enough to fit on a single server in memory (still ~100GB). So we ditched the cluster-thingie and moved everything to CouchDB. Pricy hardware, but the saved man-hours recouped that within a month or two. (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)

ddorian43 12y ago
But why throw it in couchdb when you could use a rdbms ?
- collyw 12y ago
  It must be web scale.
- msiebuhr 12y ago
  We don't need a RDBMS; we get along quite well with key/value + keyscans, making it a lot easier to switch to a database with roughly similar features later (CouchBase, Cassandra, HBase, possibly others)
  - collyw 12y ago
    OK, slightly ignorant question here, but SQL is kind of a standard for RDBMS's. As far as I am aware no similar starts exists in the NoSQL world, so I would have assumed that it would make switching harder. If you are just using it for basic stuff, then "SELECT * FROM table" is pretty standard across RDBMS's.
  - ddorian43 12y ago
    Reading this: (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...) You need a rdbms. Field names won't be repeated for each query and you will push later the time to 'start over'.
wisty 12y ago
That's odd. When I last checked, CouchDB took a lot more memory. Unless you are doing something very odd with how you model your data ("select (key, value) from attributes where recored_id = ?" ... why are you even using SQL then?). I expect that slimming down the data model was the main factor, or you simply didn't want to use SQL at all (or your data wasn't really structured enough).