3 ms·

I like your analogy, very fitting. To their defense however they are far from the only ones to do that... I lost about 2h worth of production data with HBase in

by annnnd 13y ago

I like your analogy, very fitting. To their defense however they are far from the only ones to do that... I lost about 2h worth of production data with HBase in just the same way - fortunately I didn't want to trust it completely anyway and had my own logs of all transactions on filesystem, but it definitely shattered my trust in that DB (not to mention it was a pain to setup and had no secondary indexes). I use MongoDB now in production and I am happy about it. Not huge dataset by any means so MongoDB fits the bill perfectly. It has a few idiosyncrasies (doesn't release disk space after deleting records - what?!?) and you definitely want to read the manual on settings. But it is incredibly easy to use (documents instead of relational data) and allows me to focus on app development instead of my storage backend.

ddorian43 13y ago
doesn't hbase persist in the transaction log every update, using append on hdfs ? or did you have a version of hdfs that didn't have append ?
- annnnd 13y ago
  HDFS didn't have append at the time, not sure how it is now. It did have some filesystem journalling though (if I remember correctly), we just didn't know we should turn it on.
- linuxhansl 13y ago
  HBase never needed nor used append. "Append" refers to the ability to reopen an existing file from a new client and append data to it. HBase writes only immutable files, which means they are written once then only read. Even the WAL is no exception; it is written once, and then only opened for read when needed for recovery or replication. HBase needs hflush to make sure that the WAL edits are resident at at least 3 (default) HDFS data node machines. Not sure how exactly grand parent lost data. Each edit is first written to the WAL then committed to the in memory store. The in memory store is flushed to disk into a new file at a certain size. If a server crashes and had unflushed data in the memory store that part of the data is replayed from the WAL on another server. See also here: http://hadoop-hbase.blogspot.com/2012/05/hbase-hdfs-and-durable-sync.html http://hadoop-hbase.blogspot.com/2012/05/hbase-hdfs-and-dura...

by annnnd 13y ago

ddorian43 13y ago

doesn't hbase persist in the transaction log every update, using append on hdfs ? or did you have a version of hdfs that didn't have append ?

annnnd 13y ago
HDFS didn't have append at the time, not sure how it is now. It did have some filesystem journalling though (if I remember correctly), we just didn't know we should turn it on.
linuxhansl 13y ago
HBase never needed nor used append. "Append" refers to the ability to reopen an existing file from a new client and append data to it. HBase writes only immutable files, which means they are written once then only read. Even the WAL is no exception; it is written once, and then only opened for read when needed for recovery or replication. HBase needs hflush to make sure that the WAL edits are resident at at least 3 (default) HDFS data node machines. Not sure how exactly grand parent lost data. Each edit is first written to the WAL then committed to the in memory store. The in memory store is flushed to disk into a new file at a certain size. If a server crashes and had unflushed data in the memory store that part of the data is replayed from the WAL on another server. See also here: http://hadoop-hbase.blogspot.com/2012/05/hbase-hdfs-and-durable-sync.html http://hadoop-hbase.blogspot.com/2012/05/hbase-hdfs-and-dura...