3 ms·

By coincidence I was looking around for a datastore. I almost settled on couchbase when I saw this; easy cluster deployment is my main draw after looking at how

By coincidence I was looking around for a datastore. I almost settled on couchbase when I saw this; easy cluster deployment is my main draw after looking at how bad setting up master-master system is in other solutions (hbase, mongo, couchdb all have eterogeneus nodes and weird failure modes) One quick question since I saw the devs around and I can't find a final answer on it on google: are there any strong roadblock or performance drawbacks against storing principally medium size binary data like images, say, < 100mb blobs? Yeah I could use something more appropriate but I'd be back to figuring out hadoop installation or fighting s3 eventual consistency.

ansible 11y ago
I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...
- LoSboccacc 11y ago
  but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space. I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.
  - manigandham 11y ago
    I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.
    - LoSboccacc 11y ago
      > I have a strong sense that you're vastly over-engineering something. and you're 100% right it wouldn't be interesting otherwise :D (I'm not gonna use this on real work no worries)
smoyer 11y ago
As a long-time CouchDB user, I'm going to assume that by "weird failure modes", you mean "conflicts". There are some awesome aspects to using CouchDB, but getting used to conflicts certainly took some time. In short, your application needs to include the code to resolve problems that arise due to network partitions or concurrent writes to different nodes. On the other hand, it's a really stable platform and the write-only database files worked really well (we lost zero data in over 5 years of operation ... even with tests that included powering down a node while in use). But ... I also believe that CockroachDB is going to be the project to finally solve these problems and it's got an SQL interface! I'm waiting for SQL joins, and then I suspect I'll be moving at least some of my clustered PostgreSQL instances to CockroachDB.
- LoSboccacc 11y ago
  no I mean things like this: http://docs.couchdb.org/en/stable/cluster/nodes.html#removing-a-node http://docs.couchdb.org/en/stable/cluster/nodes.html#removin... http://docs.couchdb.org/en/stable/cluster/sharding.html#moving http://docs.couchdb.org/en/stable/cluster/sharding.html#movi... http://docs.couchdb.org/en/stable/cluster/sharding.html#reshard-no-preshard http://docs.couchdb.org/en/stable/cluster/sharding.html#resh... sharding needs to be taken care manually, there is nowhere on the docs handling what happens in the event of a failover and no how to replace a failed populated node with an empty one so that the cluster can recover. those are quite big concerns if sharding with a replica set for fault tolerance is the main objective.
hnkimb3558 11y ago
(employee/founder here) Anything under 64K is perfectly reasonable to store in a CockroachDB column. Between 64K and maybe 1M is trending towards trouble. Values greater than this are going to go through CockroachDB like a goat through a python. Why is this the case? For starters, at the level of RocksDB, values greater than 64K are not jammed into SSTables (to avoid constantly rewriting them during compactions of the LSM tree). Instead, individual files are created. Also, CockroachDB has quite a lot of write amplification, which is generally OK for structured relational data, but becomes progressively more terrible for large blobs. Write amplification comes from the Raft log, as well as RocksDB's write-ahead log. What we really need is an integrated storage system for immutable blobs, something we're taking very seriously. Roughly half of the original team which built Colossus at Google are working on CockroachDB, so there's some knowledge of how to go about building such a system. While we're not sure where it would fall on our roadmap, the idea is that large blob values would be efficiently replicated and maintained through a separate subsystem. The blob column itself would just contain a pointer to the blob. The value in tight integration (a single CockroachDB cluster providing both OLTP SQL database as well as a distributed blob store) would be one deployment & admin console, and transactionally consistent blob column values (e.g. no fighting s3 eventual consistency).
- LoSboccacc 11y ago
  Thanks! That's fantastic to hear even if it's going to materialize later/eventually/never at least it's great to know the need is recognized. Disregarding write amplification issues for a sec, would it make things better to split binaries in 64k chunks and have them in a chunk table keyed by name and offset?
  - hnkimb3558 11y ago
    I'm not really sure which strategy would benchmark best between 64K chunks, 1M chunks, or even 8M chunks. I think this requires some experimentation. Pushing them all through as 64K chunks has a lot of overhead, and you'd reap the full write amplification. Could you tell me a bit more about your use case? You can email me spencer at cockroachlabs.com. I've been meaning to work on the "CockroachDB Egg Store" (my disgusting name for a blob storage subsystem) as a Free Fridays side project for a while, but have been distracted with all manner of other enticing options. There aren't enough hours in a week...

ansible 11y ago

I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...

LoSboccacc 11y ago
but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space. I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.
- manigandham 11y ago
  I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.
  - LoSboccacc 11y ago
    > I have a strong sense that you're vastly over-engineering something. and you're 100% right it wouldn't be interesting otherwise :D (I'm not gonna use this on real work no worries)

smoyer 11y ago

As a long-time CouchDB user, I'm going to assume that by "weird failure modes", you mean "conflicts". There are some awesome aspects to using CouchDB, but getting used to conflicts certainly took some time. In short, your application needs to include the code to resolve problems that arise due to network partitions or concurrent writes to different nodes. On the other hand, it's a really stable platform and the write-only database files worked really well (we lost zero data in over 5 years of operation ... even with tests that included powering down a node while in use). But ... I also believe that CockroachDB is going to be the project to finally solve these problems and it's got an SQL interface! I'm waiting for SQL joins, and then I suspect I'll be moving at least some of my clustered PostgreSQL instances to CockroachDB.

LoSboccacc 11y ago
no I mean things like this: http://docs.couchdb.org/en/stable/cluster/nodes.html#removing-a-node http://docs.couchdb.org/en/stable/cluster/nodes.html#removin... http://docs.couchdb.org/en/stable/cluster/sharding.html#moving http://docs.couchdb.org/en/stable/cluster/sharding.html#movi... http://docs.couchdb.org/en/stable/cluster/sharding.html#reshard-no-preshard http://docs.couchdb.org/en/stable/cluster/sharding.html#resh... sharding needs to be taken care manually, there is nowhere on the docs handling what happens in the event of a failover and no how to replace a failed populated node with an empty one so that the cluster can recover. those are quite big concerns if sharding with a replica set for fault tolerance is the main objective.

hnkimb3558 11y ago

(employee/founder here) Anything under 64K is perfectly reasonable to store in a CockroachDB column. Between 64K and maybe 1M is trending towards trouble. Values greater than this are going to go through CockroachDB like a goat through a python. Why is this the case? For starters, at the level of RocksDB, values greater than 64K are not jammed into SSTables (to avoid constantly rewriting them during compactions of the LSM tree). Instead, individual files are created. Also, CockroachDB has quite a lot of write amplification, which is generally OK for structured relational data, but becomes progressively more terrible for large blobs. Write amplification comes from the Raft log, as well as RocksDB's write-ahead log. What we really need is an integrated storage system for immutable blobs, something we're taking very seriously. Roughly half of the original team which built Colossus at Google are working on CockroachDB, so there's some knowledge of how to go about building such a system. While we're not sure where it would fall on our roadmap, the idea is that large blob values would be efficiently replicated and maintained through a separate subsystem. The blob column itself would just contain a pointer to the blob. The value in tight integration (a single CockroachDB cluster providing both OLTP SQL database as well as a distributed blob store) would be one deployment & admin console, and transactionally consistent blob column values (e.g. no fighting s3 eventual consistency).

LoSboccacc 11y ago
Thanks! That's fantastic to hear even if it's going to materialize later/eventually/never at least it's great to know the need is recognized. Disregarding write amplification issues for a sec, would it make things better to split binaries in 64k chunks and have them in a chunk table keyed by name and offset?
- hnkimb3558 11y ago
  I'm not really sure which strategy would benchmark best between 64K chunks, 1M chunks, or even 8M chunks. I think this requires some experimentation. Pushing them all through as 64K chunks has a lot of overhead, and you'd reap the full write amplification. Could you tell me a bit more about your use case? You can email me spencer at cockroachlabs.com. I've been meaning to work on the "CockroachDB Egg Store" (my disgusting name for a blob storage subsystem) as a Free Fridays side project for a while, but have been distracted with all manner of other enticing options. There aren't enough hours in a week...