5 ms·

I still don’t get why people try writing their own data store, especially in a language that's simply not very well suited to that task (and we're an almost 100

I still don’t get why people try writing their own data store, especially in a language that's simply not very well suited to that task (and we're an almost 100% Golang shop here). Seems to be a rite of passage. The requirements are literally <100LOC of wrapping an HTTP interface around Redis with using TTL - which has been battle tested for years and is both rock solid and ridiculously fast. Public service announcement: Don't write your own data store. Repeat after me: Don't write your own data store, except if you want to experimentally find out how to build data stores. It's arbitrarily hard and gets even more so at every layer. Plus, you leave behind an unmaintainable mess for the people after you. There's already a great OSS data store optimized for every use case and storage medium I could possible imagine.

deleted 11y ago
[deleted]
Maascamp 11y ago
Not sure if this applies to their use case since they mention FIFO in the context of it being a simple eviction policy, but if you _require_ FIFO semantics then both Redis and memcached are out of the question since they use nondeterministic LRU policies (memcached's LRU is particularly egregious in its nondeterminism due to the way slab allocation works).
- krenoten 11y ago
  this is rarely a concern for things that are ephemeral in nature to begin with. it's also a 20 line code change to add a similar ttl buffer into memcached.
tmaly 11y ago
I used Redis with Go and I was very happy with the results. I did have one case where I used boltdb. I run things on a lean vps, and I had a large amount of data I tried to stuff in a trie. Well it worked fine on my macbook with plenty of memory, but it ran out of memory on my vps. Using bolt helped in this case with the memory mapped file.
im_down_w_otp 11y ago
There are two datastores I would really like that don't exist. 1) An efficient, high-performance distributed persistence store for arbitrarily large CRDT's. 2) A Kafka-like highly-available distributed binary log w/ cheap topics, that doesn't require external coordination, and doesn't lose acknowledged writes (which I'll happily give up any shape of linearization guarantee for).
- endymi0n 11y ago
  can't help with 1), but as for 2), maybe Kudu is an option for you? http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafka.html http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafk...
  - im_down_w_otp 11y ago
    Uses Raft. Not highly-available enough unfortunately. Availability > Consistency for the use case that matters to me (basically a service message bus where all ACK'd writes will eventually be read by some consumer(s), but I don't care when or in what order, just that they all eventually get consumed, and where as long as any node in the cluster is up, it'll ACK a write).
    - Xorlev 11y ago
      > just that they all eventually get consumed Unless you have some sort of replication or consensus, this is still just best-effort delivery. For example, the message could be acknowledged & fsynced followed by drive death. At scale, this would happen. However, with asynchronous best-effort replication, broker death would be much less likely to lead to a loss of messages. In general, it's all about playing the odds. Acknowledged writes can still be lost in the best of systems if the entire DC catches fire. If you tune Kafka settings, you essentially get this behavior. You have to ensure ZK is up, but it will allow all topics to be published to so long as a single broker is alive.
      - im_down_w_otp 11y ago
markbnj 11y ago
Sort of came here to say this. Interesting article, especially some of the design choices (such as http for the interface), but I am most curious as to why redis, or memcached, or some other k/v store was not good enough?
- atomic77 11y ago
  I wanted to like this article. But this: > Considering the first point we decided to give up external caches like Redis, Memcached or Couchbase mainly because of additional time needed on the network. Even mysql+innodb can easily handle 10k read/write queries against a simple pk table with "millions of entries" on desktop-grade hardware. If the article was "here's a fun experiment to make a cache server in Go", fair enough.
maleck13 11y ago
Can you explain further why golang is bad for data stores, do you mean cache in particular? Seems contrary to many of the new datastores that I see being developed: etcd[1] , cockroachdb[2], influxdb[3] just to name a couple that are written in go. [1]https://github.com/coreos/etcd https://github.com/coreos/etcd [2]https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach [3]https://github.com/influxdata/influxdb https://github.com/influxdata/influxdb
- endymi0n 11y ago
  Go is awesome, but has very few facilities for effectively working very close to the machine and memory - you can't skip the runtime or GC. Of course, this mostly affects high throughput, high mutation rate, high memory use data stores. Eventually, you'll end up fighting the garbage collector all the time. ETCD isn't directly affected as it isn't so affected as it's not really about larger data or performance, cockroachdb is still more proof-of-concept of a pretty gigantic (though promising) architecture and Influx has honestly been an imperformant mess for us in Prod so far. It's not that it's not possible to write a decent data store in Go, but eventually, as your GC runs wild from the purposefully simple and naive stdlib implementations of maps for example, the neccessary solutions of aggressive pooling or mmapping off-heap space and writing your own allocators for it (while constantly casting something from and to *unsafe) will suck you deep down the rabbit hole and wish you had written it in C(++) or Rust in the first place.
jakub_h 11y ago
That sounds a little bit harsh (even if I ignore the claims of unsuitability of Go for that purpose). What if I want to write, for example, a Go port of DataDraw? Perhaps with profiling-based memory layout optimization thrown in? And maybe even caching/lazy/caching+lazy collections/maps. If I were obeying your recommendations verbatim, I'd probably still be using mainframe libraries.
andersen1488 11y ago
> Don't write your own data store I wish more people had this attitude. The amount of times I have seen people re-implementing something that PhD's have been perfecting since the 70's... it's cringe worthy.

deleted 11y ago

[deleted]

Maascamp 11y ago

Not sure if this applies to their use case since they mention FIFO in the context of it being a simple eviction policy, but if you _require_ FIFO semantics then both Redis and memcached are out of the question since they use nondeterministic LRU policies (memcached's LRU is particularly egregious in its nondeterminism due to the way slab allocation works).

krenoten 11y ago
this is rarely a concern for things that are ephemeral in nature to begin with. it's also a 20 line code change to add a similar ttl buffer into memcached.

tmaly 11y ago

I used Redis with Go and I was very happy with the results. I did have one case where I used boltdb. I run things on a lean vps, and I had a large amount of data I tried to stuff in a trie. Well it worked fine on my macbook with plenty of memory, but it ran out of memory on my vps. Using bolt helped in this case with the memory mapped file.

im_down_w_otp 11y ago

There are two datastores I would really like that don't exist. 1) An efficient, high-performance distributed persistence store for arbitrarily large CRDT's. 2) A Kafka-like highly-available distributed binary log w/ cheap topics, that doesn't require external coordination, and doesn't lose acknowledged writes (which I'll happily give up any shape of linearization guarantee for).

endymi0n 11y ago
can't help with 1), but as for 2), maybe Kudu is an option for you? http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafka.html http://blog.rodeo.io/2016/01/24/kudu-as-a-more-flexible-kafk...
- im_down_w_otp 11y ago
  Uses Raft. Not highly-available enough unfortunately. Availability > Consistency for the use case that matters to me (basically a service message bus where all ACK'd writes will eventually be read by some consumer(s), but I don't care when or in what order, just that they all eventually get consumed, and where as long as any node in the cluster is up, it'll ACK a write).
  - Xorlev 11y ago
    > just that they all eventually get consumed Unless you have some sort of replication or consensus, this is still just best-effort delivery. For example, the message could be acknowledged & fsynced followed by drive death. At scale, this would happen. However, with asynchronous best-effort replication, broker death would be much less likely to lead to a loss of messages. In general, it's all about playing the odds. Acknowledged writes can still be lost in the best of systems if the entire DC catches fire. If you tune Kafka settings, you essentially get this behavior. You have to ensure ZK is up, but it will allow all topics to be published to so long as a single broker is alive.
    - im_down_w_otp 11y ago

markbnj 11y ago

Sort of came here to say this. Interesting article, especially some of the design choices (such as http for the interface), but I am most curious as to why redis, or memcached, or some other k/v store was not good enough?

atomic77 11y ago
I wanted to like this article. But this: > Considering the first point we decided to give up external caches like Redis, Memcached or Couchbase mainly because of additional time needed on the network. Even mysql+innodb can easily handle 10k read/write queries against a simple pk table with "millions of entries" on desktop-grade hardware. If the article was "here's a fun experiment to make a cache server in Go", fair enough.

maleck13 11y ago

Can you explain further why golang is bad for data stores, do you mean cache in particular? Seems contrary to many of the new datastores that I see being developed: etcd[1] , cockroachdb[2], influxdb[3] just to name a couple that are written in go. [1]https://github.com/coreos/etcd https://github.com/coreos/etcd [2]https://github.com/cockroachdb/cockroach https://github.com/cockroachdb/cockroach [3]https://github.com/influxdata/influxdb https://github.com/influxdata/influxdb

endymi0n 11y ago
Go is awesome, but has very few facilities for effectively working very close to the machine and memory - you can't skip the runtime or GC. Of course, this mostly affects high throughput, high mutation rate, high memory use data stores. Eventually, you'll end up fighting the garbage collector all the time. ETCD isn't directly affected as it isn't so affected as it's not really about larger data or performance, cockroachdb is still more proof-of-concept of a pretty gigantic (though promising) architecture and Influx has honestly been an imperformant mess for us in Prod so far. It's not that it's not possible to write a decent data store in Go, but eventually, as your GC runs wild from the purposefully simple and naive stdlib implementations of maps for example, the neccessary solutions of aggressive pooling or mmapping off-heap space and writing your own allocators for it (while constantly casting something from and to *unsafe) will suck you deep down the rabbit hole and wish you had written it in C(++) or Rust in the first place.

jakub_h 11y ago

That sounds a little bit harsh (even if I ignore the claims of unsuitability of Go for that purpose). What if I want to write, for example, a Go port of DataDraw? Perhaps with profiling-based memory layout optimization thrown in? And maybe even caching/lazy/caching+lazy collections/maps. If I were obeying your recommendations verbatim, I'd probably still be using mainframe libraries.

andersen1488 11y ago

> Don't write your own data store I wish more people had this attitude. The amount of times I have seen people re-implementing something that PhD's have been perfecting since the 70's... it's cringe worthy.