4 ms·

This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architectur

This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]! Aside: There used to be a site sometime back which used to distribute compiled binaries of Go code for all platforms? Is it still up any chance? [0] - https://github.com/cockroachdb/cockroach#architecture https://github.com/cockroachdb/cockroach#architecture

Bogdanovich 11y ago
It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.
- dave_ops 11y ago
  Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.
  - Bogdanovich 11y ago
    There is no benefit in that except faster startup time. As a downside you'll get a lot head/tail db keys updates.
    - dave_ops 11y ago
      Fast start times are a valuable thing for a service component. Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens. Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.
      - Bogdanovich 11y ago
        I added some more benchmarks including packing with 200M small 64 byte messages (20Gb) and consumption of that queue. There is no slowdown because of mass delete. https://github.com/bogdanovich/siberite/blob/master/docs/benchmarks.md#queue-packing-and-unpacking-64-byte-message-size https://github.com/bogdanovich/siberite/blob/master/docs/ben...
- dave_ops 11y ago
  Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs. But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.
justinsaccount 11y ago
http://nsq.io/ http://nsq.io/ http://nsq.io/overview/internals.html http://nsq.io/overview/internals.html The service you are thinking of might be https://github.com/ddollar/godist https://github.com/ddollar/godist You can see the download links on https://github.com/ddollar/forego https://github.com/ddollar/forego use it.

Bogdanovich 11y ago

It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.

dave_ops 11y ago
Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.
- Bogdanovich 11y ago
  There is no benefit in that except faster startup time. As a downside you'll get a lot head/tail db keys updates.
  - dave_ops 11y ago
    Fast start times are a valuable thing for a service component. Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens. Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.
    - Bogdanovich 11y ago
      I added some more benchmarks including packing with 200M small 64 byte messages (20Gb) and consumption of that queue. There is no slowdown because of mass delete. https://github.com/bogdanovich/siberite/blob/master/docs/benchmarks.md#queue-packing-and-unpacking-64-byte-message-size https://github.com/bogdanovich/siberite/blob/master/docs/ben...
dave_ops 11y ago
Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs. But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.

justinsaccount 11y ago

http://nsq.io/ http://nsq.io/ http://nsq.io/overview/internals.html http://nsq.io/overview/internals.html The service you are thinking of might be https://github.com/ddollar/godist https://github.com/ddollar/godist You can see the download links on https://github.com/ddollar/forego https://github.com/ddollar/forego use it.