5 ms·

Introducing Tank, a very high performance log

12 points by markpapadakis 10y ago

dozzie 10y ago
It would be much less confusing if you didn't compare a storage mechanism (append-only log) to message transporters (Kafka).
_jezell_ 10y ago
Would love to see a non JVM kafka alternative. But, I wish people would stop building infrastructure with C/C++ when there are safer alternatives like rust and go.
- LaurenceW1 10y ago
  Really? Safer? How about writing things in languages that are more mature instead of hoping on the newest bandwagon.
  - _jezell_ 10y ago
    Said the assembly programmer to the C programmer.
- dozzie 10y ago
  Given that Kafka is a message router, what don't you like about Fluentd (Ruby) or Heka (Go)?
  - _jezell_ 10y ago
    Besides the fact that ruby performance is terrible and heka was deprecated and abandoned by Mozilla due to design flaws (https://mail.mozilla.org/pipermail/heka/2016-May/001059.html https://mail.mozilla.org/pipermail/heka/2016-May/001059.html)?
    - dozzie 10y ago
      I'm not to defend Heka, as I don't use it, but those "design flaws" only concerned development, not operations. If you read the message you quote, you would see that Mozilla is still using it. And I take it you have measured Fluentd's performance, so you say so from experience, not flail around the typical opinion about language that can safely be attached to Java as well?
continuations 10y ago
How does this compare to ZeroMQ?
- markpapadakis 10y ago
  This is a message broker. You can publish messages to it, and it will retain them for as long as you want(it supports retention and compaction semantics), and you can consume the messages, starting from any message(messages are identified by sequence numbers). This is similar to Apache Kafka and - to a lesser extent - Azure Events Hub )
deadmutex 10y ago
Wish the post talked real numbers, as "very high throughput and very low latency" means very different things to different people. For example, a 1 millisecond latency is "very low latency" for web related things, but might be a lifetime when comparing it to a top of the line trading application.
- markpapadakis 10y ago
  You are right, I should have elaborated on those claims, and I plan to add a relevant Wiki page to address this shortcoming. However, you can use the tank-cli bm command to benchmark client to broker to client (end to end) latency. We get sub-ms (a few hundred microseconds) latency here, and even on a very busy Tank broker (it takes effort to get there), I don't foresee situations where you can get over 1ms latency for any operation. For throughput, Tank relies on sendfile() and readahead() for very high throughput - and thanks to the very tight packing of the messages, you will likely saturate your disk/network easily.
- deleted 10y ago
  [deleted]
- markpapadakis 10y ago
  You may also find this interesting: https://github.com/phaistos-networks/TANK/issues/14 https://github.com/phaistos-networks/TANK/issues/14

12 points by markpapadakis 10y ago

It would be much less confusing if you didn't compare a storage mechanism (append-only log) to message transporters (Kafka).

_jezell_ 10y ago

Would love to see a non JVM kafka alternative. But, I wish people would stop building infrastructure with C/C++ when there are safer alternatives like rust and go.

LaurenceW1 10y ago
Really? Safer? How about writing things in languages that are more mature instead of hoping on the newest bandwagon.
- _jezell_ 10y ago
  Said the assembly programmer to the C programmer.
dozzie 10y ago
Given that Kafka is a message router, what don't you like about Fluentd (Ruby) or Heka (Go)?
- _jezell_ 10y ago
  Besides the fact that ruby performance is terrible and heka was deprecated and abandoned by Mozilla due to design flaws (https://mail.mozilla.org/pipermail/heka/2016-May/001059.html https://mail.mozilla.org/pipermail/heka/2016-May/001059.html)?
  - dozzie 10y ago
    I'm not to defend Heka, as I don't use it, but those "design flaws" only concerned development, not operations. If you read the message you quote, you would see that Mozilla is still using it. And I take it you have measured Fluentd's performance, so you say so from experience, not flail around the typical opinion about language that can safely be attached to Java as well?

continuations 10y ago

How does this compare to ZeroMQ?

markpapadakis 10y ago
This is a message broker. You can publish messages to it, and it will retain them for as long as you want(it supports retention and compaction semantics), and you can consume the messages, starting from any message(messages are identified by sequence numbers). This is similar to Apache Kafka and - to a lesser extent - Azure Events Hub )

deadmutex 10y ago

Wish the post talked real numbers, as "very high throughput and very low latency" means very different things to different people. For example, a 1 millisecond latency is "very low latency" for web related things, but might be a lifetime when comparing it to a top of the line trading application.

markpapadakis 10y ago
You are right, I should have elaborated on those claims, and I plan to add a relevant Wiki page to address this shortcoming. However, you can use the tank-cli bm command to benchmark client to broker to client (end to end) latency. We get sub-ms (a few hundred microseconds) latency here, and even on a very busy Tank broker (it takes effort to get there), I don't foresee situations where you can get over 1ms latency for any operation. For throughput, Tank relies on sendfile() and readahead() for very high throughput - and thanks to the very tight packing of the messages, you will likely saturate your disk/network easily.
deleted 10y ago
[deleted]
markpapadakis 10y ago
You may also find this interesting: https://github.com/phaistos-networks/TANK/issues/14 https://github.com/phaistos-networks/TANK/issues/14