5 ms·

50MB/s is tiny, you can push 500 on commodity disks. https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-living-through-filesystems?ref=https://www.con

50MB/s is tiny, you can push 500 on commodity disks. https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-living-through-filesystems?ref=https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-... Full talk: https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs (shameless plug)

halbritt 8y ago
With TLS?
- pmalynin 8y ago
  After the asymmetric phase, TLS is pretty cheap on modern hardware. About 1500mb/s per core (at least with AES-NI). This is why all modern Macs do transparent encryption even if you didn’t explicitly turn on FDE.
  - halbritt 8y ago
    Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.
    - pmalynin 8y ago
      I'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).
      - halbritt 8y ago
        The problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
        anotherhue 8y ago
        It also prevents use of the sendfile optimisation -- meaning a lot of Kafka perf disappears.
        halbritt 8y ago
        One wonders why this was downvoted.
      - deanmoriarty 8y ago
        Java TLS implementation is 99.9% implemented in Java itself (look at JSSE), openssl was ditched a long, long time ago. Not that I would necessarily expect the throughput to be significantly different than the native version considering the good performance of JITted Java, but was just commenting on your assumption.
        pmalynin 8y ago
        Oh, I didn't know that, thanks for the clarification. I mostly work with C++, Python and Node in my day to day, so my assumptions are biased.
      - 75dvtwin 8y ago
        Take a look at conscrypt https://github.com/google/conscrypt https://github.com/google/conscrypt According to reports into Jetty folks, using conscript for TLS, gives 10x improvements https://webtide.com/conscrypting-native-ssl-for-jetty/ https://webtide.com/conscrypting-native-ssl-for-jetty/ Note, that, from what I read, conscrypt is not compiled for any of the BSDs so if you are testing there, you will have to compile conscrypt yourself
    - Twirrim 8y ago
      I've worked on services running on java 8 getting high throughput with AES-NI, maxing out 10Gig interfaces. Java is rarely a problem there. Where it is, you can even JNI out to openssl for your AES-NI encryption needs and spare yourself a little CPU power (we discovered that JNI isn't as big a performance hit as people would have you believe)
      - halbritt 8y ago
        There's a JIRA issue that's been open for three years for kafka to support openssl with JNI: https://issues.apache.org/jira/browse/KAFKA-2561 https://issues.apache.org/jira/browse/KAFKA-2561 Not sure it's going to get done at this point. I've only recently started using kafka and found the TLS issues to be somewhat surprising. AES-NI has been around for how long now? I speculate that this is because it's come out of the hadoop community and they haven't really had much luck implementing security features.
        Twirrim 8y ago
        https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html https://jaitechwriteups.blogspot.com/2017/10/kafka-with-open... It looks like with Java 9 the SSL stuff in Kafka saw a drastic speed boost.
vvanders 8y ago
Yeah, I've got a Ceph cluster in my homelab that tops out about 80mb/s and it's pretty darn painful compared to a 2/3 disk mirrored ZFS on spinning rust.
- polskibus 8y ago
  Are there any good ceph alternatives? Why can't you use zfs directly?
  - maccam94 8y ago
    Ceph is distributed storage, ZFS is direct-attached to one host at a time.
  - vvanders 8y ago
    Yeah, there's a few but I haven't had a chance to give them a spin yet. I was mostly using Ceph to explore live migrations in Proxmox, although they just released support for ZFS based migrations(you have to migrate the whole local disk though so they aren't nearly as snappy). There's some pros to Ceph, mostly that you can add/remove drives at-will and the cluster will automatically rebalance. It's just very, very bandwidth hungry for cross-cluster communication. It doesn't matter that much since it's for a homelab, I could migrate to some 10Gbe links between the cluster at some point and it'll get snappier. ZFS is just such a solid, battle tested piece of tech that it's hard to beat it unless you have a very different set of constraints.
- scurvy 8y ago
  You're doing something very wrong if that's what Ceph maxes out at. It's almost like you're trying to do this on a 1gig network.
  - vvanders 8y ago
    Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;). Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.
vaishaksuresh 8y ago
Agreed, we regularly do 500+ MB/s bytes out on 12 broker cluster and we've tested more than twice that. We've seen brokers handle close to 5GB/s before performance degraded.
halbritt 8y ago
I understand that you can increase single-broker throughput, but one wonders what use case requires that? That'd be warranted in the case of single-partition topics which are necessary to preserve strict ordering. If one has ability to create enough partitions to mitigate hot spots, I'm not sure I see the need. My org has gone a different direction. We're using spinning rust in GCP and deploying on kubernetes to make scaling out easy. Standard disk provides surprisingly good performance (relative to AWS) and we can easily ingest 10gbps with 20-30 brokers.
- anotherhue 8y ago
  This was a three node cluster, with 3 replicas.
  - halbritt 8y ago
    My question, perhaps poorly articulated was why would someone focus on improving single-node performance vs. cluster-wide performance? I wasn't implying that one would want to run a single-node cluster.
andmarios 8y ago
I think it's a good number for a sustained load within a large cluster. 50MB/sec multiplies due to replication and consumers. Also in large clusters, the bottleneck comes often from the network switches rather than the rest of the stack. Also when it comes to sustained load, many people don't realize that 50MB/sec × 10 brokers × 24 hours × 7 days × 3600 seconds leads to 300TB per week (without counting the replication). :)
- zamadatix 8y ago
  32 port 100G switches are dirt cheap. If the network is your bottleneck you likely overspent on the wrong hardware.
dikei 8y ago
You can't really compare the two number, the Dropbox's post doesn't mention how many disks they are using for each brokers. In your example, they need 16 HDDs to reach 500MB/s, which average to ~30MB per disk.

halbritt 8y ago

With TLS?

pmalynin 8y ago
After the asymmetric phase, TLS is pretty cheap on modern hardware. About 1500mb/s per core (at least with AES-NI). This is why all modern Macs do transparent encryption even if you didn’t explicitly turn on FDE.
- halbritt 8y ago
  Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.
  - pmalynin 8y ago
    I'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).
    - halbritt 8y ago
      The problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
      - anotherhue 8y ago
        It also prevents use of the sendfile optimisation -- meaning a lot of Kafka perf disappears.
      - halbritt 8y ago
        One wonders why this was downvoted.
    - deanmoriarty 8y ago
      Java TLS implementation is 99.9% implemented in Java itself (look at JSSE), openssl was ditched a long, long time ago. Not that I would necessarily expect the throughput to be significantly different than the native version considering the good performance of JITted Java, but was just commenting on your assumption.
      - pmalynin 8y ago
        Oh, I didn't know that, thanks for the clarification. I mostly work with C++, Python and Node in my day to day, so my assumptions are biased.
    - 75dvtwin 8y ago
      Take a look at conscrypt https://github.com/google/conscrypt https://github.com/google/conscrypt According to reports into Jetty folks, using conscript for TLS, gives 10x improvements https://webtide.com/conscrypting-native-ssl-for-jetty/ https://webtide.com/conscrypting-native-ssl-for-jetty/ Note, that, from what I read, conscrypt is not compiled for any of the BSDs so if you are testing there, you will have to compile conscrypt yourself
  - Twirrim 8y ago
    I've worked on services running on java 8 getting high throughput with AES-NI, maxing out 10Gig interfaces. Java is rarely a problem there. Where it is, you can even JNI out to openssl for your AES-NI encryption needs and spare yourself a little CPU power (we discovered that JNI isn't as big a performance hit as people would have you believe)
    - halbritt 8y ago
      There's a JIRA issue that's been open for three years for kafka to support openssl with JNI: https://issues.apache.org/jira/browse/KAFKA-2561 https://issues.apache.org/jira/browse/KAFKA-2561 Not sure it's going to get done at this point. I've only recently started using kafka and found the TLS issues to be somewhat surprising. AES-NI has been around for how long now? I speculate that this is because it's come out of the hadoop community and they haven't really had much luck implementing security features.
      - Twirrim 8y ago
        https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html https://jaitechwriteups.blogspot.com/2017/10/kafka-with-open... It looks like with Java 9 the SSL stuff in Kafka saw a drastic speed boost.

vvanders 8y ago

Yeah, I've got a Ceph cluster in my homelab that tops out about 80mb/s and it's pretty darn painful compared to a 2/3 disk mirrored ZFS on spinning rust.

polskibus 8y ago
Are there any good ceph alternatives? Why can't you use zfs directly?
- maccam94 8y ago
  Ceph is distributed storage, ZFS is direct-attached to one host at a time.
- vvanders 8y ago
  Yeah, there's a few but I haven't had a chance to give them a spin yet. I was mostly using Ceph to explore live migrations in Proxmox, although they just released support for ZFS based migrations(you have to migrate the whole local disk though so they aren't nearly as snappy). There's some pros to Ceph, mostly that you can add/remove drives at-will and the cluster will automatically rebalance. It's just very, very bandwidth hungry for cross-cluster communication. It doesn't matter that much since it's for a homelab, I could migrate to some 10Gbe links between the cluster at some point and it'll get snappier. ZFS is just such a solid, battle tested piece of tech that it's hard to beat it unless you have a very different set of constraints.
scurvy 8y ago
You're doing something very wrong if that's what Ceph maxes out at. It's almost like you're trying to do this on a 1gig network.
- vvanders 8y ago
  Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;). Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.

vaishaksuresh 8y ago

Agreed, we regularly do 500+ MB/s bytes out on 12 broker cluster and we've tested more than twice that. We've seen brokers handle close to 5GB/s before performance degraded.

halbritt 8y ago

I understand that you can increase single-broker throughput, but one wonders what use case requires that? That'd be warranted in the case of single-partition topics which are necessary to preserve strict ordering. If one has ability to create enough partitions to mitigate hot spots, I'm not sure I see the need. My org has gone a different direction. We're using spinning rust in GCP and deploying on kubernetes to make scaling out easy. Standard disk provides surprisingly good performance (relative to AWS) and we can easily ingest 10gbps with 20-30 brokers.

anotherhue 8y ago
This was a three node cluster, with 3 replicas.
- halbritt 8y ago
  My question, perhaps poorly articulated was why would someone focus on improving single-node performance vs. cluster-wide performance? I wasn't implying that one would want to run a single-node cluster.

andmarios 8y ago

I think it's a good number for a sustained load within a large cluster. 50MB/sec multiplies due to replication and consumers. Also in large clusters, the bottleneck comes often from the network switches rather than the rest of the stack. Also when it comes to sustained load, many people don't realize that 50MB/sec × 10 brokers × 24 hours × 7 days × 3600 seconds leads to 300TB per week (without counting the replication). :)

zamadatix 8y ago
32 port 100G switches are dirt cheap. If the network is your bottleneck you likely overspent on the wrong hardware.

dikei 8y ago

You can't really compare the two number, the Dropbox's post doesn't mention how many disks they are using for each brokers. In your example, they need 16 HDDs to reach 500MB/s, which average to ~30MB per disk.