4 ms·

Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limi

Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.

pmalynin 8y ago
I'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).
- halbritt 8y ago
  The problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
  - anotherhue 8y ago
    It also prevents use of the sendfile optimisation -- meaning a lot of Kafka perf disappears.
  - halbritt 8y ago
    One wonders why this was downvoted.
- deanmoriarty 8y ago
  Java TLS implementation is 99.9% implemented in Java itself (look at JSSE), openssl was ditched a long, long time ago. Not that I would necessarily expect the throughput to be significantly different than the native version considering the good performance of JITted Java, but was just commenting on your assumption.
  - pmalynin 8y ago
    Oh, I didn't know that, thanks for the clarification. I mostly work with C++, Python and Node in my day to day, so my assumptions are biased.
- 75dvtwin 8y ago
  Take a look at conscrypt https://github.com/google/conscrypt https://github.com/google/conscrypt According to reports into Jetty folks, using conscript for TLS, gives 10x improvements https://webtide.com/conscrypting-native-ssl-for-jetty/ https://webtide.com/conscrypting-native-ssl-for-jetty/ Note, that, from what I read, conscrypt is not compiled for any of the BSDs so if you are testing there, you will have to compile conscrypt yourself
Twirrim 8y ago
I've worked on services running on java 8 getting high throughput with AES-NI, maxing out 10Gig interfaces. Java is rarely a problem there. Where it is, you can even JNI out to openssl for your AES-NI encryption needs and spare yourself a little CPU power (we discovered that JNI isn't as big a performance hit as people would have you believe)
- halbritt 8y ago
  There's a JIRA issue that's been open for three years for kafka to support openssl with JNI: https://issues.apache.org/jira/browse/KAFKA-2561 https://issues.apache.org/jira/browse/KAFKA-2561 Not sure it's going to get done at this point. I've only recently started using kafka and found the TLS issues to be somewhat surprising. AES-NI has been around for how long now? I speculate that this is because it's come out of the hadoop community and they haven't really had much luck implementing security features.
  - Twirrim 8y ago
    https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html https://jaitechwriteups.blogspot.com/2017/10/kafka-with-open... It looks like with Java 9 the SSL stuff in Kafka saw a drastic speed boost.

Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.

pmalynin 8y ago

I'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).

halbritt 8y ago
The problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
- anotherhue 8y ago
  It also prevents use of the sendfile optimisation -- meaning a lot of Kafka perf disappears.
- halbritt 8y ago
  One wonders why this was downvoted.
deanmoriarty 8y ago
Java TLS implementation is 99.9% implemented in Java itself (look at JSSE), openssl was ditched a long, long time ago. Not that I would necessarily expect the throughput to be significantly different than the native version considering the good performance of JITted Java, but was just commenting on your assumption.
- pmalynin 8y ago
  Oh, I didn't know that, thanks for the clarification. I mostly work with C++, Python and Node in my day to day, so my assumptions are biased.
75dvtwin 8y ago
Take a look at conscrypt https://github.com/google/conscrypt https://github.com/google/conscrypt According to reports into Jetty folks, using conscript for TLS, gives 10x improvements https://webtide.com/conscrypting-native-ssl-for-jetty/ https://webtide.com/conscrypting-native-ssl-for-jetty/ Note, that, from what I read, conscrypt is not compiled for any of the BSDs so if you are testing there, you will have to compile conscrypt yourself

Twirrim 8y ago

I've worked on services running on java 8 getting high throughput with AES-NI, maxing out 10Gig interfaces. Java is rarely a problem there. Where it is, you can even JNI out to openssl for your AES-NI encryption needs and spare yourself a little CPU power (we discovered that JNI isn't as big a performance hit as people would have you believe)

halbritt 8y ago
There's a JIRA issue that's been open for three years for kafka to support openssl with JNI: https://issues.apache.org/jira/browse/KAFKA-2561 https://issues.apache.org/jira/browse/KAFKA-2561 Not sure it's going to get done at this point. I've only recently started using kafka and found the TLS issues to be somewhat surprising. AES-NI has been around for how long now? I speculate that this is because it's come out of the hadoop community and they haven't really had much luck implementing security features.
- Twirrim 8y ago
  https://jaitechwriteups.blogspot.com/2017/10/kafka-with-openssl.html https://jaitechwriteups.blogspot.com/2017/10/kafka-with-open... It looks like with Java 9 the SSL stuff in Kafka saw a drastic speed boost.