24 ms·

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other p

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

manigandham 6y ago
Nice, always like seeing new projects. Is this based on Seastar? Will this be open-source?
- agallego 6y ago
  Yeah. Based on seastar. W.r.t OSS doing research on it atm. I had a VP at a major cloud say to me "we will try to use your tech" in person. So doing research before I make the decision
snidane 6y ago
Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.
- agallego 6y ago
  Thanks! Yeh I think the same, specially with the rise of core count. I think an i3.metal is at 96vcpus. I estimate a machine like that should hold around 400K partitions just fine with us. I should measure this weekend.
- sz4kerto 6y ago
  Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.
  - agallego 6y ago
    The JVM is fast. (though I'd say the best thing about the JVM is the amazingly large test suite it maintains). I think he was alluding to the fact it is much easier to develop predictable systems in a language that forces you to deal with the constraints up front. - i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. Not to mention that a lot of libraries that are immediately avail for c++ (see io_uring) take a while to get ported to java. The cost of JNI for a libaio wrapper is also expensive. Last i bench(few years back), switching to a crc32 with JNI switch alone was 30 microseconds - an eternity - before doing the work. In any case, we use seastar (seastar.io), which I'm not sure can actually be ported to java. The pinned thread per core makes a lot of sense for minimizing latency and cache pollution, etc. Externally, the feeling that apps in java are slow is real, less because JVM is slow per se, but because writing low latency apps in java is not the idiomatic way and those that do see to extract every ounce of performance of the hardware often look else where since the work is just about the same.
    - StreamBright 6y ago
      True. I think the rejection of operating systems managing resources the way it is today (dynamic resource binding) is coming. Pinned threads make a ton of sense in HPC.
    - Matthias247 6y ago
      > i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. If you want to do this, you can gain a lot of performance with having custom allocators and pools on the JVM as well. E.g. frameworks like Netty have pooling strategies for ByteBuffers. If you go that route, you can also gain a lot of performance on the JVM, might really be competitive. Unfortunately the JVM still enforces too many heap allocations since value-types are not a thing yet, but it still performs well. One of interesting things I discovered about C++/Rust vs C#/Java is that the former language family wants you to do a lot of optimizations upfront, wich typically results in good performance. But sometimes you also spend too much time into optimizing something something that won't matter in practice. Whereas the managed languages are a bit easier to work with by default, but will thereby only yield mediocre performance. However you still have the chance to look into the bottlenecks and improve them by large margins using the right approaches. In C# now even more so than in Java thanks to tools like Span and value types.
  - chillacy 6y ago
    It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.
    - agallego 6y ago
      +1K on this. tail latencies are bad.
    - sz4kerto 6y ago
      Depends. Java's GC can be tuned an infinitum. It's not a simple task and requires knowledge, but that's the beauty of it: if you need low-ish latencies, then you can tune the GC to target that instead of throughput. For example we're using relatively large heaps (partly due to inefficiencies in our code), but we still want to stay under 500 msec/request or so. So we told the G1 GC to target 150 msecs per collection, and then adjusted our heap size / code accordingly. It works well. If you need really hard limits on collection then that's a tricky problem, but that's also tricky when you're managing memory yourself.
      - lllr_finger 6y ago
        Once you start talking hundreds-to-thousands requests/second, 500ms is an incredibly long time and you're well past simple tweaks to GC. Tuning GC to a high degree is non-deterministic black magic, which is not what you're looking for at that point. Simple tweaks can go a long way for a lot of developers, but GC performance has been a problem at the last 3 organizations I've been at - and I'm not in the valley or at a FAANG - so it isn't exactly an uncommon scenario for developers.
        apta 6y ago
        The JVM now ships with ZGC, a low latency GC targetted at exactly that. https://wiki.openjdk.java.net/display/zgc/Main https://wiki.openjdk.java.net/display/zgc/Main
    - StreamBright 6y ago
      GC without tuning. With optimization, not so much.
  - lllr_finger 6y ago
    LMAX also spent a ton of time doing non-standard things with data structures to optimize cache and prevent heap allocation: https://martinfowler.com/articles/lmax.html https://martinfowler.com/articles/lmax.html Unnecessary allocations will be your bottle neck at some point when shutting data around.
  - snidane 6y ago
    You might be correct - it may be only my ignorant experience interacting with Apache Big Data technologies of present day and wondering what the hell they are doing under the hood to take them so long to process simple stuff in bulk. Perhaps open source engineers are not "paid" enough to spend countless hours carefully optimizing their programs on JVM. Properly paid engineers working on proprietary technology can surely bend and twist JVM and make it perform. It might be possible, but costly due to complexity and unpredictable nature of it. It reminds me of world of SQL where you have to run your query with many small modifications and hope that the optimizer will generate a sensible query plan while the query is still readable. That's the cost of building on top unpredictable systems. You wonder why you can't simply get access to the lower level - physical plans - and program on that level directly, since you know what your desired outcome on that level is anyway. I still wonder why new projects for performant data processing are not written for example in Rust since for this task it has all the upsides and none of the downsides of JVM.
- StreamBright 6y ago
  Because most people using JVM in production understand that your claim is without merit and performance in the JVM land can be tuned to the workload and finally it is never a single dimension decision to chose a language/ runtime. JVM had many additional properties that make it an excellent choice for big data use cases.
jonathanoliver 6y ago
@agallego I just signed up on your wait list through the vectorized.io website. I'm super interested in this project.
- agallego 6y ago
  on it.
skyde 6y ago
so this use raft for topic data not just metadata ?
- agallego 6y ago
  correct. we use one protocol, just raft - both for metadata and for data. Raft is really easy to parallelize and dispatch to multiple followers async. I measured recently on 3 i3.8xlarge instances which give you 1.2GB/s - and i got around 1.18GB/s sustained -https://twitter.com/emaxerrno/status/1260415381321084929 https://twitter.com/emaxerrno/status/1260415381321084929 Also, what's nice about using raft is that if there is a bug, we know it's w/ our implementation and not w/ the protocol. so it gives users sound reasoning.
  - skyde 6y ago
    are you hiring :-) I would love to work on this. I consider myself a RAFT expert and worked in kafka for the past 5 years.
    - agallego 6y ago
      alex@vectorized.io - let's chat. i'm looking for a senior member atm.
pjmlp 6y ago
How 10x faster? Benchmarks available? If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?
- agallego 6y ago
  p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1<<31)) \ --throughput $((1024*64)) \ --producer-props "acks=1" \ "client.id=alex.client" \ bootstrap.servers=172.31.31.28:9092 \ batch.size=81960 \ buffer.memory=$((1024 * 1024)) &> ~/nohup1.txt &
  - pjmlp 6y ago
    Thanks.
  - StreamBright 6y ago
    Did you try to tune GC (as most Kafka production systems would have GC configured)? Which JVM version?
    - koffiezet 6y ago
      Just the fact that you have to ask all these things proves the point?
      - pjmlp 6y ago
        Not really, after all most native code compilers have endless amount of configuration options as well, while Go still falls behind many use cases. Also I started to see a trend in books and blog posts regarding how to write Go code towards better performance, so it isn't a given that it excels at performance out of the box. All of which comes back to the original point that many times isn't the language, rather how it is written and what tools one makes use of. If Go is so much better than Java, Google would have replaced it already on Android with Go (battery life and such), instead they went with a mix of AOT/JIT with PGO, introducing Kotlin, while gomobile efforts were never given any serious consideration not even for the NDK.
maxdo 6y ago
No sources, no benchmarks, no production use-cases. Thanks but no.

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

manigandham 6y ago

Nice, always like seeing new projects. Is this based on Seastar? Will this be open-source?

agallego 6y ago
Yeah. Based on seastar. W.r.t OSS doing research on it atm. I had a VP at a major cloud say to me "we will try to use your tech" in person. So doing research before I make the decision

snidane 6y ago

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

agallego 6y ago
Thanks! Yeh I think the same, specially with the rise of core count. I think an i3.metal is at 96vcpus. I estimate a machine like that should hold around 400K partitions just fine with us. I should measure this weekend.
sz4kerto 6y ago
Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.
- agallego 6y ago
  The JVM is fast. (though I'd say the best thing about the JVM is the amazingly large test suite it maintains). I think he was alluding to the fact it is much easier to develop predictable systems in a language that forces you to deal with the constraints up front. - i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. Not to mention that a lot of libraries that are immediately avail for c++ (see io_uring) take a while to get ported to java. The cost of JNI for a libaio wrapper is also expensive. Last i bench(few years back), switching to a crc32 with JNI switch alone was 30 microseconds - an eternity - before doing the work. In any case, we use seastar (seastar.io), which I'm not sure can actually be ported to java. The pinned thread per core makes a lot of sense for minimizing latency and cache pollution, etc. Externally, the feeling that apps in java are slow is real, less because JVM is slow per se, but because writing low latency apps in java is not the idiomatic way and those that do see to extract every ounce of performance of the hardware often look else where since the work is just about the same.
  - StreamBright 6y ago
    True. I think the rejection of operating systems managing resources the way it is today (dynamic resource binding) is coming. Pinned threads make a ton of sense in HPC.
  - Matthias247 6y ago
    > i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. If you want to do this, you can gain a lot of performance with having custom allocators and pools on the JVM as well. E.g. frameworks like Netty have pooling strategies for ByteBuffers. If you go that route, you can also gain a lot of performance on the JVM, might really be competitive. Unfortunately the JVM still enforces too many heap allocations since value-types are not a thing yet, but it still performs well. One of interesting things I discovered about C++/Rust vs C#/Java is that the former language family wants you to do a lot of optimizations upfront, wich typically results in good performance. But sometimes you also spend too much time into optimizing something something that won't matter in practice. Whereas the managed languages are a bit easier to work with by default, but will thereby only yield mediocre performance. However you still have the chance to look into the bottlenecks and improve them by large margins using the right approaches. In C# now even more so than in Java thanks to tools like Span and value types.
- chillacy 6y ago
  It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.
  - agallego 6y ago
    +1K on this. tail latencies are bad.
  - sz4kerto 6y ago
    Depends. Java's GC can be tuned an infinitum. It's not a simple task and requires knowledge, but that's the beauty of it: if you need low-ish latencies, then you can tune the GC to target that instead of throughput. For example we're using relatively large heaps (partly due to inefficiencies in our code), but we still want to stay under 500 msec/request or so. So we told the G1 GC to target 150 msecs per collection, and then adjusted our heap size / code accordingly. It works well. If you need really hard limits on collection then that's a tricky problem, but that's also tricky when you're managing memory yourself.
    - lllr_finger 6y ago
      Once you start talking hundreds-to-thousands requests/second, 500ms is an incredibly long time and you're well past simple tweaks to GC. Tuning GC to a high degree is non-deterministic black magic, which is not what you're looking for at that point. Simple tweaks can go a long way for a lot of developers, but GC performance has been a problem at the last 3 organizations I've been at - and I'm not in the valley or at a FAANG - so it isn't exactly an uncommon scenario for developers.
      - apta 6y ago
        The JVM now ships with ZGC, a low latency GC targetted at exactly that. https://wiki.openjdk.java.net/display/zgc/Main https://wiki.openjdk.java.net/display/zgc/Main
  - StreamBright 6y ago
    GC without tuning. With optimization, not so much.
- lllr_finger 6y ago
  LMAX also spent a ton of time doing non-standard things with data structures to optimize cache and prevent heap allocation: https://martinfowler.com/articles/lmax.html https://martinfowler.com/articles/lmax.html Unnecessary allocations will be your bottle neck at some point when shutting data around.
- snidane 6y ago
  You might be correct - it may be only my ignorant experience interacting with Apache Big Data technologies of present day and wondering what the hell they are doing under the hood to take them so long to process simple stuff in bulk. Perhaps open source engineers are not "paid" enough to spend countless hours carefully optimizing their programs on JVM. Properly paid engineers working on proprietary technology can surely bend and twist JVM and make it perform. It might be possible, but costly due to complexity and unpredictable nature of it. It reminds me of world of SQL where you have to run your query with many small modifications and hope that the optimizer will generate a sensible query plan while the query is still readable. That's the cost of building on top unpredictable systems. You wonder why you can't simply get access to the lower level - physical plans - and program on that level directly, since you know what your desired outcome on that level is anyway. I still wonder why new projects for performant data processing are not written for example in Rust since for this task it has all the upsides and none of the downsides of JVM.
StreamBright 6y ago
Because most people using JVM in production understand that your claim is without merit and performance in the JVM land can be tuned to the workload and finally it is never a single dimension decision to chose a language/ runtime. JVM had many additional properties that make it an excellent choice for big data use cases.

jonathanoliver 6y ago

@agallego I just signed up on your wait list through the vectorized.io website. I'm super interested in this project.

agallego 6y ago
on it.

skyde 6y ago

so this use raft for topic data not just metadata ?

agallego 6y ago
correct. we use one protocol, just raft - both for metadata and for data. Raft is really easy to parallelize and dispatch to multiple followers async. I measured recently on 3 i3.8xlarge instances which give you 1.2GB/s - and i got around 1.18GB/s sustained -https://twitter.com/emaxerrno/status/1260415381321084929 https://twitter.com/emaxerrno/status/1260415381321084929 Also, what's nice about using raft is that if there is a bug, we know it's w/ our implementation and not w/ the protocol. so it gives users sound reasoning.
- skyde 6y ago
  are you hiring :-) I would love to work on this. I consider myself a RAFT expert and worked in kafka for the past 5 years.
  - agallego 6y ago
    alex@vectorized.io - let's chat. i'm looking for a senior member atm.

pjmlp 6y ago

How 10x faster? Benchmarks available? If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?

agallego 6y ago
p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1<<31)) \ --throughput $((1024*64)) \ --producer-props "acks=1" \ "client.id=alex.client" \ bootstrap.servers=172.31.31.28:9092 \ batch.size=81960 \ buffer.memory=$((1024 * 1024)) &> ~/nohup1.txt &
- pjmlp 6y ago
  Thanks.
- StreamBright 6y ago
  Did you try to tune GC (as most Kafka production systems would have GC configured)? Which JVM version?
  - koffiezet 6y ago
    Just the fact that you have to ask all these things proves the point?
    - pjmlp 6y ago
      Not really, after all most native code compilers have endless amount of configuration options as well, while Go still falls behind many use cases. Also I started to see a trend in books and blog posts regarding how to write Go code towards better performance, so it isn't a given that it excels at performance out of the box. All of which comes back to the original point that many times isn't the language, rather how it is written and what tools one makes use of. If Go is so much better than Java, Google would have replaced it already on Android with Go (battery life and such), instead they went with a mix of AOT/JIT with PGO, introducing Kotlin, while gomobile efforts were never given any serious consideration not even for the NDK.

maxdo 6y ago

No sources, no benchmarks, no production use-cases. Thanks but no.