5 ms·

There are also Toshi[1] and Sonic[2] in Rust. And Vector[3] as a Logstash alternative too. There is an issue[4] proposing to integrate Vector with Sonic and Tos

by xvilka 5y ago

There are also Toshi[1] and Sonic[2] in Rust. And Vector[3] as a Logstash alternative too. There is an issue[4] proposing to integrate Vector with Sonic and Toshi. Maybe Zinc can pursue this goal too. Always good to see people who realize that Java is unwieldy monster that will eat all your memory. Native is a way to go for big systems. [1] https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi [2] https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic [3] https://vector.dev/ https://vector.dev/ [4] https://github.com/vectordotdev/vector/issues/988 https://github.com/vectordotdev/vector/issues/988

fithisux 5y ago
You can always write it in Java/Kotlin and make it native with GraalVM. You do not need Rust necessarily. Mixed feelings yet. I am not big fan. Golang is good too
pkulak 5y ago
Java will use as much memory as you give it. Because you told it to use that much, and using more generally gives better performance. "Native" or not has nothing to do with the garbage collector. Go is a GCed language too, but I really don't think it's as performant or tuneable as Java's.
- stiray 5y ago
  Please check this, it explain difference between Java and Go garbage collectors quite nicely: https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26c60 https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26...
  - kaba0 5y ago
    It was criticized quite heavily on the relevant hn thread. But basically, go can get away with a less advanced GC due to having and relying on value types quite often. But Java and certain workloads do require the state-of-the-art GCs that the JVM provides. And a search engine likely constitutes such a one.
  - pkulak 5y ago
    Yeah... I guess. But I could write the same article about how Java doesn't need value types and extreme escape analysis because it has a generational garbage collector. 99% of garbage really does end up in the eden generation and is literally free to clean up, no different than a stack. Once you put aside all the words and see how things stack up in the real world, Go and Java are pretty comparable, usually with the JVM having a slight edge after the JIT is nice and warm. They go about it in different ways, but there's nothing wrong with that.
- tomohawk 5y ago
  And then some. Elasticsearch restarting due to out of memory errors was a common occurrence at my last gig where we used it heavily. The difference between Java GC and Go GC is stark. In several years, I've never had to touch a GC knob on Go, but in Java it is expected, and not usually worth the effort.
  - kaba0 5y ago
    It is not expected at all in like a decade (and even previously, it was mostly blog articles for very specific workloads, being out of date soon after publication). The default G1 GC is perfectly fine for the vast majority of workloads as is. In the rare case it is needed, the target pause time setting can be used to swift our preference of latency vs throughput.
francoismassot 5y ago
Another alternative in Rust is Quickwit[1]. Only search is currently distributed but indexing distribution will soon come up. Disclaimer: I'm a cofounder. [1] https://github.com/quickwit-inc/quickwit https://github.com/quickwit-inc/quickwit
axegon_ 5y ago
Melisearch is another really good one. To be honest, I'm also working on something kind of related in my spare time but by the looks of it, I'm another 4-5 months away from having a fully operational alpha version to open source. Though it's a slightly different concept - kind of a mixture between document oriented databases(think couchbase, mongo) and elasticsearch. Currently trying to get a half-baked working version that I want to use for another project with a friend of mine and use that in order to test some of the concepts at a larger scale. I opted for rust for a million and one reasons(performance, not dealing with garbage collectors, pleasant to work with) but if we set that aside, I'm with you when it comes to java: even without knowing what it's written in, you can immediately tell that elasticsearch is a product of java. It's incredibly clunky no matter how much resources you give it or now powerful your hardware is. My personal workstation is a dual-14 core xeon with 64gb of ram and even on it it feels like trying to run a demanding game on an old Acer eee with an intel atom CPU. 4 frames per minute if you are lucky. Java is a monster, I completely agree and it's made worse by the convoluted paradigms, syntax and the Frankenstein's monster that is the jvm. Good thing Kotlin came about - at least the syntax is nice.
pjmlp 5y ago
Java is also native, when you make use of an AOT compiler, which have existed in various forms since 2000. In that regard, a Java runtime compiled in AOT form is hardly any different from a Go runtime. As for the lack of value types, the solution is on the way, and apparently many keep forgetting that writing a JNI based library is much better than throw everything way and start from scratch in random language X.
bootcat 5y ago
Yeah as a developer/programmer - I always have this feeling that I should go into all native code like using c++, d, f, rust, crystal, nim, v, zig ( this is my choice as of now ) - just to get that last drop of hardware juice. I was also looking into luajit as it can have easy ffi and near native performance. I still think Java has a lot of value to bring or to learn and use for any programmer. The ecosystem of libraries, software, tooling, cross platform nature and with the advent of GraalVM - Java is still very alive and capable ! As for JVM guys - Kotlin seems be an easy migration. BTW - what do you think about dart ?
wildoats 5y ago
Java isn't the worst choice. It's mostly safe, reasonable to work with, and quite fast. RAM isn't a bottleneck on most servers these days. Even the cheapest EC2 tends to have 2GB which will run nearly any Java app fine. Sure, Rust is better. But it will take years to replicate what's already out there in Java. I'll take a database written in Java over C any day given equivalent choices.
- CodeGlitch 5y ago
  > I'll take a database written in Java over C any day given equivalent choices. I know it's not equivalent, but Redis is written in C and it's amazing what you can do with it. It is also incredibly efficient and what it does. I don't get the hate towards C. Is it the security issues? (These exist in Java too you know).
  - pjmlp 5y ago
    They do indeed, except in Java anything related to memory corruption, type confusion, integer overflow and implicit conversions are out of the picture.
  - wildoats 5y ago
    It's far easier to write an app safe from memory corruption in Java than C. Java is as safe as Rust. It's just 1/2 the speed and uses 2-3X the ram. It was the best C alternative for DB until a couple years ago.
- rapsey 5y ago
  > Sure, Rust is better. But it will take years to replicate what's already out there in Java. The Rust based search engine Tantivy is handily faster than Lucene and supports pretty much the same features.
  - pjmlp 5y ago
    So it also replicates Zookeeper capabilities and is available in SaaS versions like SearchStax?

by xvilka 5y ago

fithisux 5y ago

You can always write it in Java/Kotlin and make it native with GraalVM. You do not need Rust necessarily. Mixed feelings yet. I am not big fan. Golang is good too

pkulak 5y ago

Java will use as much memory as you give it. Because you told it to use that much, and using more generally gives better performance. "Native" or not has nothing to do with the garbage collector. Go is a GCed language too, but I really don't think it's as performant or tuneable as Java's.

stiray 5y ago
Please check this, it explain difference between Java and Go garbage collectors quite nicely: https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26c60 https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26...
- kaba0 5y ago
  It was criticized quite heavily on the relevant hn thread. But basically, go can get away with a less advanced GC due to having and relying on value types quite often. But Java and certain workloads do require the state-of-the-art GCs that the JVM provides. And a search engine likely constitutes such a one.
- pkulak 5y ago
  Yeah... I guess. But I could write the same article about how Java doesn't need value types and extreme escape analysis because it has a generational garbage collector. 99% of garbage really does end up in the eden generation and is literally free to clean up, no different than a stack. Once you put aside all the words and see how things stack up in the real world, Go and Java are pretty comparable, usually with the JVM having a slight edge after the JIT is nice and warm. They go about it in different ways, but there's nothing wrong with that.
tomohawk 5y ago
And then some. Elasticsearch restarting due to out of memory errors was a common occurrence at my last gig where we used it heavily. The difference between Java GC and Go GC is stark. In several years, I've never had to touch a GC knob on Go, but in Java it is expected, and not usually worth the effort.
- kaba0 5y ago
  It is not expected at all in like a decade (and even previously, it was mostly blog articles for very specific workloads, being out of date soon after publication). The default G1 GC is perfectly fine for the vast majority of workloads as is. In the rare case it is needed, the target pause time setting can be used to swift our preference of latency vs throughput.

francoismassot 5y ago

Another alternative in Rust is Quickwit[1]. Only search is currently distributed but indexing distribution will soon come up. Disclaimer: I'm a cofounder. [1] https://github.com/quickwit-inc/quickwit https://github.com/quickwit-inc/quickwit

axegon_ 5y ago

Melisearch is another really good one. To be honest, I'm also working on something kind of related in my spare time but by the looks of it, I'm another 4-5 months away from having a fully operational alpha version to open source. Though it's a slightly different concept - kind of a mixture between document oriented databases(think couchbase, mongo) and elasticsearch. Currently trying to get a half-baked working version that I want to use for another project with a friend of mine and use that in order to test some of the concepts at a larger scale. I opted for rust for a million and one reasons(performance, not dealing with garbage collectors, pleasant to work with) but if we set that aside, I'm with you when it comes to java: even without knowing what it's written in, you can immediately tell that elasticsearch is a product of java. It's incredibly clunky no matter how much resources you give it or now powerful your hardware is. My personal workstation is a dual-14 core xeon with 64gb of ram and even on it it feels like trying to run a demanding game on an old Acer eee with an intel atom CPU. 4 frames per minute if you are lucky. Java is a monster, I completely agree and it's made worse by the convoluted paradigms, syntax and the Frankenstein's monster that is the jvm. Good thing Kotlin came about - at least the syntax is nice.

pjmlp 5y ago

Java is also native, when you make use of an AOT compiler, which have existed in various forms since 2000. In that regard, a Java runtime compiled in AOT form is hardly any different from a Go runtime. As for the lack of value types, the solution is on the way, and apparently many keep forgetting that writing a JNI based library is much better than throw everything way and start from scratch in random language X.

bootcat 5y ago

Yeah as a developer/programmer - I always have this feeling that I should go into all native code like using c++, d, f, rust, crystal, nim, v, zig ( this is my choice as of now ) - just to get that last drop of hardware juice. I was also looking into luajit as it can have easy ffi and near native performance. I still think Java has a lot of value to bring or to learn and use for any programmer. The ecosystem of libraries, software, tooling, cross platform nature and with the advent of GraalVM - Java is still very alive and capable ! As for JVM guys - Kotlin seems be an easy migration. BTW - what do you think about dart ?

wildoats 5y ago

Java isn't the worst choice. It's mostly safe, reasonable to work with, and quite fast. RAM isn't a bottleneck on most servers these days. Even the cheapest EC2 tends to have 2GB which will run nearly any Java app fine. Sure, Rust is better. But it will take years to replicate what's already out there in Java. I'll take a database written in Java over C any day given equivalent choices.

CodeGlitch 5y ago
> I'll take a database written in Java over C any day given equivalent choices. I know it's not equivalent, but Redis is written in C and it's amazing what you can do with it. It is also incredibly efficient and what it does. I don't get the hate towards C. Is it the security issues? (These exist in Java too you know).
- pjmlp 5y ago
  They do indeed, except in Java anything related to memory corruption, type confusion, integer overflow and implicit conversions are out of the picture.
- wildoats 5y ago
  It's far easier to write an app safe from memory corruption in Java than C. Java is as safe as Rust. It's just 1/2 the speed and uses 2-3X the ram. It was the best C alternative for DB until a couple years ago.
rapsey 5y ago
> Sure, Rust is better. But it will take years to replicate what's already out there in Java. The Rust based search engine Tantivy is handily faster than Lucene and supports pretty much the same features.
- pjmlp 5y ago
  So it also replicates Zookeeper capabilities and is available in SaaS versions like SearchStax?