3 ms·

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same s

by jng 5y ago

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

vosper 5y ago
There’s a lot to Lucene, and there’s also a lot to ElasticSearch. And I think they’re fairly tightly coupled. But I do think that a well-funded and skilled startup team could take a run at ES. They’re a monopoly in their niche. There has to be money in disrupting them.
- jFriedensreich 5y ago
  im sure its possible in theory, but without a permissive license it would be not relevant to most applications that are interesting. now try convincing a venture backed startup to build a lucene alternative and license it permissively after seeing what happend with amazon and es...
  - PeterCorless 5y ago
    Tantivy (written in Rust) uses MIT. I can't speak to the quality of the code, but their aim is to rewrite Lucene-type functionality as a Rust crate. https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy
jFriedensreich 5y ago
yes its java and also yes there is definitely a space for a native implementation, its not that i don't want that to happen, on the contrary. but the reality is that its really hard to do and i dont see only getting rid of java to currently motivate a relevant player or large enough dev community.
jFriedensreich 5y ago
just out of interest, what was horrible about es?
- jng 5y ago
  It was brittle beyond belief. The process would run out of memory and crash really easily. You are supposed to dimension resources well, but of course you'd like to handle things more gracefully when a huge inflow of data or requests comes in, good engineering is "graceful failure" such as just rejecting requests, instead of total mayhem and destruction. It was "interesting" to discover that the monitoring plugins included by default would bring down the entire cluster. 5 beefy AWS machines brought down when pointing their own monitor at them! Yes there was significant input traffic, but I would expect more solid behavior from production-ready software. Shard migration and duplication was a beauty to watch when it worked well, but very often it wouldn't. I tried to establish a procedure to bring instances back up safely and repeatably, but sometimes I would just have to try and try again, no procedure guaranteed restoring operation in all cases. I do remember spending more than one weekend babysitting that cluster. There's a special place in my heart for that memory, and it's not a beautiful one. I was very surprised to see them IPO successfully - this test was before they were public. I assumed the open source version was just missing the kind of tools and know how to be able to work solidly without paying them or their consultants. These days, I think it's poor engineering resulting in brittleness, we shouldn't have put so much traffic to it, and probably everyone out there experiences the same behavior unless they ensure much safer margins, which probably makes it expensive but I guess who's complaining. I ended up designing and writing my own (much simpler) distributed system for what I needed (much simpler than everything ElasticSearch does, of course), and it's been in reliable operation since summer 2015. Good engineering goes a long way. Now in the process of turning that approach into a startup. If I get 1/10th as successful as ES I will be a happy camper :)
PeterCorless 5y ago
Tantivy? Written in Rust. More like Lucene than Elastic, though. https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy

by jng 5y ago

vosper 5y ago

There’s a lot to Lucene, and there’s also a lot to ElasticSearch. And I think they’re fairly tightly coupled. But I do think that a well-funded and skilled startup team could take a run at ES. They’re a monopoly in their niche. There has to be money in disrupting them.

jFriedensreich 5y ago
im sure its possible in theory, but without a permissive license it would be not relevant to most applications that are interesting. now try convincing a venture backed startup to build a lucene alternative and license it permissively after seeing what happend with amazon and es...
- PeterCorless 5y ago
  Tantivy (written in Rust) uses MIT. I can't speak to the quality of the code, but their aim is to rewrite Lucene-type functionality as a Rust crate. https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy

jFriedensreich 5y ago

yes its java and also yes there is definitely a space for a native implementation, its not that i don't want that to happen, on the contrary. but the reality is that its really hard to do and i dont see only getting rid of java to currently motivate a relevant player or large enough dev community.

jFriedensreich 5y ago

just out of interest, what was horrible about es?

jng 5y ago
It was brittle beyond belief. The process would run out of memory and crash really easily. You are supposed to dimension resources well, but of course you'd like to handle things more gracefully when a huge inflow of data or requests comes in, good engineering is "graceful failure" such as just rejecting requests, instead of total mayhem and destruction. It was "interesting" to discover that the monitoring plugins included by default would bring down the entire cluster. 5 beefy AWS machines brought down when pointing their own monitor at them! Yes there was significant input traffic, but I would expect more solid behavior from production-ready software. Shard migration and duplication was a beauty to watch when it worked well, but very often it wouldn't. I tried to establish a procedure to bring instances back up safely and repeatably, but sometimes I would just have to try and try again, no procedure guaranteed restoring operation in all cases. I do remember spending more than one weekend babysitting that cluster. There's a special place in my heart for that memory, and it's not a beautiful one. I was very surprised to see them IPO successfully - this test was before they were public. I assumed the open source version was just missing the kind of tools and know how to be able to work solidly without paying them or their consultants. These days, I think it's poor engineering resulting in brittleness, we shouldn't have put so much traffic to it, and probably everyone out there experiences the same behavior unless they ensure much safer margins, which probably makes it expensive but I guess who's complaining. I ended up designing and writing my own (much simpler) distributed system for what I needed (much simpler than everything ElasticSearch does, of course), and it's been in reliable operation since summer 2015. Good engineering goes a long way. Now in the process of turning that approach into a startup. If I get 1/10th as successful as ES I will be a happy camper :)

PeterCorless 5y ago

Tantivy? Written in Rust. More like Lucene than Elastic, though. https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy