3 ms·

From what I’ve seen working in the financial industry, hardly anybody uses lucene directly. It is typically behind elasticsearch or solr or whatever - which the

From what I’ve seen working in the financial industry, hardly anybody uses lucene directly. It is typically behind elasticsearch or solr or whatever - which then takes care of load balancing and replication and all that. For these use cases, you could swap out lucene for something faster or more “low level” in implementation without affecting the typical application using it much, as all interaction happens across some rest API.

zelphirkalt 5y ago
The point I am making is, that you would have to write that lower level or somehow different thing first. If you do it in C, then how do you make it as safe, while still having the same functionality? In huge projects this has mostly not been achieved. Large C projects (and C++ project) suffer from memory safety issues. This seems to be the general rule. Even the best make mistakes, when working with big C code bases, or there are so few of those "best", that they cannot possibly create a huge project all by themselves. If you replace a more or less safe implementation, with a buggy, memory unsafe C version of it, you are not going to make anyone happy about the performance improvement. Always assuming, that you put in "only" the same amount of time and number of people, not more, than the original implementation. The amount of things you need to keep in mind with all the memory management and other details when writing C code can also make you blind to other issues like proper input validation and implementation of invariants of application settings. You have to put a lot of energy into making sure you are not leaking memory and avoiding memory safety issues, so you lack that energy when it comes to higher level issues. This is directly affecting developer productivity.
- JetSetWilly 5y ago
  I mean, obviously using different languages has different trade offs in many dimensions. If speed were all that mattered then everything would be written in assembler. But the point at issue is that by using java you are trading off speed (potentially for other benefits). By the way - tantivy at least is implemented in rust so it shouldn’t suffer from memory safety issues just as lucene shouldn’t. And the rust language will provide some guarantees that java doesn’t - like freedom from data races.
Too 5y ago
The premise here was writing the lower piece something. Not using it.

zelphirkalt 5y ago

The point I am making is, that you would have to write that lower level or somehow different thing first. If you do it in C, then how do you make it as safe, while still having the same functionality? In huge projects this has mostly not been achieved. Large C projects (and C++ project) suffer from memory safety issues. This seems to be the general rule. Even the best make mistakes, when working with big C code bases, or there are so few of those "best", that they cannot possibly create a huge project all by themselves. If you replace a more or less safe implementation, with a buggy, memory unsafe C version of it, you are not going to make anyone happy about the performance improvement. Always assuming, that you put in "only" the same amount of time and number of people, not more, than the original implementation. The amount of things you need to keep in mind with all the memory management and other details when writing C code can also make you blind to other issues like proper input validation and implementation of invariants of application settings. You have to put a lot of energy into making sure you are not leaking memory and avoiding memory safety issues, so you lack that energy when it comes to higher level issues. This is directly affecting developer productivity.

JetSetWilly 5y ago
I mean, obviously using different languages has different trade offs in many dimensions. If speed were all that mattered then everything would be written in assembler. But the point at issue is that by using java you are trading off speed (potentially for other benefits). By the way - tantivy at least is implemented in rust so it shouldn’t suffer from memory safety issues just as lucene shouldn’t. And the rust language will provide some guarantees that java doesn’t - like freedom from data races.

Too 5y ago

The premise here was writing the lower piece something. Not using it.