4 ms·

From my experience, yes. Lucene is a "production" state of art library and Solr/Elasticsearch is very used in many scenarios. This expertise is very on demand.

by bratao 7y ago

From my experience, yes. Lucene is a "production" state of art library and Solr/Elasticsearch is very used in many scenarios. This expertise is very on demand. My company personally migrated from ElasticSearch to https://vespa.ai/ https://vespa.ai/ and could not be happier. Faster and way easier to maintain a cluster. The "Application Packages" feature present in Vespa opened many opportunities to improve our product( Curiously we use Lucene inside our custom application for a "Search map" functionality, for something like that https://www.lexisnexis.com/en-us/products/lexis-advance/search-term-maps.page https://www.lexisnexis.com/en-us/products/lexis-advance/sear... ) . I highly recommend it!

simplechris 7y ago
I am very interested in hearing about your experiences migrating to vespa. When it was opensourced we all thought it would be an absolute game-changer, but I see so few people building new products (or migrating existing products to) vespa.
- bratao 7y ago
  You can contact me. I'm also planning to do a blog post comparing with Solr and Elasticsearch. I think that naturally it takes some time to adopt a solution like that. And the ecosystem it still at it infancy. But, randomly, a project using Vespa appeared in my GitHub timeline today (https://github.com/rdoume/News_API https://github.com/rdoume/News_API). So, the adoption is increasing. For me Vespa is a absolute game-change, in features and as someone said here, ES looks like it intentionally complicated to maintain. With nodes randomly getting unhealthy. Vespa is like Redis to me. I completely forgot about maintaining it and works great. It makes a world of difference in our product, and I take every opportunity to evangelize it.
  - otisg 7y ago
    That's interesting to hear. Would love to read about how Vespa compares to Solr and ES. This may be of interest to you: https://sematext.com/opensee/report/project/trend?q=ElasticSearch,Solr,Vespa,Lucene https://sematext.com/opensee/report/project/trend?q=ElasticS... Would you happen to know how Vespa compares to ES in terms of memory or CPU footprint? Have you done apples to apples comparison by any chance?
    - bratao 7y ago
      I do not have a completely fair comparison. But a migration from Elasticsearch 5 (2016) to the Vespa 7 (2019) we reduced half of our nodes, and cut in half the average response time. Another amazing feature during the migration, is that Vespa allows you to reduce or increase the number of nodes dynamically. And it take full care about the data distribution. In ES we (used to) had to follow the limits of the pre-configured number of shards/replicas during the Index creation.
      - ksec 7y ago
        I have been wondering why Vespa isn't getting much traction. Everyone still defaults to ES, even in new project.
        lovelearning 7y ago
        It should be marketed better, I feel. Some SEO for "Solr/lucene vs X" queries might help. I have been spending the last 3-4 months studying open source and commercial search systems, but it's only in this thread that I discovered Vespa.
onethumb 7y ago
This sounds very interesting! What is the size & scale of your data, if you don't mind sharing? (How many documents, total storage footprint, etc) Thanks!
atombender 7y ago
I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's approach to schema and configuration changes. To make any kind of change, or indeed set up an index, you have to create that "application package" containing your schema and configuration in the form of files, and then use separate REST APIs to "upload", "prepare" and "activate" it. There's a CLI tool to help perform those steps, at least. It's nice that they're more consistent and rigid about schema and config evolution than Elasticsearch. But it's not exactly operator-friendly, at least not for first-time users with no pre-existing operations based around Vespa. The package design also makes it more cumbersome to perform programmatic updates for a schema. I once worked on a SaaS project where we indexed data in Elasticsearch — arbitrary documents where we didn't know the schema ahead of time, because we just accepted any JSON document posted by the client. With ES, we could just use its dynamic mapping support, which automatically creates field definitions when new fields arrive (using regex-based templates). Do you know how long a package update takes in Vespa, to add, say, a single field? The Vespa documentation is also pretty terrible, in my opinion. They explain a lot of things, but it's confusingly written, uses a lot of homegrown terminology, and neglects to collect all the reference documentation in one place. For example, you can't find an overview of the entire API — fragments of it are just scattered across a dozen or so unrelated pages. Lastly, Vespa is Java. One of the biggest challenges maintaining Elasticsearch is controlling its resource consumption. You have to give it a lot of RAM, and it's never clear how much it needs and what configuration settings and usage patterns affect its memory use. Tuning it is something of a dark art. I don't know exactly how Vespa is implemented (is it all pure Java?), but I'm worried that, being a JVM app, it has the same shortcomings.
- andreer 7y ago
  Just an FYI to your last paragraph: The core indexing/ranking/storage components of Vespa are C++, and run in a separate process (no jni). In my own attempt to compare the two, I found the memory consumption of Vespa was easier to predict and understand (there are formulas for it in the documentation).

by bratao 7y ago

simplechris 7y ago

I am very interested in hearing about your experiences migrating to vespa. When it was opensourced we all thought it would be an absolute game-changer, but I see so few people building new products (or migrating existing products to) vespa.

bratao 7y ago
You can contact me. I'm also planning to do a blog post comparing with Solr and Elasticsearch. I think that naturally it takes some time to adopt a solution like that. And the ecosystem it still at it infancy. But, randomly, a project using Vespa appeared in my GitHub timeline today (https://github.com/rdoume/News_API https://github.com/rdoume/News_API). So, the adoption is increasing. For me Vespa is a absolute game-change, in features and as someone said here, ES looks like it intentionally complicated to maintain. With nodes randomly getting unhealthy. Vespa is like Redis to me. I completely forgot about maintaining it and works great. It makes a world of difference in our product, and I take every opportunity to evangelize it.
- otisg 7y ago
  That's interesting to hear. Would love to read about how Vespa compares to Solr and ES. This may be of interest to you: https://sematext.com/opensee/report/project/trend?q=ElasticSearch,Solr,Vespa,Lucene https://sematext.com/opensee/report/project/trend?q=ElasticS... Would you happen to know how Vespa compares to ES in terms of memory or CPU footprint? Have you done apples to apples comparison by any chance?
  - bratao 7y ago
    I do not have a completely fair comparison. But a migration from Elasticsearch 5 (2016) to the Vespa 7 (2019) we reduced half of our nodes, and cut in half the average response time. Another amazing feature during the migration, is that Vespa allows you to reduce or increase the number of nodes dynamically. And it take full care about the data distribution. In ES we (used to) had to follow the limits of the pre-configured number of shards/replicas during the Index creation.
    - ksec 7y ago
      I have been wondering why Vespa isn't getting much traction. Everyone still defaults to ES, even in new project.
      - lovelearning 7y ago
        It should be marketed better, I feel. Some SEO for "Solr/lucene vs X" queries might help. I have been spending the last 3-4 months studying open source and commercial search systems, but it's only in this thread that I discovered Vespa.

onethumb 7y ago

This sounds very interesting! What is the size & scale of your data, if you don't mind sharing? (How many documents, total storage footprint, etc) Thanks!

atombender 7y ago

I've looked at Vespa a bit. It looks pretty good. It's also readily apparent that it's an ancient system that's grown out of an in-house project, and that its design has accrued a lot of oddities over the years from lack of careful, co-ordinated design. It includes a bunch of esoteric features (like the "predicate" function) that have obviously grown out of Yahoo's own internal architecture. One curiosity is Vespa's approach to schema and configuration changes. To make any kind of change, or indeed set up an index, you have to create that "application package" containing your schema and configuration in the form of files, and then use separate REST APIs to "upload", "prepare" and "activate" it. There's a CLI tool to help perform those steps, at least. It's nice that they're more consistent and rigid about schema and config evolution than Elasticsearch. But it's not exactly operator-friendly, at least not for first-time users with no pre-existing operations based around Vespa. The package design also makes it more cumbersome to perform programmatic updates for a schema. I once worked on a SaaS project where we indexed data in Elasticsearch — arbitrary documents where we didn't know the schema ahead of time, because we just accepted any JSON document posted by the client. With ES, we could just use its dynamic mapping support, which automatically creates field definitions when new fields arrive (using regex-based templates). Do you know how long a package update takes in Vespa, to add, say, a single field? The Vespa documentation is also pretty terrible, in my opinion. They explain a lot of things, but it's confusingly written, uses a lot of homegrown terminology, and neglects to collect all the reference documentation in one place. For example, you can't find an overview of the entire API — fragments of it are just scattered across a dozen or so unrelated pages. Lastly, Vespa is Java. One of the biggest challenges maintaining Elasticsearch is controlling its resource consumption. You have to give it a lot of RAM, and it's never clear how much it needs and what configuration settings and usage patterns affect its memory use. Tuning it is something of a dark art. I don't know exactly how Vespa is implemented (is it all pure Java?), but I'm worried that, being a JVM app, it has the same shortcomings.

andreer 7y ago
Just an FYI to your last paragraph: The core indexing/ranking/storage components of Vespa are C++, and run in a separate process (no jni). In my own attempt to compare the two, I found the memory consumption of Vespa was easier to predict and understand (there are formulas for it in the documentation).