9 ms·

Call me maybe: Elasticsearch 1.5.0

276 points by tylertreat 11y ago

phamilton 11y ago
TL;DR; All previous data loss scenarios still exist, however the window during which data can be lost is much smaller.
- joseraul 11y ago
  And all known defects are now documented. http://www.elastic.co/guide/en/elasticsearch/resiliency/current/ http://www.elastic.co/guide/en/elasticsearch/resiliency/curr...
Meekro 11y ago
Elasticsearch has its flaws, but are there really any alternatives that do what it does? In my case, I need to do fulltext search across millions of documents -- nearly a TB of text in all, and still growing. Many database engines can do basic fulltext, but what else can scale like Elasticsearch does while offering powerful fulltext features like fuzzy matching and "more like this?"
- nl 11y ago
  There is Solr of course. But I think the recommendations the post are entirely reasonable: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.
  - NDizzle 11y ago
    This makes me feel good because that's the same conclusion I came to with my ES implementation almost 2 years ago.
- mavdi 11y ago
  Exactly... ElasticSearch doesn't market itself as the best fault tolerant DB. If you're using ES as your main data persistence point, you're missing the point of it.
- tylertreat 11y ago
  I think the main point is that Elasticsearch should not be used as a primary store. It seems obvious (or maybe not?), but some people do use it as such. Just like Redis should only be used as a cache.
- rch 11y ago
  Riak search works well. The indexing is just Solr, but the cluster and data itself is managed differently.
  - sargun 11y ago
    Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
  - sargun 11y ago
    Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
- threeseed 11y ago
  There are a few options e.g. Microsoft FAST or you can roll your own. However Lucene based systems can leverage the wide variety of addons and is battle tested so it's hard to recommend against it. Just remember the golden rule of architecture. Always assume every component will catastrophically fail at some point and design accordingly. In the case of ElasticSearch/Solr it means don't use it as a primary data store, backup regularly and have a mechanism to insert deltas.
  - vosper 11y ago
    Rolling your own is surely the worst possible option - there's very little chance you'll build something better than ElasticSearch on your own, and a very high chance that you'll build something much worse.
  - mewwts 11y ago
    In order to use any remains of FAST you would need a sharepoint installation. Which is not anywhere near free or open source, though. I think the DB + ES/Solr is the best alternative. Possibly doing snapshots often from ES could help eliminate the need for a db?
- curun1r 11y ago
  I haven't used it in ~5 years, but Sphinx (http://sphinxsearch.com http://sphinxsearch.com) might be a reasonable competitor. Back then it was pretty pleasant to use...especially it's indexing abilities, which were significantly faster than ES. I can't speak to it's distributed ability since our index was never that large, but the documentation purports to have that capability. Sphinx has some curious features, like being able to run as a MySQL storage engine (no doubt the result of the author being ex-MySQL), but it's also got most of the features that you come to expect in search engine, so I think it's fair to call it a competitor. That said, we still use ES.
  - gpsarakis 11y ago
    Sphinx does indeed a very good job and is continuously improved. It is very fast and with small memory footprint. We also built https://techusearch.com https://techusearch.com which uses Sphinx indexes (think of a SaaS version of Elasticsearch backed by Sphinx).
- platform 11y ago
  It is commercially licensed, but MarkLogic scales horizontally and is capable of full text search and structured queries (via xquery) https://developer.marklogic.com/learn/2009-07-search-api-walkthrough https://developer.marklogic.com/learn/2009-07-search-api-wal... They also have Alert queries, which are similar to ElasticSearch percolation feature. MarkLogic is also an ACID compliant store with configurable HA, as well as asynch replication to a remote site. it could potentially serve if correctly configured, as the primary store. --- ArangoDB might have these capabilities as well. And again, it is an ACID compliant database so the data committed, the indices, and the resulting searches should respect transactional boundaries at any moment of time. I had not looked at ArangoDB deep enough to understand what limitations it might have though. It does not have similarity search capability, and hierarchical data like ElasticSearch has https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html ---- I checked RethinkDB, and it appears that it suggests to integrate with ElasticSearch, meaning that elastic search will have a copy of the data from Rethink (and therefore subject to commit protocol of ElasticSearch) --- Somebody already mentioned Riak with search capability. I think this one is much more similar to Marklogic's capability (and it is free) The advantage of these types of systems is that your primary store and text indexes are sitting on top of the single copy of data (or replicated invisibly within the infrastructure) http://docs.basho.com/riak/latest/dev/using/search/ http://docs.basho.com/riak/latest/dev/using/search/
- deleted 11y ago
  [deleted]
- digitalzombie 11y ago
  ES is probably to easiest to cluster. There's Solr which is a much more matured technology but wasn't cluster in mind, I think it uses zookeeper. Another Lucene base tech is RavenDb but IIRC it's for microsoft. what else offer something like ES? Any lucene based nosql, ES is lucened base anyway. You can roll your own and use Lucene library.
temuze 11y ago
I'm pretty pleased with Elasticsearch's progress on durability. The snapshot/restore feature has been pretty nice to work with! That said, having a "single source of truth" and regularly refreshing Elasticsearch is a huge pain in the butt. I currently maintain a ~500 line syncing script that takes ~15 minutes to run. Adding a new field in a doc_type means: - Adding a column in Postgres - Adding a field in the doc_type mapping (I think being explicit with the field is better practice) - Adding code to the syncer to update the field. Ouch. Also, the syncing scripts required a steep learning curve. At first, I was upserting everything and the syncing script took forever. To solve this, I made the syncer fetch all the documents in Postgres and all the documents in Elasticsearch then update the specific changes (thankfully, the dataset is small enough that it easily fits in memory). I'd really love to scrap this portion of my infrastructure...
- tacticus 11y ago
  Could you use something like the notify[0] and listen commands to help with this? http://www.postgresql.org/docs/9.0/static/sql-notify.html http://www.postgresql.org/docs/9.0/static/sql-notify.html
  - temuze 11y ago
    Maybe? It would be interesting to see what a Postgres - Elasticsearch foreign data wrapper would look like...
- ComNik 11y ago
  Just today I successfully finished an experiment: An existing web-app with Postgres as the source of truth pushing update events to a Kafka queue. From there, a screenful of Go forwards those events into Elasticsearch. It doesn't solve every problem and it might be a lot of new parts if you're not going to use Kafka for anything else. I will be using it for other things like caching and push events. This kind of syncing problem seems to crop up in a lot of places. A nice thing is that I don't have to care about Elasticsearch durability much, because I can simply rerun the ingester from the beginning of the log, as long as Kafka doesn't lose data.
w8rbt 11y ago
The 'Recap' section has good advice to address the issue of data loss: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.
- machbio 11y ago
  I do something similar but my data is not that important, as most of you guys work on - I have a very high rate of data being input from users for curating purposes in a bioinformatics lab..So I load the data onto a mongodb cluster and run the elasticsearch-river-mongodb [1] to keep the elasticsearch in sync with the mongodb.. I do not know whether it solves this problem - but I am happy that I have data safe in mongodb - any inputs regarding my method would be appreciated.. [1]https://github.com/richardwilly98/elasticsearch-river-mongodb.git https://github.com/richardwilly98/elasticsearch-river-mongod...
  - benburton 11y ago
    Watch out, rivers are being deprecated: https://www.elastic.co/blog/deprecating_rivers https://www.elastic.co/blog/deprecating_rivers
  - cwyers 11y ago
    "Data safe in Mongo" seems... less reassuring than I think you meant it to be: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
  - nosequel 11y ago
    > I am happy that I have data safe in mongodb Oh no. So you are protecting your data from unsafe Elasticsearch by storing it in an even more unsafe MongoDB[1]? You should probably put a database that knows how to store data somewhere in your stack and learn to use that. [1]: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
teraflop 11y ago
> How often the translog is fsynced to disk. Defaults to 5s. [...] In this test we kill random nodes and restart them. [...] In Elasticsearch, write acknowledgement takes place before the transaction is flushed to disk, which means you can lose up to five seconds of writes by default. In this particular run, ES lost about 10% of acknowledged writes. Something bothers me about this: if the bug was merely a failure to call fsync() before acknowledging an operation, then killing processes shouldn't be enough to cause data loss. Once you write to a file and the syscall returns, the written data goes into the OS's buffers, and even if the process is killed it won't be lost. The only time fsync matters is if the entire machine dies (because of power loss or a kernel panic, for instance) before those buffers can be flushed. So is the data actually not even making it to the OS before being acked to the client? Or is Jepsen doing something more sophisticated, like running each node in a VM with its own block device instead of sharing the host's filesystem?
- batbomb 11y ago
  Write acknowledgement in this case means committed to memory, i.e. the transaction was recorded to a translog buffer (which is fsync'ed to disk every 5 seconds), and changes were applied to the database in memory. The translog (aka write ahead log) has a separate thread which will do fsync every 5 seconds. Unless ES blocks on the translog fsync, you can have lost transactions in the 5 second window.
  - teraflop 11y ago
    The only thing that fsync() does is take data which is already in the operating system buffers associated with a file descriptor, and write it physically to the disk. If the thread that runs every 5 seconds is really just fsync'ing, then when the process gets killed, any un-synced data is still in those buffers and will be available to read when it restarts, exactly the same as if fsync() had been called. The OS is responsible for maintaining a consistent view of the filesystem such that, unless the machine totally fails (or you circumvent the FS and actually inspect the live block device), fsync() appears to be a no-op. It sounds like what you're saying is that the documentation is wrong, and that the translog thread is actually pulling data from Elasticsearch's internal buffers and writing it to files. In that case, the documentation which refers to that operation as "fsync" is very badly misleading because it disguises what failure modes it's actually protecting against.
beachstartup 11y ago
waaaay back in the mid 2000s when search engines were large, complex, expensive pieces of enterprise software not unlike a commercial database, it was always assumed (and the customer told) that the search engine cuts corners for the sake of speed, in both indexing and searching, and that it should never be counted on as a database, or any kind of authoritative data store. it does one thing: search staggering amounts of information quickly at acceptable levels of performance and accuracy. looks like expectations have moved beyond that, which is good, but it's not an easy problem to solve... especially with billions or trillions of documents in the index. leaving out all the the important stuff that made a database slow is what made a search engine fast.
larrywright 11y ago
I bet the people who work for these database companies have nightmares about getting bug reports from Kyle. But seriously, his work is super impressive. Kudos to Stripe for funding this.
- VeejayRampay 11y ago
  Isn't a bug report always a good thing? I mean the real nightmare is your users sending outraged emails about corrupted data and you have no idea what's going on. With the bug report (and especially from Kyle) at least you got something to chew on. As people say, a reported bug is a bug half-fixed.
jodah 11y ago
Key takeaway for me: "crash-restart durable storage is a prerequisite for most consensus algorithms, I have a hunch we’ll see stronger guarantees as Elasticsearch moves towards consensus on writes." It seems like ElasticSearch is inching towards a proper consensus algorithm, at least optionally, which makes me wonder yet again, why not just implement RAFT? While I won't speculate I will point out that the answer for other systems appears to have been related to ego (ie, distributed systems are easy).
- jpgvm 11y ago
  The funny thing is there is a community plugin that fixes almost all of this by using Zookeeper for discovery and master election. The not so funny thing is Elastic employed the primary developer of said plugin and basically shut it down.
  - kimchy 11y ago
    actually, it doesn't fixes the mentioned bug. I am only posting this here to make sure people won't go and try and install it thinking it does... There is a difference between cluster level master election and replication semantics in Elasticsearch (and other similar systems). Even if you use Zookeeper for cluster level master election, one still needs to handle point to point replication (which doesn't go through zk).
    - jpgvm 11y ago
      I should have been more specific. The ZK plugin fixes the issues related to network split-brains and other nasty partition conditions. The newly discovered translog issue is also a problem and is not solved by ZK but is also a lot less scary.
dang 11y ago
Also https://news.ycombinator.com/item?id=9475620 https://news.ycombinator.com/item?id=9475620.
nyir 11y ago
Kind of OT, but losing data in these cases is not even what I'm most concerned about with ES: I had to recreate the ES cluster way too many times now that I'm really glad that PostgreSQL keeps on running, or just restarts without bricking the data files, even after OOM or out of disk errors.
Jarlakxen 11y ago
One more thing to add to the Elastic reliability: > We recommend installing the Java 8 update 20 or later, or Java 7 update 55 or later. Previous versions of Java 7 are known to have bugs that can cause index corruption and data loss. Elasticsearch will refuse to start if a known-bad version of Java is used. http://www.elastic.co/guide/en/elasticsearch/reference/1.5/setup.html http://www.elastic.co/guide/en/elasticsearch/reference/1.5/s...

276 points by tylertreat 11y ago

phamilton 11y ago

TL;DR; All previous data loss scenarios still exist, however the window during which data can be lost is much smaller.

joseraul 11y ago
And all known defects are now documented. http://www.elastic.co/guide/en/elasticsearch/resiliency/current/ http://www.elastic.co/guide/en/elasticsearch/resiliency/curr...

Meekro 11y ago

Elasticsearch has its flaws, but are there really any alternatives that do what it does? In my case, I need to do fulltext search across millions of documents -- nearly a TB of text in all, and still growing. Many database engines can do basic fulltext, but what else can scale like Elasticsearch does while offering powerful fulltext features like fuzzy matching and "more like this?"

nl 11y ago
There is Solr of course. But I think the recommendations the post are entirely reasonable: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.
- NDizzle 11y ago
  This makes me feel good because that's the same conclusion I came to with my ES implementation almost 2 years ago.
mavdi 11y ago
Exactly... ElasticSearch doesn't market itself as the best fault tolerant DB. If you're using ES as your main data persistence point, you're missing the point of it.
tylertreat 11y ago
I think the main point is that Elasticsearch should not be used as a primary store. It seems obvious (or maybe not?), but some people do use it as such. Just like Redis should only be used as a cache.
rch 11y ago
Riak search works well. The indexing is just Solr, but the cluster and data itself is managed differently.
- sargun 11y ago
  Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
- sargun 11y ago
  Riak Search 2.0, or Yokozuna is what you're thinking about. DISCLAIMER: I work for Basho.
threeseed 11y ago
There are a few options e.g. Microsoft FAST or you can roll your own. However Lucene based systems can leverage the wide variety of addons and is battle tested so it's hard to recommend against it. Just remember the golden rule of architecture. Always assume every component will catastrophically fail at some point and design accordingly. In the case of ElasticSearch/Solr it means don't use it as a primary data store, backup regularly and have a mechanism to insert deltas.
- vosper 11y ago
  Rolling your own is surely the worst possible option - there's very little chance you'll build something better than ElasticSearch on your own, and a very high chance that you'll build something much worse.
- mewwts 11y ago
  In order to use any remains of FAST you would need a sharepoint installation. Which is not anywhere near free or open source, though. I think the DB + ES/Solr is the best alternative. Possibly doing snapshots often from ES could help eliminate the need for a db?
curun1r 11y ago
I haven't used it in ~5 years, but Sphinx (http://sphinxsearch.com http://sphinxsearch.com) might be a reasonable competitor. Back then it was pretty pleasant to use...especially it's indexing abilities, which were significantly faster than ES. I can't speak to it's distributed ability since our index was never that large, but the documentation purports to have that capability. Sphinx has some curious features, like being able to run as a MySQL storage engine (no doubt the result of the author being ex-MySQL), but it's also got most of the features that you come to expect in search engine, so I think it's fair to call it a competitor. That said, we still use ES.
- gpsarakis 11y ago
  Sphinx does indeed a very good job and is continuously improved. It is very fast and with small memory footprint. We also built https://techusearch.com https://techusearch.com which uses Sphinx indexes (think of a SaaS version of Elasticsearch backed by Sphinx).
platform 11y ago
It is commercially licensed, but MarkLogic scales horizontally and is capable of full text search and structured queries (via xquery) https://developer.marklogic.com/learn/2009-07-search-api-walkthrough https://developer.marklogic.com/learn/2009-07-search-api-wal... They also have Alert queries, which are similar to ElasticSearch percolation feature. MarkLogic is also an ACID compliant store with configurable HA, as well as asynch replication to a remote site. it could potentially serve if correctly configured, as the primary store. --- ArangoDB might have these capabilities as well. And again, it is an ACID compliant database so the data committed, the indices, and the resulting searches should respect transactional boundaries at any moment of time. I had not looked at ArangoDB deep enough to understand what limitations it might have though. It does not have similarity search capability, and hierarchical data like ElasticSearch has https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html https://docs.arangodb.com/2.3/IndexHandling/Fulltext.html ---- I checked RethinkDB, and it appears that it suggests to integrate with ElasticSearch, meaning that elastic search will have a copy of the data from Rethink (and therefore subject to commit protocol of ElasticSearch) --- Somebody already mentioned Riak with search capability. I think this one is much more similar to Marklogic's capability (and it is free) The advantage of these types of systems is that your primary store and text indexes are sitting on top of the single copy of data (or replicated invisibly within the infrastructure) http://docs.basho.com/riak/latest/dev/using/search/ http://docs.basho.com/riak/latest/dev/using/search/
deleted 11y ago
[deleted]
digitalzombie 11y ago
ES is probably to easiest to cluster. There's Solr which is a much more matured technology but wasn't cluster in mind, I think it uses zookeeper. Another Lucene base tech is RavenDb but IIRC it's for microsoft. what else offer something like ES? Any lucene based nosql, ES is lucened base anyway. You can roll your own and use Lucene library.

temuze 11y ago

I'm pretty pleased with Elasticsearch's progress on durability. The snapshot/restore feature has been pretty nice to work with! That said, having a "single source of truth" and regularly refreshing Elasticsearch is a huge pain in the butt. I currently maintain a ~500 line syncing script that takes ~15 minutes to run. Adding a new field in a doc_type means: - Adding a column in Postgres - Adding a field in the doc_type mapping (I think being explicit with the field is better practice) - Adding code to the syncer to update the field. Ouch. Also, the syncing scripts required a steep learning curve. At first, I was upserting everything and the syncing script took forever. To solve this, I made the syncer fetch all the documents in Postgres and all the documents in Elasticsearch then update the specific changes (thankfully, the dataset is small enough that it easily fits in memory). I'd really love to scrap this portion of my infrastructure...

tacticus 11y ago
Could you use something like the notify[0] and listen commands to help with this? http://www.postgresql.org/docs/9.0/static/sql-notify.html http://www.postgresql.org/docs/9.0/static/sql-notify.html
- temuze 11y ago
  Maybe? It would be interesting to see what a Postgres - Elasticsearch foreign data wrapper would look like...
ComNik 11y ago
Just today I successfully finished an experiment: An existing web-app with Postgres as the source of truth pushing update events to a Kafka queue. From there, a screenful of Go forwards those events into Elasticsearch. It doesn't solve every problem and it might be a lot of new parts if you're not going to use Kafka for anything else. I will be using it for other things like caching and push events. This kind of syncing problem seems to crop up in a lot of places. A nice thing is that I don't have to care about Elasticsearch durability much, because I can simply rerun the ingester from the beginning of the log, as long as Kafka doesn't lose data.

w8rbt 11y ago

The 'Recap' section has good advice to address the issue of data loss: My recommendations for Elasticsearch users are unchanged: store your data in a database with better safety guarantees, and continuously upsert every document from that database into Elasticsearch. If your search engine is missing a few documents for a day, it’s not a big deal; they’ll be reinserted on the next run and appear in subsequent searches. Not using Elasticsearch as a system of record also insulates you from having to worry about ES downtime during elections.

machbio 11y ago
I do something similar but my data is not that important, as most of you guys work on - I have a very high rate of data being input from users for curating purposes in a bioinformatics lab..So I load the data onto a mongodb cluster and run the elasticsearch-river-mongodb [1] to keep the elasticsearch in sync with the mongodb.. I do not know whether it solves this problem - but I am happy that I have data safe in mongodb - any inputs regarding my method would be appreciated.. [1]https://github.com/richardwilly98/elasticsearch-river-mongodb.git https://github.com/richardwilly98/elasticsearch-river-mongod...
- benburton 11y ago
  Watch out, rivers are being deprecated: https://www.elastic.co/blog/deprecating_rivers https://www.elastic.co/blog/deprecating_rivers
- cwyers 11y ago
  "Data safe in Mongo" seems... less reassuring than I think you meant it to be: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...
- nosequel 11y ago
  > I am happy that I have data safe in mongodb Oh no. So you are protecting your data from unsafe Elasticsearch by storing it in an even more unsafe MongoDB[1]? You should probably put a database that knows how to store data somewhere in your stack and learn to use that. [1]: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...

teraflop 11y ago

> How often the translog is fsynced to disk. Defaults to 5s. [...] In this test we kill random nodes and restart them. [...] In Elasticsearch, write acknowledgement takes place before the transaction is flushed to disk, which means you can lose up to five seconds of writes by default. In this particular run, ES lost about 10% of acknowledged writes. Something bothers me about this: if the bug was merely a failure to call fsync() before acknowledging an operation, then killing processes shouldn't be enough to cause data loss. Once you write to a file and the syscall returns, the written data goes into the OS's buffers, and even if the process is killed it won't be lost. The only time fsync matters is if the entire machine dies (because of power loss or a kernel panic, for instance) before those buffers can be flushed. So is the data actually not even making it to the OS before being acked to the client? Or is Jepsen doing something more sophisticated, like running each node in a VM with its own block device instead of sharing the host's filesystem?

batbomb 11y ago
Write acknowledgement in this case means committed to memory, i.e. the transaction was recorded to a translog buffer (which is fsync'ed to disk every 5 seconds), and changes were applied to the database in memory. The translog (aka write ahead log) has a separate thread which will do fsync every 5 seconds. Unless ES blocks on the translog fsync, you can have lost transactions in the 5 second window.
- teraflop 11y ago
  The only thing that fsync() does is take data which is already in the operating system buffers associated with a file descriptor, and write it physically to the disk. If the thread that runs every 5 seconds is really just fsync'ing, then when the process gets killed, any un-synced data is still in those buffers and will be available to read when it restarts, exactly the same as if fsync() had been called. The OS is responsible for maintaining a consistent view of the filesystem such that, unless the machine totally fails (or you circumvent the FS and actually inspect the live block device), fsync() appears to be a no-op. It sounds like what you're saying is that the documentation is wrong, and that the translog thread is actually pulling data from Elasticsearch's internal buffers and writing it to files. In that case, the documentation which refers to that operation as "fsync" is very badly misleading because it disguises what failure modes it's actually protecting against.

beachstartup 11y ago

waaaay back in the mid 2000s when search engines were large, complex, expensive pieces of enterprise software not unlike a commercial database, it was always assumed (and the customer told) that the search engine cuts corners for the sake of speed, in both indexing and searching, and that it should never be counted on as a database, or any kind of authoritative data store. it does one thing: search staggering amounts of information quickly at acceptable levels of performance and accuracy. looks like expectations have moved beyond that, which is good, but it's not an easy problem to solve... especially with billions or trillions of documents in the index. leaving out all the the important stuff that made a database slow is what made a search engine fast.

larrywright 11y ago

I bet the people who work for these database companies have nightmares about getting bug reports from Kyle. But seriously, his work is super impressive. Kudos to Stripe for funding this.

VeejayRampay 11y ago
Isn't a bug report always a good thing? I mean the real nightmare is your users sending outraged emails about corrupted data and you have no idea what's going on. With the bug report (and especially from Kyle) at least you got something to chew on. As people say, a reported bug is a bug half-fixed.

jodah 11y ago

Key takeaway for me: "crash-restart durable storage is a prerequisite for most consensus algorithms, I have a hunch we’ll see stronger guarantees as Elasticsearch moves towards consensus on writes." It seems like ElasticSearch is inching towards a proper consensus algorithm, at least optionally, which makes me wonder yet again, why not just implement RAFT? While I won't speculate I will point out that the answer for other systems appears to have been related to ego (ie, distributed systems are easy).

jpgvm 11y ago
The funny thing is there is a community plugin that fixes almost all of this by using Zookeeper for discovery and master election. The not so funny thing is Elastic employed the primary developer of said plugin and basically shut it down.
- kimchy 11y ago
  actually, it doesn't fixes the mentioned bug. I am only posting this here to make sure people won't go and try and install it thinking it does... There is a difference between cluster level master election and replication semantics in Elasticsearch (and other similar systems). Even if you use Zookeeper for cluster level master election, one still needs to handle point to point replication (which doesn't go through zk).
  - jpgvm 11y ago
    I should have been more specific. The ZK plugin fixes the issues related to network split-brains and other nasty partition conditions. The newly discovered translog issue is also a problem and is not solved by ZK but is also a lot less scary.

dang 11y ago

Also https://news.ycombinator.com/item?id=9475620 https://news.ycombinator.com/item?id=9475620.

nyir 11y ago

Kind of OT, but losing data in these cases is not even what I'm most concerned about with ES: I had to recreate the ES cluster way too many times now that I'm really glad that PostgreSQL keeps on running, or just restarts without bricking the data files, even after OOM or out of disk errors.

Jarlakxen 11y ago

One more thing to add to the Elastic reliability: > We recommend installing the Java 8 update 20 or later, or Java 7 update 55 or later. Previous versions of Java 7 are known to have bugs that can cause index corruption and data loss. Elasticsearch will refuse to start if a known-bad version of Java is used. http://www.elastic.co/guide/en/elasticsearch/reference/1.5/setup.html http://www.elastic.co/guide/en/elasticsearch/reference/1.5/s...