4 ms·

Could you elaborate on the issues you have with ElasticSearch? Performance/Scaling? Usability?

by mavam 11y ago

threeseed 11y ago
I've attempted to load huge amounts of data into ElasticSearch. It is a bit fiddly to disable indexing and the sheer write performance is poor against dedicated databases. It is a testament to how good ElasticSearch is that people do use it as a database.
- lobster_johnson 11y ago
  ES performs well, but we don't use it as a primary data source, and I would advise heavily against using it as one. Things are getting better with the new transaction log, but it's still extremely prone to falling over and to getting consistency issues, and some of the built-in automation can bite you in the ass. Recently we had a node — fortunately an expendable testing box — run out of disk space. ES stopped working, which is fine, but when we restarted it, several of the shards were corrupt, and there was no way to get it working again without dropping the index and recreating it. In another instance, we started getting client downtime because ES had suddenly detected low disk space on one node (its "high watermark" setting is extremely conservative and not based on a statistical regression of when it's going to run out), and had started relocating shards. When it was done, one of the shards were stuck in "RELOCATING" but marked as done; only restarting ES fixed that issue. When we freed up some disk space, it gave us more downtime when it decided to move the shards back. In theory, ES was behaving correctly, but the automation behaviour that kicked in was unnecessary and not desirable in that use case. ES has lots of good parts, but it also has a surprising number of bad parts — ugly stuff you just won't find in Postgres, for example.
lobster_johnson 11y ago
My issues with ES are entirely about ergonomics. My pet peeve is the mapping system. ES is often described as schemaless, but this is of course untrue; ES has a schema, and a very strict one, but its default mode is to automatically learn the schema from the data you feed it. By learn I mean that the first time it receives a document, it will guess the shape of the data and create a schema from it. You can't ever change mappings, only add new ones, which means that running in "dynamic" mode can (and, mostly, will) result in bad mappings. This comes from (1) that ES' guesswork is by definition incomplete with regard to data types, (2) that ES can't, by definition, guess things like text analysis settings, and (3) sometimes input data is just inconsistent with itself. #2 means that even if your data is completely consistent, if you submit a text field, it will get the default index settings. #1 means that, for example, if you have something you think is a date, which ES doesn't recognize as a date, then it will be assumed to be a string. Dates are among ES' weakest areas (though 2.x makes some improvements here). You can mitigate some of these problems with the "_default_" mapping, which can catch fields and assign defaults based on name patterns. But it's not a complete solution. Another problem is that for historical reasons, ES indexes share field names across all document types, and don't allow conflicting mappings. For example, if you have "articles" and "sections" and both have a "title" field, ES requires that the mapping be the same — because internally, they are stored in the same Lucene index. This is partly a feature (it allows you to search across all types on the field "title" and guarantee that it will work consistently) but really a principle-of-least-surprise-violating misfeature (you can accomplish the same thing with a synthetic field and "copy_to", which is more explicit and less surprising). Another problem: Posting a conflicting mapping to ES does nothing; it will silently ignore it. There's no way to determine if your own mapping changes will have any effect or not. Here's why it's a problem: Say you made some changes to your app. You have an ES index in production. Now you want to determine if your new mappings can be applied without an index migration, or whether they will conflict. Turns out you can't. Some mapping properties (e.g. name, type, analyzer, store flag) cannot be changed and should conflict. Some don't. Since there's no set of rules encoded anywhere about what are conflicts and not, you can't progammatically diff your mappings. This is exacerbated by the fact that ES will "collapse" mapping properties that equal the default. For example, let's say you post a mapping with the properties {"type": "object", "store": true}. If you read that back afterwards, it will be {"type": "object"}. Why? Because "store" is true by default. So if you try to "diff" a mapping, it will look like ES has different mappings than you have. ES tends to prefer implicit over explicit, which is a bad, bad design philosophy. You can only give up and assume that anything that isn't an add (a new document type mapping) will require a migration. Which leads the next problem: ES can't migrate indexes. You're totally on your own. Which is ironic because by default, ES will store the original source (the "_source" field) of every document you index. There's no reason ES couldn't create a whole new index, based on new mappings, from a different index. But no, you have to write a client and run the data from ES into the client and then back again, just to do a mapping migration. If I were to summarize this essay, it's that there's an icky impedance mismatch between mappings, logical ES indexes and physical Lucene indexes that have existed since the beginning, and whose consequences are increasingly felt. Lucene is used both for indexing and for document storage, and the distinction isn't adequately abstracted. Here's an example of where an abstraction could have helped immensely. In ES, unlike relational databases, there's no such as "creating an index on a field". A field either has an index or it doesn't, and field names map directly to Lucene field names. There's no indirection. So if you set up a field "name", that also creates a Lucene inverted index called "name", and because the field is now tied to a physical index coding (e.g. tokenized strings), it can never be changed. You can't create more indexes on the same field, and you can't drop the index or change it. The better solution would be to map it to an internal field name, and let each field support multiple indexes. For example: Say I have a field "name". Sometimes I want to search as tokenized text. Sometimes I want to treat the whole thing as a single token (e.g. when aggregating into buckets). If there were an indirection between document fields and indexes, I could accomplish this by creating two indexes on "name". ES could figure out which index to use from the query operators I used, and it could also offer a way to explicitly specify which index to use. This type of indirection would solve the mapping strictness because ES would no longer be tied to a single type representation for every value. I have other complaints about ES, but fortunately, some of them are being resolved in 2.0 (beta) and 2.1 — they are removing tons of cruft in the APIs, and seem to have realized that a lot of the design choices (especially the ones that promote implicit over explicit information) were a bad idea.

Could you elaborate on the issues you have with ElasticSearch? Performance/Scaling? Usability?

by mavam 11y ago

Could you elaborate on the issues you have with ElasticSearch? Performance/Scaling? Usability?

I've attempted to load huge amounts of data into ElasticSearch. It is a bit fiddly to disable indexing and the sheer write performance is poor against dedicated databases. It is a testament to how good ElasticSearch is that people do use it as a database.

lobster_johnson 11y ago
ES performs well, but we don't use it as a primary data source, and I would advise heavily against using it as one. Things are getting better with the new transaction log, but it's still extremely prone to falling over and to getting consistency issues, and some of the built-in automation can bite you in the ass. Recently we had a node — fortunately an expendable testing box — run out of disk space. ES stopped working, which is fine, but when we restarted it, several of the shards were corrupt, and there was no way to get it working again without dropping the index and recreating it. In another instance, we started getting client downtime because ES had suddenly detected low disk space on one node (its "high watermark" setting is extremely conservative and not based on a statistical regression of when it's going to run out), and had started relocating shards. When it was done, one of the shards were stuck in "RELOCATING" but marked as done; only restarting ES fixed that issue. When we freed up some disk space, it gave us more downtime when it decided to move the shards back. In theory, ES was behaving correctly, but the automation behaviour that kicked in was unnecessary and not desirable in that use case. ES has lots of good parts, but it also has a surprising number of bad parts — ugly stuff you just won't find in Postgres, for example.

lobster_johnson 11y ago

My issues with ES are entirely about ergonomics. My pet peeve is the mapping system. ES is often described as schemaless, but this is of course untrue; ES has a schema, and a very strict one, but its default mode is to automatically learn the schema from the data you feed it. By learn I mean that the first time it receives a document, it will guess the shape of the data and create a schema from it. You can't ever change mappings, only add new ones, which means that running in "dynamic" mode can (and, mostly, will) result in bad mappings. This comes from (1) that ES' guesswork is by definition incomplete with regard to data types, (2) that ES can't, by definition, guess things like text analysis settings, and (3) sometimes input data is just inconsistent with itself. #2 means that even if your data is completely consistent, if you submit a text field, it will get the default index settings. #1 means that, for example, if you have something you think is a date, which ES doesn't recognize as a date, then it will be assumed to be a string. Dates are among ES' weakest areas (though 2.x makes some improvements here). You can mitigate some of these problems with the "_default_" mapping, which can catch fields and assign defaults based on name patterns. But it's not a complete solution. Another problem is that for historical reasons, ES indexes share field names across all document types, and don't allow conflicting mappings. For example, if you have "articles" and "sections" and both have a "title" field, ES requires that the mapping be the same — because internally, they are stored in the same Lucene index. This is partly a feature (it allows you to search across all types on the field "title" and guarantee that it will work consistently) but really a principle-of-least-surprise-violating misfeature (you can accomplish the same thing with a synthetic field and "copy_to", which is more explicit and less surprising). Another problem: Posting a conflicting mapping to ES does nothing; it will silently ignore it. There's no way to determine if your own mapping changes will have any effect or not. Here's why it's a problem: Say you made some changes to your app. You have an ES index in production. Now you want to determine if your new mappings can be applied without an index migration, or whether they will conflict. Turns out you can't. Some mapping properties (e.g. name, type, analyzer, store flag) cannot be changed and should conflict. Some don't. Since there's no set of rules encoded anywhere about what are conflicts and not, you can't progammatically diff your mappings. This is exacerbated by the fact that ES will "collapse" mapping properties that equal the default. For example, let's say you post a mapping with the properties {"type": "object", "store": true}. If you read that back afterwards, it will be {"type": "object"}. Why? Because "store" is true by default. So if you try to "diff" a mapping, it will look like ES has different mappings than you have. ES tends to prefer implicit over explicit, which is a bad, bad design philosophy. You can only give up and assume that anything that isn't an add (a new document type mapping) will require a migration. Which leads the next problem: ES can't migrate indexes. You're totally on your own. Which is ironic because by default, ES will store the original source (the "_source" field) of every document you index. There's no reason ES couldn't create a whole new index, based on new mappings, from a different index. But no, you have to write a client and run the data from ES into the client and then back again, just to do a mapping migration. If I were to summarize this essay, it's that there's an icky impedance mismatch between mappings, logical ES indexes and physical Lucene indexes that have existed since the beginning, and whose consequences are increasingly felt. Lucene is used both for indexing and for document storage, and the distinction isn't adequately abstracted. Here's an example of where an abstraction could have helped immensely. In ES, unlike relational databases, there's no such as "creating an index on a field". A field either has an index or it doesn't, and field names map directly to Lucene field names. There's no indirection. So if you set up a field "name", that also creates a Lucene inverted index called "name", and because the field is now tied to a physical index coding (e.g. tokenized strings), it can never be changed. You can't create more indexes on the same field, and you can't drop the index or change it. The better solution would be to map it to an internal field name, and let each field support multiple indexes. For example: Say I have a field "name". Sometimes I want to search as tokenized text. Sometimes I want to treat the whole thing as a single token (e.g. when aggregating into buckets). If there were an indirection between document fields and indexes, I could accomplish this by creating two indexes on "name". ES could figure out which index to use from the query operators I used, and it could also offer a way to explicitly specify which index to use. This type of indirection would solve the mapping strictness because ES would no longer be tied to a single type representation for every value. I have other complaints about ES, but fortunately, some of them are being resolved in 2.0 (beta) and 2.1 — they are removing tons of cruft in the APIs, and seem to have realized that a lot of the design choices (especially the ones that promote implicit over explicit information) were a bad idea.