6 ms·

My biggest gripe with the 6.0.0 GA is the removal of multiple mapping types per index. This creates a significant breaking change that will hurt the community t

by sidi 9y ago

My biggest gripe with the 6.0.0 GA is the removal of multiple mapping types per index. This creates a significant breaking change that will hurt the community tools adoption to 6.0. Their initial plan was to deprecate it and only remove it v7 onwards, which imho would've been a better balance.

33degrees 9y ago
They haven't changed their plan: mapping types per index are only unsupported for new indices, old ones that are migrated will continue to work, until 7.
- sidi 9y ago
  Currently, only read support is provided to indexes created in 5.x. As someone who used one of their pre-GA releases, they previously had a flag to enable multiple mapping types.
  - theuntergeek 9y ago
    Hmmm. Is it read only? If so, the documentation here [1] is off: > Elasticsearch 6.x >> Indices created in 5.x will continue to function in 6.x as they did in 5.x. [1] https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html https://www.elastic.co/guide/en/elasticsearch/reference/6.0/...
    - theuntergeek 9y ago
      A quick ask to the Elasticsearch developers confirms 6.0 can both read & write to indices created in 5.x.
pudo 9y ago
The removal of mapping types really kicks the bottom out of the app we're making; it's some serious docker-style "break all the things" behaviour. Seriously losing trust on this one. Are there any good alternatives to ES? Has Solr moved?
- ergo14 9y ago
  Yes, I believe solr is better than the time I used it 4 years ago, partial json api support, but I don't think it has the analytics aggregations that ES gives us.
- milesokeefe 9y ago
  Can you not use a separate index for each mapping type and just combine the indexes by aliasing them all to one "virtual" index? I think that would effectively be the same but I may be forgetting something.
  - notimetorelax 9y ago
    I believe single document updates fail if alias refers to multiple indices.
- jordanab 9y ago
  Yeah we got the same problems with ES. I do think we "overused" it a bit by often using it as a secondary index/viewmodel store of the sorts. In hindsight that probably wasn't the smartest thing to do. If we had only used it as a search index we would have been better off.
- MrBuddyCasino 9y ago
  I'm doing a migration ES => SolrCloud right now, and it kinda works but its not great. The Json support is really a leaky XML mapping and you have to tune the configs yourself to make it work, the SolrJ Java client is a piece of crap, and some things aren't documented well. But apparently migrating is still simpler than making a BigCorp pay for the ES security package.
- jcadam 9y ago
  Oh, good, instead of getting my app finished this month, I guess I get to screw around with my ES indices and make no real progress on anything. Should have went with Solr...
  - yehosef 9y ago
bpicolo 9y ago
The biggest problem with that is really how soon 6 has come after 5.
deleted 9y ago
[deleted]
dozzie 9y ago
> My biggest gripe with the 6.0.0 GA is the removal of multiple mapping types per index. ROTFL. And I opened the comments wondering what breaks this time. Every time I hear about a new release, ElasticSearch gets worse and worse option for storing logs.
- jsmthrowaway 9y ago
  ElasticSearch was never really meant for log storage anyway. It’s a full text engine, and just happened to work reasonably well for that purpose at lower volume. ELK ran with it in an attempt to go after Splunk, but it is phenomenally difficult to scale an indexing pipeline like ELK to high volume. There are far better ways to handle log analysis, particularly when your primary query is counting things that happen over T instead of finding log entries that match a query (which it always is) — streaming analysis is a much better fit than indexing, just lesser known.
  - amenod 9y ago
    What are the better alternatives for handling log analysis? (I didn't downvote you btw)
    - jsmthrowaway 9y ago
      “Dataflow” and the open source ecosystem in that neighborhood (Flink, Spark, Beam, Kafka, that family of stuff) is a much more powerful way to look at logs in real time, rather than indexing them into storage and then querying. There just isn’t something off the shelf as easy to deploy as ElasticSearch with that architecture, that I’m aware of. (There should be!) When you jump the mental gap of events being identical to logs, then start looking at fun architectures like event sourcing, you start realizing streaming analysis is a pretty powerful way of thinking. I’ve extracted insight from millions of log records per second on a single node with a similar setup, in real time, with much room to scale. The key to scaling log analysis is to get past textual parsing, which means using something structured, which usually negates the reason you were using ElasticSearch in the first place. Google’s first Dataflow talk from I/O and the paper should give you an idea of what log analysis can look like when you get past the full text indexing mindset. Note that there’s nothing wrong with ELK, but you will run into scaling difficulty far sooner than you’d expect trying to index every log event as it comes. It’s also bad to discover this when you get slashdotted, and your ES cluster whimpers trying to keep up. One thing streaming often gets you is latency in that situation instead of death, since you’re probably building atop a distributed log (and falling behind is less bad than falling over). The key here is: are you looking for specific log events or are you counting? You’re almost always counting. Why index individual events, then?

by sidi 9y ago

33degrees 9y ago

They haven't changed their plan: mapping types per index are only unsupported for new indices, old ones that are migrated will continue to work, until 7.

sidi 9y ago
Currently, only read support is provided to indexes created in 5.x. As someone who used one of their pre-GA releases, they previously had a flag to enable multiple mapping types.
- theuntergeek 9y ago
  Hmmm. Is it read only? If so, the documentation here [1] is off: > Elasticsearch 6.x >> Indices created in 5.x will continue to function in 6.x as they did in 5.x. [1] https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html https://www.elastic.co/guide/en/elasticsearch/reference/6.0/...
  - theuntergeek 9y ago
    A quick ask to the Elasticsearch developers confirms 6.0 can both read & write to indices created in 5.x.

pudo 9y ago

The removal of mapping types really kicks the bottom out of the app we're making; it's some serious docker-style "break all the things" behaviour. Seriously losing trust on this one. Are there any good alternatives to ES? Has Solr moved?

ergo14 9y ago
Yes, I believe solr is better than the time I used it 4 years ago, partial json api support, but I don't think it has the analytics aggregations that ES gives us.
milesokeefe 9y ago
Can you not use a separate index for each mapping type and just combine the indexes by aliasing them all to one "virtual" index? I think that would effectively be the same but I may be forgetting something.
- notimetorelax 9y ago
  I believe single document updates fail if alias refers to multiple indices.
jordanab 9y ago
Yeah we got the same problems with ES. I do think we "overused" it a bit by often using it as a secondary index/viewmodel store of the sorts. In hindsight that probably wasn't the smartest thing to do. If we had only used it as a search index we would have been better off.
MrBuddyCasino 9y ago
I'm doing a migration ES => SolrCloud right now, and it kinda works but its not great. The Json support is really a leaky XML mapping and you have to tune the configs yourself to make it work, the SolrJ Java client is a piece of crap, and some things aren't documented well. But apparently migrating is still simpler than making a BigCorp pay for the ES security package.
jcadam 9y ago
Oh, good, instead of getting my app finished this month, I guess I get to screw around with my ES indices and make no real progress on anything. Should have went with Solr...
- yehosef 9y ago

bpicolo 9y ago

The biggest problem with that is really how soon 6 has come after 5.

deleted 9y ago

[deleted]

dozzie 9y ago

> My biggest gripe with the 6.0.0 GA is the removal of multiple mapping types per index. ROTFL. And I opened the comments wondering what breaks this time. Every time I hear about a new release, ElasticSearch gets worse and worse option for storing logs.

jsmthrowaway 9y ago
ElasticSearch was never really meant for log storage anyway. It’s a full text engine, and just happened to work reasonably well for that purpose at lower volume. ELK ran with it in an attempt to go after Splunk, but it is phenomenally difficult to scale an indexing pipeline like ELK to high volume. There are far better ways to handle log analysis, particularly when your primary query is counting things that happen over T instead of finding log entries that match a query (which it always is) — streaming analysis is a much better fit than indexing, just lesser known.
- amenod 9y ago
  What are the better alternatives for handling log analysis? (I didn't downvote you btw)
  - jsmthrowaway 9y ago
    “Dataflow” and the open source ecosystem in that neighborhood (Flink, Spark, Beam, Kafka, that family of stuff) is a much more powerful way to look at logs in real time, rather than indexing them into storage and then querying. There just isn’t something off the shelf as easy to deploy as ElasticSearch with that architecture, that I’m aware of. (There should be!) When you jump the mental gap of events being identical to logs, then start looking at fun architectures like event sourcing, you start realizing streaming analysis is a pretty powerful way of thinking. I’ve extracted insight from millions of log records per second on a single node with a similar setup, in real time, with much room to scale. The key to scaling log analysis is to get past textual parsing, which means using something structured, which usually negates the reason you were using ElasticSearch in the first place. Google’s first Dataflow talk from I/O and the paper should give you an idea of what log analysis can look like when you get past the full text indexing mindset. Note that there’s nothing wrong with ELK, but you will run into scaling difficulty far sooner than you’d expect trying to index every log event as it comes. It’s also bad to discover this when you get slashdotted, and your ES cluster whimpers trying to keep up. One thing streaming often gets you is latency in that situation instead of death, since you’re probably building atop a distributed log (and falling behind is less bad than falling over). The key here is: are you looking for specific log events or are you counting? You’re almost always counting. Why index individual events, then?