6 ms·

The removal of mapping types really kicks the bottom out of the app we're making; it's some serious docker-style "break all the things" behaviour. Seriously los

by pudo 9y ago

The removal of mapping types really kicks the bottom out of the app we're making; it's some serious docker-style "break all the things" behaviour. Seriously losing trust on this one. Are there any good alternatives to ES? Has Solr moved?

ergo14 9y ago
Yes, I believe solr is better than the time I used it 4 years ago, partial json api support, but I don't think it has the analytics aggregations that ES gives us.
milesokeefe 9y ago
Can you not use a separate index for each mapping type and just combine the indexes by aliasing them all to one "virtual" index? I think that would effectively be the same but I may be forgetting something.
- notimetorelax 9y ago
  I believe single document updates fail if alias refers to multiple indices.
jordanab 9y ago
Yeah we got the same problems with ES. I do think we "overused" it a bit by often using it as a secondary index/viewmodel store of the sorts. In hindsight that probably wasn't the smartest thing to do. If we had only used it as a search index we would have been better off.
MrBuddyCasino 9y ago
I'm doing a migration ES => SolrCloud right now, and it kinda works but its not great. The Json support is really a leaky XML mapping and you have to tune the configs yourself to make it work, the SolrJ Java client is a piece of crap, and some things aren't documented well. But apparently migrating is still simpler than making a BigCorp pay for the ES security package.
jcadam 9y ago
Oh, good, instead of getting my app finished this month, I guess I get to screw around with my ES indices and make no real progress on anything. Should have went with Solr...
- yehosef 9y ago
  if you're getting your app finished this month.. maybe you shouldn't upgrade to 6..
  - jcadam 9y ago
    Ah, but if I don't go through the pain now, I will get to learn an entirely new definition of pain and suffering later on when I (hopefully) have a huge ES search index in production.
    - AznHisoka 9y ago
      I have a production app (with thousands of users) and we are still on version 1.x since 2013.
    - yehosef 9y ago
      You can basically use a 5.x in the same way as 6 (one "type" per index, only one parent-child relation). If you're not doing it that way now - you can start slowly while you're on 5.x and slowly migrate your methodology to the 6.x way (if it's even so different..) Then you can move to 6 more with little change, if you even need to upgrade to 6.
yehosef 9y ago
I also really don't like this change, but you have to understand why it matters for you. If you just wanted to be able to store different types in an index - you still can, just that the type will not be stored in a meta "_type" field but in a document level "type" (or whatever) field. Of course, the API does change since it's not in the URL and if you're creating custom doc ids you'll probably have to include the type in that (comment-123, post-123). So it's annoying, but I think mostly something that can worked around. If you're using multiple types for parent-child, the situation is more bleak. They are still going to have a "join field" but there can only be one type of relation. While often this is ok, there are definitely reasonable use cases where it's not. Currently the traditional parent-child hasn't been removed from the core because they need to support 5.x indices. But it will be phased out unless there is a big uproar.
courtewing 9y ago
If it helps at all, Kibana also had to be updated to go from multiple types to a single type. It was a big project, and a bunch of approaches were explored for dealing with existing data (which I assume you are). Ultimately, we settled on continuing to have different "types" in Kibana, but we treated them as data concerns rather than architectural concerns of Elasticsearch. At a high level, this meant that we added a new "type" field to documents to track the type itself, and then we prefixed fields with the type as well to preserve the ability to use the same userland id on different "types" in the index and such. The type/id prefixing thing doesn't get exposed beyond the layer that queries Elasticsearch for the kibana index. Once that change was ready in the code, we also had to consider the need for users to migrate their existing multiple-type index to this new format. The upgrade assistant in x-pack basic handles all of this automatically, but folks can certainly repurpose the same reindexing operation we perform on .kibana on your own indices. The underlying steps and reindex operation for this are outlined on the docs: https://www.elastic.co/guide/en/kibana/current/migrating-6.0-index.html https://www.elastic.co/guide/en/kibana/current/migrating-6.0... The actual data transformation happens in step 3. I hope this helps!
creativehandle 9y ago
Our ES team lead wrote this up for folks to understand this change. I'd recommend a read through, but I just linked to the section on why: https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_why_are_mapping_types_being_removed https://www.elastic.co/guide/en/elasticsearch/reference/mast...
dotancohen 9y ago
> The removal of mapping types really kicks the bottom out of the app we're making Not to rub salt in the wound, but building a product around a feature provided by a single vendor (ie, not a standard feature or something developed in-house) means that you've just committed to maintaining that software or paying someone else good money to do so. Anybody who's built products around eg Oracle has learned this at least twice!

by pudo 9y ago

ergo14 9y ago

Yes, I believe solr is better than the time I used it 4 years ago, partial json api support, but I don't think it has the analytics aggregations that ES gives us.

milesokeefe 9y ago

Can you not use a separate index for each mapping type and just combine the indexes by aliasing them all to one "virtual" index? I think that would effectively be the same but I may be forgetting something.

notimetorelax 9y ago
I believe single document updates fail if alias refers to multiple indices.

jordanab 9y ago

Yeah we got the same problems with ES. I do think we "overused" it a bit by often using it as a secondary index/viewmodel store of the sorts. In hindsight that probably wasn't the smartest thing to do. If we had only used it as a search index we would have been better off.

MrBuddyCasino 9y ago

I'm doing a migration ES => SolrCloud right now, and it kinda works but its not great. The Json support is really a leaky XML mapping and you have to tune the configs yourself to make it work, the SolrJ Java client is a piece of crap, and some things aren't documented well. But apparently migrating is still simpler than making a BigCorp pay for the ES security package.

jcadam 9y ago

Oh, good, instead of getting my app finished this month, I guess I get to screw around with my ES indices and make no real progress on anything. Should have went with Solr...

yehosef 9y ago
if you're getting your app finished this month.. maybe you shouldn't upgrade to 6..
- jcadam 9y ago
  Ah, but if I don't go through the pain now, I will get to learn an entirely new definition of pain and suffering later on when I (hopefully) have a huge ES search index in production.
  - AznHisoka 9y ago
    I have a production app (with thousands of users) and we are still on version 1.x since 2013.
  - yehosef 9y ago
    You can basically use a 5.x in the same way as 6 (one "type" per index, only one parent-child relation). If you're not doing it that way now - you can start slowly while you're on 5.x and slowly migrate your methodology to the 6.x way (if it's even so different..) Then you can move to 6 more with little change, if you even need to upgrade to 6.

yehosef 9y ago

I also really don't like this change, but you have to understand why it matters for you. If you just wanted to be able to store different types in an index - you still can, just that the type will not be stored in a meta "_type" field but in a document level "type" (or whatever) field. Of course, the API does change since it's not in the URL and if you're creating custom doc ids you'll probably have to include the type in that (comment-123, post-123). So it's annoying, but I think mostly something that can worked around. If you're using multiple types for parent-child, the situation is more bleak. They are still going to have a "join field" but there can only be one type of relation. While often this is ok, there are definitely reasonable use cases where it's not. Currently the traditional parent-child hasn't been removed from the core because they need to support 5.x indices. But it will be phased out unless there is a big uproar.

courtewing 9y ago

If it helps at all, Kibana also had to be updated to go from multiple types to a single type. It was a big project, and a bunch of approaches were explored for dealing with existing data (which I assume you are). Ultimately, we settled on continuing to have different "types" in Kibana, but we treated them as data concerns rather than architectural concerns of Elasticsearch. At a high level, this meant that we added a new "type" field to documents to track the type itself, and then we prefixed fields with the type as well to preserve the ability to use the same userland id on different "types" in the index and such. The type/id prefixing thing doesn't get exposed beyond the layer that queries Elasticsearch for the kibana index. Once that change was ready in the code, we also had to consider the need for users to migrate their existing multiple-type index to this new format. The upgrade assistant in x-pack basic handles all of this automatically, but folks can certainly repurpose the same reindexing operation we perform on .kibana on your own indices. The underlying steps and reindex operation for this are outlined on the docs: https://www.elastic.co/guide/en/kibana/current/migrating-6.0-index.html https://www.elastic.co/guide/en/kibana/current/migrating-6.0... The actual data transformation happens in step 3. I hope this helps!

creativehandle 9y ago

Our ES team lead wrote this up for folks to understand this change. I'd recommend a read through, but I just linked to the section on why: https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_why_are_mapping_types_being_removed https://www.elastic.co/guide/en/elasticsearch/reference/mast...

dotancohen 9y ago

> The removal of mapping types really kicks the bottom out of the app we're making Not to rub salt in the wound, but building a product around a feature provided by a single vendor (ie, not a standard feature or something developed in-house) means that you've just committed to maintaining that software or paying someone else good money to do so. Anybody who's built products around eg Oracle has learned this at least twice!