5 ms·

You might be able to just put whatever you want into an Elasticsearch index, but I wouldn't recommend doing that. It could severely limit how you can query your

You might be able to just put whatever you want into an Elasticsearch index, but I wouldn't recommend doing that. It could severely limit how you can query your data later, see: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html#map-ids-as-keyword https://www.elastic.co/guide/en/elasticsearch/reference/curr... Also it can cause performance problems if you have really heterogeneous data with lots of different fields https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings https://www.elastic.co/guide/en/elasticsearch/reference/curr...

BoorishBears 6y ago
Yup, reading that comment all I thought was exactly what I said in another comment here, it'll work great until it doesn't, and by then you'll suffer a lot to work around it Same with scaling, scaling ES is super easy until you realize your index sizes aren't playing nicely with sharding or something and have to start working around that. Clickhole feels like it's targeting what most people end up using ES for. Comparing it to ES and talking about what's missing is kind of missing the point imo.
- free652 6y ago
  I manage a fairly small ES cluster of 20 i3en.2xlarge instances that ingest data from 300+ apps. Yes, the only problem I see is the field type collision and it happens occasionally. Otherwise elastic doesn't require much operational time, may be an hour a week. You pretty much want to keep your indices around 50gb and the ILM works well to manage that.
  - pojzon 6y ago
    What about threadwriterejects and max-number-of-shards. If you don’t take into consideration how much data you ingest and in what format it should be afterwards and monitor that constantly: “You gonna have a bad time”. You can automate a lot of stuff around elasticsearch, but when you provide/source it within company - other teams may not be as knowlegable and can shoot themselves into the foot very easly. Ive seen it multiple times by now. People have no idea how to manage the size of their clusters.
  - tgtweak 6y ago
    Anyone accepting freeform objects into a single ES index knows the pain of field type collisions But, most of the time, it "just works". Hearing these argument about rigid schemas saving time tells me that nobody has had to support teams with 200+ apps. ^ This guy actually manages infra
LASR 6y ago
Yeah we learned this the hard way. We had a field, status: 200. After a while, we introduced some new logging that emitted status: success. Since the auto-index used a number type instead of string, we hit some issues trying to reindex the already ingested data.

BoorishBears 6y ago

Yup, reading that comment all I thought was exactly what I said in another comment here, it'll work great until it doesn't, and by then you'll suffer a lot to work around it Same with scaling, scaling ES is super easy until you realize your index sizes aren't playing nicely with sharding or something and have to start working around that. Clickhole feels like it's targeting what most people end up using ES for. Comparing it to ES and talking about what's missing is kind of missing the point imo.

free652 6y ago
I manage a fairly small ES cluster of 20 i3en.2xlarge instances that ingest data from 300+ apps. Yes, the only problem I see is the field type collision and it happens occasionally. Otherwise elastic doesn't require much operational time, may be an hour a week. You pretty much want to keep your indices around 50gb and the ILM works well to manage that.
- pojzon 6y ago
  What about threadwriterejects and max-number-of-shards. If you don’t take into consideration how much data you ingest and in what format it should be afterwards and monitor that constantly: “You gonna have a bad time”. You can automate a lot of stuff around elasticsearch, but when you provide/source it within company - other teams may not be as knowlegable and can shoot themselves into the foot very easly. Ive seen it multiple times by now. People have no idea how to manage the size of their clusters.
- tgtweak 6y ago
  Anyone accepting freeform objects into a single ES index knows the pain of field type collisions But, most of the time, it "just works". Hearing these argument about rigid schemas saving time tells me that nobody has had to support teams with 200+ apps. ^ This guy actually manages infra

LASR 6y ago

Yeah we learned this the hard way. We had a field, status: 200. After a while, we introduced some new logging that emitted status: success. Since the auto-index used a number type instead of string, we hit some issues trying to reindex the already ingested data.