3 ms·

Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexe

Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexes - preprocessing your logs so the keys have their own schema prefix, as you mention The way we've tackled this is to have an official company-wide logging schema. It's just a GitHub repo at gocardless/logging that has an exhaustive list of logging keys, with an explanation of what they should contain. This has the benefit of encouraging consistent logging practices over many teams, as well as improving the chance your logs will get indexed correctly. If the field type doesn't match, we won't index that field but it will appear in the _log field, where you can do a full text search as a fallback if you really needed to find your log. It's not perfect though, and I still hate the dynamic type assignment.

atombender 6y ago
We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps. Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes. But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.
- axw 6y ago
  Yes, you can do this using Filebeat or in Elasticsearch itself using an Ingest node script processor. Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#indices-option-es https://www.elastic.co/guide/en/beats/filebeat/current/elast... (You can incorporate variables in the destination index; variables may refer to labels and other fields.) Ingest node: https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... (search for "_index")

atombender 6y ago

We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps. Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes. But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.

axw 6y ago
Yes, you can do this using Filebeat or in Elasticsearch itself using an Ingest node script processor. Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#indices-option-es https://www.elastic.co/guide/en/beats/filebeat/current/elast... (You can incorporate variables in the destination index; variables may refer to labels and other fields.) Ingest node: https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... (search for "_index")