3 ms·

Slightly OT- I'm working with data sets that might change, but not often if at all, which are provided by Elasticsearch. I'm processing the raw data in Flask (A

Slightly OT- I'm working with data sets that might change, but not often if at all, which are provided by Elasticsearch. I'm processing the raw data in Flask (API), munging, joining, and dropping what I don't want going out to the world. I've been toying with the idea of using something like PB, Cap'n Proto, or now Bond to define and track schema changes and centralize marshaling / serializing logic. I'm not concerned about having RPC. Does this sound like crazy talk? Anyone else happen to track schemas agains schemaless data stores? (I also like the idea of not having to ship JSON everywhere if I don't want to.)

rch 12y ago
I'm doing something similar at work. I'd be happy to chat about it if you drop me a line sometime.
seanp2k2 12y ago
TL;DR by using more of the available features in ElasticSearch, you can probably replace all of your external app with ElasticSearch. A few things: - ElasticSearch is definitely not schema-less, but it can try to generate a schema (aka "mapping") for you if you don't give it one: http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapping.html http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/c... - ElasticSearch has tons of ways to customize the data you get back, so, unless you really don't want the ES cluster crunching things for you, you can do a lot of the transformation server-side. You can go so far as to have your own type + mapping for e.g. a report, which sources data from another type and transforms it: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html http://www.elasticsearch.org/guide/en/elasticsearch/referenc... - This covers both why the schema can't, by nature, be dynamic (so the argument of "schema-less / dynamic schema" is BS in practice IMO), as well as how to get data out from one index an into another (e.g. your "report" index which does scripted transformation). - Another idea would be to use the scripting module to write a custom "view": http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html http://www.elasticsearch.org/guide/en/elasticsearch/referenc... - You can use Groovy, mvel, JS, or Python for scripts. If you combine this with how ES lets you do "site plugins", you could make a JS + CSS + HTML site which is actually served by the ES cluster, which interacts with it and generates reports or whatever all without additional infrastructure. Example: https://github.com/karmi/elasticsearch-paramedic https://github.com/karmi/elasticsearch-paramedic

rch 12y ago

I'm doing something similar at work. I'd be happy to chat about it if you drop me a line sometime.

seanp2k2 12y ago

TL;DR by using more of the available features in ElasticSearch, you can probably replace all of your external app with ElasticSearch. A few things: - ElasticSearch is definitely not schema-less, but it can try to generate a schema (aka "mapping") for you if you don't give it one: http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapping.html http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/c... - ElasticSearch has tons of ways to customize the data you get back, so, unless you really don't want the ES cluster crunching things for you, you can do a lot of the transformation server-side. You can go so far as to have your own type + mapping for e.g. a report, which sources data from another type and transforms it: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html http://www.elasticsearch.org/guide/en/elasticsearch/referenc... - This covers both why the schema can't, by nature, be dynamic (so the argument of "schema-less / dynamic schema" is BS in practice IMO), as well as how to get data out from one index an into another (e.g. your "report" index which does scripted transformation). - Another idea would be to use the scripting module to write a custom "view": http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html http://www.elasticsearch.org/guide/en/elasticsearch/referenc... - You can use Groovy, mvel, JS, or Python for scripts. If you combine this with how ES lets you do "site plugins", you could make a JS + CSS + HTML site which is actually served by the ES cluster, which interacts with it and generates reports or whatever all without additional infrastructure. Example: https://github.com/karmi/elasticsearch-paramedic https://github.com/karmi/elasticsearch-paramedic