6 ms·
Elasticsearch 6.0.0 GA released
- sidi 9y agoMy biggest gripe with the 6.0.0 GA is the removal of multiple mapping types per index. This creates a significant breaking change that will hurt the community tools adoption to 6.0. Their initial plan was to deprecate it and only remove it v7 onwards, which imho would've been a better balance.
- 33degrees 9y agoThey haven't changed their plan: mapping types per index are only unsupported for new indices, old ones that are migrated will continue to work, until 7.
- sidi 9y agoCurrently, only read support is provided to indexes created in 5.x. As someone who used one of their pre-GA releases, they previously had a flag to enable multiple mapping types.
- theuntergeek 9y agoHmmm. Is it read only? If so, the documentation here [1] is off: > Elasticsearch 6.x >> Indices created in 5.x will continue to function in 6.x as they did in 5.x. [1] https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html https://www.elastic.co/guide/en/elasticsearch/reference/6.0/...
- theuntergeek 9y agoA quick ask to the Elasticsearch developers confirms 6.0 can both read & write to indices created in 5.x.
- pudo 9y agoThe removal of mapping types really kicks the bottom out of the app we're making; it's some serious docker-style "break all the things" behaviour. Seriously losing trust on this one. Are there any good alternatives to ES? Has Solr moved?
- ergo14 9y agoYes, I believe solr is better than the time I used it 4 years ago, partial json api support, but I don't think it has the analytics aggregations that ES gives us.
- milesokeefe 9y agoCan you not use a separate index for each mapping type and just combine the indexes by aliasing them all to one "virtual" index? I think that would effectively be the same but I may be forgetting something.
- notimetorelax 9y agoI believe single document updates fail if alias refers to multiple indices.
- jordanab 9y agoYeah we got the same problems with ES. I do think we "overused" it a bit by often using it as a secondary index/viewmodel store of the sorts. In hindsight that probably wasn't the smartest thing to do. If we had only used it as a search index we would have been better off.
- MrBuddyCasino 9y agoI'm doing a migration ES => SolrCloud right now, and it kinda works but its not great. The Json support is really a leaky XML mapping and you have to tune the configs yourself to make it work, the SolrJ Java client is a piece of crap, and some things aren't documented well. But apparently migrating is still simpler than making a BigCorp pay for the ES security package.
- jcadam 9y ago
- bpicolo 9y agoThe biggest problem with that is really how soon 6 has come after 5.
- deleted 9y ago[deleted]
- dozzie 9y ago> My biggest gripe with the 6.0.0 GA is the removal of multiple mapping types per index. ROTFL. And I opened the comments wondering what breaks this time. Every time I hear about a new release, ElasticSearch gets worse and worse option for storing logs.
- jsmthrowaway 9y agoElasticSearch was never really meant for log storage anyway. It’s a full text engine, and just happened to work reasonably well for that purpose at lower volume. ELK ran with it in an attempt to go after Splunk, but it is phenomenally difficult to scale an indexing pipeline like ELK to high volume. There are far better ways to handle log analysis, particularly when your primary query is counting things that happen over T instead of finding log entries that match a query (which it always is) — streaming analysis is a much better fit than indexing, just lesser known.
- amenod 9y agoWhat are the better alternatives for handling log analysis? (I didn't downvote you btw)
- jsmthrowaway 9y ago“Dataflow” and the open source ecosystem in that neighborhood (Flink, Spark, Beam, Kafka, that family of stuff) is a much more powerful way to look at logs in real time, rather than indexing them into storage and then querying. There just isn’t something off the shelf as easy to deploy as ElasticSearch with that architecture, that I’m aware of. (There should be!) When you jump the mental gap of events being identical to logs, then start looking at fun architectures like event sourcing, you start realizing streaming analysis is a pretty powerful way of thinking. I’ve extracted insight from millions of log records per second on a single node with a similar setup, in real time, with much room to scale. The key to scaling log analysis is to get past textual parsing, which means using something structured, which usually negates the reason you were using ElasticSearch in the first place. Google’s first Dataflow talk from I/O and the paper should give you an idea of what log analysis can look like when you get past the full text indexing mindset. Note that there’s nothing wrong with ELK, but you will run into scaling difficulty far sooner than you’d expect trying to index every log event as it comes. It’s also bad to discover this when you get slashdotted, and your ES cluster whimpers trying to keep up. One thing streaming often gets you is latency in that situation instead of death, since you’re probably building atop a distributed log (and falling behind is less bad than falling over). The key here is: are you looking for specific log events or are you counting? You’re almost always counting. Why index individual events, then?
- jordanab 9y agoIn a large internal project we still make heavy use of Groovy for scripting. All scripting languages besides the new Painless language were deprecated in version 5, and now in version 6 they are removed. This hampers us in our migration efforts from 5 to 6. Does anyone know if it is still possible to use Groovy through a separate (third party) plugin? I played around a bit with Painless, but can't say that I really like it. Documentation was/is kind of poor, and it seemed to me it somewhat assumed familiarity with Java and it's framework/APIs.
- milesokeefe 9y agoMight be worth trying the Groovy plugin that was needed prior to ES integrating it: https://github.com/elastic/elasticsearch-lang-groovy https://github.com/elastic/elasticsearch-lang-groovy
- jordanab 9y agoThx. I already came across that repo, and seems very outdated. I might have to check out creating my own plugin, and use this as a start.
- laichzeit0 9y agoThey deprecated all scripting langs? Ok this is something new I learned in this thread. That is an extremely lame deprecation. Is ElasticSearch the new Apple? “let’s remove as much functionality as possible for the sake of being as minimal as possible”. Groovy is a fantastic language. It really is a hidden gem. It is my language of choice for cross-platform work, especially in enterprise.
- timv 9y agoGroovy was a security nightmare. The groovy sandbox simply doesn't do enough to protect a server. Our scripting language "Painless" is faster and more secure than we could achieve with groovy, so in Elasticsearch 5.0 we made Painless the default and deprecated groovy. In 6.0, groovy is gone. We didn't do it to be minimalist, but we couldn't in good conscience continue to ship an insecure scripting language when we had an alternative. Disclosure: I work at Elastic on security.
- eric_b 9y agoThe new sequence IDs are the most interesting things in this release I think. Having an officially supported cross datacenter replication strategy would be real nice. Lots of folks will be mad, but removing multiple mapping types is a nice change too. It was a feature that never really made sense. Index-per-type was always the better strategy, even going back to the 0.23 days. As others in this thread will no doubt point out though - the ES folks are moving awfully fast. I still support 1.7.5 clusters that will take heroic efforts to update. I'd love to use the new features on those clusters, but there simply isn't a valid business case to take on the risk. This isn't like small NPM packages that you can update on a whim - some of these systems require terabytes of re-indexing to upgrade :/
- taf2 9y agoI wish after 1.x they had maintained backwards compatibility with upgrades as they had kind of promised... there is just so much money in making it hard to upgrade ;)
- stingraycharles 9y agoI don’t think it will ever make sense for a business to make it hard to upgrade. There is much more to lose.
- eric_b 9y agoThat consulting arm of Elastic needs something to do :) More charitably, I can understand why they felt the need to make a hard break, but difficult upgrades plus fast release cycle means a fair bit of friction. The no downtime upgrades will be nice, but on a big production system, I wouldn't feel comfortable upgrading major versions every 6 months.
- slantedview 9y agoRolling upgrade support was a big part of this release, which obviously makes upgrades easier.
- naiv 9y agoCross data center replication is really a much needed feature. The way Elasticsearch is going though looks promising. With sorted indices, single mapping type and the other changes we might give it another try after switching to Algolia. Is there a safe way now to query Elasticsearch directly without the need to go via proxy scripts on the server? This just adds so much overhead to the queries compared to Algolia.
- jdhawk 9y agoI really wish they'd included the ElasticSearch SQL in 6.0, but I guess that feature still isn't fully baked. Even if it didn't have the full power of the Elastic JSON Queries, for simple SELECT COUNT() ..GROUP BY, it would have been a nice addition...oh well, back to counting open and closed brackets...
- djhworld 9y agoWe're just in the process of looking into ElasticSearch for an analytics use case, most of the queries we will be doing are simple group by aggregations with count/sums etc A SQL interface would help a lot, even better if it came with a JDBC driver
- alexott 9y agoThere is some ES integration in the Apache Calcite. And it has JDBC driver... P.S. Writing this level of SQL for ES that you describe isn't very difficult - in my project we got working in implementation in 2-3 weeks. Take Calcite. (Recommended but complex, imho), Facebook's Presto SQL parser (not recommended, but simpler)
- praseodym 9y agoAgreed! Using something like elasticsearch-dsl for Python [1] makes writing queries a little more bearable, but it's not a good fit for manual ad-hoc querying (e.g. some analysis I'd be doing from Kibana). https://github.com/elastic/elasticsearch-dsl-py https://github.com/elastic/elasticsearch-dsl-py
- yehosef 9y agocheck out dremio.com - It's superduper Apache Drill with elasticsearch support (ie SQL with real joins.)
- CSDude 9y agoDremio seemed very promising, but it failed more than half of my queries with ES
- chatman 9y agoNot as exciting as Solr 7 release.
- liveoneggs 9y agoI've been using solr since 1.3 and found trying to use elastic is pretty crazy annoying. I'd love to swap the ELK stack for a solar-based setup.
- joking 9y agolucidworks made banana, a kibana port to solr, but never take off like kibana. I dont find the diferences between solr and elasticsearch so big, and always prefered the solr way of configurating things instead of the json api of elasticsearch. The solr data-import-handler has been also a big timesaver.
- lclarkmichalek 9y agoDoes Lucene 7.0 have a better story around G1 GC? ES has always warned against it, would love to see if that had changed
- dakrone 9y agoWe revisit the G1GC recommendation every once it a while. In fact, I am doing benchmarks and testing for G1GC versus CMS with Elasticsearch 6.0.0 right now, so that we have a better idea of where we stand. Disclaimer: I'm an Elasticsearch dev employed by Elastic.
- lclarkmichalek 9y agoCool, I've a pretty big cluster with some GC issues (p90 - 15s, p99 - 60s) during node failures, and would be super interested in those results! If there's anything a user can do to help, my email is on my user page :D
- notimetorelax 9y agoWe observed in past that long GC is the cause of node failures. When long GC happens node doesn’t respond, master node decides that this node had left the cluster :\
- lclarkmichalek 9y agoYa, we often see a node die of natural causes, and then the garbage produced from recovering the node and relocating the data ends up bringing down the rest of the cluster via long GC pauses.
- notimetorelax 9y agoWe run ES on G1GC, have been doing this for the last 3 years. With heaps of more than 20gb in size and a lot of churn CMS just doesn’t cut it. What helps is a high number of replicas to help with any potential corruptions, and that we never treat ES as a primary store.
- Qerub 9y agoI'm excited about the official Docker images now being available (again?) in an OSS flavor without X-Pack. Quote from https://www.elastic.co/guide/en/elasticsearch/reference/6.0/docker.html https://www.elastic.co/guide/en/elasticsearch/reference/6.0/... : > The images are available in three different configurations or "flavors". The basic flavor, which is the default, ships with X-Pack Basic features pre-installed and automatically activated with a free licence. The platinum flavor features all X-Pack functionally under a 30-day trial licence. The oss flavor does not include X-Pack, and contains only open-source Elasticsearch.
- praseodym 9y agoThe pre-installed X-Pack basic license sounds great as well: this at least allows one to use features like monitoring, dev tools and the upgrade assistant out of the box, without the 30-day trial kicking in as was the case for the 5.x Docker images.
- sbr464 9y agoVery much appreciate the improvements to handling docs with sparse fields!
- sschueller 9y agoOff topic but how do you guys Unit Test Elastic Search queries? Do you start a full node and dump test data in it? In case of SQL I can start an in memory sqllite and run my tests (Symfony PHP).
- gerbal 9y agoWe've had good results using docker-compose to run ES test suites. Run the app in one container and an ES instance in another. Test data can be loaded from fixtures or captured/snapshotted using `docker commit` to create specific test images.