9 ms·

The Elastic Stack: Future of ELK Platform

108 points by hurrycane 11y ago

hurrycane 11y ago
More details on the blog post too: https://www.elastic.co/blog/heya-elastic-stack-and-x-pack https://www.elastic.co/blog/heya-elastic-stack-and-x-pack
glial 11y ago
Is X-Pack going to be a subscription based service?
- wyaeld 11y ago
  Considering all the components currently in it are only available if you're in their subscription plan, that's basically the core business model.
  - glial 11y ago
    I wonder whether the components will be available a la carte like they are now, or if it's going to be a single bundle.
spdustin 11y ago
I got pretty excited to read about this unified endeavor, and hungrily clicked the CTA to Get 5.0 Alpha. > Don't see a button? You may have to disable your ad blocker. What a shitty experience. You're pitching a product that you're clearly intending to sell commercially (which is both fine and dandy), and your first interaction of this brave new World Stack is to tell me I'm browsing wrong! And that's immediately after I click the previous "Get 5.0 Alpha" Call To Action (which was a button that appeared just fine, mind you) and I'm faced with the bitter realization that it's not Get, it's Get In Our CRM. This kind of thing rubs me the wrong way, hard. I live in the B2B world, I totally get the pitch and the signup, but don't mislead me on the call to action, and don't make me distrust your intentions (or web development skills, for that matter) by telling me I need to open my privacy and performance preserving ad blocking gate and let you come over to my yard just to fill out a damn signup form on your fancy clipboard. It's inauthentic, and I don't think I'm the only one who feels that way.
- msellout 11y ago
  I don't care about the authenticity/ickiness/etc. that just strikes me as a terrible interface decision, one that would make me worry they've made other terrible interface decisions where it matters -- in the product itself. The idea of forcing the user to change a browser setting to get work done is a very "enterprisey" mindset. An old way of thinking that it's the user's fault if they can't use the software.
  - yeukhon 11y ago
    Kibana 4 interface looks nice first, but actually very confusing, even more confusing than the old interface.
    - kkirsche 11y ago
      Maybe you'll be happy 5 is another new interface.
      - dozzie 11y ago
        And another change of underlying stack? To what this time? Kibana 2 was PHP, then (K3) it was static HTML + user-side JavaScript, and Kibana 4 runs on Node.js.
- anonfunction 11y ago
  Not to mention when you fill it out it says: > Success! We've got your information and will send you a note when 5.0 Alpha is available. With no links to documentation or github or anything. Having me sign up and get told I'll be emailed at some unspecified time when it's available was very disappointing.
- tvmalsv 11y ago
  Also annoying is that in order to watch a video overview of many of their products (ie. Elasticsearch 2.0, Beats, Watcher, Shield, ES-Hadoop) I have to register (name, email, use case). Sorry, I'm not going to register just to watch the videos.
vacri 11y ago
Dear Elastic: For the love of internet-god, please stop your constant moving of stuff around. Now we have new logos. Last year's fun, I was using logstash-shipper to ship logs. Early on, the package got pulled completely - sucks to be you if it's in your deployment script or in your documentation. Then it had a name change. Then it moved to one domain. Then it moved to another domain. Then it got switched out for Beats. Not everyone finds setting up and maintaining an ELK stack so fascinating that they want to keep up to date with exactly where everything is this month. While you can do other things with ELK, the primary use-case is logging. Logging is supposed to be reliable and 'just work'. Every time I see the elastic website, something else has changed, and everyone is pushing the new stuff. ELK is cool and all, but it's frustrating to follow when you just poke your nose in every few months. Love, - Vacri
- AznHisoka 11y ago
  Their renaming of ElasticSearch to Elastic really irked me. If only for the simple reason that searching for questions related to them on Google is harder now.
  - msellout 11y ago
    They could have gone with "Elastico" or something easier to trademark. I think that's why they keep saying "Elastic Stack" instead of just "Elastic".
    - JorgeGT 11y ago
      Can you rally trademark Spanish words?
      - msellout 11y ago
        Probably. Trademarks are contextual. Two companies can both trademark the word "Apple". One in the context of computers, another in music distribution. So long as they don't get into each other's business, it's OK. Hmm... Yeah, Apple Records wasn't too happy with iTunes.
packetized 11y ago
This is the latest in a long string of irritations that will have me looking for a replacement logging datastore post-haste. I've already replaced Logstash with Heka; I guess ES is next.
- ktamura 11y ago
  Curious if you had a chance to look at Fluentd (disclaimer: I'm one of the maintainers of Fluentd) If you did, I'm interested in your honest feedback.
  - packetized 11y ago
    Not yet, but I'll give it a shot. How's sharding/scalability?
  - packetized 11y ago
    We already have a transport pipeline; what's needed is a decently scalable, searchable, easy-to-use datastore for short/medium/long-term log analysis.
- zenlikethat 11y ago
  Right? An ES replacement in Go would be so great.
  - sz4kerto 11y ago
    Why do you care of the language?
    - glial 11y ago
      For one thing it means you wouldn't have to screw around with the peculiarities of the JVM.
    - zenlikethat 11y ago
      Ridiculously easy deployment and lower memory footprint.
stuartaxelowen 11y ago
Does anyone have a good description of the ES query DSL? In spite of the time I've spent with it, I'm consistently unable to write a reasonable query without going to google first, even for basics.
- NDizzle 11y ago
  Here is what I have linked on one of my search pages: (for advanced customers to examine) https://www.elastic.co/guide/en/elasticsearch/reference/1.7/query-dsl-query-string-query.html#query-string-syntax https://www.elastic.co/guide/en/elasticsearch/reference/1.7/... I have SEVERAL elasticsearch-hellride.txt files stored in my documents folder with a bunch of example queries, because it's so wordy I can't keep it all in my head. I just refer to those every few months when I'm adding functionality. I wouldn't get caught up with having to google for basics. Find some things that work FOR YOU and save those, with your personal notes added in the right places. Here is what I can recommend as far as plugins go: http://www.elastichq.org/ http://www.elastichq.org/ - clean, simple interface, but not quite as powerful as: KOPF: https://github.com/lmenezes/elasticsearch-kopf https://github.com/lmenezes/elasticsearch-kopf This will give you access to a great query interface. You can use the GUIs here to build a few queries until you get a feel for what the JSON should look like. How I use elasticsearch is kind of... Well. It's just what I do. We have PDFs that are OCRed and we store the text in the document along with ~50 fields that are entered by humans. It's probably too complex, but it's financial data and people love to be super verbose with their queries. I can't rely on the OCR to be perfect for SEC filings.
  - packetized 11y ago
    Kopf is phenomenal. All the node data and index stats that I want, all super-accessible.
  - deleted 11y ago
    [deleted]
- glial 11y ago
  I typically use elasticsearch-py[0] to build and test queries. Deeply nested JSON documents are a headache. [0] http://elasticsearch-py.readthedocs.org/en/master/ http://elasticsearch-py.readthedocs.org/en/master/
willejs 11y ago
So logstash V5, is that a rewrite in go? I say this because it seems all their other tooling is now written in go, and also, logstash agent and server are very resource intensive in jruby. If theres a v5 alpha, which isnt on github, is it not open source?
- gerhardhaering 11y ago
  Apparently all they've announced is that they'll release all their products in lockstep with a unified version number from now on. I myself have hoped for a Go rewrite of Logstash for a long time, but there are apparently no plans for this. They are creating lightweight forwarders with their *beats, though. But they are only for forwarding to ElasticSearch, not a general log pipeline processor like Logstash. FWIW there is a thing that's like Logstash in Go, it's Heka by Mozilla. I am very fond of it, but for some reason not many people seem to be aware of it or deploy it.
  - LeoHexspoor 11y ago
    They have a couple of outputs from *beats though, not only ElasticSearch. Besides ElasticSearch there are outputs to console, file, Logstash and a deprecated Redis output.
  - packetized 11y ago
    I've been beating the drum for Heka ever since I found it.
forgotpwtomain 11y ago
I've been extremely unimpressed with logstash - many of the plugins in the standard repo are poorly maintained; certain plugins misbehaving - can kill the entire logstash process; it's really inexcusably bad for such core-infrastructure software tbh.
clebio 11y ago
I was excited and hopeful to try the ELK stack a while back, but, like all the other comments here, have decided it was too unreliable and brittle. I have an open issue on Logstash, spent a fair amount of time detailing it, but have gotten no feedback. And then I realized there's 600+ open issues! https://github.com/elastic/logstash/issues/4389 https://github.com/elastic/logstash/issues/4389 I'm considering using Syslog-ng and would love to hear if anyone has comments on that. Based on other comments here, will be checking out Riemann and Fluentd as well. https://syslog-ng.org/ https://syslog-ng.org/
- dozzie 11y ago
  Don't get too excited about Riemann. It's a stream processing engine, not data transport. It has its uses, but not for shipping logs. syslog-ng and its competitor, rsyslog, are fine, but they are concentrated around logs specifically. Fluentd (and logstash) can be used to transport other data, too, like monitoring or inventory.
  - clebio 11y ago
    Good info (from jedisct1 as well) -- the distinctions are indeed important!
- jedisct1 11y ago
  Fluentd is great, and works very well to inject logs into Elasticsearch. We were using it at OpenDNS. We didn't even try Logstash, as Fluentd did the job. At OVH, both turned out to be way too slow, so we wrote Flowgger which is heavily used by all our services: https://github.com/jedisct1/flowgger https://github.com/jedisct1/flowgger
- Xylakant 11y ago
  I'm not an elastic person, but I can shed some light on this: You're holding it wrong. It's not a bug. You have multiple config files in one directory - if you do that, all those files are combined to one, that means that each event gets handed to each of your individual outputs - multiplying the message. See https://www.elastic.co/guide/en/logstash/current/command-line-flags.html https://www.elastic.co/guide/en/logstash/current/command-lin... Feel free to hop on the IRC if you have further questions, there's usually somebody qualified to answer.
GordonS 11y ago
So I was interested in finding out more about Beats. From the product page I click 'View More' beside 'Beats Overview & Demo Video'... and get taken to a registration form with all fields mandatory... eh... no... forget it then. What the hell has happened to Elastic?!
- packetized 11y ago
  Please take a look at Heka (http://hekad.rtfd.org http://hekad.rtfd.org) - it's a fairly complex tool, but I'm convinced that it's infinitely better than anything Elastic will put out anytime soon for log shipping.
  - GordonS 11y ago
    Looks nice! I wasn't aware of Heka either (I've been out of the Elasticsearch world for a year or so), so thanks :)
jamesblonde 11y ago
What i didn't like about Kibana 4 was that they tried to force you to use a nodejs server. I wanted to embed Kibana in my webapp, but Elastic are trying to make you use their services and lock them into their platform. No surprise, i guess, but annoying nonetheless.
- bni 11y ago
  Has it ever been possible to embed Kibana in a webapp? Excluding iframe hacks ofcourse
  - jamesblonde 11y ago
    Yes, in v3. In v4, we're using this: https://github.com/kibana-community/kibana4-static https://github.com/kibana-community/kibana4-static
- Xylakant 11y ago
  I'm not exactly fond of the node requirement, but I fail to see how it locks you any more to their stack than before. I see why they needed some sort of app behind it and well, node is probably the first you pick when you're a js developer by trade.

108 points by hurrycane 11y ago

More details on the blog post too: https://www.elastic.co/blog/heya-elastic-stack-and-x-pack https://www.elastic.co/blog/heya-elastic-stack-and-x-pack

glial 11y ago

Is X-Pack going to be a subscription based service?

wyaeld 11y ago
Considering all the components currently in it are only available if you're in their subscription plan, that's basically the core business model.
- glial 11y ago
  I wonder whether the components will be available a la carte like they are now, or if it's going to be a single bundle.

spdustin 11y ago

I got pretty excited to read about this unified endeavor, and hungrily clicked the CTA to Get 5.0 Alpha. > Don't see a button? You may have to disable your ad blocker. What a shitty experience. You're pitching a product that you're clearly intending to sell commercially (which is both fine and dandy), and your first interaction of this brave new World Stack is to tell me I'm browsing wrong! And that's immediately after I click the previous "Get 5.0 Alpha" Call To Action (which was a button that appeared just fine, mind you) and I'm faced with the bitter realization that it's not Get, it's Get In Our CRM. This kind of thing rubs me the wrong way, hard. I live in the B2B world, I totally get the pitch and the signup, but don't mislead me on the call to action, and don't make me distrust your intentions (or web development skills, for that matter) by telling me I need to open my privacy and performance preserving ad blocking gate and let you come over to my yard just to fill out a damn signup form on your fancy clipboard. It's inauthentic, and I don't think I'm the only one who feels that way.

msellout 11y ago
I don't care about the authenticity/ickiness/etc. that just strikes me as a terrible interface decision, one that would make me worry they've made other terrible interface decisions where it matters -- in the product itself. The idea of forcing the user to change a browser setting to get work done is a very "enterprisey" mindset. An old way of thinking that it's the user's fault if they can't use the software.
- yeukhon 11y ago
  Kibana 4 interface looks nice first, but actually very confusing, even more confusing than the old interface.
  - kkirsche 11y ago
    Maybe you'll be happy 5 is another new interface.
    - dozzie 11y ago
      And another change of underlying stack? To what this time? Kibana 2 was PHP, then (K3) it was static HTML + user-side JavaScript, and Kibana 4 runs on Node.js.
anonfunction 11y ago
Not to mention when you fill it out it says: > Success! We've got your information and will send you a note when 5.0 Alpha is available. With no links to documentation or github or anything. Having me sign up and get told I'll be emailed at some unspecified time when it's available was very disappointing.
tvmalsv 11y ago
Also annoying is that in order to watch a video overview of many of their products (ie. Elasticsearch 2.0, Beats, Watcher, Shield, ES-Hadoop) I have to register (name, email, use case). Sorry, I'm not going to register just to watch the videos.

vacri 11y ago

Dear Elastic: For the love of internet-god, please stop your constant moving of stuff around. Now we have new logos. Last year's fun, I was using logstash-shipper to ship logs. Early on, the package got pulled completely - sucks to be you if it's in your deployment script or in your documentation. Then it had a name change. Then it moved to one domain. Then it moved to another domain. Then it got switched out for Beats. Not everyone finds setting up and maintaining an ELK stack so fascinating that they want to keep up to date with exactly where everything is this month. While you can do other things with ELK, the primary use-case is logging. Logging is supposed to be reliable and 'just work'. Every time I see the elastic website, something else has changed, and everyone is pushing the new stuff. ELK is cool and all, but it's frustrating to follow when you just poke your nose in every few months. Love, - Vacri

AznHisoka 11y ago
Their renaming of ElasticSearch to Elastic really irked me. If only for the simple reason that searching for questions related to them on Google is harder now.
- msellout 11y ago
  They could have gone with "Elastico" or something easier to trademark. I think that's why they keep saying "Elastic Stack" instead of just "Elastic".
  - JorgeGT 11y ago
    Can you rally trademark Spanish words?
    - msellout 11y ago
      Probably. Trademarks are contextual. Two companies can both trademark the word "Apple". One in the context of computers, another in music distribution. So long as they don't get into each other's business, it's OK. Hmm... Yeah, Apple Records wasn't too happy with iTunes.

packetized 11y ago

This is the latest in a long string of irritations that will have me looking for a replacement logging datastore post-haste. I've already replaced Logstash with Heka; I guess ES is next.

ktamura 11y ago
Curious if you had a chance to look at Fluentd (disclaimer: I'm one of the maintainers of Fluentd) If you did, I'm interested in your honest feedback.
- packetized 11y ago
  Not yet, but I'll give it a shot. How's sharding/scalability?
- packetized 11y ago
  We already have a transport pipeline; what's needed is a decently scalable, searchable, easy-to-use datastore for short/medium/long-term log analysis.
zenlikethat 11y ago
Right? An ES replacement in Go would be so great.
- sz4kerto 11y ago
  Why do you care of the language?
  - glial 11y ago
    For one thing it means you wouldn't have to screw around with the peculiarities of the JVM.
  - zenlikethat 11y ago
    Ridiculously easy deployment and lower memory footprint.

stuartaxelowen 11y ago

Does anyone have a good description of the ES query DSL? In spite of the time I've spent with it, I'm consistently unable to write a reasonable query without going to google first, even for basics.

NDizzle 11y ago
Here is what I have linked on one of my search pages: (for advanced customers to examine) https://www.elastic.co/guide/en/elasticsearch/reference/1.7/query-dsl-query-string-query.html#query-string-syntax https://www.elastic.co/guide/en/elasticsearch/reference/1.7/... I have SEVERAL elasticsearch-hellride.txt files stored in my documents folder with a bunch of example queries, because it's so wordy I can't keep it all in my head. I just refer to those every few months when I'm adding functionality. I wouldn't get caught up with having to google for basics. Find some things that work FOR YOU and save those, with your personal notes added in the right places. Here is what I can recommend as far as plugins go: http://www.elastichq.org/ http://www.elastichq.org/ - clean, simple interface, but not quite as powerful as: KOPF: https://github.com/lmenezes/elasticsearch-kopf https://github.com/lmenezes/elasticsearch-kopf This will give you access to a great query interface. You can use the GUIs here to build a few queries until you get a feel for what the JSON should look like. How I use elasticsearch is kind of... Well. It's just what I do. We have PDFs that are OCRed and we store the text in the document along with ~50 fields that are entered by humans. It's probably too complex, but it's financial data and people love to be super verbose with their queries. I can't rely on the OCR to be perfect for SEC filings.
- packetized 11y ago
  Kopf is phenomenal. All the node data and index stats that I want, all super-accessible.
- deleted 11y ago
  [deleted]
glial 11y ago
I typically use elasticsearch-py[0] to build and test queries. Deeply nested JSON documents are a headache. [0] http://elasticsearch-py.readthedocs.org/en/master/ http://elasticsearch-py.readthedocs.org/en/master/

willejs 11y ago

So logstash V5, is that a rewrite in go? I say this because it seems all their other tooling is now written in go, and also, logstash agent and server are very resource intensive in jruby. If theres a v5 alpha, which isnt on github, is it not open source?

gerhardhaering 11y ago
Apparently all they've announced is that they'll release all their products in lockstep with a unified version number from now on. I myself have hoped for a Go rewrite of Logstash for a long time, but there are apparently no plans for this. They are creating lightweight forwarders with their *beats, though. But they are only for forwarding to ElasticSearch, not a general log pipeline processor like Logstash. FWIW there is a thing that's like Logstash in Go, it's Heka by Mozilla. I am very fond of it, but for some reason not many people seem to be aware of it or deploy it.
- LeoHexspoor 11y ago
  They have a couple of outputs from *beats though, not only ElasticSearch. Besides ElasticSearch there are outputs to console, file, Logstash and a deprecated Redis output.
- packetized 11y ago
  I've been beating the drum for Heka ever since I found it.

forgotpwtomain 11y ago

I've been extremely unimpressed with logstash - many of the plugins in the standard repo are poorly maintained; certain plugins misbehaving - can kill the entire logstash process; it's really inexcusably bad for such core-infrastructure software tbh.

clebio 11y ago

I was excited and hopeful to try the ELK stack a while back, but, like all the other comments here, have decided it was too unreliable and brittle. I have an open issue on Logstash, spent a fair amount of time detailing it, but have gotten no feedback. And then I realized there's 600+ open issues! https://github.com/elastic/logstash/issues/4389 https://github.com/elastic/logstash/issues/4389 I'm considering using Syslog-ng and would love to hear if anyone has comments on that. Based on other comments here, will be checking out Riemann and Fluentd as well. https://syslog-ng.org/ https://syslog-ng.org/

dozzie 11y ago
Don't get too excited about Riemann. It's a stream processing engine, not data transport. It has its uses, but not for shipping logs. syslog-ng and its competitor, rsyslog, are fine, but they are concentrated around logs specifically. Fluentd (and logstash) can be used to transport other data, too, like monitoring or inventory.
- clebio 11y ago
  Good info (from jedisct1 as well) -- the distinctions are indeed important!
jedisct1 11y ago
Fluentd is great, and works very well to inject logs into Elasticsearch. We were using it at OpenDNS. We didn't even try Logstash, as Fluentd did the job. At OVH, both turned out to be way too slow, so we wrote Flowgger which is heavily used by all our services: https://github.com/jedisct1/flowgger https://github.com/jedisct1/flowgger
Xylakant 11y ago
I'm not an elastic person, but I can shed some light on this: You're holding it wrong. It's not a bug. You have multiple config files in one directory - if you do that, all those files are combined to one, that means that each event gets handed to each of your individual outputs - multiplying the message. See https://www.elastic.co/guide/en/logstash/current/command-line-flags.html https://www.elastic.co/guide/en/logstash/current/command-lin... Feel free to hop on the IRC if you have further questions, there's usually somebody qualified to answer.

GordonS 11y ago

So I was interested in finding out more about Beats. From the product page I click 'View More' beside 'Beats Overview & Demo Video'... and get taken to a registration form with all fields mandatory... eh... no... forget it then. What the hell has happened to Elastic?!

packetized 11y ago
Please take a look at Heka (http://hekad.rtfd.org http://hekad.rtfd.org) - it's a fairly complex tool, but I'm convinced that it's infinitely better than anything Elastic will put out anytime soon for log shipping.
- GordonS 11y ago
  Looks nice! I wasn't aware of Heka either (I've been out of the Elasticsearch world for a year or so), so thanks :)

jamesblonde 11y ago

What i didn't like about Kibana 4 was that they tried to force you to use a nodejs server. I wanted to embed Kibana in my webapp, but Elastic are trying to make you use their services and lock them into their platform. No surprise, i guess, but annoying nonetheless.

bni 11y ago
Has it ever been possible to embed Kibana in a webapp? Excluding iframe hacks ofcourse
- jamesblonde 11y ago
  Yes, in v3. In v4, we're using this: https://github.com/kibana-community/kibana4-static https://github.com/kibana-community/kibana4-static
Xylakant 11y ago
I'm not exactly fond of the node requirement, but I fail to see how it locks you any more to their stack than before. I see why they needed some sort of app behind it and well, node is probably the first you pick when you're a js developer by trade.