5 ms·

Working with the ELK stack

87 points by greenonion 12y ago

octotoad 12y ago
I'm currently working on deploying an ELK box for visualising a mix of financial transaction data and corresponding network application backend syslog records. So far so good, but frontend Kibana 3 performance can be really painful at times given the large amount of "documents" that can be involved (anywhere up to 5000 events per minute). It seems like displaying multiple histogram panels may be quicker in real time, given the shorter timeframe, but it would be nice to be able work with something like a month's worth of data without major performance hits. Not sure if this is something specific to Kibana design, Elasticsearch indexing/search configuration or certain JavaScript engine behaviour.
- charkost 12y ago
  We have (at skroutz.gr) kibana 3 dashboards with 61,269,264 documents using the last month as time window and still no performance hit. Check our cluster resources in our blog post to get an idea.
ehurrell 12y ago
The ELK stack is great, powerful, relatively easy to use and set up and very configurable. I'd definitely consider it for any dashboard prototyping, given that I've worked places where the 'MVP' dashboard could've been mocked up in days with ELK rather than the months it took otherwise.
- giorgos_tsif 12y ago
  so true about the 'MVP' dashboard
ed_blackburn 12y ago
I'm in an environment that has dismissed the idea of ELK for SCOM. I'll be interested to see how this SCOM world progresses. I'm currently underwhelmed by the SCOM narrative for both pushing data to it and more alarmingly how to get web based dashboards published without great expense. Anybody got or know of any success stories? (Sulks off pining for ELK.)
nissimk 12y ago
Is there a lightweight solution for centralized logging with full text search and a nice UI? ELK seems like it requires some serious RAM and cores even if you are just talking about elasticsearch. Is there a similar solution that works well in just a couple of gigs of RAM for a slower message flow rate? Is anybody here successfully using ELK in a low ram environment with low message flow?
- simonw 12y ago
  Have you tried it yourself on your dataset? I've been experimenting with ElasticSearch on small datasets with a single node on my laptop and it doesn't seem to be hugely resource demanding.
- ihsw 12y ago
  Please define low RAM. I've got a t2.small server on AWS that's taking in a low/moderate logging from three servers (syslog x3, nginx x3, zero-low traffic redis) and it's usually in the 750MB of RAM with absolutely no notable CPU load. Granted Kibana/ElasticSearch gets pretty much no traffic other than checking it once or twice a day, just to get a glance at 4xx/5xx errors. So yeah, you just defined my current use-case down to the letter -- centralized logging, fancy interface for filtering/searching/visualizing said log data.
- mbell 12y ago
  I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.
  - robbles 12y ago
    Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria? We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.
greenonion 12y ago
Hi, OP here, we 're really excited that this made the HN front page! Feel free to ask us anything.
- kiyoto 12y ago
  For the logging layer, have you considered/looked at Fluentd? Disclaimer: I am a Fluentd maintainer and would love to hear feedback from folks.
lkrubner 12y ago
I am working at a company that has 75,000 customers, and we keep track of a fairly large set of personal data about these 75,000 customers. The data is typically kept in PostGres, but for reports, we assumed we could dump it out in a denormalized form to ElasticSearch. We would not dump all the data, of course, we only take 18 items that are considered very important. We had never done much analysis of who our customers were, and what their level of engagement was. We had a new person come in, focused on business intelligence, and they were desperate to get some data about our customers. So I wrote a short Python script that pulled the data we wanted out of PostGres and stored in ElasticSearch. I then made it available to the team via Kibana. I assumed everyone would be fascinated to look at the data and perhaps see various trends. But that didn't work. Kibana was unusable. With 75,000 records it never loaded. Not in anyone's browser. So I cut that in half, to 36,000 records. And still, it never loaded. So I kept cutting the amount. And eventually I got down to 10,000 records. Then it loaded, but was so slow no one could use it. So finally I cut it down to 7,000 records. Now it loaded, and it was fast enough that we could use it. To do the real analysis, I ended writing another script that dumped out the 75,000 records as a CSV file, then I uploaded it to a spreadsheet on Google Docs. This worked fine. I am curious why Google spreadsheets can render 75,000 records, but Kibana can not? I am also curious what the real use case is for Kibana? If it can't handle large datasets, then its ability to make pretty charts seems useless -- we could never get the data in there to make the chart. I assume that other people will do what I did, and use a spreadsheet instead.
- greenonion 12y ago
  While I do not know the specifications of your Elasticsearch cluster, I am really surprised by your experience. We (at skroutz.gr) use a moderately small cluster (four nodes) and easily process > 4 million rows a day and view them on Kibana dashboards in real-time. We also have success running aggregations on data going back as far as 3 months.
- spencera 12y ago
  Kibana dev here, also surprised by your experience. I have 1 million web log documents stored in elasticsearch right now, running on my macbook, and Kibana runs quite fast IMO. Would love to debug this further if you wouldn't mind filing an issue on Github.
Karunamon 12y ago
I would be seriously beginning to worry, were I working over at Splunk. I know the ES guys have said, over and over, that they're not gunning for Splunk customers but, as the ELK stack becomes a more mature tool, the reasons to use Splunk and pay their downright extortionate prices decrease more and more. I'm assisting with a ~500GB/day cluster right now (with that number expected to quadruple in the next year or so), and ELK has proven to be an amazingly resilient and flexible tool.
- Iaks 12y ago
  I've worked with both systems quite a bit, and really like them both. The licensing obviously makes ELK a no-brainer for my personal use. But, to be fair to Splunk the average required knowledge for their end-user is significantly lower. For the crowd poking around here it seems obvious that ELK is the right choice, but when I am unleashing a team of helpdesk techs on a set of data I would probably have a much more nuanced consideration about the tools. I'm kind of pointing out the obvious, but it's my two cents for whatever they're worth.
helfire 12y ago
Just started playing with ELK last weekend, this guide got me up and running pretty quickly http://evanhazlett.com/2014/11/Logging-with-ELK-and-Docker/ http://evanhazlett.com/2014/11/Logging-with-ELK-and-Docker/
Thaxll 12y ago
The problem I have with ELK is Kibana, it's not a tool designed to watch logs but to do reporting.
- mdaniel 12y ago
  That depends on how one defines "watch logs". Looking at the text, no; looking at the metrics, yes. We had a dashboard that would graph the number of errors grouped by the component over a given span of time. If we saw the chart grow beyond our tolerance, that's when we would break out elasticsearch-head or grep or $tool and try to dig into the details. Do you have a tool in mind that does what you are describing and thus would supplement Kibana?
makmanalp 12y ago
Slightly off topic, but wow - thank you for the Turkish stemmer! There have been a few attempts at this, but I'm not sure if any worked super well. For those who don't know, Turkish is agglutinative (http://en.wikipedia.org/wiki/Agglutination http://en.wikipedia.org/wiki/Agglutination) meaning most of the meaning in words are added as suffixes to the words, so stemming is even more crucial and difficult - removing a suffix can completely change the meaning. Example: "Kitap" is "Book" but "Kitaplik" is "Bookshelf" (or more literally "Book-container"). Here it is: https://github.com/skroutz/elasticsearch-analysis-turkishstemmer https://github.com/skroutz/elasticsearch-analysis-turkishste...
- giorgos_tsif 12y ago
  Thank you, maybe we will write a blogpost describing this experience! (We are Greeks we do not know Turkish!) There is a ruby version also here: https://github.com/skroutz/turkish_stemmer https://github.com/skroutz/turkish_stemmer
Dowwie 12y ago
Hey thank you for sharing your experiences. I am a little confused about this architecture decision if the goal is to offer business insight, though. I don't understand why usage metrics that are used to calculate business analytics would be logged in a data store that is separate from business transactions and reporting related data. How would you run cohort studies or track funnels?
elementai 12y ago
We've just finished migrating our ELK stack from cloud VPS'es to bare metal to cut costs. I was and still impressed how smooth that went including upgrading from 0.9x to 1.4
Epicism 12y ago
Has anyone been able to do any sort of correlation or alerting with ELK?

87 points by greenonion 12y ago

I'm currently working on deploying an ELK box for visualising a mix of financial transaction data and corresponding network application backend syslog records. So far so good, but frontend Kibana 3 performance can be really painful at times given the large amount of "documents" that can be involved (anywhere up to 5000 events per minute). It seems like displaying multiple histogram panels may be quicker in real time, given the shorter timeframe, but it would be nice to be able work with something like a month's worth of data without major performance hits. Not sure if this is something specific to Kibana design, Elasticsearch indexing/search configuration or certain JavaScript engine behaviour.

charkost 12y ago
We have (at skroutz.gr) kibana 3 dashboards with 61,269,264 documents using the last month as time window and still no performance hit. Check our cluster resources in our blog post to get an idea.

ehurrell 12y ago

The ELK stack is great, powerful, relatively easy to use and set up and very configurable. I'd definitely consider it for any dashboard prototyping, given that I've worked places where the 'MVP' dashboard could've been mocked up in days with ELK rather than the months it took otherwise.

giorgos_tsif 12y ago
so true about the 'MVP' dashboard

ed_blackburn 12y ago

I'm in an environment that has dismissed the idea of ELK for SCOM. I'll be interested to see how this SCOM world progresses. I'm currently underwhelmed by the SCOM narrative for both pushing data to it and more alarmingly how to get web based dashboards published without great expense. Anybody got or know of any success stories? (Sulks off pining for ELK.)

nissimk 12y ago

Is there a lightweight solution for centralized logging with full text search and a nice UI? ELK seems like it requires some serious RAM and cores even if you are just talking about elasticsearch. Is there a similar solution that works well in just a couple of gigs of RAM for a slower message flow rate? Is anybody here successfully using ELK in a low ram environment with low message flow?

simonw 12y ago
Have you tried it yourself on your dataset? I've been experimenting with ElasticSearch on small datasets with a single node on my laptop and it doesn't seem to be hugely resource demanding.
ihsw 12y ago
Please define low RAM. I've got a t2.small server on AWS that's taking in a low/moderate logging from three servers (syslog x3, nginx x3, zero-low traffic redis) and it's usually in the 750MB of RAM with absolutely no notable CPU load. Granted Kibana/ElasticSearch gets pretty much no traffic other than checking it once or twice a day, just to get a glance at 4xx/5xx errors. So yeah, you just defined my current use-case down to the letter -- centralized logging, fancy interface for filtering/searching/visualizing said log data.
mbell 12y ago
I haven't found ELK to be terribly resource intensive. We're currently loading ~20,000 events / min into an ELK stack that runs on a single m1.xlarge box, we do use redis as a logstash broker to deal with unusually bursty load, but it doesn't seem to have much issue keeping up with normal load.
- robbles 12y ago
  Did you need a customized server configuration of Elasticsearch to enable this kind of throughput? And do you split your indexes up by day or some other criteria? We're running a similar configuration and I'd like to know the limitations before we'd need to start using a clustered setup.

greenonion 12y ago

Hi, OP here, we 're really excited that this made the HN front page! Feel free to ask us anything.

kiyoto 12y ago
For the logging layer, have you considered/looked at Fluentd? Disclaimer: I am a Fluentd maintainer and would love to hear feedback from folks.

lkrubner 12y ago

I am working at a company that has 75,000 customers, and we keep track of a fairly large set of personal data about these 75,000 customers. The data is typically kept in PostGres, but for reports, we assumed we could dump it out in a denormalized form to ElasticSearch. We would not dump all the data, of course, we only take 18 items that are considered very important. We had never done much analysis of who our customers were, and what their level of engagement was. We had a new person come in, focused on business intelligence, and they were desperate to get some data about our customers. So I wrote a short Python script that pulled the data we wanted out of PostGres and stored in ElasticSearch. I then made it available to the team via Kibana. I assumed everyone would be fascinated to look at the data and perhaps see various trends. But that didn't work. Kibana was unusable. With 75,000 records it never loaded. Not in anyone's browser. So I cut that in half, to 36,000 records. And still, it never loaded. So I kept cutting the amount. And eventually I got down to 10,000 records. Then it loaded, but was so slow no one could use it. So finally I cut it down to 7,000 records. Now it loaded, and it was fast enough that we could use it. To do the real analysis, I ended writing another script that dumped out the 75,000 records as a CSV file, then I uploaded it to a spreadsheet on Google Docs. This worked fine. I am curious why Google spreadsheets can render 75,000 records, but Kibana can not? I am also curious what the real use case is for Kibana? If it can't handle large datasets, then its ability to make pretty charts seems useless -- we could never get the data in there to make the chart. I assume that other people will do what I did, and use a spreadsheet instead.

greenonion 12y ago
While I do not know the specifications of your Elasticsearch cluster, I am really surprised by your experience. We (at skroutz.gr) use a moderately small cluster (four nodes) and easily process > 4 million rows a day and view them on Kibana dashboards in real-time. We also have success running aggregations on data going back as far as 3 months.
spencera 12y ago
Kibana dev here, also surprised by your experience. I have 1 million web log documents stored in elasticsearch right now, running on my macbook, and Kibana runs quite fast IMO. Would love to debug this further if you wouldn't mind filing an issue on Github.

Karunamon 12y ago

I would be seriously beginning to worry, were I working over at Splunk. I know the ES guys have said, over and over, that they're not gunning for Splunk customers but, as the ELK stack becomes a more mature tool, the reasons to use Splunk and pay their downright extortionate prices decrease more and more. I'm assisting with a ~500GB/day cluster right now (with that number expected to quadruple in the next year or so), and ELK has proven to be an amazingly resilient and flexible tool.

Iaks 12y ago
I've worked with both systems quite a bit, and really like them both. The licensing obviously makes ELK a no-brainer for my personal use. But, to be fair to Splunk the average required knowledge for their end-user is significantly lower. For the crowd poking around here it seems obvious that ELK is the right choice, but when I am unleashing a team of helpdesk techs on a set of data I would probably have a much more nuanced consideration about the tools. I'm kind of pointing out the obvious, but it's my two cents for whatever they're worth.

helfire 12y ago

Just started playing with ELK last weekend, this guide got me up and running pretty quickly http://evanhazlett.com/2014/11/Logging-with-ELK-and-Docker/ http://evanhazlett.com/2014/11/Logging-with-ELK-and-Docker/

Thaxll 12y ago

The problem I have with ELK is Kibana, it's not a tool designed to watch logs but to do reporting.

mdaniel 12y ago
That depends on how one defines "watch logs". Looking at the text, no; looking at the metrics, yes. We had a dashboard that would graph the number of errors grouped by the component over a given span of time. If we saw the chart grow beyond our tolerance, that's when we would break out elasticsearch-head or grep or $tool and try to dig into the details. Do you have a tool in mind that does what you are describing and thus would supplement Kibana?

makmanalp 12y ago

Slightly off topic, but wow - thank you for the Turkish stemmer! There have been a few attempts at this, but I'm not sure if any worked super well. For those who don't know, Turkish is agglutinative (http://en.wikipedia.org/wiki/Agglutination http://en.wikipedia.org/wiki/Agglutination) meaning most of the meaning in words are added as suffixes to the words, so stemming is even more crucial and difficult - removing a suffix can completely change the meaning. Example: "Kitap" is "Book" but "Kitaplik" is "Bookshelf" (or more literally "Book-container"). Here it is: https://github.com/skroutz/elasticsearch-analysis-turkishstemmer https://github.com/skroutz/elasticsearch-analysis-turkishste...

giorgos_tsif 12y ago
Thank you, maybe we will write a blogpost describing this experience! (We are Greeks we do not know Turkish!) There is a ruby version also here: https://github.com/skroutz/turkish_stemmer https://github.com/skroutz/turkish_stemmer

Dowwie 12y ago

Hey thank you for sharing your experiences. I am a little confused about this architecture decision if the goal is to offer business insight, though. I don't understand why usage metrics that are used to calculate business analytics would be logged in a data store that is separate from business transactions and reporting related data. How would you run cohort studies or track funnels?

elementai 12y ago

We've just finished migrating our ELK stack from cloud VPS'es to bare metal to cut costs. I was and still impressed how smooth that went including upgrading from 0.9x to 1.4

Epicism 12y ago

Has anyone been able to do any sort of correlation or alerting with ELK?