4 ms·

As a possible middle-ground: I'm using Graylog with a lot of success, but my log volumes are rather modest (medium sized corporate network switches/routers/fire

As a possible middle-ground: I'm using Graylog with a lot of success, but my log volumes are rather modest (medium sized corporate network switches/routers/firewalls/servers log aggregation and processing)

fxtentacle 7y ago
We have 5TB per day, which may or may not break Graylog. Before evaluating ES, we used Logstash, but that couldn't keep up at around 2TB+ per day.
- virgilp 7y ago
  > Before evaluating ES, we used Logstash Huh? I thought one would use Logstash to ingest & massage data going into ES. I view it as a sort of "quick&dirty stream processing engine". How would one go about using Logstash _instead of_ ES, as opposed to "in addition to ES" ?
  - fxtentacle 7y ago
    Sorry, I worded that badly. We used to use a locally-hosted ELK stack, so yes, including ES as the data backend for Kibana, but there Logstash or more precisely filebeat and logstash-forwarder became the bottleneck. That was a horrible experience because then filebeat would keep the old (unsent, but deleted) log files open, so production servers crashed running out of HDD space even though "du -h -s /" was a lot smaller than the HDD size. It took me way too long to figure out that deleted but still open files were the problem. So our decision to move from using ES invisibly as part of Logstash towards a full dedicated ES cluster was forced by the Logstash pipeline getting too slow. I then evaluated Elastic Cloud and using Amazon EC2 for deploying ES, both of which were ultimately rejected. The first due to fears of data loss, the second due to costs.
    - virgilp 7y ago
      I'm confused again - wouldn't Elastic Cloud cost more than EC2? It makes no sense to be the other way around. And secondly - why use logstash if you had filebeat - why not send data to ES directly? (I mean, specifically for your usecase, where you don't seem to need to do much processing prior to ingestion). Yeah, I saw that big parts of Logstash are in ruby so I expect underwhelming performance from it - but Filebeat should in theory be able to ingest large volumes of data into ES directly.
      - fxtentacle 7y ago
        We used a central logstash server to receive the data from filebeat and logstash-forwarder running on multiple other servers. So in the initial deployment, logstash only set up the host and service tags needed for Kibana plus it was like a central gateway for external servers to write into the locally running ES storage. As for why the price that elastic.co quoted us was lower than EC2, I have no idea. My guess would be that they were hoping to get a reference customer onboard for their (then new) offering. Or maybe they are internally running bare-metal and thereby purchasing their CPU power cheaper than EC2 pricing. I mean you don't really need a redundant fault-tolerant virtual machine if the database on top is redundant and fault-tolerant, too.
    - deleted 7y ago
      [deleted]
- pepemon 7y ago
  I'm piping 5TB/day through 2 Graylog hosts with plain stupid DNS round-robin, all good.
jbergens 7y ago
We used Greylog in previous company and it worked perfectly. Scaled to our needs but we might have used a cluster.

fxtentacle 7y ago

We have 5TB per day, which may or may not break Graylog. Before evaluating ES, we used Logstash, but that couldn't keep up at around 2TB+ per day.

virgilp 7y ago
> Before evaluating ES, we used Logstash Huh? I thought one would use Logstash to ingest & massage data going into ES. I view it as a sort of "quick&dirty stream processing engine". How would one go about using Logstash _instead of_ ES, as opposed to "in addition to ES" ?
- fxtentacle 7y ago
  Sorry, I worded that badly. We used to use a locally-hosted ELK stack, so yes, including ES as the data backend for Kibana, but there Logstash or more precisely filebeat and logstash-forwarder became the bottleneck. That was a horrible experience because then filebeat would keep the old (unsent, but deleted) log files open, so production servers crashed running out of HDD space even though "du -h -s /" was a lot smaller than the HDD size. It took me way too long to figure out that deleted but still open files were the problem. So our decision to move from using ES invisibly as part of Logstash towards a full dedicated ES cluster was forced by the Logstash pipeline getting too slow. I then evaluated Elastic Cloud and using Amazon EC2 for deploying ES, both of which were ultimately rejected. The first due to fears of data loss, the second due to costs.
  - virgilp 7y ago
    I'm confused again - wouldn't Elastic Cloud cost more than EC2? It makes no sense to be the other way around. And secondly - why use logstash if you had filebeat - why not send data to ES directly? (I mean, specifically for your usecase, where you don't seem to need to do much processing prior to ingestion). Yeah, I saw that big parts of Logstash are in ruby so I expect underwhelming performance from it - but Filebeat should in theory be able to ingest large volumes of data into ES directly.
    - fxtentacle 7y ago
      We used a central logstash server to receive the data from filebeat and logstash-forwarder running on multiple other servers. So in the initial deployment, logstash only set up the host and service tags needed for Kibana plus it was like a central gateway for external servers to write into the locally running ES storage. As for why the price that elastic.co quoted us was lower than EC2, I have no idea. My guess would be that they were hoping to get a reference customer onboard for their (then new) offering. Or maybe they are internally running bare-metal and thereby purchasing their CPU power cheaper than EC2 pricing. I mean you don't really need a redundant fault-tolerant virtual machine if the database on top is redundant and fault-tolerant, too.
  - deleted 7y ago
    [deleted]
pepemon 7y ago
I'm piping 5TB/day through 2 Graylog hosts with plain stupid DNS round-robin, all good.

jbergens 7y ago

We used Greylog in previous company and it worked perfectly. Scaled to our needs but we might have used a cluster.