20 ms·

With all the Hipster tech being released recently, the headline statement holds true for a lot of things, unfortunately. We recently discussed new logging tool

With all the Hipster tech being released recently, the headline statement holds true for a lot of things, unfortunately. We recently discussed new logging tools at work. It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. The log ingestion and search performance was roughly the same... EDIT: To give everyone a sense of scale, those $200 each bare metal servers are 2x Intel Xeon 6-core + 256GB RAM + 15x 10TB 7200 rpm. We retain logs for 30 days and handle 4-5TB per day.

certera 7y ago
Why not run ES on the bare metal servers?
- fxtentacle 7y ago
  Tried and it was horribly slow. We add about 5TB of logs per day, and the ES index wasn't able to keep up, despite having 256 GB RAM and 2x Intel Xeon 6-core
  - camkego 7y ago
    That seems like a really good price for bare metal of that magnitude, where are you getting those from?
    - fxtentacle 7y ago
      https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx On their German website, you can bid for used-but-still-good servers and there you'll find great offers. There, you can also find hardware that they customized for others and then didn't need anymore. For example, I rented some GPU rendering servers for $110 monthly each from there. https://www.hetzner.de/sb https://www.hetzner.de/sb Plus I can't praise their service and ops team enough. After we rented the new storage servers, I sent them an email to ask for it, and they then put the new storage servers on the same router inside the same datacenter as our other servers with them. Now we effectively have 2x redundant 1GBit LAN between our servers with them, plus an 10GBit internet uplink shared between our servers. For that level of performance, they are ridiculously cheap. BUT - and that is a big gotcha - you will be responsible for diagnosing hardware issues and asking for a replacement yourself. That's why we have RAID and two storage systems with them, because we tend to see one HDD fail every 3-6 months.
Johnny555 7y ago
What kind of searches are you doing where the search performance of grep is the same as with Elasticsearch?
- fxtentacle 7y ago
  One common query was to check the FTP server on our side for access from a customer server, so that we can help with troubleshooting as to why they couldn't connect. Turns out, in many cases less-technical customers mistyped the hostname or left out the .com at the end. tail -n 10000 /var/log/ftp.log | grep -i $CUSTOMER_IP | tail -n 5
  - Johnny555 7y ago
    If that's your typical use-case, I can see why that's as fast (or faster) as Elasticsearch, but it's not clear why you'd have gone with Elasticsearch in the first place. When I last used Elasticsearch, we indexed ~10TB of log data a day, kept 14+ days, and a typical query was looking for log records that matched a unique session ID over the past 10 days or so, not an easy task for grep. But we didn't pay $50K/month for that cluster, it was closer to $12K/month. Before we used ES, someone had written a parallel grep that would grep multiple files at once, and would run multiple greps at once through chunks of the file, but still it could take 30 minutes or longer to churn through the logs on a 32 core machine - ES took that query time down to 100 milliseconds. The ES cluster easily paid for itself in employee time savings.
    - londons_explore 7y ago
      If you always want to search by one thing, you can manually index by that thing. In your case, arrange your log files by the first 6 hex characters of the user ID (/var/log/xxx/xxx/date.log), and grep will typically only then have a few megabytes to scan. If you need real indexes, or just want something industry standard and maintainable rather than 'some guys grep script', then elastic search is probably the way to go.
      - 7y ago
idnefju 7y ago
Horizontal scaling is also where ES shines. If you don't need that among other things, then there are alternatives yea
threeseed 7y ago
ElasticSearch also does a lot more than your CLI does. Splunk style dashboards, multi-user access, full text searching e.g. stemming, ability to support non-structured formats.
- fxtentacle 7y ago
  I fully agree, but our business need was to stream log files (e.g. tail -f) and to search for snippets in them and to produce statistic counters (e.g. |grep|wc) While I can imagine situations where ElasticSearch is the best solution, I have seen it mostly used in situations that would be better served with simple command-line tools, similar to what the author demonstrated with awk vs. Hadoop.
- enriquto 7y ago
  > ElasticSearch also does a lot more than your CLI does. This is not necessarily a good thing. In fact, it is probably a really bad thing if you do not need the additional stuff.
DrPhish 7y ago
As a possible middle-ground: I'm using Graylog with a lot of success, but my log volumes are rather modest (medium sized corporate network switches/routers/firewalls/servers log aggregation and processing)
- fxtentacle 7y ago
  We have 5TB per day, which may or may not break Graylog. Before evaluating ES, we used Logstash, but that couldn't keep up at around 2TB+ per day.
  - virgilp 7y ago
    > Before evaluating ES, we used Logstash Huh? I thought one would use Logstash to ingest & massage data going into ES. I view it as a sort of "quick&dirty stream processing engine". How would one go about using Logstash _instead of_ ES, as opposed to "in addition to ES" ?
    - fxtentacle 7y ago
      Sorry, I worded that badly. We used to use a locally-hosted ELK stack, so yes, including ES as the data backend for Kibana, but there Logstash or more precisely filebeat and logstash-forwarder became the bottleneck. That was a horrible experience because then filebeat would keep the old (unsent, but deleted) log files open, so production servers crashed running out of HDD space even though "du -h -s /" was a lot smaller than the HDD size. It took me way too long to figure out that deleted but still open files were the problem. So our decision to move from using ES invisibly as part of Logstash towards a full dedicated ES cluster was forced by the Logstash pipeline getting too slow. I then evaluated Elastic Cloud and using Amazon EC2 for deploying ES, both of which were ultimately rejected. The first due to fears of data loss, the second due to costs.
      - virgilp 7y ago
        I'm confused again - wouldn't Elastic Cloud cost more than EC2? It makes no sense to be the other way around. And secondly - why use logstash if you had filebeat - why not send data to ES directly? (I mean, specifically for your usecase, where you don't seem to need to do much processing prior to ingestion). Yeah, I saw that big parts of Logstash are in ruby so I expect underwhelming performance from it - but Filebeat should in theory be able to ingest large volumes of data into ES directly.
bobnamob 7y ago
Might be worth taking a look at ripgrep[1]. [1] https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep
- fxtentacle 7y ago
  That tool looks great :) but since we're already seeing <1s search times and the tool is only used by internal support employees, I'm mostly going with "never touch a running system" these days. While for a database like ES you'd put all of the data into one big pile and then filter by keywords, e.g. host=ftp service=ftp query=IP, for logfiles you usually search on a much smaller set. They are rotated by day and logs are broken down by host and service by rsyslog, so instead of filtering the full 150TB - which is what ES has to do - my grep only needs to look at the 1-2 GB of data inside the file where host, service, and date match.
  - pepemon 7y ago
    Do you understand something about such thing as ES indices?
- reagent_finder 7y ago
  RG is awesome for the basic use case. Using it across platforms just makes you tear your hair out, though.
  - foota 7y ago
    I'm curious, why? Looks pretty easy to get the binaries.
  - burntsushi 7y ago
    Why? Can you share your frustrations?
wdroz 7y ago
if you are using systemd, you don't even need to grep "manually" that much. journalctl -u nginx -u ssh --since today
- fxtentacle 7y ago
  I had never heard of that before. Thank you for pointing this out :)
  - pepemon 7y ago
    And yet you're rsyslogging and grepping what is supposed to be structured and typed. Have you heard about backpressure, controlled retention, compliant log redundancy and all of this advanced stuff that you'll never get with aggregation via rsyslogd/syslog-ng?
    - Spivak 7y ago
      rsyslog is far more powerful than you're making it out. You have to actually tell it what to do but it's more expressive than filebeat and logstash. * rsyslog in the use-case he's describing is just a method of pushing some subset of the logs generated on a client system to a directory on the collector which has trade-offs but the benefit is having really simple failure modes. * both rsyslog and journald store structured data: rsyslog with lumberjack, and journald just always. And rsyslog can parse and structure the logs in-flight so you save computing power on the collector. * rsyslog behaves exactly like filebeat when it comes to reliable delivery and can persist unsent messages to memory then disk. rsyslog's rate limiting, backoff, and retry options are super powerful.
      - pepemon 7y ago
        Yes, you are right, sorry. I was too fast in my assumptions, rsyslog (don't know much of syslog-ng) has feature parity with ELK in terms of log delivery and processing. But I think that grep and its permutations aren't right tools of choice for log analysis anyway.
    - 7y ago
bses 7y ago
> We recently discussed new logging tools at work. It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. The log ingestion and search performance was roughly the same... If grep has the same search performance as elasticsearch, you should not be using elasticsearch and any comparison is bullshit. > It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. $50K monthly is $69.44 an hour, or 8 instances of the most expensive thing from https://aws.amazon.com/elasticsearch-service/pricing/ https://aws.amazon.com/elasticsearch-service/pricing/ (with 488GB RAM) Please cite references, your numbers seem made up. Where can you get 256GB of RAM for $200/month?
- jdc 7y ago
  Really? Try Googling '256gb ram dedicated'
  - bses 7y ago
    You're right, I replied too quickly. > We recently discussed new logging tools at work. It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. The log ingestion and search performance was roughly the same... If grep has the same search performance as elasticsearch, you should not be using elasticsearch and any comparison is bullshit.
    - jdc 7y ago
      Right, though some people really do reach for heavy-weight tools on datasets that are way too small to justify their use. But yeah, even ripgrep should get out-performed by any db/search daemon with a decent index.
- fxtentacle 7y ago
  https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx They get slightly cheaper for large customers like us, plus you can get hardware upgrades like the 2x CPUs by paying for the difference upfront. So yes, it's $400 monthly + $2000 once, but those $2k don't really make much of a difference over the years that we've been running the system ^^
didibus 7y ago
Where are you getting such a bare metal machine for 200$ a month?
avip 7y ago
Your setup is redundant and expensive. logrotate directly to s3 and use Athena.
jammygit 7y ago
What cloud advocates always say is that the $50k monthly will save you money from not needing to hire a team to manage it for you, and that over the course of 10+ years you will be ahead. Is that true in anyone's experience? Every once in a while somebody posts about their competing bare-metal system and it looks like a lot of people have managed to cut their server costs by 99% (based on the numbers they post) by avoiding the cloud as a service Honestly curious
- gscott 7y ago
  I have a friend with a rack of co-located servers I manage. I drive from San Diego to LA where they are located maybe once every 6 months or more. I don't believe I have been to the rack for about a year. The rack with 20mbs of bandwidth costs around $2,500 a month.
  - lrem 7y ago
    Did you really mean twenty megabits?
    - xchaotic 7y ago
      if it’s unmetered 20mbps I’d happily take that over ingress / egress cost at cloud provider such as AWS
      - davedx 7y ago
        Yeah AWS bandwidth egress is extortionate. Digital Ocean is something like 5-7x less expensive. It's ridiculous.
      - lrem 7y ago
        That's kind of shocking when compared to European prices. Over a full order of magnitude of difference, comparing to the list price of the first Google result.
    - ChrisCinelli 7y ago
carlsborg 7y ago
You are underestimating the value of indexed logs and a Kibana view into them. For example, if you wanted a histogram of error logs for the last 12 days, and then zoom in and see the actual logs.
- travbrack 7y ago
  You can do this with graylog
- fxtentacle 7y ago
  You are underestimating how much costs the issues caused by Kibana and a slow ES can produce. I used to have these nice Kibana diagrams set up with auth and a https proxy so that our team could easily check if the server error rates looked normal. I quickly found out two things: 1. Most of our employees never ever looked at it, dismissing it as "too technical" just from seeing a screenshot. 2. The ELK stack uses backpressure, meaning the servers running filebeat run out of HDD space. BTW, with a bit of grep in front, the actual per-service working data sets are small enough so that I can load them into R on a 256GB RAM server and then produce nice diagrams. You can also script R from the command line and then send the diagrams around as a cronjob.
  - pepemon 7y ago
    Have you used the file input? With such throughput you don't need it at all, it hangs both of its host because of iowait and remote endpoint too if beats proto is used. All of my log traffic is sent as GELF UDP or syslog UDP.
- eecc 7y ago
  Sure it’s nice and maybe it’s worth 50k/month if you’re running Uber... but grep and sed together can get you quite far. Admittedly for the histogram you’d have to cut and paste into a spreadsheet but yeah, hardly something you need in real time (probably more for a postmortem presentation to management.)
koevet 7y ago
Out of curiosity, is the 200$/month price tag for colocation?
amelius 7y ago
But those "hipster" tools can scale. Perhaps you simply haven't hit the right scale yet?
- ddalex 7y ago
  You don't need to scale. You're not Google, Amazon or FB to have a billion daily active users. And if you manage to make it to that scale, you can certainly hire engineers to refactor those systems. Fix the problem only when you encounter it. Parsing 1Tb of logs daily with a 14 day retention? A desktop with a NAS attached can do that. The corps that developed these fancy tools and then open sourced them and labeled them hipster ish names? Of course they want engineers out there to use them, it's called vendor lockin.
  - amelius 7y ago
    Yes, you are right. But there are a lot more usecases, e.g. imagine you run a website for a (small or large) government, or a website for a popular television show. Or perhaps you are a developer in a telecom company in some country. Scale is everywhere. > you can certainly hire engineers to refactor those systems. Wouldn't it be better if those engineers used standard tools like Hadoop?
    - smoe 7y ago
      I worked for the largest online news site of Switzerland. We were nowhere near the scale where such tools would provide significant benefit over the "boring", "outdated" technology we we using. We also had a fraction of the operational cost than our much much smaller competitor within the same publisher that went full hipster on their tech. Thing is, it for a lot of use cases it is a whole lot harder to get to large scales from a business and product perspective than it is for good engineers to adapt to it.
deleted 7y ago
[deleted]
prepend 7y ago
Do you mean “bare metal” as just basic aws images using only rsyslog and grep? You call out physical disks so it makes me think like physical machines. How are two physical machines $400/month? Is that assuming you already have the hardware laying around? How much would it cost for basic aws ec2 instances running rsyslog and grep?
iso1631 7y ago
I've been fighting elasticsearch to parse some netflow traffic. Found a great tool for logging the traffic and it even set up an elasticsearch instance and kibana for me, pretty much plug-and-play. It logs lots of data, but I'm only interested in subnet level at this stage. I'm not interested in direction either. A (multiple) text files with say a csv format (other fields are fine, cut -f will strip those out) datetime,collector,srcip,dstip,bytes I could easilly pipe into sed to strip the 4th octet, but in reality I'd probably parse with perl. Would take about 10 minutes and output a nice simple spreadsheet showing me which subnets are busiest, which times are busy etc, and apply accounting on a subnet basis (or with a bit more perl trivally assign different IPs to different accounting buckets) I can only assume elk is a completely different way of thinking. Collegues think that grep is "hard", but click-click-click in kibana is easy. However, I know I'm a grumpy old fart. I find lots of new ways to reinvent the wheel tiring and pointless, but it feels like shouting into the wind. Recently things that have made me simply sign include ubuntu switching from /etc/network/interfaces to netplan, or from debian-installer to subiquity. Moving from initd to systemd. I'm sure there are good reasons for changing all of these, but for my use cases it just increases the workload.
- 4d617832 7y ago
  Can you tell me which tool you consider for getting netflows? I am trying building something that is really close to what you describe but fail to find the time to do so.
  - iso1631 7y ago
    Using https://gitlab.com/thart/flowanalyzer https://gitlab.com/thart/flowanalyzer at the moment However thinking it would be far easier to write something from scratch
neurostimulant 7y ago
> those $200 each bare metal servers are 2x Intel Xeon 6-core + 256GB RAM + 15x 10TB 7200 rpm. That's really cheap for such a big server. I think the same specs on Hetzner and OVH cost at least twice of that. Where do you rent those servers?
- pepemon 7y ago
  I think this guy have no idea what he's talking about.
- duncanawoods 7y ago
  Sounds similar to a Hetzner SX292 - Xeon 6-core, 256 RAM, 15x10TB. Only one CPU though at $285.
afpx 7y ago
You should really be packaging up what you built and selling it as a competing service.
- pepemon 7y ago
  The irony is that there is no service. It's a duck-and-tape solution reminding me my first years in Linux ops industry.
mamon 7y ago
We reached the point where NOT being hosted on AWS is a competitive advantage :)
wwright 7y ago
How does the functionality compare, though? With ELK, you have a variety of options for monitoring (I have personal experience with Kibana, Grafana, and the API) and visualizing. You explicitly say “rsyslog and grep”; do you build dashboard systems on top of grep? What does the performance look like? How many dashboards does that scale to? What’s your strategy for availability?
dajohnson89 7y ago
how much dev time would both approaches take?
pepemon 7y ago
Where do you get such low prices for dedicated rigs? Well anyway, looks like you haven't tried to roll your own ES cluster on bare metal hosts - I've got more throughput than yours with 2 Graylog hosts backed by 5x bare metal ES instances. With all benefits of web interface for less tech savvy people, structured logs, analytics and data exposure through unified API. All of these for less than 2k/mo. If I lived in wonderland with server prices like yours, our check would be less than 1k/mo.
- choeger 7y ago
  Not quite that price, but with 308EUR/month not too far off: https://www.hetzner.com/dedicated-rootserver/sx292 https://www.hetzner.com/dedicated-rootserver/sx292 I guess if you buy and host the machines yourself, you should be able to beat hetzner's price.
- leonardteo 7y ago
  I personally use OVH.com and Hetzner (for EU based installations). Don't expect support, but you get good machines, unlimited traffic (OVH). If you can set up your infrastructure so that it's JBOS (Just a Bunch of Servers) where you can hot swap them if there's any hardware failure, etc. it's great price/performance.
oxfordmale 7y ago
Bespoke solutions are always going to outperform generic ones. You need to choose the right tools for each job. ElasticSearch definitely has it advantages if you have to write complex queries, however, might be less suitable for grep like queries.
nasalgoat 7y ago
ELK could handle that workload pretty well and isn't too hard to set up. Plus you get a nice web interface. But I still find myself logging in and grepping for things.
api 7y ago
The cost savings with bare metal servers vs managed cloud can be ludicrous... like take away 2-3 zeroes. It's particularly true for bandwidth and raw CPU power where major cloud providers massively overcharge. The real answer is like most other things: it depends. If you have a highly variable workload or you really get a lot of value from specialized managed services then cloud will save you and generally work better. If you have a less variable workload then a more DIY approach can yield truly massive cost savings and even better uptime and performance in some cases.

certera 7y ago

Why not run ES on the bare metal servers?

fxtentacle 7y ago
Tried and it was horribly slow. We add about 5TB of logs per day, and the ES index wasn't able to keep up, despite having 256 GB RAM and 2x Intel Xeon 6-core
- camkego 7y ago
  That seems like a really good price for bare metal of that magnitude, where are you getting those from?
  - fxtentacle 7y ago
    https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx On their German website, you can bid for used-but-still-good servers and there you'll find great offers. There, you can also find hardware that they customized for others and then didn't need anymore. For example, I rented some GPU rendering servers for $110 monthly each from there. https://www.hetzner.de/sb https://www.hetzner.de/sb Plus I can't praise their service and ops team enough. After we rented the new storage servers, I sent them an email to ask for it, and they then put the new storage servers on the same router inside the same datacenter as our other servers with them. Now we effectively have 2x redundant 1GBit LAN between our servers with them, plus an 10GBit internet uplink shared between our servers. For that level of performance, they are ridiculously cheap. BUT - and that is a big gotcha - you will be responsible for diagnosing hardware issues and asking for a replacement yourself. That's why we have RAID and two storage systems with them, because we tend to see one HDD fail every 3-6 months.

Johnny555 7y ago

What kind of searches are you doing where the search performance of grep is the same as with Elasticsearch?

fxtentacle 7y ago
One common query was to check the FTP server on our side for access from a customer server, so that we can help with troubleshooting as to why they couldn't connect. Turns out, in many cases less-technical customers mistyped the hostname or left out the .com at the end. tail -n 10000 /var/log/ftp.log | grep -i $CUSTOMER_IP | tail -n 5
- Johnny555 7y ago
  If that's your typical use-case, I can see why that's as fast (or faster) as Elasticsearch, but it's not clear why you'd have gone with Elasticsearch in the first place. When I last used Elasticsearch, we indexed ~10TB of log data a day, kept 14+ days, and a typical query was looking for log records that matched a unique session ID over the past 10 days or so, not an easy task for grep. But we didn't pay $50K/month for that cluster, it was closer to $12K/month. Before we used ES, someone had written a parallel grep that would grep multiple files at once, and would run multiple greps at once through chunks of the file, but still it could take 30 minutes or longer to churn through the logs on a 32 core machine - ES took that query time down to 100 milliseconds. The ES cluster easily paid for itself in employee time savings.
  - londons_explore 7y ago
    If you always want to search by one thing, you can manually index by that thing. In your case, arrange your log files by the first 6 hex characters of the user ID (/var/log/xxx/xxx/date.log), and grep will typically only then have a few megabytes to scan. If you need real indexes, or just want something industry standard and maintainable rather than 'some guys grep script', then elastic search is probably the way to go.
    - 7y ago

idnefju 7y ago

Horizontal scaling is also where ES shines. If you don't need that among other things, then there are alternatives yea

threeseed 7y ago

ElasticSearch also does a lot more than your CLI does. Splunk style dashboards, multi-user access, full text searching e.g. stemming, ability to support non-structured formats.

fxtentacle 7y ago
I fully agree, but our business need was to stream log files (e.g. tail -f) and to search for snippets in them and to produce statistic counters (e.g. |grep|wc) While I can imagine situations where ElasticSearch is the best solution, I have seen it mostly used in situations that would be better served with simple command-line tools, similar to what the author demonstrated with awk vs. Hadoop.
enriquto 7y ago
> ElasticSearch also does a lot more than your CLI does. This is not necessarily a good thing. In fact, it is probably a really bad thing if you do not need the additional stuff.

DrPhish 7y ago

As a possible middle-ground: I'm using Graylog with a lot of success, but my log volumes are rather modest (medium sized corporate network switches/routers/firewalls/servers log aggregation and processing)

fxtentacle 7y ago
We have 5TB per day, which may or may not break Graylog. Before evaluating ES, we used Logstash, but that couldn't keep up at around 2TB+ per day.
- virgilp 7y ago
  > Before evaluating ES, we used Logstash Huh? I thought one would use Logstash to ingest & massage data going into ES. I view it as a sort of "quick&dirty stream processing engine". How would one go about using Logstash _instead of_ ES, as opposed to "in addition to ES" ?
  - fxtentacle 7y ago
    Sorry, I worded that badly. We used to use a locally-hosted ELK stack, so yes, including ES as the data backend for Kibana, but there Logstash or more precisely filebeat and logstash-forwarder became the bottleneck. That was a horrible experience because then filebeat would keep the old (unsent, but deleted) log files open, so production servers crashed running out of HDD space even though "du -h -s /" was a lot smaller than the HDD size. It took me way too long to figure out that deleted but still open files were the problem. So our decision to move from using ES invisibly as part of Logstash towards a full dedicated ES cluster was forced by the Logstash pipeline getting too slow. I then evaluated Elastic Cloud and using Amazon EC2 for deploying ES, both of which were ultimately rejected. The first due to fears of data loss, the second due to costs.
    - virgilp 7y ago
      I'm confused again - wouldn't Elastic Cloud cost more than EC2? It makes no sense to be the other way around. And secondly - why use logstash if you had filebeat - why not send data to ES directly? (I mean, specifically for your usecase, where you don't seem to need to do much processing prior to ingestion). Yeah, I saw that big parts of Logstash are in ruby so I expect underwhelming performance from it - but Filebeat should in theory be able to ingest large volumes of data into ES directly.

bobnamob 7y ago

Might be worth taking a look at ripgrep[1]. [1] https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep

fxtentacle 7y ago
That tool looks great :) but since we're already seeing <1s search times and the tool is only used by internal support employees, I'm mostly going with "never touch a running system" these days. While for a database like ES you'd put all of the data into one big pile and then filter by keywords, e.g. host=ftp service=ftp query=IP, for logfiles you usually search on a much smaller set. They are rotated by day and logs are broken down by host and service by rsyslog, so instead of filtering the full 150TB - which is what ES has to do - my grep only needs to look at the 1-2 GB of data inside the file where host, service, and date match.
- pepemon 7y ago
  Do you understand something about such thing as ES indices?
reagent_finder 7y ago
RG is awesome for the basic use case. Using it across platforms just makes you tear your hair out, though.
- foota 7y ago
  I'm curious, why? Looks pretty easy to get the binaries.
- burntsushi 7y ago
  Why? Can you share your frustrations?

wdroz 7y ago

if you are using systemd, you don't even need to grep "manually" that much. journalctl -u nginx -u ssh --since today

fxtentacle 7y ago
I had never heard of that before. Thank you for pointing this out :)
- pepemon 7y ago
  And yet you're rsyslogging and grepping what is supposed to be structured and typed. Have you heard about backpressure, controlled retention, compliant log redundancy and all of this advanced stuff that you'll never get with aggregation via rsyslogd/syslog-ng?
  - Spivak 7y ago
    rsyslog is far more powerful than you're making it out. You have to actually tell it what to do but it's more expressive than filebeat and logstash. * rsyslog in the use-case he's describing is just a method of pushing some subset of the logs generated on a client system to a directory on the collector which has trade-offs but the benefit is having really simple failure modes. * both rsyslog and journald store structured data: rsyslog with lumberjack, and journald just always. And rsyslog can parse and structure the logs in-flight so you save computing power on the collector. * rsyslog behaves exactly like filebeat when it comes to reliable delivery and can persist unsent messages to memory then disk. rsyslog's rate limiting, backoff, and retry options are super powerful.
    - pepemon 7y ago
      Yes, you are right, sorry. I was too fast in my assumptions, rsyslog (don't know much of syslog-ng) has feature parity with ELK in terms of log delivery and processing. But I think that grep and its permutations aren't right tools of choice for log analysis anyway.
  - 7y ago

bses 7y ago

> We recently discussed new logging tools at work. It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. The log ingestion and search performance was roughly the same... If grep has the same search performance as elasticsearch, you should not be using elasticsearch and any comparison is bullshit. > It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. $50K monthly is $69.44 an hour, or 8 instances of the most expensive thing from https://aws.amazon.com/elasticsearch-service/pricing/ https://aws.amazon.com/elasticsearch-service/pricing/ (with 488GB RAM) Please cite references, your numbers seem made up. Where can you get 256GB of RAM for $200/month?

jdc 7y ago
Really? Try Googling '256gb ram dedicated'
- bses 7y ago
  You're right, I replied too quickly. > We recently discussed new logging tools at work. It was either a redundant Amazon EC2 cluster with ElasticSearch for $50K monthly, or two large bare metal servers with rsyslog and grep for $400 monthly. The log ingestion and search performance was roughly the same... If grep has the same search performance as elasticsearch, you should not be using elasticsearch and any comparison is bullshit.
  - jdc 7y ago
    Right, though some people really do reach for heavy-weight tools on datasets that are way too small to justify their use. But yeah, even ripgrep should get out-performed by any db/search daemon with a decent index.
fxtentacle 7y ago
https://www.hetzner.com/dedicated-rootserver/matrix-sx https://www.hetzner.com/dedicated-rootserver/matrix-sx They get slightly cheaper for large customers like us, plus you can get hardware upgrades like the 2x CPUs by paying for the difference upfront. So yes, it's $400 monthly + $2000 once, but those $2k don't really make much of a difference over the years that we've been running the system ^^

didibus 7y ago

Where are you getting such a bare metal machine for 200$ a month?

avip 7y ago

Your setup is redundant and expensive. logrotate directly to s3 and use Athena.

jammygit 7y ago

What cloud advocates always say is that the $50k monthly will save you money from not needing to hire a team to manage it for you, and that over the course of 10+ years you will be ahead. Is that true in anyone's experience? Every once in a while somebody posts about their competing bare-metal system and it looks like a lot of people have managed to cut their server costs by 99% (based on the numbers they post) by avoiding the cloud as a service Honestly curious

gscott 7y ago
I have a friend with a rack of co-located servers I manage. I drive from San Diego to LA where they are located maybe once every 6 months or more. I don't believe I have been to the rack for about a year. The rack with 20mbs of bandwidth costs around $2,500 a month.
- lrem 7y ago
  Did you really mean twenty megabits?
  - xchaotic 7y ago
    if it’s unmetered 20mbps I’d happily take that over ingress / egress cost at cloud provider such as AWS
    - davedx 7y ago
      Yeah AWS bandwidth egress is extortionate. Digital Ocean is something like 5-7x less expensive. It's ridiculous.
    - lrem 7y ago
      That's kind of shocking when compared to European prices. Over a full order of magnitude of difference, comparing to the list price of the first Google result.
  - ChrisCinelli 7y ago

carlsborg 7y ago

You are underestimating the value of indexed logs and a Kibana view into them. For example, if you wanted a histogram of error logs for the last 12 days, and then zoom in and see the actual logs.

travbrack 7y ago
You can do this with graylog
fxtentacle 7y ago
You are underestimating how much costs the issues caused by Kibana and a slow ES can produce. I used to have these nice Kibana diagrams set up with auth and a https proxy so that our team could easily check if the server error rates looked normal. I quickly found out two things: 1. Most of our employees never ever looked at it, dismissing it as "too technical" just from seeing a screenshot. 2. The ELK stack uses backpressure, meaning the servers running filebeat run out of HDD space. BTW, with a bit of grep in front, the actual per-service working data sets are small enough so that I can load them into R on a 256GB RAM server and then produce nice diagrams. You can also script R from the command line and then send the diagrams around as a cronjob.
- pepemon 7y ago
  Have you used the file input? With such throughput you don't need it at all, it hangs both of its host because of iowait and remote endpoint too if beats proto is used. All of my log traffic is sent as GELF UDP or syslog UDP.
eecc 7y ago
Sure it’s nice and maybe it’s worth 50k/month if you’re running Uber... but grep and sed together can get you quite far. Admittedly for the histogram you’d have to cut and paste into a spreadsheet but yeah, hardly something you need in real time (probably more for a postmortem presentation to management.)

koevet 7y ago

Out of curiosity, is the 200$/month price tag for colocation?

amelius 7y ago

But those "hipster" tools can scale. Perhaps you simply haven't hit the right scale yet?

ddalex 7y ago
You don't need to scale. You're not Google, Amazon or FB to have a billion daily active users. And if you manage to make it to that scale, you can certainly hire engineers to refactor those systems. Fix the problem only when you encounter it. Parsing 1Tb of logs daily with a 14 day retention? A desktop with a NAS attached can do that. The corps that developed these fancy tools and then open sourced them and labeled them hipster ish names? Of course they want engineers out there to use them, it's called vendor lockin.
- amelius 7y ago
  Yes, you are right. But there are a lot more usecases, e.g. imagine you run a website for a (small or large) government, or a website for a popular television show. Or perhaps you are a developer in a telecom company in some country. Scale is everywhere. > you can certainly hire engineers to refactor those systems. Wouldn't it be better if those engineers used standard tools like Hadoop?
  - smoe 7y ago
    I worked for the largest online news site of Switzerland. We were nowhere near the scale where such tools would provide significant benefit over the "boring", "outdated" technology we we using. We also had a fraction of the operational cost than our much much smaller competitor within the same publisher that went full hipster on their tech. Thing is, it for a lot of use cases it is a whole lot harder to get to large scales from a business and product perspective than it is for good engineers to adapt to it.

deleted 7y ago

[deleted]

prepend 7y ago

Do you mean “bare metal” as just basic aws images using only rsyslog and grep? You call out physical disks so it makes me think like physical machines. How are two physical machines $400/month? Is that assuming you already have the hardware laying around? How much would it cost for basic aws ec2 instances running rsyslog and grep?

iso1631 7y ago

I've been fighting elasticsearch to parse some netflow traffic. Found a great tool for logging the traffic and it even set up an elasticsearch instance and kibana for me, pretty much plug-and-play. It logs lots of data, but I'm only interested in subnet level at this stage. I'm not interested in direction either. A (multiple) text files with say a csv format (other fields are fine, cut -f will strip those out) datetime,collector,srcip,dstip,bytes I could easilly pipe into sed to strip the 4th octet, but in reality I'd probably parse with perl. Would take about 10 minutes and output a nice simple spreadsheet showing me which subnets are busiest, which times are busy etc, and apply accounting on a subnet basis (or with a bit more perl trivally assign different IPs to different accounting buckets) I can only assume elk is a completely different way of thinking. Collegues think that grep is "hard", but click-click-click in kibana is easy. However, I know I'm a grumpy old fart. I find lots of new ways to reinvent the wheel tiring and pointless, but it feels like shouting into the wind. Recently things that have made me simply sign include ubuntu switching from /etc/network/interfaces to netplan, or from debian-installer to subiquity. Moving from initd to systemd. I'm sure there are good reasons for changing all of these, but for my use cases it just increases the workload.

4d617832 7y ago
Can you tell me which tool you consider for getting netflows? I am trying building something that is really close to what you describe but fail to find the time to do so.
- iso1631 7y ago
  Using https://gitlab.com/thart/flowanalyzer https://gitlab.com/thart/flowanalyzer at the moment However thinking it would be far easier to write something from scratch

neurostimulant 7y ago

> those $200 each bare metal servers are 2x Intel Xeon 6-core + 256GB RAM + 15x 10TB 7200 rpm. That's really cheap for such a big server. I think the same specs on Hetzner and OVH cost at least twice of that. Where do you rent those servers?

pepemon 7y ago
I think this guy have no idea what he's talking about.
duncanawoods 7y ago
Sounds similar to a Hetzner SX292 - Xeon 6-core, 256 RAM, 15x10TB. Only one CPU though at $285.

afpx 7y ago

You should really be packaging up what you built and selling it as a competing service.

pepemon 7y ago
The irony is that there is no service. It's a duck-and-tape solution reminding me my first years in Linux ops industry.

mamon 7y ago

We reached the point where NOT being hosted on AWS is a competitive advantage :)

wwright 7y ago

How does the functionality compare, though? With ELK, you have a variety of options for monitoring (I have personal experience with Kibana, Grafana, and the API) and visualizing. You explicitly say “rsyslog and grep”; do you build dashboard systems on top of grep? What does the performance look like? How many dashboards does that scale to? What’s your strategy for availability?

dajohnson89 7y ago

how much dev time would both approaches take?

pepemon 7y ago

Where do you get such low prices for dedicated rigs? Well anyway, looks like you haven't tried to roll your own ES cluster on bare metal hosts - I've got more throughput than yours with 2 Graylog hosts backed by 5x bare metal ES instances. With all benefits of web interface for less tech savvy people, structured logs, analytics and data exposure through unified API. All of these for less than 2k/mo. If I lived in wonderland with server prices like yours, our check would be less than 1k/mo.

choeger 7y ago
Not quite that price, but with 308EUR/month not too far off: https://www.hetzner.com/dedicated-rootserver/sx292 https://www.hetzner.com/dedicated-rootserver/sx292 I guess if you buy and host the machines yourself, you should be able to beat hetzner's price.
leonardteo 7y ago
I personally use OVH.com and Hetzner (for EU based installations). Don't expect support, but you get good machines, unlimited traffic (OVH). If you can set up your infrastructure so that it's JBOS (Just a Bunch of Servers) where you can hot swap them if there's any hardware failure, etc. it's great price/performance.

oxfordmale 7y ago

Bespoke solutions are always going to outperform generic ones. You need to choose the right tools for each job. ElasticSearch definitely has it advantages if you have to write complex queries, however, might be less suitable for grep like queries.

nasalgoat 7y ago

ELK could handle that workload pretty well and isn't too hard to set up. Plus you get a nice web interface. But I still find myself logging in and grepping for things.

api 7y ago

The cost savings with bare metal servers vs managed cloud can be ludicrous... like take away 2-3 zeroes. It's particularly true for bandwidth and raw CPU power where major cloud providers massively overcharge. The real answer is like most other things: it depends. If you have a highly variable workload or you really get a lot of value from specialized managed services then cloud will save you and generally work better. If you have a less variable workload then a more DIY approach can yield truly massive cost savings and even better uptime and performance in some cases.