9 ms·

This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding

by piaste 5y ago

This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or powerful as Kibana, it's as close to zero-configuration as you can hope. It also has a bunch more features that show it's a monitoring system first and foremost. Sure, it's a SAAS that we can't self-host, but diagnostic logs aren't business-critical so if we got locked out or priced out tomorrow, we would suffer no real disruption while looking for a replacement.

vimda 5y ago
At an old job we used NewRelic right up until they tried to 3x our annual bill :/
- devin 5y ago
  You are not alone.
deleted 5y ago
[deleted]
OldOneEye 5y ago
At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it would be enough. Not to ingest, that it did fine, but to query them under a reasonable time frame without a timeout, that's where it couldn't handle the high volume. Also, their support service, while trying to win us over, that is, in their best moment, was nothing really stellar. Favouring sending us sales/presales people to solve technical problems. It didn't leave us a good aftertaste.
- pfarrell 5y ago
  Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.
  - baq 5y ago
    there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.
    - sethammons 5y ago
      I love splunk. Our clusters process 10s of billions of structured log events daily. We have search, reports, PagerDuty integration, dashboards, etc. It is crazy expensive but is the best system I've used in this space. We are having to save costs with so much data, so we are lowering retention time and moving the data to snowflake for data older than a week. More and more, we are leveraging Looker for reporting out of Snowflake and relying more on Prometheus monitoring for alerting. But Splunk would still be my ideal service if we had less total data.
      - fn1 5y ago
        I second that. I love splunk as well. Costs can also be reduced by spending some development-effort into abbreviating logs and being smart about deciding what to log and where.
      - VectorLock 5y ago
        > Our clusters process 10s of billions of structured log events daily. Whats that run you?
    - mrweasel 5y ago
      Primary issue with using Splunk is that pretty much all other solution will seem inferiour. Great product, terrible business partner.
Sevii 5y ago
How are diagnostic logs not business critical? If you have an outage while your non-mission critical logs are offline what are you going to do?
- piaste 5y ago
  If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments) NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.
  - sateesh 5y ago
    SSH to machine and grepping through is much harder when the number of machine is > ~5 (say). SSH to host doesn't scale as you keep adding the hosts.
    - piaste 5y ago
      We use Docker Swarm so we can SSH into a manager and run `docker service logs` regardless of which hosts are actually running the services. I assume other orchestration systems have equivalent features.
    - VectorLock 5y ago
      When someone says "we ssh and grep" you know pretty much how small their operation is.

by piaste 5y ago

vimda 5y ago

At an old job we used NewRelic right up until they tried to 3x our annual bill :/

devin 5y ago
You are not alone.

deleted 5y ago

[deleted]

OldOneEye 5y ago

At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it would be enough. Not to ingest, that it did fine, but to query them under a reasonable time frame without a timeout, that's where it couldn't handle the high volume. Also, their support service, while trying to win us over, that is, in their best moment, was nothing really stellar. Favouring sending us sales/presales people to solve technical problems. It didn't leave us a good aftertaste.

pfarrell 5y ago
Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.
- baq 5y ago
  there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.
  - sethammons 5y ago
    I love splunk. Our clusters process 10s of billions of structured log events daily. We have search, reports, PagerDuty integration, dashboards, etc. It is crazy expensive but is the best system I've used in this space. We are having to save costs with so much data, so we are lowering retention time and moving the data to snowflake for data older than a week. More and more, we are leveraging Looker for reporting out of Snowflake and relying more on Prometheus monitoring for alerting. But Splunk would still be my ideal service if we had less total data.
    - fn1 5y ago
      I second that. I love splunk as well. Costs can also be reduced by spending some development-effort into abbreviating logs and being smart about deciding what to log and where.
    - VectorLock 5y ago
      > Our clusters process 10s of billions of structured log events daily. Whats that run you?
  - mrweasel 5y ago
    Primary issue with using Splunk is that pretty much all other solution will seem inferiour. Great product, terrible business partner.

Sevii 5y ago

How are diagnostic logs not business critical? If you have an outage while your non-mission critical logs are offline what are you going to do?

piaste 5y ago
If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments) NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.
- sateesh 5y ago
  SSH to machine and grepping through is much harder when the number of machine is > ~5 (say). SSH to host doesn't scale as you keep adding the hosts.
  - piaste 5y ago
    We use Docker Swarm so we can SSH into a manager and run `docker service logs` regardless of which hosts are actually running the services. I assume other orchestration systems have equivalent features.
  - VectorLock 5y ago
    When someone says "we ssh and grep" you know pretty much how small their operation is.