14 ms·

Monitoring Is a Pain

• XorNot 3y ago
  The rule I follow personally for log levels is: DEBUG - anything which would otherwise be a 1 line code comment INFO - all business process relevant actions. Anytime something is committed to a database is a fairly good place to start. WARN - abnormal outcomes which collectively might indicate a problem. i.e. a warning log should be printed when something succeeds but takes much longer then it should, or if I'm going to fail an action which will be retried later. Anything which logs at this level should be suppressible by reconfiguring the application to define the behavior as normal. ERROR - Anything which should work, didn't, and isn't recoverable. Actions requested by the user which don't succeed should log at this level, since we're failing to meet a user request.
  • fatnoah 3y ago
    That's pretty close to my definition as well, though I add FATAL as well, but mostly for extreme cases of "there's no way the system can function" type of errors, such as missing/invalid configuration or similar. My philosophy is that INFO is the default runtime configuration. At smallish scales (< 1B requests/month) I've seen this approach work very well, especially when the Eng team is on board with high-quality log messaging. It was an investment to get to that state at my last startup, but the massive improvement to QoL for Eng and Ops teams was totally worth it.
  • kerblang 3y ago
    Not sure the descriptions are ideal here, but the 4 levels selected are completely adequate for most purposes in business information systems. If someone really wanted, you could cut it down to INFO & ERROR - I secretly think of DEBUG & WARN as tolerable fluff that keeps fussy people satisfied that things are complicated enough without making a mess. Per the article's concerns, no, I would not try to use this for auditing, metrics, or anything like that - just as a tool for devs to find out "OK what the heck is going on here?" I just collect all my logs on a big, fast file server where I can grep the crap out of them, with way more success than I've ever had with Fancy Stack Of Whiz Bang Thingies W/ Web Interface, although I'm always willing to watch someone attempt another Fancy Stack Fail... I recommend pressuring devs to make log messages coherent with good spelling and grammar - like anything else, it's "Write to be read," i.e. treat your reader with some respect, it's not that hard. Logs often end up thrown in the trash by sysadmins because they look like trash. It's especially hard to convince them that the logs are good enough that they can diagnose problems themselves instead of covering their eyes and asking me to look at the logs for them, and I don't entirely blame them.
  • twic 3y ago
    Our logging is tied to our monitoring system. So for us, it is: DEBUG - throw it away INFO - put it in a log file WARNING - put it on a dashboard ERROR - put it in PagerDuty This does make it very easy to decide on the level. Do i want to be woken up at five in the morning for this? No? Then it's definitely not ERROR!
    • marcosdumay 3y ago
      Well, then I'd say ERROR is a very bad name. Because it's a well defined word with a precise meaning. If you tell somebody "an error happened here" they will have no trouble understanding it, and what they understand won't be "we need to wake up somebody to deal with this NOW!". Those names are proof that your logging structure was created for debugging, not for production.
      • twic 3y ago
        I'm afraid this is an entirely fantastical objection. I don't believe we have ever had the confusion you describe, and this structure was created entirely for production use.
      • Supermancho 3y ago
        > Well, then I'd say ERROR is a very bad name. The name is irrelevant. The issue being addressed is a poor assertion. Log levels are not meaningless in a standardized system. It's trivial to map a set of levels to a useful set of conditions. Like any monitoring (or logging), the system has to be well-regulated and maintained to be useful. In good faith, log levels are useful.
• 0xbadcafebee 3y ago
  This isn't an article, this is just whinging that things aren't perfect. "Log Levels Don't Mean Anything" If you're not handling log levels correctly, you've just thrown away half the utility of logging. The author doesn't even make an argument here, he just lists a bunch of different log levels and moves on. You should absolutely rely on logging levels in your applications. "No consensus on what logs are even for" They're for an application to tell you what is going on. Come on dude. "Maybe I'm wrong about monitoring." You're not wrong (well, about being a pain), you just should get a different job. Monitoring is hard, yes. But it's very beneficial, despite being highly imperfect. It just doesn't benefit you to build your own monitoring stack. Pay someone else to do it, accept that it's not going to be the Platonic ideal of computer science theory, and move on with life.
  • KaiserPro 3y ago
    > They're for an application to tell you what is going on. Come on dude. I mean yeah, but the format is unique to every app. trying to rationalise them to reduce them to a tertiary value (ok, degraded, fucked) is as unique to each app. Plus, with every update, it'll change, means that its loads of work
    • wspeirs 3y ago
      You just need a tool that's adaptable and will let you parse-as-you-search. Without a schema, your logs can change whenever, and you can still easily derive value from them.
      • KaiserPro 3y ago
        but thats debugging, not monitoring. If you don't know that your service is down, then you don't know where and what to look for. for monitoring, you need to be looking for a specific parameter or string. if that changes, its very difficult to generate an automated alert.
        • wspeirs 3y ago
• ckozlowski 3y ago
  This was a good read. But the question that I kept waiting to be asked was "What do I need to monitor?" In my opinion, the way to avoid many of the problems and complexities that author lists is to start with the goals. What are the business/mission goals? What is the workload (not the monitoring tool) trying to accomplish? What does the thing do? Once you have that, you can go about selecting the right tool for the job. The author notes that there's no consensus on what logs are for. Or is buried under signal-to-noise ratio for metrics. But when you start with a goal, and then determine a KPI you want to obtain to measure that, you'll then be able to ascertain where a log, metric, trace, or combination thereof can reflect that. And you only need create the ones needed to measure that. The scope of the problem becomes far more manageable. It make take a couple tries to get right, and a user may find that they need to add more over time. But the monitoring doesn't need to change often unless the deliverables or architecture of the workload does. And creating those should be part of the development process. By starting with goals and working from there, monitoring becomes a manageable task. The myriad of other logs and metrics created can be safely stored and kept quiet until they're needed to troubleshoot or review an incident. That falls into observability. Observability is something you have. Monitoring is something you do. I haven't addressed the difficulties the author states in creating various traceability mechanisms, how to parse all of these logs from various platforms, or some of the other technical challenges he states. There are usually solutions for these (usually...), some easier than others. But by narrowing the scope down to what's needed to achieve your given insights, the problem set becomes only what's required to achieve that aim, and not trying to build for every conceivable scenario.
  • ChoHag 3y ago
    [dead]
  • poulsbohemian 3y ago
    >"What do I need to monitor?" I did this stuff professionally for about a decade... the short version is, consider the nodes of the graph. Stuff breaks because the front end can't reach the back, the back can't take a message off the queue, the database stopped taking connections, etc. IE: most apps of any size or complexity have reliance on other systems or infrastructure components, and these becomes things that break, timeout, don't scale, etc. That's a good starting point.
• KaiserPro 3y ago
  The Way(tm) that I was taught/experienced is as follows: o Logs are there for ignoring. You need them precisely twice: once when you are developing, and once when the thing's gone to shit, but they are never verbose enough when you need them. o Using logs to derive metrics is an expensive fools errand pushed by splunk and the cloud equivalents(ie cloudwatch and the like). Its slow, inaccurate and horrendously expensive. o using logs for monitoring is a fools errand. Its always too slow, and really really fucking brittle. o metrics are king. o pull model metrics is an antipattern o Graphite + grafana is still actually quite good, although time resolution isn't there. o You need to raid your metrics stores o We had a bunch of metrics servers in a raid 1, which were then in a raid 0 for performance, all behind loadbalancers and DNS Cnames with a really low TTL. o Cloudwatch metrics are utterly shite o Cloudwatch is actually entirely shit. o tracing is great, and brilliant for performance monitoring. o Xray from AWS is good, but only really for lambdas. o tracing is fragile and doesn't really plug and play end to end, unless you have the engineering discipline to enforce "the one true" tracing system everywhere but what do you monitor? http://widgetsandshit.com/teddziuba/2011/03/monitoring-theory.html http://widgetsandshit.com/teddziuba/2011/03/monitoring-theor... this still is canonical. In short, everything should have a minimum set of graphs, CPU, Memory, connections, upstream service response times hits per second and query time, at a minimum. You can then aggregate those metrics into a "service health" gauge, where you set a minimum level of service (ie no response time greater than 600ms, and no 5xx/4xx errors or similar) red == the service isn't performing within spec, yellow == its close to being outside spec, green == its inside spec. if you are running a monolith, then each subsection needs to have a "gauge". for microservice people, every microservice. You can aggregate all those gauges into "business services" to make a dashboard that even CEOs can understand.
  • brad0 3y ago
    Could you elaborate on Cloudwatch metrics?
    • KaiserPro 3y ago
      A few things that I dislike: The temporal resolution is unpredictable (as in the reduction of resolution with age is separate from the view resolution.) Its slow as shit as in it (used) to take a good while to update with this time bucket's value, and that value seems to be able to change wildly (more than just an average) The min/max/average/count/$other parameters are poorly documented and inconsistently applied across services. therefore what chance do you have for implementing them properly. It's expensive(well can be) deletion is different from hiding. its less of a problem with grafana/custom renderers, but its nasty. the lack of functions for slicing/dicing and combining annoy me. (again that might have changed in the last year or so)
• compumike 3y ago
  > I'm running more complicated stacks to capture massive amounts of data in order to use it less and less. > […] > My experience is companies do not anticipate that the cost of monitoring an application can easily exceed the cost of hosting the application even for simple applications. These problems resonate with me for sure, and after tackling them at other companies, they were part of our team’s inspiration for building Heii On-Call [1] as a lightweight, minimal monitoring platform for apps / APIs / Cron Jobs / etc. Instead of “ingest all the metrics/logs” we have guides showing, for example, how to set up the most minimal possible Prometheus / Alertmanager, with the smallest footprint of CPU/memory/disk requirements. [2] And for many of our users they don’t even use any of our Prometheus or Datadog integrations, and just stick with our dead simple out-of-the-box HTTP monitoring. [3] Note that these address the “now” problem of monitoring: is it working now, or not? While most of the article talks about the “past” problem of monitoring (long term storage of logs, metrics; pretty charts). The “past” part is the part that the author is pointing to as being rarely useful and the source of the cost and complexity. I think there’s probably still value in dumping whatever logs and metrics you want into S3 just in case you need to go digging one day, but for the most part, this is pretty low value. The relatively higher value and lower implementation complexity is in the “now”. [1] https://heiioncall.com/ https://heiioncall.com/ [2] https://heiioncall.com/guides/minimalistic-monitoring-and-alerting-for-your-kubernetes-cluster-with-prometheus-and-alertmanager https://heiioncall.com/guides/minimalistic-monitoring-and-al... [3] https://heiioncall.com/docs https://heiioncall.com/docs
  • wspeirs 3y ago
    >Instead of “ingest all the metrics/logs” we have guides showing, for example, how to set up the most minimal possible Prometheus / Alertmanager, with the smallest footprint of CPU/memory/disk requirements. [2] I'll bite... why _not_ "ingest all the metrics/logs"? I see 2 possible reasons: 1) don't have enough resources (CPU/memory/disk) to consume "all the logs"; 2) you won't be able to find what you need. I think both are a failure of the tools in the space.
• SoftTalker 3y ago
  I've outsourced monitoring to my users. They let me know when something isn't working.
  • awestroke 3y ago
    Your increased churn rate is probably cheaper than a Datadog plan
    • politelemon 3y ago
      I'm hearing way too many stories very recently about Datadog's extortionate pricing. I wonder if there's something going on and if they're on an aggressive campaign.
      • mikeshi42 3y ago
        It's been happening for a while now - I assume it's more in the news now as companies are starting to tighten budgets for the first time in a long time. We've [1] been taking advantage of it - counter-positioning against DD by offering low cost and really straightforward pricing while NR and DD both have chosen non-sensical (to me and many others) pricing models. [1] https://www.hyperdx.io https://www.hyperdx.io
  • spentu 3y ago
    I also have done this for some projects. Sadly they seem to be unreliable to degree of using alerts with ms Teams..
  • knorker 3y ago
    Crowd sourced consensus driven monitoring based on multiple instances of general AI.
  • hagen1778 3y ago
    How do you silence noisy alerts?
• dbg31415 3y ago
  I think you can monitor things "as a developer" and you can, and should... but let's face it, nobody is really watching that stuff. Better to tell the marketing teams to set up their own monitors and metrics. It could be Google Analytics, it could be UptimeRobot. Or whatever they want to use. Good for them to set monitors of every page that gets more than 1% traffic. Good to monitor for the HTTP Response Code (especially if there's a WAF involved), simple content check (like the page title, to make sure something didn't get redirected), "too many hops test" for more than 2 hops to get to a destination, page load speed, and of course uptime. Most tools also will alert you if your SSL Cert or Domain is about to expire. Kinda useful. Good to monitor on every major vanity URL redirect as well... so like mysite.com/some-promo. Setting this stuff up is SO EASY... but few people do it correctly / consistently -- so it's good to ask them to check once a month if they have all the monitors they need. Just grab the GA data and set a monitor on any page URL that gets more than 1% traffic... easy to just drop the data from GA into a CSV and import it into UptimeRobot. 2 minutes a month and you have amazing monitoring. Really important for monitoring to be run from a 3rd party location -- not from the same server, or behind the same firewall as your corporate network, but run from a location similar to what your real users are seeing. From the dev side... having good logs so you can diagnose issues. But Let's face it, nobody has time to monitor and respond to things unless it's been escalated. So just let the marketing team / ecommerce team escalate the things they care about, and just make sure you've got enough logs and ways to diagnose the issues they report.
• aynyc 3y ago
  Monitoring is at the infrastructure level. CPU/Memory/Disk space/network. Logging is at the application level. Log just enough that you can replicate a problem, usually it means just the input. The rest is marketing.
  • WFHRenaissance 3y ago
    There are some basic performance metrics that come to mind that are also important: Latency Uptime Throughput And if you're working with a distributed system, you should really be using metrics to track the health of your system at the application level.
    • aynyc 3y ago
      Latency, uptime and throughput for an application are non-functional requirements that should be built into your application. People like to generalize that and pawn that off to logging dashboard but I personally think that's a mistake.
  • crabbone 3y ago
    This is misguided. Think about, eg. Ceph. From the side of the user, it gives user the storage, from the side of Ceph admin, it's an application. Now, imagine Ceph is used in Kubernetes to create PVC. Now, Kubernetes is your storage provider... and so on.
  • wise0wl 3y ago
    I have to completely disagree. Application level metrics can also be emitted at the log level, but are much easier to work with as a metric. We have total request count, average request latency per route, and per HTTP response. This is extremely useful for finding performance regressions when new code is released.
  • klysm 3y ago
    Usually it’s input + state to repro - good luck logging that
• trabant00 3y ago
  Good luck trying to monitor using debugging tools like logs and metrics. Devs thinking they can do Ops because reasons. What you really need is state monitoring. You know, the old Nagios style that everybody thinks is out of date because, again, reasons. Just start by having e2e tests for everything and add intermediate tests when you discover points of failure and bottlenecks. And don't give me that "Nagios does not scale" bullshit. You don't need to and can't monitor 9 trillion things. You also have Icinga and other options that scale for state monitoring. In place of argumentation which would be too long, let me give you an analogy. Monitoring is telling you the fire alarm went off. The exact reason is left for a investigation and that's where saving data like logs and metrics helps. But first you need to evacuate the building and call the fire department. If instead of detecting the smoke (e2e) you try to monitor all the data like pressure in gas pipes, all the pipes fittings, how many lighters are in the building and where, etc, you will wake up every night 10 times for nothing and when a fire starts you will burn sound asleep. Oh, and don't try to scale Prometheus, that's what things like victoriametrics are for.
  • wspeirs 3y ago
    >If instead of detecting the smoke (e2e) you try to monitor all the data like pressure in gas pipes, all the pipes fittings, how many lighters are in the building and where, etc, you will wake up every night 10 times for nothing Without this you'll always be awoken by smoke, and at that point it's too late... there's already a fire. However, if you can monitor other things (gas pipes, lighters, etc), you might be able to remediate a problem before it starts smoking or burning. There's no one-size-fits-all. Start with some obvious failure (e2e tests/checks; aka smoke), and when you root-cause, add in additional checks to hopefully catch things before the smoke. However, this also requires you update (or remove) these additional checks as your software and infra change... that's what most people forget, and then alert fatigue sets in, and everything becomes a non-alert.
    • teddyh 3y ago
      > when you root-cause, add in additional checks to hopefully catch things before the smoke. No, what you should do is update the software to handle or avoid the problem gracefully. Keep the smoke detectors, but don’t keep adding tiny things to monitor and keep in sync with your ever-changing software.
• phillipcarter 3y ago
  > The problem with me and tracing is nobody uses it. When I monitor the teams usage of traces, it is always a small fraction of the development team that ever logs in to use them. I don't know why the tool hasn't gotten more popularity among developers. I think there's two primary reasons: 1. Historically it was too hard to get easy value early. That's probably changed with OpenTelemetry since it has so much automatic instrumentation you can install. Even 3 years ago it just wasn't feasible to install an agent or write 5 lines of code and then get most of the libraries you use instrumented for you. Now it is, and it's fully portable across tools. 2. Cost, which still isn't solved yet. Tail-based sampling is still too hard, but you need it if you want to do stuff like always capture 100% of error traces but get only a statistically representative sample of 200 OK traces. There's some solutions for that today (e.g., my employer Honeycomb has such a tool), but it's still locked away with vendors and too hard to use. I do expect the narrative on tracing to change in the coming years, though. OpenTelemetry is growing in popularity and it has enough features and stability for a wide range of people to adopt it. I think the cost issue will also be dealt with over time.
  • wspeirs 3y ago
    >Cost, which still isn't solved yet. I'd argue it _is_ solved... store your logs in S3. At ~$0.02/GB you can store a _lot_ of logs for like $20. The problem is that _most_ solutions (Honeycomb included) are SaaS-based solutions, and so they have to charge a margin on top of whatever provider is charging them. You just need a tool (like log-store.com) that can search through logs when they're stored in S3!
    • phillipcarter 3y ago
      At the small volume of data where you could "just put stuff in S3 and search it later", the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else.
      • OldManRyan 3y ago
        Depends on how much you value not being vendor locked in. I'll spend a week of dev time in order to save a month or two in migration pain later down the line when my vendor jacks up cost. See Datadog as an example.
• awinter-py 3y ago
  Strong yes, particularly this line: > the cost of monitoring an application can easily exceed the cost of hosting the application even for simple applications (especially if you are looking at saas monitoring for a single-box or serverless thing) I suspect SIEM is the most established use case for ingesting logs to something actionable; wonder if there are concepts there which port generally. Datomic's approach to change capture seemed very good when it came out -- I wish this were a thing I could cheaply turn on and off on SQL columns. As things are, every ORM seems to resist having a table with no primary key so 'log important things to the DB' is hard in practice. In postgres, mvcc makes it expensive to edit a scalar, so it is impractical to store counts. (There is an old uber blogpost I think about why they use mysql for this use case). Specifically for alerting, I wish I could define application logic statistics in my application rather than PromQL / datadog json. Would be much easier to express 'warn me if this ever fails', 'expect 50% failure', 'warn me if too many retries' type stuff. On the 'looking at logs' side, the hosted monitoring tools I've used are often uncomfortably slow, even when my data is small. I understand it's a hard problem but I miss graphite / whisper. (not even getting into mixpanel / posthog type product-level observability)
  • rjbwork 3y ago
    >I wish this were a thing I could cheaply turn on and off on SQL columns. You can turn it on and off for whole tables in systems that have decided to implement SQL:2011 such as MSSQL, Oracle, MariaDB, etc. and then issue time travel queries to see what the state of the database was at any given point in time. Keyword here is "Temporal Table".
    • awinter-py 3y ago
      did not know about this -- thank you for the rec
  • lukeasrodgers 3y ago
    What do you mean "expensive to edit a scalar"? If we're thinking of the same Uber article on mysql -> postgres, IIRC it had to do with write amplification related to updates to indexed columns. I don't think postgres has any problems storing counts, but would be interested to know if I'm wrong.
• doctorpangloss 3y ago
  > You can now track all sorts of things across the stack and compare things like "how successful was a marketing campaign". "Hey we need to know if Big Customer suddenly gets 5xxs on their API integration so we can tell their account manager." "Can you tell us if a customer stops using the platform so we know to reach out to them with a discount code?" These are all requests I've gotten and so many more, at multiple jobs. But now you can ask ChatGPT GPT4 to author PromQL and Jaeger queries for you, which it is incredibly good at. The real, actual pain point - not cost, not maintenance, etc. - between the bottom line affecting stuff like customer success management and metrics was always turning the English language problem into the thing that goes into the computer. And that is seriously going away, the only people who think it's not are the experienced ones who, for some reason, months into the distribution of this insane technology, haven't tried asking for a PromQL query yet!
  • gofreddygo 3y ago
    > the English language problem This is so true, and IMO underlooked for other flashy capabilities. GPT transformers (even bad ones) have nailed english grammar. This opens up a lot of doors for a lot of small businesses.
• iDemonix 3y ago
  Using Prometheus in federated mode has been fine for me, I'm running it almost exactly as depicted in the article and I'm not sure why OP describes it as a complex on - you just need to have some idea of what metrics you'd like to store longer term.
• AndrewKemendo 3y ago
  Solving this is the entire purpose of Solarwinds and we see what kinds of risk trade offs you get… I lived this for years and there’s just no easy solutions here. I don’t care what kind of application you’re building, if you want to be able to reliably deliver good service and iterate in any type of scientific fashion, you must-have great observability. This is not a suggestion. It is simply creating the situation where you can run experiments and understand the impact of changes to your system. But that means you need to significantly invest in not only the technical tooling, but the people, education and procedural things that are necessary to think in a data oriented way. That is something that is excessively hard to convince people to do because it’s so seemingly boring. The real siren here is push button observability, and automation like with solar winds. Everyone would love to be able to have State machine level determinism, and monitoring to tell you which distinct state you’re in. I’ve yet to find an org that can comfortably manage and exploit that level of granular observability. This is an amazing article that every engineer should read.
  • SteveNuts 3y ago
    I’ve never before heard Solarwinds being discussed even in the same realm of observability. Last time I used it it _barely_ can tell you up/down status of a hardware device via ICMP or SNMP. Has it gotten 10,000x better in the last few years?
    • AndrewKemendo 3y ago
      I didn’t know that it was possible not to connect those two given that it’s literally the entire Raison Detre of Solarwinds. It is on every single part of their marketing materials and is exactly what they are selling to and promising CIOs and CISOs. I mean just look at: https://www.solarwinds.com/ https://www.solarwinds.com/
      • SteveNuts 3y ago
        Have you ever... used it?
• buro9 3y ago
  Monitoring all the things can be hard. There are a lot of different solutions, that have different trade-offs for different volumes of data, cardinality of data, how out of order / async the data may arrive, whether you need push or pull delivery, etc, etc. These apply to you and only you know the parameters and considerations of your application, and the environment it runs in (perhaps you come with the baggage of "this is instrumented in x and I can't change that" in addition to volume and cardinality considerations, perhaps it's a point-of-sale in a shop that only connects to the internet once a day for upload of the data). But for the simplest solution, you're probably talking Prometheus + Loki/Elastic + Tempo + Pyroscope/Polar Signals + Grafana. This is simple as in "all pieces are proven and work in the small and get you fairly far"... but you still have to compose the solution out of those parts. And later you may wish to graduate to Grafana Cloud when the pain of running it all yourself is taking more time and effort than running the systems you're supposed to be working on. Oh, and if you really want to lean in on traces for everything, take look at Honeycomb too. As to "the monitoring costs more than the app"... yup, at Cloudflare for a very very long time there was little monitoring of DNS requests or billing of it... because the act of counting it and recording it cost more than just serving the request. This principle scales well (or badly if you're looking at your credit card bill)... and is also why those who make these systems really have to keep in mind how to make the instrumentation cost less (eBPF?) as well as the ingestion and retention cost less (various methods, including aggregation, sampling, etc).
  • hinkley 3y ago
    My company does a lot of things the dumbest way possible but they’re always been good at tracing service requests and telemetry. Any time correlation IDs were missing was considered a regression. And we are quick to agree to action items that increase visibility from a postmortem. The one complaint I have is we don’t move things out of Splunk into graphs as fast as I’m comfortable with. But the thing I see we are all missing for telemetry is a sense of coverage. We changed backends last year and as we were wrapping that up I found big chunks of code that were missing coverage. Meanwhile we found a bunch of stats that nobody looks at, meaning the SNR goes down over time. There are only so many dashboards I can watch. So some sense of code blocks missing telemetry, and data that is collected but has no queries, queries that exist but never run (that dashboard nobody remembers making and isn’t in the runbooks anywhere) would be a great goal for a next level of maturity.
• JohnnyTsunami 3y ago
  Well, I built technology that alleviates this pain. :0
  • JohnnyTsunami 3y ago
    Either: If you are using $XYZ software, take a look at Aronetics' Thor. This technology is currently running within the a NCR Government office for evaluation with great success. We're leveraging this blessing from Uncle Sam as another go-to market strategy. Thor has completely eliminated the threat presented to your data, it's a defense and counter-offense tool. As further exposure, when could you set aside time to learn more? Or Our digital hammer Thor is a defensive technology that closes gaps utilizing Zero Trust providing precise tamper-proof assurance to take action against tomorrows unknown threats so you have operational truth and trust of your compute device today.
• jillesvangurp 3y ago
  It's hard because it's being pushed and done by people with different agendas that also aren't necessarily the right agendas to be prioritizing. - mistake #1, assuming it's a technical thingy that some techie person should do and they should just get on with it and do it. Whatever it is. The mistake here is that without guidance this person is going to be selecting some random tools mostly focused on operational things (logging, infrastructure). These have low value for other stakeholders. Yes you need that but these are also commodities. - mistake #2, each of the stakeholders adds their preferred tools to the mix and the end result is a bunch of poorly integrated tools with a lot of complexity. - mistake #3, assuming that selecting a tool means the job is done. It's not. Most of these tools assume that you have some notion of how to use the tool and what you want out of the tool. Getting the tool is a the beginning, not the end. The combination is lethal. Lots of complexity, lots of data, lots of cost, and not a whole lot of value being delivered. The fix is an observability mindset that exists in product, tech, and business departments. Select 1 tool, not 2, 5, or 10. The more tools, the more fragmented the effort and the more duplication of effort and mistakes. Have a plan, engineer for the system to be observable, have a notion of things that are important to track, etc. Don't dump the responsibility on some techie but actually reflect a bit on what you are looking to get out of this and what this is worth to you.
  • juancn 3y ago
    That, and the tooling sucks. The good tools are expensive, both Sumologic and Wavefront are amazing, but past a certain scale it's hard to justify the cost. On the open source side Grafana and its stack (tempo, loki and prometheus) all just suck, they kinda scale and check all the boxes, but in a really stupid way (data nodes should probably be query nodes, rather than moving everything from storage and caching, everything is supposed to be a dashboard, don't get me started on the query languages). The Grafana UI was built by the enemy and explorations are incredibly annoying. I'm more of an exploratory user, not a dashboard one (think level 3 on-call).
    • lazyasciiart 3y ago
      We just moved to grafana from New Relic and it makes me angry just to try and look at monitoring, we have lost so much utility. And the company spent a year on the transition and a shit ton of work, so it’s not (just) that we had good workflows set up that haven’t come across. (I am also a heavy exploratory user)
• acaloiar 3y ago
  I generally agree with the sentiments here, yet I think it's important to understand the value of the ".0001%" of logs that _are_ used. Just because log utilization rates are often very low, doesn't meant that the .0001% isn't immensely valuable.
• jrms 3y ago
  That's why I'm still using munin, even for kubernetes clusters, even after 20 years in the business, if I should need a plugin that I wrote back then it would still work! And I can monitor a whole k8s cluster (around 70 nodes) using 100MB of mem and 3G for storage (up to 1 year). Keep it simple!!
  • klysm 3y ago
    It’s only simple because it’s something you already understand - that word doesn’t really mean anything
    • jrms 3y ago
      That's correct! Keep it simple, for yourself!!
  • milar 3y ago
    I love munin and I’d rather do Perl with it than the other shit
• slaymaker1907 3y ago
  In terms of controlling what data is actually logged, SQL Server takes a completely different approach instead of using log levels. Extended Events are strongly typed and reduce data by having predicates that can look at the data of the event to determine if it should be saved or not. While log levels are handy, I think it would be great if more log frameworks had this kind of flexibility since it lets you log some specific “DEBUG” style events in production rather than turning on all DEBUG logs for a particular logger (for example, logging all client connections could be expensive, so maybe you just log the ones which aren’t system accounts).
• perpil 3y ago
  If you want to do monitoring and logging at scale, the majority of the best practices are captured in this article: https://aws.amazon.com/builders-library/instrumenting-distributed-systems-for-operational-visibility/ https://aws.amazon.com/builders-library/instrumenting-distri... These are the key insights we learned running hundreds of services at Amazon scale and you'll learn something whether you're just starting, you think you know it all or you need some actionable ways of improving your operational excellence.
  • jeffbee 3y ago
    Notable they are not discussing centralized collection and indexing of debug logs, they implicitly leave the logs on the disk of the machine where they were produced, and go out to read them when called for. This is an important lesson because centralizing and indexing logs is very foolish unless you are a Splunk shareholder.
    • perpil 3y ago
      I'm not sure that is a takeaway here. The processes evolved, but at AWS, centralized logging w/CloudWatch Logs was becoming more common than having tools that ran commands on end hosts against locally stored ephemeral logs. Sometimes an agent would aggregate the data locally before publishing, but using tools like CloudWatch Insights was distributed and fast enough (multiple gigabytes/sec) that no indexes were necessary to get quick answers to adhoc questions. My read is they are not advocating for publishing debug logs which gets expensive quick. They are suggesting you have the ability to enable verbose logging if needs be for a temporary use case. The two types of logs they describe are service logs which have one log entry per request (customer, request id, e2e latency, response code, etc) and applications logs that go into more detail about what happened on a particular request. Service logs are less expensive to store and more information dense. You are primarily querying service logs for operational visibility. Service logs are inexpensive and fast. You query application logs to trace a specific request and that is slower and more costly. For cost savings, I know of some teams that would reprocess applications logs after x weeks to strip information that only provided short term value.
    • 3y ago
• giovannibonetti 3y ago
  When it comes to monitoring, the first thing I look for is a good data store. In the open-source world, probably the best one is Clickhouse, and there are multiple solutions built on top of it like SigNoz [1] and Uptrace [2]. [1] https://github.com/SigNoz/signoz https://github.com/SigNoz/signoz [2] https://github.com/uptrace/uptrace https://github.com/uptrace/uptrace
  • serverlessmom 3y ago
    Definitely +1 for SigNoz, much more cohesive view of metrics, logs, and traces
• mvdtnz 3y ago
  I just think that this is an incredibly whiny post. These are all solved problems but he brushes away the solutions with a hand wave in order to continue being relentlessly negative. > Maybe you start with an ELK stack, but running Elasticsearch is actually a giant pain in the ass. And that's the last he says of it. But I've seen ELK stacks in organizations from 30 employees / 5,000 users all the way to 5,000 employees / millions of users and it scales well. I'm sure it continues to scale above that. Or for Prometheus scaling options, > Adopt a hierarchical federation which is one Prometheus server scraping higher-level metrics from another server. [...] The complexity jump here cannot be overstated. You go from store everything and let god figure it out to needing to understand both what metrics matter and which ones matter less, how to do aggregations inside of Prometheus and you need to add out-of-band monitoring for all these new services. I've done it, it's doable but it is a pain in the ass. It's not a pain, it's like an afternoon of work. I don't see why you need to be deciding which metrics matter - collect all of them. And if you don't know how to do aggregations in Prom, learn! You'll need to know this to use it anyway.
  • lolinder 3y ago
    A lot of the article is less "monitoring is a pain" and more "distributed systems are a pain". Everything is harder in a distributed system, not just monitoring, and the author's fixing of the blame on monitoring tools is a distraction from what's actually causing their headaches.
  • lazyant 3y ago
    It can be both true that ELK scales well and it's a PITA to manage.
• tonymet 3y ago
  Monitoring is more about process than logs . The logs (including text logging, telemetry) are just signals. The way to improve monitoring is by continuing to refine s/n ratio & critical alert health through continuous review of signals and identifying leading indicators. Every week you should identify the critical objectives and the leading indicators that represent healthy and unhealthy states for those. Also identify false positives and noisy alerts and work to reduce noise. For example, how many people are checking out, adding to cart, viewing the website. What are indicators of health and error conditions. What are true signals and what are noise? The biggest issue i see is people setting up logging and alerting without continuing to revise and improve the process.
  • jimbokun 3y ago
    Yes. I found the article jumped too quickly to specific tools and their shortcomings. The process needs to be monitoring/alerting need -> best tools and processes to meet those needs.
  • 8note 3y ago
    I'm not sure if weekly is right, but at least when an alert goes off, it's worth checking if it was useful. Checking for things that don't alert is much harder, and I think also needs a review of what you aren't measuring to make an update
• bob1029 3y ago
  We decided to outsource this pain to Azure. Az Monitor & Application Insights are far from perfect (I can't stand their UI for querying logs), but it really does "just work" out of the box unless you are using something very non-ms with it. If you are throwing a happy path thing like .NET6 on top of Az function runners, you can expect to not have to configure anything, aside from an option at app creation time. Our primary mission around monitoring is to proactively respond to customers before they call us regarding exceptions in their environments. Az Monitor w/ basic alerts are all we really need to get that job done. After spending 20 minutes with these tools, I felt really dumb for rolling our in house solution the last ~8 years. All we had to do was take a look and try. There is absolutely no way in hell that my team can compete with the resources of a trillion dollar hyperscaler like this. The long game I am looking towards is the partnership between MS & OpenAI. "AIOps" is already listed as a "Respond" option under Az Monitor in their documentation. Anyone who has been standing on principle against Microsoft has to at least see some of the light coming through this tunnel by now, train or no train.
• radiator 3y ago
  I don't like this article. - we are all doing monitoring wrong? No, sorry. - You set up these tools with the mentality of "set and forget" but they actually require ever increasing amounts of maintenance? The tools and your monitoring rules require maintainance like everything in IT. - Logs OK, don't use logs for monitoring. - Metrics Yes, do use time series for monitoring. - Prometheus is really designed to be running on one server. You can scale vertically ... Actually you can operate more than one Prometheis behind a load balancer, avoiding the situation "when there is a Prometheus problem you lose visibility into your entire stack at once", and without needing to federate them.
• GnarfGnarf 3y ago
  I write my app’s logs in HTML. It allows me to format as tables, large fonts for alerts, colour for errors, embed images, all sorts of visual aids. Easy to scan and identify the important stuff at a glance. There is one common function to format as low-level HTML, and a trivial markup code to simplify calling it.
  • ducharmdev 3y ago
    Are the logs actually written as HTML, or are you logging objects (like structured logs) that are transformed into HTML when displayed? If the former, I would imagine that'd make searchability harder for larger volumes of logs.
    • GnarfGnarf 3y ago
      The logs are written as raw HTML. They can be searched by including HTML id's, tags, etc. They are plain text so they are searchable. The color option really makes things stand out. The formatting possibilities has made debugging so much easier.
      • ducharmdev 3y ago
        I guess what I was getting at, when you are logging lots of data in something like Azure app insights, Splunk, etc., you don't have the luxury of being able to just scan your logs with your eyeballs, because there's simply too much information. Instead you log objects, and are querying logs almost like you would SQL.
• paulvnickerson 3y ago
  Check out Grafana's suite of tools, especially Mimir [1] for a simple and easy to use monitoring platform. [1] https://grafana.com/oss/mimir https://grafana.com/oss/mimir
  • hagen1778 3y ago
    It consists of 6 separate components to run, plus 4 additional optional components [1]. And I don't even count 5 additional caches [2]. How is that "simple and easy"? [1] https://grafana.com/docs/mimir/latest/references/architecture/components/ https://grafana.com/docs/mimir/latest/references/architectur... [2] https://github.com/grafana/mimir/blob/9a52522a05f25d6d6bc2e8e059ab1acbbee3bc51/operations/helm/charts/mimir-distributed/small.yaml#L91-L108 https://github.com/grafana/mimir/blob/9a52522a05f25d6d6bc2e8...
    • paulvnickerson 3y ago
      Currently we run it in "monolithic" mode, in which all components are grouped together, and it works just fine: https://grafana.com/docs/mimir/latest/references/architecture/deployment-modes/#monolithic-mode https://grafana.com/docs/mimir/latest/references/architectur...
      • hagen1778 3y ago
        How is it different from running a Prometheus server then? Prometheus is monolithic as well. It is cool and not cool in the same time. With monolithic approach, the process becomes very simple to run and efficient at the same time, since there is no penalty on interprocess communication and network is not involved. But you can't scale monolithic applications horizontally. Horizontal scaling is very likely the reason why Mimir exists at all. If Prometheus supported horizontal scaling out of the box, there may be no need in Thanos, Cortex or Mimir.
        • Jedd 3y ago
          Have a read about the design intent and architecture to get deeper info on how it's different - but basically, indexes aren't retained locally, which means you've got a MUCH bigger scale (several orders of magnitude) than Prometheus can cope with. You're right that you can't 'scale monoliths horizontally' but Mimir was designed to handle much more data than Prometheus was, on the same compute/memory footprint (obviously storage is a direct function). Mimir will consume multiple Prometheus with remote_write's and de-dup those series. Because it puts older (default 2h) data out to block storage, costs plummet without impacting performance too savagely - the vast majority of queries are going to be hitting fresh data. We've been running monolithic (Nomad, docker) but are experimenting with their 'in-between model' which has 3 components - read, write, backend - which should help us scale more easily than juggling the full microservices deployment method you mentioned above. > If Prometheus supported horizontal scaling ... Well, yes, precisely so.
• fpanzer 3y ago
  OP needs to install naemon and chill. Monitoring means alerting, and not collecting metrics. Monitoring means being notified of outages. Metrics are for troubleshooting and post mortems. If you monitor CPU or RAM usage, you are doing it wrong. Oh and by the way, you want your ram to be full.
• waffletower 3y ago
  This article focuses on the author's dislike of logging. Never does the author mention log storage expiration, an obvious and effective tool to reduce the largest complaint, storage cost. The author also doesn't mention differences of scale for services -- the log resolution for service X, may be inappropriate for service Y, etc.
• hagen1778 3y ago
  Monitoring is a pain because we need to monitor more and more complex things. And not only that. Mostly, the problems described by OP are connected to the scale. So the main source of the "pain" is the volume of data we need to process with monitoring systems. That's why Graphite is fading off - it just can't keep up with the growth of data volume. Prometheus is a great tool, but now it is not enough of one server. So you are either looking for workarounds with federation, or running more complex monitoring systems like Thanos, Cortex, VictoriaMetrics.
• waffletower 3y ago
  Would be useful to add data presence/freshness monitoring to the taxonomy of tools here.
• rsync 3y ago
  Higher value needs to be placed on "feel" and the ineffable heuristics that human brains automatically build up as they operate a system. This is one of the reasons I strongly dislike the idea of remote or telepresence inspections of factories. You need to just walk the factory floor and use all of your senses (including smell). If you do this enough, you'll know when something is wrong. The pitch of the machines, the vibrations, the hum ... and you'll catch it before your rules do. A personal example: rsync.net has almost zero fraud detection for payments and signups. However, I, personally, have been watching the real-time order flow[1] for decades and I just know. A fraudulent signup just sticks out like a flashing red light. I could not teach this to you and I could not codify my heuristics. I have spent decades thinking about how to augment - or replace - these real-time human observations with warnings and alerts and thresholds and I am deeply skeptical of their value relative to human interaction with the functioning system. [1] ... with tail ...
  • verve_rat 3y ago
    How do you detect fraud when you are asleep?
    • rsync 3y ago
      We don't auto-create accounts - we do them in batches throughout the day. I am not typically involved in that but I do see the order flow - usually early in the morning. I can usually spot them before they are created or if they have been created already we just disable it and make some effort to notify the cardholder that something fishy is going on (and ask them to accept our refund rather than issue an annoying chargeback which only complicates the refund process). This process doesn't scale. That's fine with me and everyone here.
• jeffbee 3y ago
  One thing I like to evangelize is the similarity between logging and tracing. A trace could be assembled from properly-formatted distributed log messages, and some tracing systems are implemented this way. If you have a working tracing setup, and you trace every request as some people do, then using a span event [1] can be an easier way to get information about what happened during the request than it would be to find all the associated debug logs. 1: https://opentelemetry.io/docs/concepts/signals/traces/#span-events https://opentelemetry.io/docs/concepts/signals/traces/#span-... A span event is data that is associated with a point in time rather than an interval.
• llama052 3y ago
  Honestly we've had great luck with HA prometheus instances shipping data to Thanos. All in kubernetes so we get to use pre-built operators. We don't really have to touch it other than updates. We are a 3 man team and we are holding around ~4m active series with around a year of downsampled data in a storage bucket. Doesn't really cost us anything other than some servers on a cluster. We also use Loki for logs and Tempo for tracing in the same pattern. One of the reasons I like Kubernetes, it's rigid enough compared to the wild west of VM land, and utilizing operators and tooling can get you solutions that just work provided you have enough core knowledge to fill in blanks.
• geoffbp 3y ago
  The key is to develop a monitoring framework. pick the tool/s you want, enrich and standardise what can be done with them (dashboards, alerting, etc) then make the path to using them easy
• lrem 3y ago
  I share the love of tracing. Internally at Google I sometimes stare at traces traversing hundreds of RPCs. It's about the only tool where I can very quickly go "wait, this shouldn't be the costly part" and "ugh, why are these sequential", then look exactly what the particular requests were and where the QoS was set wrong.
• jrockway 3y ago
  I have achieved zen with logs. Levels are easy. You have three, DEBUG, INFO, and ERROR. DEBUG is stuff that's interesting to you, the author of the code. INFO is things that are interesting to the person/team that runs your software in production. ERROR is for things that require human attention. My opinion is that INFO logs are the primary user interface for your server-side software, and demand the level of attention that you would expect a frontend designer to devote to your brand or whatever. That's where the software gets to interface with the person that's responsible for its care and feeding. Garbage in there wastes their time. Everything in INFO should be logged for a reason. (Yes, sometimes there's spam. Usually request/response data. Most of the time the requests go great, but you don't know they've gone great until you've logged the fact that they were made. It's a necessary evil.) Meanwhile, DEBUG should probably be pretty chatty. If your program is consuming resources like CPU, it better justify itself in the DEBUG logs. A program sitting there using 100% of the CPU and not logging anything at level DEBUG is very very suspicious. My philosophy is to "show your work" on every critical operation. Log when something starts, what decisions it makes along the way, and when it's done. A message like "failed to contact server example.com: i/o timeout after 10 minutes" is 10 minutes too late to be interesting. "connect to example.com: started", "connect to example.com: i/o timeout after 10 minutes" is much better, because 10 minutes before the error you can read the logs and realize example.com is NOT where your database is hosted. (You can really go overboard here and I would never stop you. connect to example.com: resolve DNS: start, connect to example.com: resolve DNS: A 1.2.3.4, connect to example.com: dial 1.2.3.4: start, .... In production, I compromise and log "start", "ongoing long HTTP request" (at 10 seconds), and "finished" with the status code.) Every branch is a place to log which branch you selected and why. If there's a bug and you only know the outcome, but not the software's "thought process" that led you there, you have nothing to go on. "Hey, the FrobTheBaz endpoint is returning 400s." That is useless information. You have to read the code and guess. You do not want to be guessing during an outage. Errors are a whole can of worms. If someone reading the logs can't fix the problem, it's not an error. "Error: cannot write data to the database". That's a big problem. Disconnect that popcorn popper and plug the power cable to your database back in! "Error: login for root from spambot@1.2.3.4: invalid signature" is a problem for your firewall. What is someone reading the logs going to do about that? Call up 1.2.3.4 and ask them to stop guessing SSH keys? At best, that's an INFO log. There are 3.4 with 38 zeros possible keys. They aren't ever going to guess it. If you decide "this isn't even worth logging", I can definitely agree with you. (I'd log it, though. An increase in failure attempts is good ancillary information when something else draws your attention to the system.) Finally, newline-delimited JSON ("jsonl") is the only correct format for logs. Nobody wants to write a parser to figure out "how many 400s did we serve today". Just let JQ [https://jqlang.github.io/jq/ https://jqlang.github.io/jq/] do it for you. If you want pretty colors for viewing, use something like jlog: https://github.com/jrockway/json-logs https://github.com/jrockway/json-logs. You don't need to write VT100 control characters to your log file. I know for a fact you don't even have a VT100. All other formats of telemetry are just logs with fancier user interfaces. Metrics? Just write them out to your logs: "meter:http_bytes_sent delta:16384 x-request-id:813db394-75a1-4b22-94ad-e7850d9cdd25". You still have the ability to tail your logs and convert the metrics to a proper time series database (with less cardinality than "per request" of course), but you can now also do investigations in context. For example, I've implemented exactly this system at work. A customer shows up, saying that we're using too much memory. I ask for the logs and see something like "span start: compaction", "meter:self_rss_bytes delta:2000000000", "span end: compaction, error:nil", "meter:self_rss_bytes delta:-1000000000". Hey, we leaked a gig of RAM in compaction. You would never figure out the memory leak with "compaction done after 5 minutes" and a graph of memory usage across your fleet. While you weren't looking I also threw traces in there! Have a logging primitive like Span("operate(foo)", func { ... }) that logs the start and the end, and throw a generated ID in every message. Concatenate the span IDs which each recursive call to Span (I abuse context.Context for this, sorry Rob Pike), and log them with every message in the span. Now you have distributed tracing. As always, sure, parse those logs and throw the log-less spans in to Jaeger so you can efficiently search for rare events and have a nice data flow graph or whatever. Log sare so simple and so great. The problem you run into is by not producing enough of them. Disks cost nothing. You can delete the debug logs after a short period of time, or not even produce them until you suspect an issue. Don't sweat the log levels; pick the 3 I picked and try to enforce the rules in code review. Aim the log messages at the right audience; dev team, ops team, emergencies. Show your work. Log contextual information. Put metrics in the logs, and aggregate them for dashboards and monitoring. Annotate logs with trace IDs, and put those into Jaeger. You'll never have to wonder why your system is broken again; you printed out everything it's doing. The aggregation will tell you you're not meeting your error budget, and the logs for a representative request will tell you why that one failed. Once you know what's broken, the fix is always easy. Whenever I tell people my system, they tell me it can't scale. When I worked at Google, I wrote exactly this system for monitoring Google Fiber (well, we didn't have log levels really...), and we could answer questions about the entire CPE fleet as easily as we could answer questions about one device or one bug. Over many years, I've finally gotten this into the "real world" (like weeks ago), and it's been amazing. Good logs are the best thing you can do for someone that has to run your software. For a lot of folks, that's you! For other folks, that's your customers. Can you customer set up Prometheus, ElasticSearch, and Jaeger, and dump that all out and send it to you in Slack? Unlikely. Can they dump a file that's 10MB of JSON lines into Slack? They sure can. And now you can diagnose the bug and fix it without bothering them for more info. It's good for everyone.
• solatic 3y ago
  If you're not excluding debugging and info logs at the SaaS level, you're doing logging with a SaaS provider wrong. Log everything including debug, exclude debug and info, set up alerts for error logs, wait until an error comes in, re-enable debug logging for 15 minutes if you need to, then enable the exclusion again. SaaS observability gives you the tools to control these costs. It's much, much cheaper to log up to debug if you're not indexing them most of the time. If you don't understand features like exclusion rules, you should go learn them.