6 ms·

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everyth

by buro9 6y ago

A lot of excellent information in that blog post and linked from it... but if you're wondering where to start: 1. Write good logs... not too noisy when everything is running well, meaningful enough to let you know the key state or branch of code when things deviate from the good path. Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number), and that the message will help you debug. 2. Instrument metrics using Prometheus, there are libraries that make this easy: https://prometheus.io/docs/instrumenting/clientlibs/ https://prometheus.io/docs/instrumenting/clientlibs/ . Counts get you started, but you probably want to think in aggregation and to ask about the rate of things and percentiles. Use histograms for this https://prometheus.io/docs/practices/histograms/ https://prometheus.io/docs/practices/histograms/ . Use labels to create a more complex picture, i.e. A histogram of HTTP request times with a label of HTTP method means you can see all reqs, just the POST, or maybe the HEAD, GET together, etc... and then create rates over time, percentiles, etc. Do think about cardinality of label values, HTTP methods is good, but request identifiers are bad in high traffic environments... labels should group not identify. Start with those things, tracing follows good logging and metrics as it takes a little more effort to instrument an entire system whereas logging and metrics are valuable even when only small parts of a system are instrumented. Once you've instrumented... Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) https://grafana.com/products/cloud/ https://grafana.com/products/cloud/ so you can see the results of your work immediately. If it's a big project, you have a lot of options and I assume you know them already, this is when you start looking at Cortex and Thanos, Datadog and Loki, tracing with Jaegar.

kostarelo 6y ago
> Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number), and that the message will help you debug. if you include all these information and the logs are not structured, you won't get much information out of them.
deepGem 6y ago
From the post "The key, he says, is using the right transaction identifiers so that calls can be traced across components, services, and queues". I think this is a key feature not many people implement especially in today's world of over blown micro services, having a transaction id from the time the request hits the reverse-proxy till the database write is so helpful in debugging, saves a ton of time.
- KaiserPro 6y ago
  100% if you manage to get opentrace to work, it is a brilliant debug tool
  - eatonphil 6y ago
    You can also just propagate a uuid throughout your system. I've used both uuids and opentrace. Just don't get hung up if opentrace seems too complex.
    - dkarl 6y ago
      I agree with this wholeheartedly. You can even define a standard and let downstream services opt in over time. Simple wins like this should not be put off because "someday" we're going to implement a complex distributed tracing solution.
      - dillonmckay 6y ago
        So, in terms of full implementation, is the constraint the dev time to implement, or getting various factions to agree to something? I guess, is it political, or technical? Asking for a friend. Thanks!
        nitrogen 6y ago
        Often it's political, but political friction can feed into technical friction if there are also a half dozen different half wrappers around half-baked HTTP libraries. Also, if someone has included OpenTracing as a shadow library in a company-wide library (JVM territory), but you want it as a top-level dependency, you have to write translators.
viraptor 6y ago
> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).
- rudasn 6y ago
  Twice I fell into the trap of datadog, having paid more than $300 for a single month in each case. The simplicity of it, dashboards, notebooks, logs etc, is what makes it so appealing though.
  - count 6y ago
    If it saves you more than an hour or two, $300 seems super worth it for a system making you money...
    - viraptor 6y ago
      If you're in an established company making money - yes. If you're bootstrapping a service and counting on $50 total monthly cost while initial users are signing up - no.
    - smoe 6y ago
      It depends where you are in the world. When I was working in Switzerland, most SaaS pricing were no-brainers for us. But since I work in Latin America for small companies with local costumers, all the different services and tools you might want to use, with prices targeted at "western" customers, much more quickly add up to the equivalent of having multiple people on staff full time. Still it is often not worth to roll your own, so it is nice to have alternatives for different price points and company scales.
      - rudasn 6y ago
        > It depends where you are in the world. Exactly this. We operate in Eastern Europe with local clients, offering on-prem SaaS. If I added all my clients' servers on datadog it would very easily eat through our profit margins. > Still it is often not worth to roll your own I tried hard not to, but at at the end, after spending 1 week trying to setup netdata and failed, I decided not to spend another week trying to setup grafana/influx/prometheus (lot's of docs to go through), and just have some bash scripts send metrics on a $10 digital ocean node service that sends me emails/sms when something "looks bad" (eg. high cpu temperature, stopped docker containers, etc). I gave up on aggregated logging for the time being, since I can just ssh on each server and check journal and docker logs if I need to (as long as the hard drives don't crash).
dkersten 6y ago
I’d add to no. 1 by saying include a correlation id or request id so you have a way to filter the logs into a single linear stream related to the same action.
- jrott 6y ago
  Absolutely constantly being able to get a single linear stream is when tracing becomes super powerful.
- Bombthecat 6y ago
  Also adding a context tag can very powerful, especially in times of Microservices and some kind of event driven stuff like monthly payments. Imagine: user registers, the post gets and id, and context registering. Then he adds a credit card. (a new id, context credit payments) after 14 days the bill goes out, same id, same context.
ysoft 6y ago
I don't think loki is production ready. Needs more work. It's going to be great with grafana though
- yellow_mixer 6y ago
  What else do you think it needs. I’ve been playing around with v1.5 the past few days in Grafana and I like it’s simplicity. Grafana now has a Loki datasource which is nice.
chandraonline 6y ago
I would add that if you add a traceId to #1 and use something like https://www.jaegertracing.io/ https://www.jaegertracing.io/. You get even more.
sciurus 6y ago
I agree with 2. I have a presentation at https://www.polibyte.com/2019/02/10/pytennessee-presentation/ https://www.polibyte.com/2019/02/10/pytennessee-presentation... which goes into how to get started with Prometheus. Not as in "how to set it up", but more about what to instrument and why, how to name things, etc. Despite the title, there's very little in it that's specific to Python.
gnufx 6y ago
Why should you have to use Prometheus? There are plenty of options, and good reasons why you might want to push data rather than pull. Measurements should minimize perturbation of the system being measured, and the (computer) system generating data is likely best placed to determine when and how, when that matters -- e.g. in HPC, where jitter is important.
gdcohen 6y ago
Gavin from Zebrium here. Completely concur with #1. We are big advocates of writing good logs and not having to worry about structured vs unstructured (and even if you structure your logs, you'll still probably have to deal with unstructured logs in third party components). Our approach to deal with logs is to use ML to structure them after the fact (and we can deal with changing log structures). You can read about it in a couple of our blogs like: https://www.zebrium.com/blog/using-ml-to-auto-learn-changing-log-structures https://www.zebrium.com/blog/using-ml-to-auto-learn-changing... and https://www.zebrium.com/blog/please-dont-make-me-structure-logs https://www.zebrium.com/blog/please-dont-make-me-structure-l....

by buro9 6y ago

kostarelo 6y ago

> Don't worry about structured vs unstructured too much, just ensure you include a timestamp, file, log level, func name (or line number), and that the message will help you debug. if you include all these information and the logs are not structured, you won't get much information out of them.

deepGem 6y ago

From the post "The key, he says, is using the right transaction identifiers so that calls can be traced across components, services, and queues". I think this is a key feature not many people implement especially in today's world of over blown micro services, having a transaction id from the time the request hits the reverse-proxy till the database write is so helpful in debugging, saves a ton of time.

KaiserPro 6y ago
100% if you manage to get opentrace to work, it is a brilliant debug tool
- eatonphil 6y ago
  You can also just propagate a uuid throughout your system. I've used both uuids and opentrace. Just don't get hung up if opentrace seems too complex.
  - dkarl 6y ago
    I agree with this wholeheartedly. You can even define a standard and let downstream services opt in over time. Simple wins like this should not be put off because "someday" we're going to implement a complex distributed tracing solution.
    - dillonmckay 6y ago
      So, in terms of full implementation, is the constraint the dev time to implement, or getting various factions to agree to something? I guess, is it political, or technical? Asking for a friend. Thanks!
      - nitrogen 6y ago
        Often it's political, but political friction can feed into technical friction if there are also a half dozen different half wrappers around half-baked HTTP libraries. Also, if someone has included OpenTracing as a shadow library in a company-wide library (JVM territory), but you want it as a top-level dependency, you have to write translators.

viraptor 6y ago

> Grafana Cloud offers a hosted Grafana, Prometheus metrics scraping and storage, and Log tailing and storage (via Loki) I haven't looked at their pricing before, but for small-ish environments, their standard plan looks really good and simple. None of the "per host, but also per function, and extra for each feature, and extra for usage" approach like other providers (datadog, I'm looking at you).

rudasn 6y ago
Twice I fell into the trap of datadog, having paid more than $300 for a single month in each case. The simplicity of it, dashboards, notebooks, logs etc, is what makes it so appealing though.
- count 6y ago
  If it saves you more than an hour or two, $300 seems super worth it for a system making you money...
  - viraptor 6y ago
    If you're in an established company making money - yes. If you're bootstrapping a service and counting on $50 total monthly cost while initial users are signing up - no.
  - smoe 6y ago
    It depends where you are in the world. When I was working in Switzerland, most SaaS pricing were no-brainers for us. But since I work in Latin America for small companies with local costumers, all the different services and tools you might want to use, with prices targeted at "western" customers, much more quickly add up to the equivalent of having multiple people on staff full time. Still it is often not worth to roll your own, so it is nice to have alternatives for different price points and company scales.
    - rudasn 6y ago
      > It depends where you are in the world. Exactly this. We operate in Eastern Europe with local clients, offering on-prem SaaS. If I added all my clients' servers on datadog it would very easily eat through our profit margins. > Still it is often not worth to roll your own I tried hard not to, but at at the end, after spending 1 week trying to setup netdata and failed, I decided not to spend another week trying to setup grafana/influx/prometheus (lot's of docs to go through), and just have some bash scripts send metrics on a $10 digital ocean node service that sends me emails/sms when something "looks bad" (eg. high cpu temperature, stopped docker containers, etc). I gave up on aggregated logging for the time being, since I can just ssh on each server and check journal and docker logs if I need to (as long as the hard drives don't crash).

dkersten 6y ago

I’d add to no. 1 by saying include a correlation id or request id so you have a way to filter the logs into a single linear stream related to the same action.

jrott 6y ago
Absolutely constantly being able to get a single linear stream is when tracing becomes super powerful.
Bombthecat 6y ago
Also adding a context tag can very powerful, especially in times of Microservices and some kind of event driven stuff like monthly payments. Imagine: user registers, the post gets and id, and context registering. Then he adds a credit card. (a new id, context credit payments) after 14 days the bill goes out, same id, same context.

ysoft 6y ago

I don't think loki is production ready. Needs more work. It's going to be great with grafana though

yellow_mixer 6y ago
What else do you think it needs. I’ve been playing around with v1.5 the past few days in Grafana and I like it’s simplicity. Grafana now has a Loki datasource which is nice.

chandraonline 6y ago

I would add that if you add a traceId to #1 and use something like https://www.jaegertracing.io/ https://www.jaegertracing.io/. You get even more.

sciurus 6y ago

I agree with 2. I have a presentation at https://www.polibyte.com/2019/02/10/pytennessee-presentation/ https://www.polibyte.com/2019/02/10/pytennessee-presentation... which goes into how to get started with Prometheus. Not as in "how to set it up", but more about what to instrument and why, how to name things, etc. Despite the title, there's very little in it that's specific to Python.

gnufx 6y ago

Why should you have to use Prometheus? There are plenty of options, and good reasons why you might want to push data rather than pull. Measurements should minimize perturbation of the system being measured, and the (computer) system generating data is likely best placed to determine when and how, when that matters -- e.g. in HPC, where jitter is important.

gdcohen 6y ago

Gavin from Zebrium here. Completely concur with #1. We are big advocates of writing good logs and not having to worry about structured vs unstructured (and even if you structure your logs, you'll still probably have to deal with unstructured logs in third party components). Our approach to deal with logs is to use ML to structure them after the fact (and we can deal with changing log structures). You can read about it in a couple of our blogs like: https://www.zebrium.com/blog/using-ml-to-auto-learn-changing-log-structures https://www.zebrium.com/blog/using-ml-to-auto-learn-changing... and https://www.zebrium.com/blog/please-dont-make-me-structure-logs https://www.zebrium.com/blog/please-dont-make-me-structure-l....