3 ms·

another factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "fina

another factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "final" scrape and the actual process exit where any recorded metrics won't get propagated. this may give you a false impression about whether there are any errors occurring during the shutdown sequence. it's also possible, if you're not careful, to lose the last few seconds of logs from when your service is shutting down. for example, if you write to a log file that is watched by a sidecar process such as Promtail or Vector, and on startup the service truncates and starts writing to that same path, you've got a race condition that can cause you to lose logs from the shutdown.

tmpz22 1y ago
Is it me or are observability stacks kind of ridiculous. Logs, metrics, and traces, each with their own databases, sidecars, visualization stacks. Language-specific integration libraries written by whoever felt like it. MASSIVE cloud bills. Then after you go through all that effort most of the data is utterly ignored and rarely are the business insights much better then the trailer park version ssh'ing into a box and greping a log file to find the error output. Like we put so much effort into this ecosystem but I don't think it has paid us back with any significant increase in uptime, performance, or ergonomics.
- nkraft11 1y ago
  I can say that going from a place that had all of that observability tooling set up to one that was at the "ssh'ing into a box and greping a log" stage, you best believe I missed company A immensely. Even knowing which box to ssh into, which log file to grep, and which magic words to search far was nigh impossible if you weren't the dev that set up the machine and wrote the bug in the first place.
  - MortyWaves 1y ago
    I completely agree with you but I also think, like many aspects of "tech" certain segments of it have been monopolised and turned into profit generators for certain organisations. DevOps, Agile/Scrum, Observability, Kubernetes, are all examples of this. This dilutes the good and helpful stuff with marketing bullshit. Grafana seemingly inventing new time series databases and engines every few months is absolutely painful to try keep up to date with in order to make informed decisions. So much so I've started using rrdtool/smokeping again.
    - bbkane 1y ago
      You might look into https://openobserve.ai/ https://openobserve.ai/ - you can self host it and it's a single binary that ingests logs/metrics/traces. I've found it useful for my side projects.
- 01HNNWZ0MV43FF 1y ago
  Programs are for people. That's why we got JSON, a bunch of debuggers, Python, and so on. Programming is only like 10 percent of programming
- evil-olive 1y ago
  if you're working on a system simple enough that "SSH to the box and grep the log file" works, then by all means have at it. but many systems are more complicated than that. the observability ecosystem exists for a reason, there is a real problem that it's solving. for example, your app might outgrow running on a single box. now you need to SSH into N different hosts and grep the log file from all of them. or you invent your own version of log-shipping with a shell script that does SCP in a loop. going a step further, you might put those boxes into an auto-scaling group so that they would scale up and down automatically based on demand. now you really want some form of automatic log-shipping, or every time a host in the ASG gets terminated, you're throwing away the logs of whatever traffic it served during its lifetime. or, maybe you notice a performance regression and narrow it down to one particular API endpoint being slow. often it's helpful to be able to graph the response duration of that endpoint over time. has it been slowing down gradually, or did the response time increase suddenly? if it was a sudden increase, what else happened around the same time? maybe a code deployment, maybe a database configuration change, etc. perhaps the service you operate isn't standalone, but instead interacts with services written by other teams at your company. when something goes wrong with the system as a whole, how do you go about root-causing the problem? how do you trace the lifecycle of a request or operation through all those different systems? when something goes wrong, you SSH to the box and look at the log file...but how do you know something went wrong to begin with? do you rely solely on user complaints hitting your support@ email? or do you have monitoring rules that will proactively notify you if a "huh, that should never happen" thing is happening?
- HdS84 1y ago
  Overall, I think centralized logging and metrics are super valuable. But stacks are all missing the mark. For example, every damn log message has hundreds of fields,. Most of which never change. Why not push this information once, on service startup an not with every log message? OK, obviously the current system provides huge bills to the benefit of the company or's offering these services.
  - valyala 1y ago
    > For example, every damn log message has hundreds of fields,. Most of which never change. Why not push this information once, on service startup an not with every log message? If the log field doesn't change with every log entry, then good databases for logs (such as VictoriaLogs) compress such a field by 1000x and more times, so its' storage space usage can be ignored, and it doesn't affect query performance in any way. Storing many fields per every log entry simplifies further analysis of these logs, since you can get all the needed information from a single log entry instead of jumping over big number of interconnected logs. This also improves analysis of logs at scale by filtering and grouping the logs by any subset of numerous fields. Such logs with big number of fields are named "wide events". See the following excellent article about this type of logs - https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/ https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide... .
- openWrangler 1y ago
  It's not just you - OSS toolstacks can be sprawling and involve long manual processes while costs from most enterprise vendors are too steep for fully mapped observability. Coroot is an open source project I'm working with to try and to tackle this. eBPF automatically gathers your data into a centralized service map, and then the tool provides RCA insights (with things like mapped incident timeframes) to help implement fixes quicker and improve uptime. GitHub here and we'd love any feedback if you think it can help: https://github.com/coroot/coroot https://github.com/coroot/coroot
utrack 1y ago
Jfyi, I'm doing exactly this (and more) in a platform library; it covers the issues I've encountered during the last 8+ years I've been working with Go highload apps. During this time developing/improving the platform and rolling was a hobby of mine in every company :) It (will) cover the stuff like "sync the logs"/"wait for ingresses to catch up with the liveness handler"/etc. https://github.com/utrack/caisson-go/blob/main/caiapp/caiapp.go#L87 https://github.com/utrack/caisson-go/blob/main/caiapp/caiapp... https://github.com/utrack/caisson-go/tree/main/closer https://github.com/utrack/caisson-go/tree/main/closer The docs are sparse and some things aren't covered yet; however I'm planning to do the first release once I'm back from a holiday. In the end, this will be a meta-platform (carefully crafted building blocks), and a reference platform library, covering a typical k8s/otel/grpc+http infrastructure.
- peterldowns 1y ago
  I'll check this out, thanks for sharing. I think all of us golang infra/platform people probably have had to write our own similar libraries. Thanks for sharing yours!
RainyDayTmrw 1y ago
I never understood why Prometheus and related use a "pull" model for data, when most things use a "push" model.
- evil-olive 1y ago
  Prometheus doesn't necessarily lock you into the "pull" model, see [0]. however, there are some benefits to the pull model, which is why I think Prometheus does it by default. with a push model, your service needs to spawn a background thread/goroutine/whatever that pushes metrics on a given interval. if that background thread crashes or hangs, metrics from that service instance stop getting reported. how do you detect that, and fire an alert about it happening? "cloud-native" gets thrown around as a buzzword, but this is an example where it's actually meaningful. Prometheus assumes that whatever service you're trying to monitor, you're probably already registering each instance in a service-discovery system of some kind, so that other things (such as a load-balancer) know where to find it. you tell Prometheus how to query that service-discovery system (Kubernetes, for example [1]) and it will automatically discover all your service instances, and start scraping their /metrics endpoints. this provides an elegant solution to the "how do you monitor a service that is up and running, except its metrics-reporting thread has crashed?" problem. if it's up and running, it should be registered for service-discovery, and Prometheus can trivially record (this is the `up` metric) if it discovers a service but it's not responding to /metrics requests. and this greatly simplifies the client-side metrics implementation, because you don't need a separate metrics thread in your service. you don't need to ensure it runs forever and never hangs and always retries and all that. you just need to implement a single HTTP GET endpoint, and have it return text in a format simple enough that you can sprintf it yourself if you need to. for a more theoretical understanding, you can also look at it in terms of the "supervision trees" popularized by Erlang. parents monitor their children, by pulling status from them. children are not responsible for pushing status reports to their parents (or siblings). with the push model, you have a supervision graph instead of a supervision tree, with all the added complexity that entails. 0: https://prometheus.io/docs/instrumenting/pushing/ https://prometheus.io/docs/instrumenting/pushing/ 1: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config https://prometheus.io/docs/prometheus/latest/configuration/c...
PrayagS 1y ago
> another factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "final" scrape and the actual process exit where any recorded metrics won't get propagated. this may give you a false impression about whether there are any errors occurring during the shutdown sequence. Have you come across any convenient solution for this? If my scrape interval is 15 seconds, I don't exactly have 30 seconds to record two scrapes. This behavior has sort of been the reason why our services still use statsd since the push-based model doesn't see this problem.

tmpz22 1y ago

Is it me or are observability stacks kind of ridiculous. Logs, metrics, and traces, each with their own databases, sidecars, visualization stacks. Language-specific integration libraries written by whoever felt like it. MASSIVE cloud bills. Then after you go through all that effort most of the data is utterly ignored and rarely are the business insights much better then the trailer park version ssh'ing into a box and greping a log file to find the error output. Like we put so much effort into this ecosystem but I don't think it has paid us back with any significant increase in uptime, performance, or ergonomics.

nkraft11 1y ago
I can say that going from a place that had all of that observability tooling set up to one that was at the "ssh'ing into a box and greping a log" stage, you best believe I missed company A immensely. Even knowing which box to ssh into, which log file to grep, and which magic words to search far was nigh impossible if you weren't the dev that set up the machine and wrote the bug in the first place.
- MortyWaves 1y ago
  I completely agree with you but I also think, like many aspects of "tech" certain segments of it have been monopolised and turned into profit generators for certain organisations. DevOps, Agile/Scrum, Observability, Kubernetes, are all examples of this. This dilutes the good and helpful stuff with marketing bullshit. Grafana seemingly inventing new time series databases and engines every few months is absolutely painful to try keep up to date with in order to make informed decisions. So much so I've started using rrdtool/smokeping again.
  - bbkane 1y ago
    You might look into https://openobserve.ai/ https://openobserve.ai/ - you can self host it and it's a single binary that ingests logs/metrics/traces. I've found it useful for my side projects.
01HNNWZ0MV43FF 1y ago
Programs are for people. That's why we got JSON, a bunch of debuggers, Python, and so on. Programming is only like 10 percent of programming
evil-olive 1y ago
if you're working on a system simple enough that "SSH to the box and grep the log file" works, then by all means have at it. but many systems are more complicated than that. the observability ecosystem exists for a reason, there is a real problem that it's solving. for example, your app might outgrow running on a single box. now you need to SSH into N different hosts and grep the log file from all of them. or you invent your own version of log-shipping with a shell script that does SCP in a loop. going a step further, you might put those boxes into an auto-scaling group so that they would scale up and down automatically based on demand. now you really want some form of automatic log-shipping, or every time a host in the ASG gets terminated, you're throwing away the logs of whatever traffic it served during its lifetime. or, maybe you notice a performance regression and narrow it down to one particular API endpoint being slow. often it's helpful to be able to graph the response duration of that endpoint over time. has it been slowing down gradually, or did the response time increase suddenly? if it was a sudden increase, what else happened around the same time? maybe a code deployment, maybe a database configuration change, etc. perhaps the service you operate isn't standalone, but instead interacts with services written by other teams at your company. when something goes wrong with the system as a whole, how do you go about root-causing the problem? how do you trace the lifecycle of a request or operation through all those different systems? when something goes wrong, you SSH to the box and look at the log file...but how do you know something went wrong to begin with? do you rely solely on user complaints hitting your support@ email? or do you have monitoring rules that will proactively notify you if a "huh, that should never happen" thing is happening?
HdS84 1y ago
Overall, I think centralized logging and metrics are super valuable. But stacks are all missing the mark. For example, every damn log message has hundreds of fields,. Most of which never change. Why not push this information once, on service startup an not with every log message? OK, obviously the current system provides huge bills to the benefit of the company or's offering these services.
- valyala 1y ago
  > For example, every damn log message has hundreds of fields,. Most of which never change. Why not push this information once, on service startup an not with every log message? If the log field doesn't change with every log entry, then good databases for logs (such as VictoriaLogs) compress such a field by 1000x and more times, so its' storage space usage can be ignored, and it doesn't affect query performance in any way. Storing many fields per every log entry simplifies further analysis of these logs, since you can get all the needed information from a single log entry instead of jumping over big number of interconnected logs. This also improves analysis of logs at scale by filtering and grouping the logs by any subset of numerous fields. Such logs with big number of fields are named "wide events". See the following excellent article about this type of logs - https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/ https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide... .
openWrangler 1y ago
It's not just you - OSS toolstacks can be sprawling and involve long manual processes while costs from most enterprise vendors are too steep for fully mapped observability. Coroot is an open source project I'm working with to try and to tackle this. eBPF automatically gathers your data into a centralized service map, and then the tool provides RCA insights (with things like mapped incident timeframes) to help implement fixes quicker and improve uptime. GitHub here and we'd love any feedback if you think it can help: https://github.com/coroot/coroot https://github.com/coroot/coroot

utrack 1y ago

Jfyi, I'm doing exactly this (and more) in a platform library; it covers the issues I've encountered during the last 8+ years I've been working with Go highload apps. During this time developing/improving the platform and rolling was a hobby of mine in every company :) It (will) cover the stuff like "sync the logs"/"wait for ingresses to catch up with the liveness handler"/etc. https://github.com/utrack/caisson-go/blob/main/caiapp/caiapp.go#L87 https://github.com/utrack/caisson-go/blob/main/caiapp/caiapp... https://github.com/utrack/caisson-go/tree/main/closer https://github.com/utrack/caisson-go/tree/main/closer The docs are sparse and some things aren't covered yet; however I'm planning to do the first release once I'm back from a holiday. In the end, this will be a meta-platform (carefully crafted building blocks), and a reference platform library, covering a typical k8s/otel/grpc+http infrastructure.

peterldowns 1y ago
I'll check this out, thanks for sharing. I think all of us golang infra/platform people probably have had to write our own similar libraries. Thanks for sharing yours!

RainyDayTmrw 1y ago

I never understood why Prometheus and related use a "pull" model for data, when most things use a "push" model.

evil-olive 1y ago
Prometheus doesn't necessarily lock you into the "pull" model, see [0]. however, there are some benefits to the pull model, which is why I think Prometheus does it by default. with a push model, your service needs to spawn a background thread/goroutine/whatever that pushes metrics on a given interval. if that background thread crashes or hangs, metrics from that service instance stop getting reported. how do you detect that, and fire an alert about it happening? "cloud-native" gets thrown around as a buzzword, but this is an example where it's actually meaningful. Prometheus assumes that whatever service you're trying to monitor, you're probably already registering each instance in a service-discovery system of some kind, so that other things (such as a load-balancer) know where to find it. you tell Prometheus how to query that service-discovery system (Kubernetes, for example [1]) and it will automatically discover all your service instances, and start scraping their /metrics endpoints. this provides an elegant solution to the "how do you monitor a service that is up and running, except its metrics-reporting thread has crashed?" problem. if it's up and running, it should be registered for service-discovery, and Prometheus can trivially record (this is the `up` metric) if it discovers a service but it's not responding to /metrics requests. and this greatly simplifies the client-side metrics implementation, because you don't need a separate metrics thread in your service. you don't need to ensure it runs forever and never hangs and always retries and all that. you just need to implement a single HTTP GET endpoint, and have it return text in a format simple enough that you can sprintf it yourself if you need to. for a more theoretical understanding, you can also look at it in terms of the "supervision trees" popularized by Erlang. parents monitor their children, by pulling status from them. children are not responsible for pushing status reports to their parents (or siblings). with the push model, you have a supervision graph instead of a supervision tree, with all the added complexity that entails. 0: https://prometheus.io/docs/instrumenting/pushing/ https://prometheus.io/docs/instrumenting/pushing/ 1: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config https://prometheus.io/docs/prometheus/latest/configuration/c...

PrayagS 1y ago

> another factor to consider is that if you have a typical Prometheus `/metrics` endpoint that gets scraped every N seconds, there's a period in between the "final" scrape and the actual process exit where any recorded metrics won't get propagated. this may give you a false impression about whether there are any errors occurring during the shutdown sequence. Have you come across any convenient solution for this? If my scrape interval is 15 seconds, I don't exactly have 30 seconds to record two scrapes. This behavior has sort of been the reason why our services still use statsd since the push-based model doesn't see this problem.