5 ms·

I’m of the opposite opinion. It’s better to ingest everything and then filter out the stuff you don’t want at the observability platform. The problem of filter

I’m of the opposite opinion. It’s better to ingest everything and then filter out the stuff you don’t want at the observability platform. The problem of filtering out debug logs is you don’t need them, until you do. And then trying to recreate an event you can’t even debug is often impossible. So it’s easier to then retrieve those debug logs if they’re already there but hidden.

gavinray 1y ago
"Better to have it and not need it; than to need it, and not have it..."
- jkogara 1y ago
  Or more succinctly, albeit less eloquently: "Better to be looking at it than looking for it."
- 9dev 1y ago
  Until you’re working with personal information of EU customers, where the opposite maxime applies: "Only store what you absolutely need" Seriously, storing petabytes of logs is a guarantee for someone on your team writing sensitive data to logs, and/or violate regulations.
- jodrellblank 1y ago
  “You can’t have everything. Where would you put it?” - Steven Wright. “Better to have hoarding disorder than to need a fifty year old carrier bag full of rotting bus tickets and not have one” really should need more justification than a quote about how convenient it is to have what you need. The reason caches exist as a thing is so you can have what you probably need handy because you can’t have everything handy and have to choose. The amount of things you might possibly want or need one day - including unforeseen needs - is unbounded, and refusing to make a decision is not good engineering, it’s a cop-out. Apart from cost, the more time and money you spend indexing, cataloging, searching it. How many companies are going to run an internal Google-2002 sized infrastructure just to search their old hoarded data?
  - gavinray 1y ago
    I'm not sure what poor engineering practices you have seen, but in my painfully-gotten experience, application of this principle usually amounts to having varying levels of a debug log flag that dump this info either to stdout via JSONL that's piped somewhere, or as attributes in OTEL spans. This has never been a source of significant issues for me.
  - hnlmorg 1y ago
    This is a really easy problem to solve. Step one: add log severity to your log messages (pretty much every log library supports this out of the box). Step two: add a log archive (you should have this anyway so that logs can be retained past the initial retention period of your log querying tools. Eg you might have a compliance requirement to keep logs for two years but you obviously wouldn’t want anything that old stored in your expensive fast log search) Step three: create a way to ingest your archived logs (again, something your business should have, otherwise what’s the bloody point in having an archive) Step four: have a rule that pushes logs of high severity straight into your log ingestion pipeline, and logs of lower severity into your archive. Step four seems to be the piece that most people are oblivious too. But it’s generally really easy to implement. Particularly so if you’re using a reputable observability platform. People who think “log everything” means “log PII” or “stick everything in the same log ingestion pipeline” are simply doing logging wrong. I’m not normally one to say “you’re doing it wrong” but when it comes to logging, these tools are long since mature now. The problem isn’t the tooling, it’s people’s awareness of it.
- lelanthran 1y ago
  > "Better to have it and not need it; than to need it, and not have it..." Having it is pointless if your SNR is so low that it costs more money than simply waiting for the bug the next time it comes up. IMO, if a bug never surfaces again, that's not a bug I care about anyway. Keeping all generated data in case someone wants to see the record from a bug 3 months ago is absolutely pointless - if it hasn't surfaced again in the last three weeks, you absolutely have more high-priority things to look at! I want to see this mythical company, where a paid employee is dedicated by the company to look at a log from 3 months ago, to solve a bug that hasn't resurfaced in that three month period!
hinkley 1y ago
Once a bug is closed the value of those logs starts to decay. And the fact is that we get punished for working on things that aren’t in the sprint, and working on “done done” stories is one of those ways. Even if you want to clean up your mess, there’s incentive not to. And many of us very clearly don’t like to clean up our own messes, so a ready excuse gets them out of the conversation about a task they don’t want to be voluntold to do.
- pstuart 1y ago
  My approach for this is to add dev logging IN ALL CAPS so that it stands out as ugly and "need adjusting", which is to delete it before merging to main.
  - hinkley 1y ago
    On my last project I was able to convince the team to clean up feature toggles before closing out epics. But I didn’t make much headway on logs. I came at them sideways and got all but one of my coworkers to stop trying to generate charts from Splunk and use Grafana instead. And I squeezed him by adding stats for things he liked to look at b
- hnlmorg 1y ago
  In DevOps (et al) the value of those logs doesn’t decay in the same way it does in pure dev. Also, as I pointed out elsewhere, modern observability platforms enable a way to have those debug logs available as an archive that can be optionally ingested after an incident but without filling up your regular quota of indexed logs. Thus giving you the best of both worlds (all logging but without the expense and flooding your daily logs with debug messages)
  - hinkley 1y ago
    > In DevOps (et al) the value of those logs doesn’t decay in the same way it does in pure dev. I’ve been on-call, and I think you’re cherry picking. The world has too many devs who still debug with log statements. Those logs never had any value to anyone but the original author. I’ve also seen too many devs who are perfectly happy trying to write vastly complex Splunk queries to generate charts, and those charts tend to break in a production incident becausea bunch of people load them at once and blow up Splunk’s rate limiting. I’ve almost never had this problem with grafana. It’s true that you can make a dashboard with long-term trends that will fall over, but you wouldn’t use that dashboard for triage, unless you make one that tries to do both and the solution is split it into two dashboards. If you want to make a successfully scaling organization, you need a way for new members to join your core of troubleshooters, without pulling resources away from solving the trouble. So they can’t demand time, resources or attention that are in short supply from the core group. Grafana fits that yardstick much better than log analyzers.
UltraSane 1y ago
You really need to define how much you are willing to spend on logging/observability as a percentage of total budget. IMHO 5% is bare minimum 10% is better. I've worked for a company that had a dedicated storage array just for logging with Splunk and it was amazing and very much worth the money. Good automatic tiering for logs is very useful as the most recent logs tend to be the most useful. I like NVMe -> hard disk -> tape library. LTO tape storage is cheap enough you don't need to delete data until it is VERY old.
nikolayasdf123 1y ago
that looks like tail-error sampling
jgalt212 1y ago
> then filter out the stuff you don’t want This is often easier said than done. And there's ginormous costs associated with logging everything. Money that can be better spent elsewhere. Also, logging everything creates yet another security hole to worry about.
- phillipcarter 1y ago
  > And there's ginormous costs associated with logging everything If you use a tool that defaults the log spew to a cheap archive, sampling to the fast store, and a way to pull from the archive on-demand much of that is resolved. FWIW I think most orgs get big scared at seeing $$$ in their cloud bills, but don't properly account for time spent by engineers rummaging around for data they need but don't have.
  - nijave 1y ago
    >but don't properly account for time spent by engineers rummaging around for data they need but don't have This is a tricky one that's come up recently. How you you quantify the value of $$$ observability platform? Anecdotally I know robust tracing data can help me find problems in 5-15 minutes that would have taken hours or days with manual probing and scouring logs. Even then you have the additional challenge of quantifying the impact of the original issue.
    - phillipcarter 1y ago
      At the end of the day it's just vibes. If the company is one that sees: - Reliability as a cost center - Vendor costs are to be limited - CIO-driven rather than CTO-driven Then it's going to be a given that they prioritize costs that are easy to see, and will do things like force a dev team to work for a month to shave ~2k/month off of a cloud bill. In my experience, these orgs will also sometimes do a 180 when they learn that their SLAs involve paying out to customers at a premium during incidents, which is always very funny to observe. Then you talk to some devs and they say things like "we literally told them this would happen years ago and it fell on deaf ears" or something.
    - 1y ago

gavinray 1y ago

"Better to have it and not need it; than to need it, and not have it..."

jkogara 1y ago
Or more succinctly, albeit less eloquently: "Better to be looking at it than looking for it."
9dev 1y ago
Until you’re working with personal information of EU customers, where the opposite maxime applies: "Only store what you absolutely need" Seriously, storing petabytes of logs is a guarantee for someone on your team writing sensitive data to logs, and/or violate regulations.
jodrellblank 1y ago
“You can’t have everything. Where would you put it?” - Steven Wright. “Better to have hoarding disorder than to need a fifty year old carrier bag full of rotting bus tickets and not have one” really should need more justification than a quote about how convenient it is to have what you need. The reason caches exist as a thing is so you can have what you probably need handy because you can’t have everything handy and have to choose. The amount of things you might possibly want or need one day - including unforeseen needs - is unbounded, and refusing to make a decision is not good engineering, it’s a cop-out. Apart from cost, the more time and money you spend indexing, cataloging, searching it. How many companies are going to run an internal Google-2002 sized infrastructure just to search their old hoarded data?
- gavinray 1y ago
  I'm not sure what poor engineering practices you have seen, but in my painfully-gotten experience, application of this principle usually amounts to having varying levels of a debug log flag that dump this info either to stdout via JSONL that's piped somewhere, or as attributes in OTEL spans. This has never been a source of significant issues for me.
- hnlmorg 1y ago
  This is a really easy problem to solve. Step one: add log severity to your log messages (pretty much every log library supports this out of the box). Step two: add a log archive (you should have this anyway so that logs can be retained past the initial retention period of your log querying tools. Eg you might have a compliance requirement to keep logs for two years but you obviously wouldn’t want anything that old stored in your expensive fast log search) Step three: create a way to ingest your archived logs (again, something your business should have, otherwise what’s the bloody point in having an archive) Step four: have a rule that pushes logs of high severity straight into your log ingestion pipeline, and logs of lower severity into your archive. Step four seems to be the piece that most people are oblivious too. But it’s generally really easy to implement. Particularly so if you’re using a reputable observability platform. People who think “log everything” means “log PII” or “stick everything in the same log ingestion pipeline” are simply doing logging wrong. I’m not normally one to say “you’re doing it wrong” but when it comes to logging, these tools are long since mature now. The problem isn’t the tooling, it’s people’s awareness of it.
lelanthran 1y ago
> "Better to have it and not need it; than to need it, and not have it..." Having it is pointless if your SNR is so low that it costs more money than simply waiting for the bug the next time it comes up. IMO, if a bug never surfaces again, that's not a bug I care about anyway. Keeping all generated data in case someone wants to see the record from a bug 3 months ago is absolutely pointless - if it hasn't surfaced again in the last three weeks, you absolutely have more high-priority things to look at! I want to see this mythical company, where a paid employee is dedicated by the company to look at a log from 3 months ago, to solve a bug that hasn't resurfaced in that three month period!

hinkley 1y ago

Once a bug is closed the value of those logs starts to decay. And the fact is that we get punished for working on things that aren’t in the sprint, and working on “done done” stories is one of those ways. Even if you want to clean up your mess, there’s incentive not to. And many of us very clearly don’t like to clean up our own messes, so a ready excuse gets them out of the conversation about a task they don’t want to be voluntold to do.

pstuart 1y ago
My approach for this is to add dev logging IN ALL CAPS so that it stands out as ugly and "need adjusting", which is to delete it before merging to main.
- hinkley 1y ago
  On my last project I was able to convince the team to clean up feature toggles before closing out epics. But I didn’t make much headway on logs. I came at them sideways and got all but one of my coworkers to stop trying to generate charts from Splunk and use Grafana instead. And I squeezed him by adding stats for things he liked to look at b
hnlmorg 1y ago
In DevOps (et al) the value of those logs doesn’t decay in the same way it does in pure dev. Also, as I pointed out elsewhere, modern observability platforms enable a way to have those debug logs available as an archive that can be optionally ingested after an incident but without filling up your regular quota of indexed logs. Thus giving you the best of both worlds (all logging but without the expense and flooding your daily logs with debug messages)
- hinkley 1y ago
  > In DevOps (et al) the value of those logs doesn’t decay in the same way it does in pure dev. I’ve been on-call, and I think you’re cherry picking. The world has too many devs who still debug with log statements. Those logs never had any value to anyone but the original author. I’ve also seen too many devs who are perfectly happy trying to write vastly complex Splunk queries to generate charts, and those charts tend to break in a production incident becausea bunch of people load them at once and blow up Splunk’s rate limiting. I’ve almost never had this problem with grafana. It’s true that you can make a dashboard with long-term trends that will fall over, but you wouldn’t use that dashboard for triage, unless you make one that tries to do both and the solution is split it into two dashboards. If you want to make a successfully scaling organization, you need a way for new members to join your core of troubleshooters, without pulling resources away from solving the trouble. So they can’t demand time, resources or attention that are in short supply from the core group. Grafana fits that yardstick much better than log analyzers.

UltraSane 1y ago

You really need to define how much you are willing to spend on logging/observability as a percentage of total budget. IMHO 5% is bare minimum 10% is better. I've worked for a company that had a dedicated storage array just for logging with Splunk and it was amazing and very much worth the money. Good automatic tiering for logs is very useful as the most recent logs tend to be the most useful. I like NVMe -> hard disk -> tape library. LTO tape storage is cheap enough you don't need to delete data until it is VERY old.

nikolayasdf123 1y ago

that looks like tail-error sampling

jgalt212 1y ago

> then filter out the stuff you don’t want This is often easier said than done. And there's ginormous costs associated with logging everything. Money that can be better spent elsewhere. Also, logging everything creates yet another security hole to worry about.

phillipcarter 1y ago
> And there's ginormous costs associated with logging everything If you use a tool that defaults the log spew to a cheap archive, sampling to the fast store, and a way to pull from the archive on-demand much of that is resolved. FWIW I think most orgs get big scared at seeing $$$ in their cloud bills, but don't properly account for time spent by engineers rummaging around for data they need but don't have.
- nijave 1y ago
  >but don't properly account for time spent by engineers rummaging around for data they need but don't have This is a tricky one that's come up recently. How you you quantify the value of $$$ observability platform? Anecdotally I know robust tracing data can help me find problems in 5-15 minutes that would have taken hours or days with manual probing and scouring logs. Even then you have the additional challenge of quantifying the impact of the original issue.
  - phillipcarter 1y ago
    At the end of the day it's just vibes. If the company is one that sees: - Reliability as a cost center - Vendor costs are to be limited - CIO-driven rather than CTO-driven Then it's going to be a given that they prioritize costs that are easy to see, and will do things like force a dev team to work for a month to shave ~2k/month off of a cloud bill. In my experience, these orgs will also sometimes do a 180 when they learn that their SLAs involve paying out to customers at a premium during incidents, which is always very funny to observe. Then you talk to some devs and they say things like "we literally told them this would happen years ago and it fell on deaf ears" or something.
  - 1y ago