10 ms·

turning on logging post-facto won't go back in time to catch a rare bug. And since you need the added resources that logging requires to be available at all ti

turning on logging post-facto won't go back in time to catch a rare bug. And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs?

jedberg 11y ago
> turning on logging post-facto won't go back in time to catch a rare bug I guess my point is that if a rare bug is rare enough, then it doesn't really matter, especially if the effect isn't catastrophic. > And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs? There are a lot more resources required to log everything all the time then to log a subset of things some of the time. Even having recent logs of everything is a huge overhead compared to more detailed logs of just some things.
- nostrademons 11y ago
  That works if you're building Reddit, Netflix, or even most Google services. It doesn't work if you're doing financial software or anything that handles money or is critical for someone else's business.
  - jedberg 11y ago
    I haven't looked into it, so I'm not sure, but I'm also not yet convinced that server logs can increase the integrity of banking transactions. I'd love to get some more info on specifics for this. For example, a lot of people just assume that a bank can't be eventually consistent, but if you make them stop and think about it, they realize that they can and are. The ATM isn't always connected to the bank's servers. That's why your card has a withdrawl limit. That's how much they are willing to risk for eventual consistency. I feel like the same is true for logging. That other checks can be put in place to mitigate risk to an acceptable level.
    - nostrademons 11y ago
      It's not that the logs themselves increase the integrity of banking transactions, it's that they find bugs that increase the integrity of banking transactions. Say that you've built your system like any good highly-available system so that you rigorously check consistency of the program invariants, log if there are any anomalies, and then either try to recover or back out with a user error if something goes wrong. You've configured your monitoring to alert if any of the consistency checks fail. Now you get an alert that you've errored out and served a 500 on one in every million requests. For Reddit, Facebook, Google, or other free services, it's no big deal: one in a million means that you're serving a couple hundred, maybe a few thousand 500s per day, max, and then the user just refreshes and gets over it. But in a financial transaction app, this is a big deal. The basic assumption you have to make is that if you have a consistency problem that your monitoring caught, you probably have consistency problems that you didn't catch. And the right thing to do is investigate until you understand what is going on and what the impact is. So you pull the logs of all surrounding requests, you pull the logs of any requests that were in flight at the same time, you pull the logs for the user that initiated the request and see exactly what they were doing. And then you try to reproduce the problem, test for it, and fix it. The difference is entirely in the tolerances built into the system, and how that dictates that you respond to errors. In some domains, a one-in-a-million error is a "well, we'll catch it next time" event. In others, a one-in-a-million error is a "drop everything you're doing, fix it, and ensure that no similar errors exist" event. BTW, the strictest tolerances usually are not in B2C companies, they are in suppliers of large B2C companies (or the government). The bank is completely willing to risk a few hundred dollars per customer on an ATM card, but if the ATM vendor loses that money without it being part of the spec they gave the bank, they've lost the contract. Most of these enterprise B2B contracts are based on trust, and if the customer observes you losing money accidentally, they'll wonder what else you're doing accidentally.
- xorcist 11y ago
  > I guess my point is that if a rare bug is rare enough, then it doesn't really matter, especially if the effect isn't catastrophic. That's only true if you run a trivial application. If you are a bank for example, it matters very much why the outcome of a transaction came out wrong, even if the effect wasn't catastrophic. Every circumstance and every bit of identity to the transaction is important. "It's rare" or "I don't believe in logging" doesn't cut it. > There are a lot more resources required to log everything all the time then to log a subset of things some of the time. Resources are a matter of budgetary concerns, mostly. What does a bug in production cost you? Are you liable? How fast? What's your SLA? Those things matter.
  - jedberg 11y ago
    > Resources are a matter of budgetary concerns, mostly. True, but most people unfortunately don't have unlimited budgets to work with. Also, there is more than a monetary cost. Every system in your architecture incurs a non-zero cost to maintenance, reliability, cognitive load, and management. For example, if you are in AWS, every unused box adds time to the API call for listing the instances. Every extra box that is there for "just in case" does too. It's important to balance these things.
  - ryandrake 11y ago
    Don't forget safety-critical software. I don't want avionics software developers thinking "that bug doesn't matter because it's very rare". Even if the effect is non-catastrophic. For certain classes of software, zero bugs is not an unreasonable or unrealistic goal.
  - daurnimator 11y ago
    > Resources are a matter of budgetary concerns, mostly. After making my own logging system, which was expensive in dev time + maintenance, I switched our company over to papertrail ( https://papertrailapp.com https://papertrailapp.com ) they handle quite large volumes of logs for less than $100 a month. They provide an interface to search the logs with low latency. I'm not sure what type of organisation can't afford it.... (cost scales with usage)

jedberg 11y ago

> turning on logging post-facto won't go back in time to catch a rare bug I guess my point is that if a rare bug is rare enough, then it doesn't really matter, especially if the effect isn't catastrophic. > And since you need the added resources that logging requires to be available at all times anyway, you might as well just leave logging on. What's your objection to having recent logs? There are a lot more resources required to log everything all the time then to log a subset of things some of the time. Even having recent logs of everything is a huge overhead compared to more detailed logs of just some things.

nostrademons 11y ago
That works if you're building Reddit, Netflix, or even most Google services. It doesn't work if you're doing financial software or anything that handles money or is critical for someone else's business.
- jedberg 11y ago
  I haven't looked into it, so I'm not sure, but I'm also not yet convinced that server logs can increase the integrity of banking transactions. I'd love to get some more info on specifics for this. For example, a lot of people just assume that a bank can't be eventually consistent, but if you make them stop and think about it, they realize that they can and are. The ATM isn't always connected to the bank's servers. That's why your card has a withdrawl limit. That's how much they are willing to risk for eventual consistency. I feel like the same is true for logging. That other checks can be put in place to mitigate risk to an acceptable level.
  - nostrademons 11y ago
    It's not that the logs themselves increase the integrity of banking transactions, it's that they find bugs that increase the integrity of banking transactions. Say that you've built your system like any good highly-available system so that you rigorously check consistency of the program invariants, log if there are any anomalies, and then either try to recover or back out with a user error if something goes wrong. You've configured your monitoring to alert if any of the consistency checks fail. Now you get an alert that you've errored out and served a 500 on one in every million requests. For Reddit, Facebook, Google, or other free services, it's no big deal: one in a million means that you're serving a couple hundred, maybe a few thousand 500s per day, max, and then the user just refreshes and gets over it. But in a financial transaction app, this is a big deal. The basic assumption you have to make is that if you have a consistency problem that your monitoring caught, you probably have consistency problems that you didn't catch. And the right thing to do is investigate until you understand what is going on and what the impact is. So you pull the logs of all surrounding requests, you pull the logs of any requests that were in flight at the same time, you pull the logs for the user that initiated the request and see exactly what they were doing. And then you try to reproduce the problem, test for it, and fix it. The difference is entirely in the tolerances built into the system, and how that dictates that you respond to errors. In some domains, a one-in-a-million error is a "well, we'll catch it next time" event. In others, a one-in-a-million error is a "drop everything you're doing, fix it, and ensure that no similar errors exist" event. BTW, the strictest tolerances usually are not in B2C companies, they are in suppliers of large B2C companies (or the government). The bank is completely willing to risk a few hundred dollars per customer on an ATM card, but if the ATM vendor loses that money without it being part of the spec they gave the bank, they've lost the contract. Most of these enterprise B2B contracts are based on trust, and if the customer observes you losing money accidentally, they'll wonder what else you're doing accidentally.
xorcist 11y ago
> I guess my point is that if a rare bug is rare enough, then it doesn't really matter, especially if the effect isn't catastrophic. That's only true if you run a trivial application. If you are a bank for example, it matters very much why the outcome of a transaction came out wrong, even if the effect wasn't catastrophic. Every circumstance and every bit of identity to the transaction is important. "It's rare" or "I don't believe in logging" doesn't cut it. > There are a lot more resources required to log everything all the time then to log a subset of things some of the time. Resources are a matter of budgetary concerns, mostly. What does a bug in production cost you? Are you liable? How fast? What's your SLA? Those things matter.
- jedberg 11y ago
  > Resources are a matter of budgetary concerns, mostly. True, but most people unfortunately don't have unlimited budgets to work with. Also, there is more than a monetary cost. Every system in your architecture incurs a non-zero cost to maintenance, reliability, cognitive load, and management. For example, if you are in AWS, every unused box adds time to the API call for listing the instances. Every extra box that is there for "just in case" does too. It's important to balance these things.
- ryandrake 11y ago
  Don't forget safety-critical software. I don't want avionics software developers thinking "that bug doesn't matter because it's very rare". Even if the effect is non-catastrophic. For certain classes of software, zero bugs is not an unreasonable or unrealistic goal.
- daurnimator 11y ago
  > Resources are a matter of budgetary concerns, mostly. After making my own logging system, which was expensive in dev time + maintenance, I switched our company over to papertrail ( https://papertrailapp.com https://papertrailapp.com ) they handle quite large volumes of logs for less than $100 a month. They provide an interface to search the logs with low latency. I'm not sure what type of organisation can't afford it.... (cost scales with usage)