4 ms·

> Whatever service you are using for logging, it costs money, and a fast way to burn money is to log the entire json object that was relatively small on your de

> Whatever service you are using for logging, it costs money, and a fast way to burn money is to log the entire json object that was relatively small on your dev env, but blew up on production. You could also, you know, run your own infrastructure and log to your heart's content.

chaghalibaghali 4y ago
There are still going to be time and effort costs involved in scaling that infrastructure as your log volume increases
- marginalia_nu 4y ago
  You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or requests per second from a worldwide user-base, then affording storage for the logs really shouldn't be a problem anymore (idk, with the possible exception of Twitter)
  - KronisLV 4y ago
    > You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. This is a good point - a RAID array of a few HDDs/SSDs scales surprisingly far and is cheaper than many of the cloud services out there, though whether you can or can't use either approach probably relies on compliance requirements and such. I will definitely add that logs can compress really well - to the point where it's been close to a year since I added Logrotate to a project that didn't have it before, for a pretty basic setup, and I haven't had the need to even look at how many archives are currently retained, given that the disk usage has changed very slightly. And that's for multiple systems that filled up the available storage in months previously. Of course, my personal gripe is that most of the logging solutions out there are rather complex - something like Graylog feels like one of the simpler self-hostable options while still being fully featured, but in my experience anything that runs ES is really resource hungry. Sometimes it feels like MariaDB/PostgreSQL would be good enough for most of the simpler low logging volume setups out there - if you don't want to manage logs as files, want to ship them somewhere, but don't want the receiving system to be too complex either.
  - jodrellblank 4y ago
    And what are you going to do when you need a human to read sixteen trillion bytes of compressed logs streamed off a single SATA disk? Once you face the fact that the performance of a single SATA disk means you can't search the logs in any quick time, and nobody can possibly read that much log data, so nobody will use it, you start to see it as a hoarding disorder not a useful tool.
    - marginalia_nu 4y ago
      It's not unheard of to need to retain years even decades worth of logs due to regulatory compliance. Nobody is reading them, they just need to exist. In that scenario you'll probably keep the current year or so fresh on a mechanical drive and past years on tape.
  - adra 4y ago
    Except you know when you.actually want to do something valuable with all those logs. You _should_ be creating logs (signals) to be valuable in some way (diagnostics, alerting, canaries statistics), etc. If you're just dumping logs into opaque blobs that are never looked at them sure write them to blobs to your heart's content and have fun hunting and pecking for reasons you're users are already screaming at you. That strategy is fine, but the limitations are clear. It's reactive.
    - marginalia_nu 4y ago
      Depends entirely on what and why you are logging. Is it audit logs for security or due to some regulatory requirement? Then huge blobs are fine. Desirable, even. Transaction logs for machine-loading so you're able to replay an application's state at any given moment in time? Yeah probably gonna end up with huge blobs again.
NeckBeardPrince 4y ago
You know that costs money, right?
- marginalia_nu 4y ago
  Not as much as you'd think, and critically, the cost is largely disconnected from how you use the infra.
  - darkwater 4y ago
    If your infra is not on-prem, yes it will cost you more money as you are generating more and more and bigger logs.
    - marginalia_nu 4y ago
      You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at which sure, I guess you could probably log a bit less. Either that or you're operating at a scale with many millions of users where you should have income and be able to afford it. ... well, either that, or you're being fleeced.
      - giaour 4y ago
        When people complain about the cost of excessive logging, they are almost certainly not thinking in terms of how much a drive costs. Services like CloudWatch are an excellent way to burn through money, though it's usually the time series storage and ingestion costs that balloon out of control.
        partdavid 4y ago
        Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.

chaghalibaghali 4y ago

There are still going to be time and effort costs involved in scaling that infrastructure as your log volume increases

marginalia_nu 4y ago
You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or requests per second from a worldwide user-base, then affording storage for the logs really shouldn't be a problem anymore (idk, with the possible exception of Twitter)
- KronisLV 4y ago
  > You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. This is a good point - a RAID array of a few HDDs/SSDs scales surprisingly far and is cheaper than many of the cloud services out there, though whether you can or can't use either approach probably relies on compliance requirements and such. I will definitely add that logs can compress really well - to the point where it's been close to a year since I added Logrotate to a project that didn't have it before, for a pretty basic setup, and I haven't had the need to even look at how many archives are currently retained, given that the disk usage has changed very slightly. And that's for multiple systems that filled up the available storage in months previously. Of course, my personal gripe is that most of the logging solutions out there are rather complex - something like Graylog feels like one of the simpler self-hostable options while still being fully featured, but in my experience anything that runs ES is really resource hungry. Sometimes it feels like MariaDB/PostgreSQL would be good enough for most of the simpler low logging volume setups out there - if you don't want to manage logs as files, want to ship them somewhere, but don't want the receiving system to be too complex either.
- jodrellblank 4y ago
  And what are you going to do when you need a human to read sixteen trillion bytes of compressed logs streamed off a single SATA disk? Once you face the fact that the performance of a single SATA disk means you can't search the logs in any quick time, and nobody can possibly read that much log data, so nobody will use it, you start to see it as a hoarding disorder not a useful tool.
  - marginalia_nu 4y ago
    It's not unheard of to need to retain years even decades worth of logs due to regulatory compliance. Nobody is reading them, they just need to exist. In that scenario you'll probably keep the current year or so fresh on a mechanical drive and past years on tape.
- adra 4y ago
  Except you know when you.actually want to do something valuable with all those logs. You _should_ be creating logs (signals) to be valuable in some way (diagnostics, alerting, canaries statistics), etc. If you're just dumping logs into opaque blobs that are never looked at them sure write them to blobs to your heart's content and have fun hunting and pecking for reasons you're users are already screaming at you. That strategy is fine, but the limitations are clear. It's reactive.
  - marginalia_nu 4y ago
    Depends entirely on what and why you are logging. Is it audit logs for security or due to some regulatory requirement? Then huge blobs are fine. Desirable, even. Transaction logs for machine-loading so you're able to replay an application's state at any given moment in time? Yeah probably gonna end up with huge blobs again.

NeckBeardPrince 4y ago

You know that costs money, right?

marginalia_nu 4y ago
Not as much as you'd think, and critically, the cost is largely disconnected from how you use the infra.
- darkwater 4y ago
  If your infra is not on-prem, yes it will cost you more money as you are generating more and more and bigger logs.
  - marginalia_nu 4y ago
    You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at which sure, I guess you could probably log a bit less. Either that or you're operating at a scale with many millions of users where you should have income and be able to afford it. ... well, either that, or you're being fleeced.
    - giaour 4y ago
      When people complain about the cost of excessive logging, they are almost certainly not thinking in terms of how much a drive costs. Services like CloudWatch are an excellent way to burn through money, though it's usually the time series storage and ingestion costs that balloon out of control.
      - partdavid 4y ago
        Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.