4 ms·

I can believe that they do not keep logs of the database access. As brain dead as it sounds. I have been in the position of implementing a client on a API I d

by worik 6y ago

I can believe that they do not keep logs of the database access. As brain dead as it sounds. I have been in the position of implementing a client on a API I do not control. The owners of the servers (colleagues but in a different country) do not seem to know what logs are. We get random failures from the server. I can pin down to the second when they occur (not closer because of network lag). I suspect that the server is failing under load, but the way I would find out is to... Read the logs. My foreign colleagues do not respond to me, ghost me entirely, when I ask them to inspect the logs. Perhaps it is a Windows/Azure thing?

jiggawatts 6y ago
Logs are typically off by default in most Enterprise software, or goes nowhere by default which is basically the same thing. Logs cost money to both collect and store. Not everyone is cheerfully burning through VC capital. Some people have budgets. Speaking of log collection, simply dumping the logs into a central repository is the same as taking the garbage to the landfill. Collecting trash just results in a big collection of trash. Extracting useful information from a huge pile of logs is non-trivial. You'd have to know at least one query language, probably several if you work on big enterprise systems. You'd have to know a bunch of esoteric things like how to convert the long decimal strings to a number so that you can interpret as any one of a dozen timestamp formats as an actual datetime in UTC, and then convert that to local timestamp so that you can tell when something actually happened. And so on. All of this is merely a prerequisite for finding a specific instance of what you know is there. You know what you'll never find in logs? Things you didn't know to look for! That's unfortunately about 90-99% of what you actually need to know, making logs typically about 1-10% as useful as you'd like. Did I mention they cost money? Have you seen how many arms and legs Splunk charges these days? Why would you pay that much for something that is less than a tenth as useful as it could be? The fundamental problem is that discipline doesn't scale. You can't expect a hundred thousand IT operations guys to all do everything all of the time, in spite of the non-technical finance guy holding on to the purse strings like he's gripping a life preserver after going overboard in a raging storm. PS: I turned logging on extensively for a recent Azure project. Some logs cost more than the service they were monitoring. I mean sure, I could turn off the unnecessary logs, but how do I know ahead of time which ones will be necessary or not? I don't have a time machine! I can't go back and turn off the logs I won't need... later. PPS: Have you noticed all logging companies charge by the gigabyte? What incentive do you imagine they have for improving the efficiency of the log transfer and storage formats?
- deleted 6y ago
  [deleted]
- worik 6y ago
  It is untrue that logs are trash. I gag at the comparison. If the would just look at logs they could answer so many questions that are costing a lot of money. I am used to a Unix world, with logs as text files. They take up very little room, and it is easy to through out old ones (keeping logs for ever is foolish for a lot of reasons). I am staggered that "I turned logging on extensively for a recent Azure project. Some logs cost more than the service they were monitoring." There is no way that this can be true unless there is some very very bad craziness going on. A real mindfuck Even a hours worth of logs would of by now saved at least a week of developer time for the people I work with (my time, they pay for). "Logs are typically off by default in most Enterprise software" That is probably true, I have no reason to doubt you and it fits with the other things I have witnessed, but there is a better way.
  - jiggawatts 6y ago
    > There is no way that this can be true unless there is some very very bad craziness going on. A real mindfuck System Center Operations Manager and all derived products such as the various Azure Monitor and Azure Logs offerings have craziness going on. For example, a typical metric is stored with a 300:1 write amplification because the database schema they use repeats the fully qualified machine name, counter name, and counter instance name per measurement. This prevents the collection of a useful set of metrics at a useful frequency. Logs are similarly generated using suuuuuuper verbose JSON, capturing everything except the critical fields you actually wanted, and then stored uncompressed record-by-record, keeping the full field names for each and every row. What incentive do you think they have to improve on the efficiency of the wire format or the storage format if they charge by the gigabyte?
    - worik 6y ago
      Perhaps it is too much to ask for the HTTP request strings dumped to a file? If it is too much for production, turn it on for development servers. My foreign friends seem to know not what that means

by worik 6y ago

jiggawatts 6y ago

Logs are typically off by default in most Enterprise software, or goes nowhere by default which is basically the same thing. Logs cost money to both collect and store. Not everyone is cheerfully burning through VC capital. Some people have budgets. Speaking of log collection, simply dumping the logs into a central repository is the same as taking the garbage to the landfill. Collecting trash just results in a big collection of trash. Extracting useful information from a huge pile of logs is non-trivial. You'd have to know at least one query language, probably several if you work on big enterprise systems. You'd have to know a bunch of esoteric things like how to convert the long decimal strings to a number so that you can interpret as any one of a dozen timestamp formats as an actual datetime in UTC, and then convert that to local timestamp so that you can tell when something actually happened. And so on. All of this is merely a prerequisite for finding a specific instance of what you know is there. You know what you'll never find in logs? Things you didn't know to look for! That's unfortunately about 90-99% of what you actually need to know, making logs typically about 1-10% as useful as you'd like. Did I mention they cost money? Have you seen how many arms and legs Splunk charges these days? Why would you pay that much for something that is less than a tenth as useful as it could be? The fundamental problem is that discipline doesn't scale. You can't expect a hundred thousand IT operations guys to all do everything all of the time, in spite of the non-technical finance guy holding on to the purse strings like he's gripping a life preserver after going overboard in a raging storm. PS: I turned logging on extensively for a recent Azure project. Some logs cost more than the service they were monitoring. I mean sure, I could turn off the unnecessary logs, but how do I know ahead of time which ones will be necessary or not? I don't have a time machine! I can't go back and turn off the logs I won't need... later. PPS: Have you noticed all logging companies charge by the gigabyte? What incentive do you imagine they have for improving the efficiency of the log transfer and storage formats?

deleted 6y ago
[deleted]
worik 6y ago
It is untrue that logs are trash. I gag at the comparison. If the would just look at logs they could answer so many questions that are costing a lot of money. I am used to a Unix world, with logs as text files. They take up very little room, and it is easy to through out old ones (keeping logs for ever is foolish for a lot of reasons). I am staggered that "I turned logging on extensively for a recent Azure project. Some logs cost more than the service they were monitoring." There is no way that this can be true unless there is some very very bad craziness going on. A real mindfuck Even a hours worth of logs would of by now saved at least a week of developer time for the people I work with (my time, they pay for). "Logs are typically off by default in most Enterprise software" That is probably true, I have no reason to doubt you and it fits with the other things I have witnessed, but there is a better way.
- jiggawatts 6y ago
  > There is no way that this can be true unless there is some very very bad craziness going on. A real mindfuck System Center Operations Manager and all derived products such as the various Azure Monitor and Azure Logs offerings have craziness going on. For example, a typical metric is stored with a 300:1 write amplification because the database schema they use repeats the fully qualified machine name, counter name, and counter instance name per measurement. This prevents the collection of a useful set of metrics at a useful frequency. Logs are similarly generated using suuuuuuper verbose JSON, capturing everything except the critical fields you actually wanted, and then stored uncompressed record-by-record, keeping the full field names for each and every row. What incentive do you think they have to improve on the efficiency of the wire format or the storage format if they charge by the gigabyte?
  - worik 6y ago
    Perhaps it is too much to ask for the HTTP request strings dumped to a file? If it is too much for production, turn it on for development servers. My foreign friends seem to know not what that means