3 ms·

>Nobody invests in their “test in prod” tooling. Firstly, what is logging then? How is this not tooling to ensure things are running smoothly? `less +F` anyon

>Nobody invests in their “test in prod” tooling. Firstly, what is logging then? How is this not tooling to ensure things are running smoothly? `less +F` anyone? Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing.

mgkimsal 6y ago
"Firstly, what is logging then?" "Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing." This presume that you, the person expected to fix problem X actually have access to logs and the servers where problems are happening. I've been at multiple projects/companies where this simply Isn't Allowed(tm). "You wrote this code, you need to fix it! It's broken!" "Let me get on the server and take a look at the logs to see what's going on". "That violates our security policy! You can't do that!" I was tasked with 'finding a problem' and was told to look in the logs. We knew what day the problem was, but I couldn't get anyone to confirm if the log files were in UTC or something else (turns out it was something else). HOWEVER, I never actually got the actual log files. I didn't have access directly (that takes about a week to go through the chain of command to approve), so someone just sent me small snippets from where they thought the problem might be. So even orgs that tick all their checkboxes of stats/analytics/logging... sometimes seem to forget that access to the collected info is a requirement too.
- aunty_helen 6y ago
  Feel your pain <3 You can tell it's been a few years now but I have worked at Big Corp Inc. before. I know what it's like to ask someone in charge of 8k people to personally approve my server permission escalation even though he's never met me, doesn't know anything about the IT systems, has never visited the site I work at or probably even been in the country I reside. Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever google call their data storage). Then you only need that permission. As for the non-UTC servers... -_- but if it's a problem, you can always append the tz info to the log date format.
  - jagraff 6y ago
    The problem that the above user had was that the logs were already written without tz info, so they had no way to know. Assuming it was set to the machine's tz producing the logs, you could probably figure out the tz from the machine's ip address (assuming that's produced in the logs as well..)
    - mgkimsal 6y ago
      what I learned was that the logs are timestamped to whatever machine they're on, wherever it is. The servers in London - GMT. The servers in west coast - Pacfic time. Servers in DC - Eastern time. Not ideal.
      - jagraff 6y ago
        Right - assuming the server IP, or at least some other network identifying factor, is in the log, you could write some sort of regex to parse the logs and identify the correct time. Of course, that depends on someone being able to actually parse the logs with your regex
        perl4ever 6y ago
        I once found a solution to determining the timezone when it wasn't available - a field with only a date was unnecessarily and incorrectly converted to UT, meaning that the zone could be derived from the offset from midnight and applied to other fields. If enough things are screwed up, you can possibly find the solution to one issue in another.
truculent 6y ago
Logging is tooling but it is the bare minimum and requires very little investment to do. Good logging requires a little more investment but often isn’t done. However, it still generally only provides insight into the known knowns and sometimes a known unknown, which is largely not where bugs happen. Better tooling would enable understanding and further aid reproducibility over mere observability.

mgkimsal 6y ago

"Firstly, what is logging then?" "Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing." This presume that you, the person expected to fix problem X actually have access to logs and the servers where problems are happening. I've been at multiple projects/companies where this simply Isn't Allowed(tm). "You wrote this code, you need to fix it! It's broken!" "Let me get on the server and take a look at the logs to see what's going on". "That violates our security policy! You can't do that!" I was tasked with 'finding a problem' and was told to look in the logs. We knew what day the problem was, but I couldn't get anyone to confirm if the log files were in UTC or something else (turns out it was something else). HOWEVER, I never actually got the actual log files. I didn't have access directly (that takes about a week to go through the chain of command to approve), so someone just sent me small snippets from where they thought the problem might be. So even orgs that tick all their checkboxes of stats/analytics/logging... sometimes seem to forget that access to the collected info is a requirement too.

aunty_helen 6y ago
Feel your pain <3 You can tell it's been a few years now but I have worked at Big Corp Inc. before. I know what it's like to ask someone in charge of 8k people to personally approve my server permission escalation even though he's never met me, doesn't know anything about the IT systems, has never visited the site I work at or probably even been in the country I reside. Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever google call their data storage). Then you only need that permission. As for the non-UTC servers... -_- but if it's a problem, you can always append the tz info to the log date format.
- jagraff 6y ago
  The problem that the above user had was that the logs were already written without tz info, so they had no way to know. Assuming it was set to the machine's tz producing the logs, you could probably figure out the tz from the machine's ip address (assuming that's produced in the logs as well..)
  - mgkimsal 6y ago
    what I learned was that the logs are timestamped to whatever machine they're on, wherever it is. The servers in London - GMT. The servers in west coast - Pacfic time. Servers in DC - Eastern time. Not ideal.
    - jagraff 6y ago
      Right - assuming the server IP, or at least some other network identifying factor, is in the log, you could write some sort of regex to parse the logs and identify the correct time. Of course, that depends on someone being able to actually parse the logs with your regex
      - perl4ever 6y ago
        I once found a solution to determining the timezone when it wasn't available - a field with only a date was unnecessarily and incorrectly converted to UT, meaning that the zone could be derived from the offset from midnight and applied to other fields. If enough things are screwed up, you can possibly find the solution to one issue in another.

truculent 6y ago

Logging is tooling but it is the bare minimum and requires very little investment to do. Good logging requires a little more investment but often isn’t done. However, it still generally only provides insight into the known knowns and sometimes a known unknown, which is largely not where bugs happen. Better tooling would enable understanding and further aid reproducibility over mere observability.