3 ms·

Do you have some tricks for how you handle filtering through logs? Or some ideas if there could be a tool that could help you or mitigate your most critical iss

by akavel 5y ago

Do you have some tricks for how you handle filtering through logs? Or some ideas if there could be a tool that could help you or mitigate your most critical issue[s]? I found filtering through longs a major pain even for a fully sighted person like me, so I wrote a tool to help me with that, but it's fully in a "TUI" paradigm (i.e. curses-like), so I presume it wouldn't help you much (https://github.com/akavel/up https://github.com/akavel/up). No promises, given that the tool as is scratched my itch, but I am honestly curious if something similar could reduce your PITA, including whether this specific tool could be made useful for you through some minimal effort on my side.

gostsamo 5y ago
Hi, usually grep saves the day. I will check your tool, but what I need is for a terminal command that can recognize the meta fields from a log record and put them on a line separated from the main message. Also, it must be installed everywhere I work, which is not so easy. Putting logs in a table with filtering capabilities might be best, but this means web access to the location of the logs which is again tricky.
- ryanianian 5y ago
  > what I need is for a terminal command that can recognize the meta fields from a log record and put them on a line separated from the main message Isn't this the exact use-case of structured logging? Log events have {timestamp, log level, log category, string message, ...arbitrary key/value pairs} Usually serializing each message as a single json line in a file. Since it's all on one line you can still use grep, but then since it's machine-readable you can pipe the grep to anything that can parse json. Vanilla python3 works and tends to be a part of most ops toolkits. Such tooling can split out the fields onto other lines etc or in a more reader-friendly format.
  - gostsamo 5y ago
    Yes, this has been my idea in many cases, but it is not always that I have a say over the logging format.
- akavel 5y ago
  The idea of my tool is not really to help in some specific way of processing logs, but rather to make it easier to fiddle with grep and other Linux CLI filtering tools, by shortening the feedback loop vs. normal shell REPL. I'm not sure if that sounds in any way clear; also, it might sound strange that shell REPL is IMO too slow, or that it is in any way important, but I found it enough so that I invented a way to speed it up, and it seemed to hit a nerve of quite many people, seeing from the reception the tool got. I can try to explain more if you are interested and/or don't really understand what I might be even talking about. I tried to explain it in the readme, but for many reasons I have no slightest freaking idea to what extent it might sound understandable to you - not the least due to the fact, that for fully-sighted people one important way I tried to convey the idea is through an animated gif of a terminal window showing how the tool is used. As someone said, with really innovative ideas, it's often necessary to push them down people's throats to make them understand those; this gif is part of that effort, and fortunately seems quite effective, but even it is not enough for some people, and I'm by default assuming that it's for obvious reasons completely inaccessible to you. I was pondering just now whether trying to copy this animation to asciinema could make it in any way more accessible to you, but as of now I have high doubts if that would work at all (including that I have no idea whether the asciinema site is at all accessible to you, as well as that I'm highly suspicious the terminal ANSI sequences generated by a library I'm using are "tiniest diffs", so although the result might look indistinguishable to fully-sighted people, a screen reader might [or might not???] take them at face value and read them as a mess of random jumps and single-character changes on the screen). That said, I'm more than 100% happy if you managed to understand what the tool is doing, and just don't think it could be useful to you, whether as is or with some accessibility improvement attempts. Or if you don't understand but don't feel like wanting to dive deeper either.

by akavel 5y ago

gostsamo 5y ago

Hi, usually grep saves the day. I will check your tool, but what I need is for a terminal command that can recognize the meta fields from a log record and put them on a line separated from the main message. Also, it must be installed everywhere I work, which is not so easy. Putting logs in a table with filtering capabilities might be best, but this means web access to the location of the logs which is again tricky.

ryanianian 5y ago
> what I need is for a terminal command that can recognize the meta fields from a log record and put them on a line separated from the main message Isn't this the exact use-case of structured logging? Log events have {timestamp, log level, log category, string message, ...arbitrary key/value pairs} Usually serializing each message as a single json line in a file. Since it's all on one line you can still use grep, but then since it's machine-readable you can pipe the grep to anything that can parse json. Vanilla python3 works and tends to be a part of most ops toolkits. Such tooling can split out the fields onto other lines etc or in a more reader-friendly format.
- gostsamo 5y ago
  Yes, this has been my idea in many cases, but it is not always that I have a say over the logging format.
akavel 5y ago
The idea of my tool is not really to help in some specific way of processing logs, but rather to make it easier to fiddle with grep and other Linux CLI filtering tools, by shortening the feedback loop vs. normal shell REPL. I'm not sure if that sounds in any way clear; also, it might sound strange that shell REPL is IMO too slow, or that it is in any way important, but I found it enough so that I invented a way to speed it up, and it seemed to hit a nerve of quite many people, seeing from the reception the tool got. I can try to explain more if you are interested and/or don't really understand what I might be even talking about. I tried to explain it in the readme, but for many reasons I have no slightest freaking idea to what extent it might sound understandable to you - not the least due to the fact, that for fully-sighted people one important way I tried to convey the idea is through an animated gif of a terminal window showing how the tool is used. As someone said, with really innovative ideas, it's often necessary to push them down people's throats to make them understand those; this gif is part of that effort, and fortunately seems quite effective, but even it is not enough for some people, and I'm by default assuming that it's for obvious reasons completely inaccessible to you. I was pondering just now whether trying to copy this animation to asciinema could make it in any way more accessible to you, but as of now I have high doubts if that would work at all (including that I have no idea whether the asciinema site is at all accessible to you, as well as that I'm highly suspicious the terminal ANSI sequences generated by a library I'm using are "tiniest diffs", so although the result might look indistinguishable to fully-sighted people, a screen reader might [or might not???] take them at face value and read them as a mess of random jumps and single-character changes on the screen). That said, I'm more than 100% happy if you managed to understand what the tool is doing, and just don't think it could be useful to you, whether as is or with some accessibility improvement attempts. Or if you don't understand but don't feel like wanting to dive deeper either.