6 ms·

Uno: a uniq like CLI tool for log data

100 points by daenney 8y ago

hirundo 8y ago
_
- empath75 8y ago
  the cli tool is free.
- whalesalad 8y ago
  Really aggressive feedback for a free tool like this. Maybe go for a walk and come back and read what you wrote out loud.
deleted 8y ago
[deleted]
waterhouse 8y ago
Protip: "sed -E 's/[0-9]+/N/g'" is, by itself, what I'd call 'unreasonably effective' at converting log lines into a canonicalized form, such that each line of printf or printf-equivalent source code corresponds to only one or a few distinct output lines. Then you can pipe it into "sort | uniq" (possibly "uniq -c" if you want counts, possibly for logspam-quantification purposes), and otherwise conceivably into a "profile" of a logfile, and then you can diff between a "normal" profile and something that may be less normal.
nn3 8y ago
Appears to be just l = readline() print l while True: n = readline() if editdistance(n, l) > 30%: print n l = n The simple tool doesn't appear to handle interleaving? Normally uniq handles that by sorting first, but this would need an edit distance aware sort first. I suppose that wouldn't be that hard to build either. Perhaps it could be just handled with a short window in the tool.
- sciurus 8y ago
  What makes you think it doesn't handle interleaving? They demonstrate a mode where, using the previous day's logs as a baseline, uno can read the current days logs and not print anything that occurred the previous day. That's more sophisticated than you're giving them credit for.
shepik 8y ago
That looked very interesting. But I had so search through the video and zoom in to learn how exactly to separate events that stand out from the baseline (instead of, you know, just reading that command from the webpage). So, i did all that and downloaded the tool and run that command and found out that the feature is "experimental" and not included in the binaries. It's a bit strange: the tool got my attention, and i was ready to experiment, but i couldn't. If the reason is that it's a paid or saas only feature, that's totally fine but i think authors should make it a bit more obvious from the web page. Otherwise, i'd suggest to put "experimental" binaries there, too.
- michaelcampbell 8y ago
  (Note: the preloading feature is experimental and not included in the public version) Right under the download links.
  - shepik 8y ago
    Yes, i noticed this after downloading, and even quoted it it my comment. The feature is called "baseline" everywhere in the text, not "preloading feature", and that's why i got confused. For example, "As expected, when I start uno with the baseline and don’t do anything with my laptop, it is almost silent since everything that comes in has been seen before."
Mizza 8y ago
Good idea, but it's non-Free. Should be fairly simple to recreate the ideas as a Free tool.
- daenney 8y ago
  Open sourcing it is being worked on, so if you wait a bit you won't have to recreate it :). But there was a desire to already put it out there so people can toy with it while it gets ready.
  - Mizza 8y ago
    Ah, okay, great! I was a bit disappointed as this actually would have been very handy to me today! Looking forward to the FOSS release.
  - nwmcsween 8y ago
    What language is it written in I may just recreate it.
vsipuli 8y ago
Is there a Homebrew package for this tool? Or another similar tool?
- guessmyname 8y ago
  Download the static binary [1] into /usr/local/bin/ That’s what Homebrew would do if there was a package anyway. For a similar tool, I recommend GoAccess [2] and LNAV [3]. [1] https://s3-eu-west-1.amazonaws.com/unomaly/releases/uno_mac/uno https://s3-eu-west-1.amazonaws.com/unomaly/releases/uno_mac/... [2] brew install goaccess — https://goaccess.io/ https://goaccess.io/ [3] brew install lnav — http://lnav.org/ http://lnav.org/
cryptonector 8y ago
Here's something I'd very much like: syslog(3) and anything like it that takes printf-style format strings and arguments to log the format string and the arguments in a structured way. E.g., syslog(LOG_WARN, "this thing %s didn't go well", thing); should log something like this, using JSON just for this example to show the actual structure: [<timestamp>,"<hostname>",<pid>,"<ident>","<facility>",<priority>,"this thing %s didn't go well", "<thing>"] This would make it much easier to recognize structure in log messages and extract details (syslog(3) format string arguments). Granted, this would have to have happened 20 years ago for it to be widely available now, but still, gotta start somewhere.
- isostatic 8y ago
  Main issue with syslog (or rather CSVs/TSVs) -- quotes, commas, newlines, backslashes are a bit of a pain when parsing. Apparently knowing how to use things like grep and awk aren't cool nowadays though. Too many people who grew up on windows. I attempted to install centos earlier, after 18 years of dealing with debian-installer, it amazed me how backwards the installer was. All about the clickity-click.
  - jlgaddis 8y ago
    FWIW, you can get back to utopia by passing in the "text" parameter at the very first menu you see (where the default choice is something like "Check media and install CentOS").
    - isostatic 8y ago
      From what I could see you can’t even set the network interfaces up in text mode (which is proper text mode, not ncurses)
      - jlgaddis 8y ago
        You can configure network interfaces (manually or using DHCP) in "text" mode. What you call "proper text mode" is referred to as "cmdline" mode. This mode, also known as "noninteractive" mode, is intended for 100% fully automated / "hands off" installations where everything has already has been defined and/or configured ahead of time. If you need or want to configure some configuration option or network interface or whatever manually at install-time, well, you'll want to pick either "graphical" or "text" instead. (You'll notice that the "luxuries" of text mode -- tmux, ncurses, spacing and alignment of text -- are missing in cmdline mode. If you are able to use cmdline mode, it is only because your installation process has been totally and completely automated, tested extensively, and works flawlessly. If all of that is true, well, you almost certainly aren't staring at the console watching the installation as it takes place and, if no one is watching anyways, why bother spending the effort to make "cmdline" mode anything more than the minimum required -- especially if no one is going to even notice!?)
robax 8y ago
This tool shows a lot of promise to me and I've already installed it on our stage box. Thanks! Some of our log lines contain large json blobs that uno only ignores when I set the difference ratio >50%. Maybe you could add some sort of json detection and parsing identify and ignore similar JSON? Seems tricky but just an idea. Also, the go routine seems to panic on the linux and mac binaries when I run "uno --help". Tools are needed that turn log analysis into more science than art. We currently use the ELK stack to analyze our logs and it works pretty well but still feels like spelunking. Do any of you HNers have any logging tools you'd recommend?
- guessmyname 8y ago
  I often use LNAV [1] and GoAccess [2]. [1] http://lnav.org/ http://lnav.org/ [2] https://goaccess.io/ https://goaccess.io/
jlgaddis 8y ago
> Looking at a sample of the logs it’s obvious that there is both repetition and structure in this data – similarly structured messages repeat with different parameters, dates, numbers, file names, etc. This seems like a great candidate for some automation, let’s see what we can do! While "uno" is, perhaps, "cooler", logcheck [0] has been around for ages and does just this -- go through your log files, weed out all the stuff that you don't care about, and mails you anything else (which, by definition, is all the stuff you do care about). Logcheck doesn't use "fuzziness" or uniqueness or any intelligence or anything like that -- just regexes -- but I end up with one single mail per day that only takes me a minute or two to sift through. Additionally, logcheck is open source. [0]: http://logcheck.org http://logcheck.org
- CupOfJava 8y ago
  Do you have to setup the regexes ahead of time or does logcheck automatically compute the regexes?
  - michaelmior 8y ago
    > Normal entries are entries which match one of the many included regular expression files contain in the database. I haven't used logcheck, but I assume you can add your own regexes, but looks like you have something to start with.
    - jlgaddis 8y ago
      It ships with a ton of "rules" [0]. You can choose to use just those, just your own, or some or all of both. [0]: https://salsa.debian.org/debian/logcheck/tree/master/rulefiles/linux https://salsa.debian.org/debian/logcheck/tree/master/rulefil...
- justaj 8y ago
  I'm not really sure it's still an active project: └─$ git clone git://git.debian.org/git/logcheck/logcheck.git Cloning into 'logcheck'... fatal: unable to connect to git.debian.org: moszumanska.debian.org[0: 5.153.231.21]: errno=No route to host moszumanska.debian.org[1: 2001:41c8:1000:21::21:21]: errno=Network is unreachable Also none of their mailing lists seem to exist anymore: http://logcheck.org/lists.html http://logcheck.org/lists.html
CupOfJava 8y ago
Does it understand stacktraces, or do I have to preprocess the mutliline stacktraces into single lines?

100 points by daenney 8y ago

hirundo 8y ago

empath75 8y ago
the cli tool is free.
whalesalad 8y ago
Really aggressive feedback for a free tool like this. Maybe go for a walk and come back and read what you wrote out loud.

deleted 8y ago

[deleted]

waterhouse 8y ago

Protip: "sed -E 's/[0-9]+/N/g'" is, by itself, what I'd call 'unreasonably effective' at converting log lines into a canonicalized form, such that each line of printf or printf-equivalent source code corresponds to only one or a few distinct output lines. Then you can pipe it into "sort | uniq" (possibly "uniq -c" if you want counts, possibly for logspam-quantification purposes), and otherwise conceivably into a "profile" of a logfile, and then you can diff between a "normal" profile and something that may be less normal.

nn3 8y ago

Appears to be just l = readline() print l while True: n = readline() if editdistance(n, l) > 30%: print n l = n The simple tool doesn't appear to handle interleaving? Normally uniq handles that by sorting first, but this would need an edit distance aware sort first. I suppose that wouldn't be that hard to build either. Perhaps it could be just handled with a short window in the tool.

sciurus 8y ago
What makes you think it doesn't handle interleaving? They demonstrate a mode where, using the previous day's logs as a baseline, uno can read the current days logs and not print anything that occurred the previous day. That's more sophisticated than you're giving them credit for.

shepik 8y ago

That looked very interesting. But I had so search through the video and zoom in to learn how exactly to separate events that stand out from the baseline (instead of, you know, just reading that command from the webpage). So, i did all that and downloaded the tool and run that command and found out that the feature is "experimental" and not included in the binaries. It's a bit strange: the tool got my attention, and i was ready to experiment, but i couldn't. If the reason is that it's a paid or saas only feature, that's totally fine but i think authors should make it a bit more obvious from the web page. Otherwise, i'd suggest to put "experimental" binaries there, too.

michaelcampbell 8y ago
(Note: the preloading feature is experimental and not included in the public version) Right under the download links.
- shepik 8y ago
  Yes, i noticed this after downloading, and even quoted it it my comment. The feature is called "baseline" everywhere in the text, not "preloading feature", and that's why i got confused. For example, "As expected, when I start uno with the baseline and don’t do anything with my laptop, it is almost silent since everything that comes in has been seen before."

Mizza 8y ago

Good idea, but it's non-Free. Should be fairly simple to recreate the ideas as a Free tool.

daenney 8y ago
Open sourcing it is being worked on, so if you wait a bit you won't have to recreate it :). But there was a desire to already put it out there so people can toy with it while it gets ready.
- Mizza 8y ago
  Ah, okay, great! I was a bit disappointed as this actually would have been very handy to me today! Looking forward to the FOSS release.
- nwmcsween 8y ago
  What language is it written in I may just recreate it.

vsipuli 8y ago

Is there a Homebrew package for this tool? Or another similar tool?

guessmyname 8y ago
Download the static binary [1] into /usr/local/bin/ That’s what Homebrew would do if there was a package anyway. For a similar tool, I recommend GoAccess [2] and LNAV [3]. [1] https://s3-eu-west-1.amazonaws.com/unomaly/releases/uno_mac/uno https://s3-eu-west-1.amazonaws.com/unomaly/releases/uno_mac/... [2] brew install goaccess — https://goaccess.io/ https://goaccess.io/ [3] brew install lnav — http://lnav.org/ http://lnav.org/

cryptonector 8y ago

Here's something I'd very much like: syslog(3) and anything like it that takes printf-style format strings and arguments to log the format string and the arguments in a structured way. E.g., syslog(LOG_WARN, "this thing %s didn't go well", thing); should log something like this, using JSON just for this example to show the actual structure: [<timestamp>,"<hostname>",<pid>,"<ident>","<facility>",<priority>,"this thing %s didn't go well", "<thing>"] This would make it much easier to recognize structure in log messages and extract details (syslog(3) format string arguments). Granted, this would have to have happened 20 years ago for it to be widely available now, but still, gotta start somewhere.

isostatic 8y ago
Main issue with syslog (or rather CSVs/TSVs) -- quotes, commas, newlines, backslashes are a bit of a pain when parsing. Apparently knowing how to use things like grep and awk aren't cool nowadays though. Too many people who grew up on windows. I attempted to install centos earlier, after 18 years of dealing with debian-installer, it amazed me how backwards the installer was. All about the clickity-click.
- jlgaddis 8y ago
  FWIW, you can get back to utopia by passing in the "text" parameter at the very first menu you see (where the default choice is something like "Check media and install CentOS").
  - isostatic 8y ago
    From what I could see you can’t even set the network interfaces up in text mode (which is proper text mode, not ncurses)
    - jlgaddis 8y ago
      You can configure network interfaces (manually or using DHCP) in "text" mode. What you call "proper text mode" is referred to as "cmdline" mode. This mode, also known as "noninteractive" mode, is intended for 100% fully automated / "hands off" installations where everything has already has been defined and/or configured ahead of time. If you need or want to configure some configuration option or network interface or whatever manually at install-time, well, you'll want to pick either "graphical" or "text" instead. (You'll notice that the "luxuries" of text mode -- tmux, ncurses, spacing and alignment of text -- are missing in cmdline mode. If you are able to use cmdline mode, it is only because your installation process has been totally and completely automated, tested extensively, and works flawlessly. If all of that is true, well, you almost certainly aren't staring at the console watching the installation as it takes place and, if no one is watching anyways, why bother spending the effort to make "cmdline" mode anything more than the minimum required -- especially if no one is going to even notice!?)

robax 8y ago

This tool shows a lot of promise to me and I've already installed it on our stage box. Thanks! Some of our log lines contain large json blobs that uno only ignores when I set the difference ratio >50%. Maybe you could add some sort of json detection and parsing identify and ignore similar JSON? Seems tricky but just an idea. Also, the go routine seems to panic on the linux and mac binaries when I run "uno --help". Tools are needed that turn log analysis into more science than art. We currently use the ELK stack to analyze our logs and it works pretty well but still feels like spelunking. Do any of you HNers have any logging tools you'd recommend?

guessmyname 8y ago
I often use LNAV [1] and GoAccess [2]. [1] http://lnav.org/ http://lnav.org/ [2] https://goaccess.io/ https://goaccess.io/

jlgaddis 8y ago

> Looking at a sample of the logs it’s obvious that there is both repetition and structure in this data – similarly structured messages repeat with different parameters, dates, numbers, file names, etc. This seems like a great candidate for some automation, let’s see what we can do! While "uno" is, perhaps, "cooler", logcheck [0] has been around for ages and does just this -- go through your log files, weed out all the stuff that you don't care about, and mails you anything else (which, by definition, is all the stuff you do care about). Logcheck doesn't use "fuzziness" or uniqueness or any intelligence or anything like that -- just regexes -- but I end up with one single mail per day that only takes me a minute or two to sift through. Additionally, logcheck is open source. [0]: http://logcheck.org http://logcheck.org

CupOfJava 8y ago
Do you have to setup the regexes ahead of time or does logcheck automatically compute the regexes?
- michaelmior 8y ago
  > Normal entries are entries which match one of the many included regular expression files contain in the database. I haven't used logcheck, but I assume you can add your own regexes, but looks like you have something to start with.
  - jlgaddis 8y ago
    It ships with a ton of "rules" [0]. You can choose to use just those, just your own, or some or all of both. [0]: https://salsa.debian.org/debian/logcheck/tree/master/rulefiles/linux https://salsa.debian.org/debian/logcheck/tree/master/rulefil...
justaj 8y ago
I'm not really sure it's still an active project: └─$ git clone git://git.debian.org/git/logcheck/logcheck.git Cloning into 'logcheck'... fatal: unable to connect to git.debian.org: moszumanska.debian.org[0: 5.153.231.21]: errno=No route to host moszumanska.debian.org[1: 2001:41c8:1000:21::21:21]: errno=Network is unreachable Also none of their mailing lists seem to exist anymore: http://logcheck.org/lists.html http://logcheck.org/lists.html

CupOfJava 8y ago

Does it understand stacktraces, or do I have to preprocess the mutliline stacktraces into single lines?