4 ms·
Kind of looks like grok. Grok lets you name patterns, then build up larger patterns by those names, and then also name the groups that it matches to those sub-
by drivers99 8y ago
Kind of looks like grok. Grok lets you name patterns, then build up larger patterns by those names, and then also name the groups that it matches to those sub-patterns so you can refer to them in the data. It's built on top of regex, as each pattern can be defined by a mix of other patterns and/or regex.
For example this grok pattern (taken from [1] )
%{TIMESTAMP_ISO8601:timestamp} \[%{IPV4:ip};%{WORD:environment}\] %{LOGLEVEL:log_level} %{GREEDYDATA:message}
refers to a pattern called TIMESTAMP_ISO8601 and calls it "timestamp" in the resulting output data structure.
In logstash, TIMESTAMP_ISO8601 is predefined in a patterns file, such as [2], which is made of up of a mix of regex and other patterns like YEAR, MONTHNUM, etc.
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
MONTHNUM is a regex (optional 0 followed by 1-9; or 10 through 12):
MONTHNUM (?:0?[1-9]|1[0-2])
I'm not sure what all of Rosie's base patterns are. This appears to be a valid regex though, from the example: [:alpha:]+ (regex character class and "+" meaning 1 or more). It's C instead of java which is useful in more/different places.
[1] https://www.elastic.co/blog/do-you-grok-grok https://www.elastic.co/blog/do-you-grok-grok
[2] https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns https://github.com/logstash-plugins/logstash-patterns-core/b...
- eutropia 8y agoIs there anyway to make use of grok on the command line like the post does with rosie patterns? I rather like the idea of `| grep 'net.ipv4'`
- busterarm 8y agoI wish. I think the closest that you can get is to run Logstash locally with the stdout output plugin. You'll have to do some work with the logstash config to set up your grok input filters each time, but it should give you what you want otherwise.
- aequitas 8y agoGrok is implemented in other languages too. For example https://github.com/garyelephant/pygrok/ https://github.com/garyelephant/pygrok/ It makes developing and unit testing Logstash Grok patterns a lot quicker compared to spinning up a full Logstash instance for every line of code changed.
- Twirrim 8y agofor things like that, I write bash aliases. If you'll excuse the useless use of cat: $ cat /etc/hosts | netgrep 127.0.0.1 255.255.255.255 .. ..