4 ms·

Why don't languages have "grok" patterns in their standard libraries? It seems to only exist in log parsing ecosystems but this really helps with getting rid o

Why don't languages have "grok" patterns in their standard libraries? It seems to only exist in log parsing ecosystems but this really helps with getting rid of little bugs and wrong parsing of specific regex patterns. Instead of doing "^\d+(\.\d+){3}$" for IP checking which is clearly wrong, you'd do "%{IPV4:ip}" which is so much better. List of known patterns : https://github.com/hpcugent/logstash-patterns/blob/master/files/grok-patterns https://github.com/hpcugent/logstash-patterns/blob/master/fi... Even for PHP a third party library only has 15 stars.

pimlottc 4y ago
Perhaps because many devs, like me, haven't heard of "grok patterns"[0] before. Because of that, it took me a while to understand what your post was saying, since I was reading "grok" as a normal word. It's not a bad idea, though you'd want to make it a more formal standard; since it's just used internally in one project, it may be subject to change based on that project's own needs. There could also be more documentation, showing examples of strings that are accepted and not accepted by each pattern, as well as advice for generating compliant strings. 0: https://logz.io/blog/logstash-grok/ https://logz.io/blog/logstash-grok/
- mekster 4y ago
  It may have originated from a project but many other log parser projects such as Vector and fluentd have such support. https://vector.dev/docs/reference/vrl/examples/#parse_grok https://vector.dev/docs/reference/vrl/examples/#parse_grok https://github.com/fluent/fluent-plugin-grok-parser https://github.com/fluent/fluent-plugin-grok-parser
andy81 4y ago
.NET has something close in standard libraries, in that you can try converting a string to System.Net.IPAddress and other similar classes. It's for parsing rather than search though, you'd still need regex if e.g. searching unstructured logs for IPs. e.g. (powershell syntax for easy reproduction) returns parsed, or throw if invalid. [uri]"\\ServerName1\Folder\UserName" [version]'1.2.3.4' [ipaddress]'1.1.1.1'
- gregmac 4y ago
  Many of these have a `TryParse` method [1] which makes for nice syntax, and of course then you get a variable of the correct type. if (int.TryParse(userInput, out var number)) { total += number; } This is cleaner code compared to try/catch and usually faster (though debatable if it matters) [2]. [1] https://learn.microsoft.com/en-us/dotnet/api/system.net.ipaddress.tryparse?view=net-7.0 https://learn.microsoft.com/en-us/dotnet/api/system.net.ipad... [2] https://stackoverflow.com/q/161942 https://stackoverflow.com/q/161942
gregmac 4y ago
Having never heard of this, it just looks like a list of regex patterns. Am I missing something? There's lots of lists of patterns available [1] If I came across this in code, I still have similar issues to regex. I have to question the source and what unit testing exists, that the tests cover the particular cases we care about, and if composed it needs tests anyway. For context, I always split my regexs to a function and add unit tests for them, and ask the same in PRs. It is one of the easiest tests to write and most beneficial: if I don't discover a case I missed while I come up with example inputs, it protects any future modifications by future devs. It also empowers devs that don't know regexs well to be able to modify them (because they know it won't break) and to approve a PR with one. I don't see how grok patterns avoid the need for tests (especially if composed), and if there are tests anyway, they're just a level of indirection. In the worst case, when there's inevitably a bug (or app-specific missing situation) in one of these, it's harder to fix than just having a regex directly. [1] https://github.com/mmkjony/awesome-regex-1 https://github.com/mmkjony/awesome-regex-1 , https://regexlib.com/DisplayPatterns.aspx?cattabindex=5&categoryId=6 https://regexlib.com/DisplayPatterns.aspx?cattabindex=5&cate...
- mekster 4y ago
  You're not missing anything but why reinvent a known regex pattern on every project or even on every instance of its usage in a project when the pattern can be defined to be used by anyone that never changes? In your case, putting the grok patterns in your regex specific function seems to make it more readable.
b3morales 4y ago
Apple's framework has [NSDataDetector][0] > A specialized regular expression object that matches natural language text for predefined data patterns. It's targeted at the kind of information end users are interested in: dates, email addresses, like that. [0]:https://developer.apple.com/documentation/foundation/nsdatadetector https://developer.apple.com/documentation/foundation/nsdatad...
Minor49er 4y ago
They don't have them because most of these checks are expensive and offer little benefit outside of things like log parsing A common example is email address validation. Trying to validate that an email address is well-formed with regex will often miss valid addresses (the example you linked would miss some in the HOSTNAME portion alone). It would be better to simply send a confirmation link to the address and wait for it to be clicked since that would ensure that the address is valid
- mekster 4y ago
  That's just email alone. I didn't think some milliseconds is worth more than correctness. Grok pattern is like commenting on your regexp, except you'll also get logical validation instead of bunch of characters you may make a mistake one day.
  - Minor49er 4y ago
    It was a simple example. As always, it depends on the application, though most applications do not need this level of checking Further, what happens if the grok pattern is wrong? Or if it changes?

pimlottc 4y ago

Perhaps because many devs, like me, haven't heard of "grok patterns"[0] before. Because of that, it took me a while to understand what your post was saying, since I was reading "grok" as a normal word. It's not a bad idea, though you'd want to make it a more formal standard; since it's just used internally in one project, it may be subject to change based on that project's own needs. There could also be more documentation, showing examples of strings that are accepted and not accepted by each pattern, as well as advice for generating compliant strings. 0: https://logz.io/blog/logstash-grok/ https://logz.io/blog/logstash-grok/

mekster 4y ago
It may have originated from a project but many other log parser projects such as Vector and fluentd have such support. https://vector.dev/docs/reference/vrl/examples/#parse_grok https://vector.dev/docs/reference/vrl/examples/#parse_grok https://github.com/fluent/fluent-plugin-grok-parser https://github.com/fluent/fluent-plugin-grok-parser

andy81 4y ago

.NET has something close in standard libraries, in that you can try converting a string to System.Net.IPAddress and other similar classes. It's for parsing rather than search though, you'd still need regex if e.g. searching unstructured logs for IPs. e.g. (powershell syntax for easy reproduction) returns parsed, or throw if invalid. [uri]"\\ServerName1\Folder\UserName" [version]'1.2.3.4' [ipaddress]'1.1.1.1'

gregmac 4y ago
Many of these have a `TryParse` method [1] which makes for nice syntax, and of course then you get a variable of the correct type. if (int.TryParse(userInput, out var number)) { total += number; } This is cleaner code compared to try/catch and usually faster (though debatable if it matters) [2]. [1] https://learn.microsoft.com/en-us/dotnet/api/system.net.ipaddress.tryparse?view=net-7.0 https://learn.microsoft.com/en-us/dotnet/api/system.net.ipad... [2] https://stackoverflow.com/q/161942 https://stackoverflow.com/q/161942

gregmac 4y ago

Having never heard of this, it just looks like a list of regex patterns. Am I missing something? There's lots of lists of patterns available [1] If I came across this in code, I still have similar issues to regex. I have to question the source and what unit testing exists, that the tests cover the particular cases we care about, and if composed it needs tests anyway. For context, I always split my regexs to a function and add unit tests for them, and ask the same in PRs. It is one of the easiest tests to write and most beneficial: if I don't discover a case I missed while I come up with example inputs, it protects any future modifications by future devs. It also empowers devs that don't know regexs well to be able to modify them (because they know it won't break) and to approve a PR with one. I don't see how grok patterns avoid the need for tests (especially if composed), and if there are tests anyway, they're just a level of indirection. In the worst case, when there's inevitably a bug (or app-specific missing situation) in one of these, it's harder to fix than just having a regex directly. [1] https://github.com/mmkjony/awesome-regex-1 https://github.com/mmkjony/awesome-regex-1 , https://regexlib.com/DisplayPatterns.aspx?cattabindex=5&categoryId=6 https://regexlib.com/DisplayPatterns.aspx?cattabindex=5&cate...

mekster 4y ago
You're not missing anything but why reinvent a known regex pattern on every project or even on every instance of its usage in a project when the pattern can be defined to be used by anyone that never changes? In your case, putting the grok patterns in your regex specific function seems to make it more readable.

b3morales 4y ago

Apple's framework has [NSDataDetector][0] > A specialized regular expression object that matches natural language text for predefined data patterns. It's targeted at the kind of information end users are interested in: dates, email addresses, like that. [0]:https://developer.apple.com/documentation/foundation/nsdatadetector https://developer.apple.com/documentation/foundation/nsdatad...

Minor49er 4y ago

They don't have them because most of these checks are expensive and offer little benefit outside of things like log parsing A common example is email address validation. Trying to validate that an email address is well-formed with regex will often miss valid addresses (the example you linked would miss some in the HOSTNAME portion alone). It would be better to simply send a confirmation link to the address and wait for it to be clicked since that would ensure that the address is valid

mekster 4y ago
That's just email alone. I didn't think some milliseconds is worth more than correctness. Grok pattern is like commenting on your regexp, except you'll also get logical validation instead of bunch of characters you may make a mistake one day.
- Minor49er 4y ago
  It was a simple example. As always, it depends on the application, though most applications do not need this level of checking Further, what happens if the grok pattern is wrong? Or if it changes?