3 ms·
How would you go about parsing logs in R? The problem with logs is that lines have different number of columns depending on what that line is logging. What a
by wodenokoto 7y ago
How would you go about parsing logs in R?
The problem with logs is that lines have different number of columns depending on what that line is logging.
What a given line is logging can usually be determined by a regex (e.g. ends with a ip-address, starts with this words, etc., etc)
I'm genuinely curious, as this is a use-case where I have a lot of difficulty using python or R. I can see how grep + awk are strong as a preprocessor, as they can scan through a larger than memory file, and select columns from rows that match your criteria.
- tylermw 7y agoI was just doing this yesterday--the "base R" way would be to simply use readLines() and grep through the same way you would in the command line. You're right though, grep + awk are super useful and parsing larger-than-memory logs is where they really shine and the shortcomings of R become apparent. However, if your data fits in memory and you have a non-trivial analysis to perform, R is a great choice. I had a file with tabular data interspersed with metadata at random points and it was straightforward to parse and store the data in a custom data structure.