3 ms·
Ask HN: Regex on a File or Stream
I just ran into this seemingly not that hard issue of trying to match a multi-line regex against a 3Gb text file. What is the right tool for this? grep and perl failed running into PCRE limits.
- burntsushi 3y agoripgrep should be able to handle it with the -U/--multiline flag.
- zaktoo2 3y agoCould you paste the regex portion of it please? Possibly some efficiencies to be gained there. You could also split the file into smaller chunks and then check the boundaries of the chunks.
- buzzdenver 3y agoYes, breaking it up would work, but that is not a solution for streams. The regex is dead simple: /Authorization: Basic (.*)\ngrant_type=refresh_token/ "." does not match newline, so I'm basically looking two lines that conform to a template. Specific cases can be transformed with some grep/awk magic, but IMO the concept of pattern matching against a stream is interesting regardless.
- zaktoo2 3y agoI missed the part where it was a stream. Also, is the grant_type guaranteed to be immediately after the token?
- jepler 3y agoMaybe some other PCRE-compatible implementation offers streaming. For instance, https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-hyperscan.html https://www.intel.com/content/www/us/en/developer/articles/t... says it has this feature, but of course given who it's from it may be tied to a single brand of CPU. github seems to be https://github.com/intel/hyperscan https://github.com/intel/hyperscan
- cvalka 3y agohttps://github.com/VirusTotal/yara https://github.com/VirusTotal/yara