9 ms·
If you use GNU grep on text files, use the -a (--text) option
- deleted 7y ago[deleted]
- chaps 7y agoAlso works well on non-text files, similar to `strings`! I don't think it works as well, but can still be useful for quick checks.
- lizknope 7y agoI normally do: strings file | grep search_pattern
- cesarb 7y agoBut remember, always use "strings -a"! http://lcamtuf.blogspot.com/2014/10/psa-dont-run-strings-on-untrusted-files.html http://lcamtuf.blogspot.com/2014/10/psa-dont-run-strings-on-...
- steerablesafe 7y agoIt's the default since 2014: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=7fac9594c41ab180979bdf5927ff7f7e1d13a9e9 https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=c... edit: I would say that if you are doing forensics on an untrusted binary and you are not using a dedicated VM for it then you are not careful enough. objdump, nm are still attack vectors, not to mention debuggers and disassemblers.
- linsomniac 7y agoI mostly run into this on searching my environment: "set | grep whatever", now needs an "-a", possibly because of escape codes added to the environment a decade ago. Maybe the fix would be to only activate the "detect binary files" code if stdout isatty? Because it is a nice feature when I do a big grep to find something among my home directory or the entire filesystem. It is certainly annoying to get binary garbage in my terminal. Or maybe the binary detection could get smarter, maybe making the determination on a match-by-match basis ("This line I'm about to output is a kilobyte and half of it is non-printable", say). Though, ack-grep doesn't seem to avoid putting binary garbage on my terminal, so maybe reasonable to switch to something that isn't so clever? Most of my terminal greping is done with ack these days, so I'd probably be happy with gnu-grep disabling this cleverness.
- downerending 7y agoUsually you want this feature no matter where the output is going. Adding "-a" sucks, but it's not obvious how else this could work (and still be backward-compatible). IIRC the grep heuristic only considers a short prefix of the file. If the garbage comes later, you lose. Unfortunately, this makes things seem a bit unpredictable.
- linsomniac 7y agoI probably do not want this feature the case of a script, as this post talks about. Which is why I was suggesting if isatty(stdout). Most people do not want their shell scripts functionality to depend on whether there is a "high character" somewhere in it.
- gumby 7y ago> IIRC the grep heuristic only considers a short prefix of the file. If the garbage comes later, you lose. Unfortunately, this makes things seem a bit unpredictable. This was changed about five years ago to just keep looking. Which makes things a bit unpredictable in a different way.
- arendtio 7y agoActually, I never considered log files untrusted input, but as this example shows, it would be wise to do so.
- JoachimSchipper 7y agoFWIW, attacks like Javascript or SQL injection via logfiles are hardly unknown. Log files are plenty scary. ;-)
- mr_toad 7y agoI’ve seen ansi sequences embedded in (malformed) http requests in my server logs. Presumably they expect people to view the log in a terminal or editor with some sort of vulnerability.
- hinkley 7y agoSomeone did a POC of a CSRF against the admin interface for Cisco routers. They sent garbage packets that sufficiently confused the 'hex editor' display view in the admin web pages in such a way that it made a request to another page, changing permissions.
- avodonosov 7y agoAlso beware of CWE-117
- arendtio 7y agohttps://cwe.mitre.org/data/definitions/117.html https://cwe.mitre.org/data/definitions/117.html 'CWE-117: Improper Output Neutralization for Logs' That is something probably often forgotten when simply dumping some requests into a log, but at least it should be obvious that the source of the content is untrusted. On the other hand, a log file is a file on your server, so you would probably think of it as nothing dangerous, as everybody has cared about CWE-117, right? ;-)
- _jal 7y agoOh yes. And a target for PII redaction. Logs need a lot of attention.
- kwoff 7y agoAlso precede the file list by `--`. A very confusing thing can happen if a file happens to begin with a dash... (You can intersperse options like `-e pattern` among file names, if for some reason you wanted to do that.)
- iforgotpassword 7y agoBut that's not grep specific and generally a good idea, especially in scripts that get the file names from their command line, some input file or god knows where.
- steerablesafe 7y agoIf the command line utility doesn't support -- or equivalent then you can prepend "./" for untrusted filenames.
- gpvos 7y agoI just looked through the GNU grep history to see when it suddenly started being able to decide halfway through a file that it is binary after all; this is since 16 September 2014, so fairly recently. Before that, it just checked the first few kilobytes to decide, and didn't change its opinion afterwards. To me, this is a very nonintuitive change.
- iforgotpassword 7y agoOh yes. I was bitten by that when I piped around some tool's output that eventually went through grep. The tool's output was actually text but for some reason there was a null byte at the end which nobody noticed before. The fun part was that the way the data got chunked through the pipes was not deterministic, so sometimes you got the desired output from grep, but other times just "binary file matches", even when the raw output from the first tool was identical in both runs. That took quite some head scratching to figure out.
- gpvos 7y agoWhy is it that in recent years, with this and the more recent ls quoting fiasco, maintainers of longstanding UNIX utilities suddenly got the urge to fix what isn't broken?
- chubot 7y agoWhat's the 'ls quoting fiasco'? Actually I recently found that coreutils and ls behave fairly well with funny filenames: Here is an invalid utf-8 byte and then a valid utf-8 sequence $ x=$'\xce\xce\xbc' $ touch "$x" You can list it: $ ls ?μ And here 'ls' does better than other tools that display filenames. It shows the invalid byte and then keeps decoding with error recovery: $ ls --escape \316μ However GNU stat (which I think is also in coreutils) does something similar, but weirdly messed up: $ stat * File: ''$'\316''μ' (it looks like it's outputting a valid shell string, except with extra quotes) ----- Most command line tools are not aware of stuff like this. For example you can touch "x$ANSI_TERMINAL_CODES" and if you do "bash x??" or "python x??", then your terminal will change color because of the escape codes printed back to the terminal. I just changed Oil to use a well-defined format I called QSN (quoted string notation): http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#the-highlights http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#t... It adapts Rust's string literal syntax to express arbitrary byte strings precisely and losslessly. (JSON can't express arbitrary byte strings.) The QSN encoder does UTF-8 decoding with a specific error recovery mechanism. So it's basically like what ls and stat do, but it's more precise. (If anyone is interested in QSN, please contact me. I think it's more generally useful in a lot of places. It's something we already do but it's precise like JSON.)
- the_jeremy 7y agoshoutout to [RipGrep](https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep), which is generally faster, has more intelligent defaults (searches cwd by default, ignores files matching .gitignore), and can search through only certain text files (like your .java and .py files, say). Not affiliated, just found it worth the effort to learn some slightly different flags, though many are the same as normal grep.
- yencabulator 7y agoShoutout to git grep. I use `git grep -n -E --untracked` (via a small wrapper) a lot more than I use ripgrep. It's even faster than ripgrep for me.
- burntsushi 7y agoI'm always looking for new benchmarks. :) Do you have any easily accessible examples where git grep is materially faster than ripgrep? If not that's cool. Thanks!
- freedomben 7y agoAlso a ripgrep fan. I rolled my own search tool[1] due to dissatisfaction with the available options at the time, but I gave ripgrep an evaluation when it came out and it was really good. I ultimately still use my own tool because I'm very attached to the way grep colors/formats stuff (and I implemented the same scheme in findref) but I use ripgrep on large code bases (like the Linux kernel) since it performs a lot better due to various optimizations. [1] https://github.com/FreedomBen/findref https://github.com/FreedomBen/findref
- burntsushi 7y agoSmall note: if you want ripgrep's output to match grep's, then you can do alias rg="rg --no-heading" It will still show file paths and line numbers. Those can be disabled too, with `rg --no-heading --no-line-number --no-filename`.
- arendtio 7y agoDoes someone know if using grep on a binary file is somehow defined by POSIX? At a glance, I couldn't find a reference on the grep page: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html#top https://pubs.opengroup.org/onlinepubs/9699919799/utilities/g...
- jwilk 7y agoIt isn't. The page says: > The input files shall be text files. "Text file" is defined in https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_403 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1... : > A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character.
- arendtio 7y agoThank you :-) Interesting, especially the part about the LINE_MAX. Even though it kinda makes sense, I would never have thought that having a very long line makes a file a non-text file when all characters are 'normal' characters.
- bawolff 7y agoId kind of like that. Grepping in a directory also containing minified js (with ten thousand character long lines) is a pain
- seppel 7y agoSo, what does trigger grep to regard a file as binary/non-text? For me, it seems these are not only NUL characters. And I also think that this would not be a discussion if it was just about NUL. There is something else that grep does not like (escape characters? Invalid utf8? Wrong codepage? I don't know), but this makes the binary detection annoying.
- jwilk 7y ago
- exabrial 7y ago> 'LC_ALL=C' Wuff, reminds me of the completely incompatible difference between BSD sed and GNU sed
- eu 7y agoI normally use -I to skip binary files
- jindraj 7y agoHave you thought about environment variable GREP_OPTIONS? https://www.gnu.org/software/grep/manual/grep.html#Environment-Variables https://www.gnu.org/software/grep/manual/grep.html#Environme... You can define it at the beginning of the script.
- yellowapple 7y ago> As this causes problems when writing portable scripts, this feature will be removed in a future release of grep, and grep warns if it is used. Please use an alias or script instead.
- king_phil 7y agoI see this and suddenly it clicks! That is exactly why I couldn't import an SQL dump that I tried importing for days now, that is filtered with grep. Wow. And I was wondering all the time why mysql reported this strange error "SQL error in Binary file" when the .sql file was clearly a text file...
- spenrose 7y agoLove ack for source code and text docs: https://beyondgrep.com https://beyondgrep.com
- Skunkleton 7y agoI used ack for a while, switched to ag (I don’t remember why? FOTM maybe?), and finally ended up with ripgrep. If you haven’t tried ripgrep you definitely should. It has almost completely replaced gnu grep for me.
- petepete 7y agoI did the same but stayed with ag. For me, like with its contemporary find replacement fd, it's the UX that provides the most benefit. The speed benefit isn't really a huge factor; working how I'd expect, omitting ignored files, being able to specify file extensions to search, plus simple editor integration. Amazing.
- Skunkleton 7y agoI keep meaning to replace find, but never get around to it. I will check out fd. Ty.
- battery_cowboy 7y agoOh man, I've had this issue before and I just chose to nuke the logs and try again, thinking they were corrupted!
- viklove 7y agoI started using ripgrep a few years ago and haven't looked back. It's way faster, automatically excludes .gitignored files, and just has a bunch of common sense functionality. https://blog.burntsushi.net/ripgrep/ https://blog.burntsushi.net/ripgrep/
- zajio1am 7y agoThis 'feature' is especially irritating when one uses grep on some text files with legacy (non UTF-8) encoding, but has locale with UTF-8 encoding. The grep decides that regular text file is binary just because there are byte sequences that are not valid UTF-8 sequences.
- jodrellblank 7y agoDrum banging time: another good use for the UTF8 BOM.
- nerdponx 7y agoWhy is this even a thing? This is a serious anti-feature as far as I can see.. can someone clarify otherwise for me?
- _ZeD_ 7y agoTry `cat /usr/bin/*` in your terminal