11 ms·
Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of
by ImaCake 5y ago
Another way to speed grep up is to use something like ripgrep. Something I picked up lately, as a noob with bash scripting, is that you can run a whole bunch of things from a single bash script and they will be automatically allocated to different cores on whatever node/machine you are on. Just append `&` to each line and stick a `wait` at the end and voila, you have a very hack-y but robust way of running a bunch of processes in parallel.
- efxhoy 5y agoIf you care about errors you can use function wait_all { for job in $(jobs -p); do wait "${job}"; done } A lone wait wont tell you if any of the jobs fail.
- TacticalCoder 5y agoForking in the background is one way, you can also use GNU "parallel" ("apt-get install parallel" for those on Debian and the likes). For example I've got all my CD ripped to FLAC files, but my car only takes mp3 or wav... So I did a batch convert of FLAC to mp3, making sure to put all cores at work by piping the output of "find" into "parallel". Some commands also allows to directly parallelize (like, say, "make -j 16 ..." to build using 16 cores/hyperthreads).
- chockchocschoir 5y agoAnother tip is using wget2 instead of wget if you're mirroring a site (but this is more I/O tip than computationally heavy) https://gitlab.com/gnuwget/wget2/-/wikis/home https://gitlab.com/gnuwget/wget2/-/wikis/home Sadly, wget2 doesn't support WARC last time I checked, but wget2 comes with a `--max-threads` parameter that together with `--mirror` and `--tries` makes it trivial to mirror even the slowest websites out there. Edit: your parallel to `parallel` made me think of wget2 as I often see scripts that use `parallel` together with `wget` when `wget2` can be much better to use alone instead of pairing the two. Just wanted to add some context.
- burntsushi 5y agoDo note that unlike GNU (or BSD) grep, ripgrep will automatically use multiple threads to execute a search on a directory.
- chakkepolja 5y agoIs there a reason classic grep or find utilities do not use multithreading? It takes 5-6 seconds to search for some files on my machine. Would'nt it be an easy win if `find` had spawned 4 threads?
- burntsushi 5y agoI don't know. I'm not a maintainer of those projects. There's no obvious reason to me other than that it's hard to adapt a legacy code base that assumes a single thread to something that uses multiple threads. IIRC, there have been multiple attempts to make GNU grep multi-threaded. I believe folks have even submitted patches to the project. I don't follow it closely enough to know why they haven't been accepted. Sometimes the answer to "why don't they do this" is just uninteresting. :-)