6 ms·
How can it crunch through 1GB in 1s? Even just reading the data would take longer on any system I know.
by maxxxxx 8y ago
How can it crunch through 1GB in 1s? Even just reading the data would take longer on any system I know.
- Twirrim 8y agoThe SM0512G in this machine is capable of over 1.5GB/s in sequential read. It's also possible that the file is cached in memory (I ran grep a few times through the file before I carried out the specific measurements).
- avar 8y agoOn my puny laptop with an SSD I get ~400MB/s from cold cache, and ~5GB/s after the first run. So the answer is likely "it's in the FS cache". That's a very common use-case with grep. Either grepping a file you recently wrote, or running grep multiple times as you refine the regex, at which point the files will be in the FS cache.
- LeoPanthera 8y agoThis is answered in the article.
- scrooched_moose 8y agoNo it isn't. Unless grep is capable of skipping entire blocks (4096 bytes) it still has to pull that data off the disk.
- BeeOnRope 8y agoNo because most blocks will be found in the file system cache.
- kijin 8y agoNVMe SSD's can read about 3GB/s these days.
- Twirrim 8y agoFaster than that even, but you're not likely to see them in laptops at the moment :D Samsung's PM1725b (https://www.samsung.com/semiconductor/ssd/enterprise-ssd/MZPLL12THMLA/ https://www.samsung.com/semiconductor/ssd/enterprise-ssd/MZP...) has a Seq. Read of 6300 MB/s and Seq. Write of 3300 MB/s.
- karlding 8y agoAccording to Apple's MacBook Pro marketing page [0], their SSD supports sequential read speeds of up to 3.2GB/s. So this isn't as far-fetched as you imply, even if we're discounting other factors like the filesystem cache. [0] https://www.apple.com/macbook-pro/ https://www.apple.com/macbook-pro/
- maxxxxx 8y agoYes, I see these specs all the time but I never see them materialize in the real world. Agree about the cache but that's also an easy way to fool yourself when you are measuring performance.
- dspillett 8y ago> easy way to fool yourself when you are measuring performance In this case isn't the "already in RAM" test a more accurate reflection of performance anyway, as we are talking about the performance of grep and not the IO subsystem? There are many cases where grep's performance won't be bottlenecked by IO, or at least not impacted directly by a bottleneck there. Anywhere when the input is coming from another program, essentially, and even if that task is IO bound it might be sending output to grep much faster than it is consuming input (perhaps gunzip <file | grep searchtext). And in the case of searching a log file interactively, it is common that you won't just run grep of the file just once in a short space of time, instead doing it a couple of times as you refine your search, so for most of the runs it will be in cache (assuming the file is not to large).
- Rapzid 8y agoThe data could also be in the VFS cache/buffers for a number of reasons depending on the system setup and access pattern; including being written.
- kllrnohj 8y agoThis usage is literally ideal for pretty much any file I/O - it's a straight sequential read. Even spinning rust will achieve >400MB/s on this type of load. Take a look at the sequential burst speeds at QD1, first chart: https://www.anandtech.com/show/13761/the-samsung-970-evo-plus-ssd-review/7 https://www.anandtech.com/show/13761/the-samsung-970-evo-plu... Nearly ever SSD listed achieves well over 1GB/s in an actual benchmark, not just on a spec sheet. And these are just boring old off the shelf consumer drives. Nothing crazy.
- cyphar 8y agoAs TFA mentions, > #1 trick: GNU grep is fast because it AVOIDS LOOKING AT EVERY INPUT BYTE. TFA is incredibly short, and will explain it much better than I can.
- Arcuru 8y agoThat doesn't avoid it needing to read the data off disk though. AFAIK even SSDs still only read in 4K chunks. As the other reply mentioned though, it's just that MacBook SSDs are that fast.
- rujuladanh 8y agoAll SSDs are that fast (and way faster). Nothing special on Macbook ones.
- saagarjha 8y ago> it AVOIDS LOOKING AT EVERY INPUT BYTE This would not help, since the backing storage doesn't provide support for this kind of resolution. It would end up reading in the entire file anyways, unless your input string is on the order of an actual block.
- pathseeker 8y agoDMA would like a word with you
- Rapzid 8y agoOthers have mentioned the SSD and VFS cache, however spinning disks in a RAID configuration can easily surpass this in raw sequential read performance.
- BeeOnRope 8y agoNot really "easily" - the second test does 1.5 GB in 0.622s for a throughput of 2.41 GB/s. If we assume something like 100 MB/s sustained for spinning disks, that's a lot of disks to get to 2.41 GB/s even ignoring overheads.