3 ms·
YouTube fed me a video I’d watched a couple years ago about a log searching tool. Compressed data, decompressed into cpu cache, and then scanned. Almost no inde
by hinkley 6y ago
YouTube fed me a video I’d watched a couple years ago about a log searching tool. Compressed data, decompressed into cpu cache, and then scanned. Almost no indexing at all, just cpu cache for speed.
We spend a lot of time arguing with the cpu. If we give it something it’s actually happy doing, it almost doesn’t matter how stupid that thing is, because it’s stupid fast.
- owaislone 6y agoInteresting. Would really appreciate if you could share the video with us. Thanks
- posnet 6y agoThis is one of the main selling points of Blosc https://blosc.org/pages/blosc-in-depth/ https://blosc.org/pages/blosc-in-depth/
- e12e 6y agoI couldn't find the talk hinted at, but this blog post seems to touch on some of the same ideas: https://www.humio.com/whats-new/blog/how-fast-can-you-grep/ https://www.humio.com/whats-new/blog/how-fast-can-you-grep/
- devbug 6y agoBrute force search is how Scaylr does it. Here’s an article they wrote about their implementation. https://www.scalyr.com/blog/searching-1tb-sec-systems-engineering-before-algorithms/ https://www.scalyr.com/blog/searching-1tb-sec-systems-engine...
- mhh__ 6y ago> Compressed data, decompressed into cpu cache, and then scanned. Almost no indexing at all, just cpu cache for speed. Where are these programs not using the CPU cache?
- chii 6y agoassume they meant decompress small enough a block to fit the CPU cache, and do in-memory scans.
- hinkley 6y agoIndexes and tree structures involve pointer walking. Nothing is even guaranteed to be in main memory, nevermind L2 cache. These guys apparently went straight from disk (linear reads) to L1/L2.