3 ms·
Grep is a mapjob which takes hours to run. You'll be waiting quite a while until anyone can afford to quickly run regex queries against billions of documents! A
by greglindahl 15y ago
Grep is a mapjob which takes hours to run. You'll be waiting quite a while until anyone can afford to quickly run regex queries against billions of documents! And by then, there will be hundreds of billions of documents.
- _delirium 15y agoGoogle Code Search does do regexes against an impressively large set of documents nearly instantly, though it's clearly much smaller than the set of all webpages. It'd be interesting to know how much Google could scale it; could they handle 100x the number of documents in the current code search? 10,000x?
- tikhonj 15y agoOne thing you should note is that Google Code Search, as far as I know, supports regular expressions that are actually regular. This means you can't have an expression like /(ab..)\1/, for example. In all, re2, the regular expression engine that Google Code uses, is a very interesting project; you should read about it on its google code page: http://code.google.com/p/re2/ http://code.google.com/p/re2/.
- greglindahl 15y agoThe issue is not so much how much cpu time the regex evaluation takes up, it's the I/O time of loading every byte of every page we've crawled. That being said, re2 does look pretty cool... having a guarantee that nothing in an re can blow up is pretty nice, on top of the overall speed improvement.