3 ms·
Something I would be interested in would be a true distributed “grep” for web pages. The service would take 2 regexes: 1 for the url and 1 for the content. It
by RcouF1uZ4gsC 4y ago
Something I would be interested in would be a true distributed “grep” for web pages.
The service would take 2 regexes: 1 for the url and 1 for the content. It would then return the first 10,000 results of the pages that match both.
The latency would likely be in hours, and I don’t think advertising would work for monetization, so it would be a pay for use service, but for deep searches, this might be valuable.
- marginalia_nu 4y agoIt would likely not be very useful even ignoring the speed aspect. A lot of work goes into making a search engine understand how a term relates to a website, and to rank the results accordingly. You might get lucky and search for something that's in the goldie locks zone between underspecified and overspecified, and actually get good results, but in most cases you'd either get nothing or just noise. Unix grep works well because you're typically grepping in hundreds or thousands of documents, not hundreds of millions or billions.