4 ms·
I recently found https://commoncrawl.org/ https://commoncrawl.org/ and started working on my own personal search engine. I'm surprised how easy it is to actuall
by frmdstryr 5y ago
I recently found https://commoncrawl.org/ https://commoncrawl.org/ and started working on my own personal search engine. I'm surprised how easy it is to actually get decent results with just metadata that show up nowhere in Google/DDG.
- guerrilla 5y agoWhat's the license for this? It isn't obvious from the website and I got bored of looking.
- buo 5y agoNot a license, but there are some terms of use: https://commoncrawl.org/terms-of-use/ https://commoncrawl.org/terms-of-use/
- closedloop129 5y agoHeads up to https://news.ycombinator.com/item?id=31410042 https://news.ycombinator.com/item?id=31410042 marginalia_nu, the author of another search engine, is looking for ways to cooperate.