3 ms·
I recently found https://commoncrawl.org/ https://commoncrawl.org/ and started working on my own personal search engine. I'm surprised how easy it is to actuall
by frmdstryr 4y ago
I recently found https://commoncrawl.org/ https://commoncrawl.org/ and started working on my own personal search engine. I'm surprised how easy it is to actually get decent results with just metadata that show up nowhere in Google/DDG.
- guerrilla 4y agoWhat's the license for this? It isn't obvious from the website and I got bored of looking.
- buo 4y agoNot a license, but there are some terms of use: https://commoncrawl.org/terms-of-use/ https://commoncrawl.org/terms-of-use/
- closedloop129 4y agoHeads up to https://news.ycombinator.com/item?id=31410042 https://news.ycombinator.com/item?id=31410042 marginalia_nu, the author of another search engine, is looking for ways to cooperate.