3 ms·
My search engine has some not-very-obvious tools for exploring the link graph, that will occasionally turn up interesting things. Similarity navigation: https:
by marginalia_nu 11mo ago
My search engine has some not-very-obvious tools for exploring the link graph, that will occasionally turn up interesting things.
Similarity navigation:
https://marginalia-search.com/site/simonwillison.net https://marginalia-search.com/site/simonwillison.net
Backlinks:
https://marginalia-search.com/site/simonwillison.net?view=links https://marginalia-search.com/site/simonwillison.net?view=li...
- ggirelli 11mo agoOut of curiosity, I see my website not being well indexed and I wondered whether it is because I include ``` User-agent: * Disallow: / ``` in my robots.txt. What should one add to allow Marginalia to crawl their website?
- marginalia_nu 11mo agoThis should work User-agent: search.marginalia.nu Allow: /
- ggirelli 11mo agoThank you! I just added it :)