4 ms·

Hey, @jarofgreen! Thank you for the feedback! 1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I wi

Hey, @jarofgreen! Thank you for the feedback! 1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms. 2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me. I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search. I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.

crowcroft 1y ago
Point 2 is true to an extent, assuming you aren't monetizing your traffic though, wouldn't earning citations still be more valuable than not showing up in Google at all? You should also consider that a large proportion of search is purely navigational. What if someone is trying to find your blog and is typing 'evgenii pendragon'. AI summaries don't show for this kind of search. Currently I can see your site is still indexed (disallowing Google won't remove a page from the index if it's already been indexed) and so you show in search results, but because you block Google in robots.txt it can't crawl content and so shows an awkward 'No information is available for this page.' message. If you continue to block Google in robots.txt eventually the links to your site will disappear entirely. Even if you really don't want to let AI summarize your content, I would at least allow crawlers to your homepage.
- EPendragon 1y ago
  That is what I first thought of when I started seeing more comments noting that it might be a good idea to allow for search engine traffic. All of these suggestions lead me to think about it some more. I definitely will need to understand the domain of SEO and search engine indexing a little deeper if I want to make an educated decision about it and not shoot myself in the foot in the long run. I appreciate your suggestion!

crowcroft 1y ago

Point 2 is true to an extent, assuming you aren't monetizing your traffic though, wouldn't earning citations still be more valuable than not showing up in Google at all? You should also consider that a large proportion of search is purely navigational. What if someone is trying to find your blog and is typing 'evgenii pendragon'. AI summaries don't show for this kind of search. Currently I can see your site is still indexed (disallowing Google won't remove a page from the index if it's already been indexed) and so you show in search results, but because you block Google in robots.txt it can't crawl content and so shows an awkward 'No information is available for this page.' message. If you continue to block Google in robots.txt eventually the links to your site will disappear entirely. Even if you really don't want to let AI summarize your content, I would at least allow crawlers to your homepage.

EPendragon 1y ago
That is what I first thought of when I started seeing more comments noting that it might be a good idea to allow for search engine traffic. All of these suggestions lead me to think about it some more. I definitely will need to understand the domain of SEO and search engine indexing a little deeper if I want to make an educated decision about it and not shoot myself in the foot in the long run. I appreciate your suggestion!