EPendragon

8 ms·

91.
Image Generation for Designers – Recraft(recraft.ai)
3 points by EPendragon 1y ago|0 comments
92.
Learn Anything(learn-anything.xyz)
2 points by EPendragon 1y ago|0 comments
93.
Free Open Source Tailwind CSS v4 Components(hyperui.dev)
3 points by EPendragon 1y ago|0 comments
94.
by EPendragon 1y ago
I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.
95.
by EPendragon 1y ago
You are bringing a great point. In some cases having your data as available as possible is the best thing you can do for your business. Letting them crawl and scrape creates means by which your product is found and advertised. In other case
96.
by EPendragon 1y ago
I agree. I mostly mean that it is good to strive towards a system of rules that will be easy to follow and difficult to break by default. That is an ideal case. In reality, it is never that simple.
97.
by EPendragon 1y ago
This is a really cool tool. I haven't seen it before. Thank you for sharing it! On their README.md they state: > This program is designed to help protect the small internet from the endless storm of requests that flood in from AI co
98.
by EPendragon 1y ago
I have not heard of humans.txt before. It is apparently used for acknowledgement and crediting the dev team who created the resource.
99.
by EPendragon 1y ago
It is so interesting to track this technology's origin back to the source. It makes sense that it would come from a background of limited resources where things would break if you overwhelm it. It didn't take much to do so.
100.
by EPendragon 1y ago
Yeah, I don't even think I 100% understand how the algorithm and moderation works here. I will consider to bring in more content that is other than my own. I did read the guidelines for HN, and I saw that it encourages members to share
101.
by EPendragon 1y ago
To be honest, I also think that the discussion has become more interesting than what I wrote in the article. I have learned quite a few things for myself. And on top of that it has been great to hear some feedback in regards to some thought
102.
by EPendragon 1y ago
It is an interesting tidbit. I personally don't need Google to remove it from indexing. It is more of a "I don't care if they index it". I mostly care about the scrapping and not indexing. I do understand that these term
103.
by EPendragon 1y ago
> Computation was still expensive I think this is still relevant today in cases where there are not many resources available: think free tiers, smallest fixed cost/fixed allocation scenarios, etc.
104.
by EPendragon 1y ago
It definitely was (and still is to a degree) self-inflicted :) It is fun to learn something when you discover an unintended consequence and then work backwards from it.
105.
by EPendragon 1y ago
That is what I first thought of when I started seeing more comments noting that it might be a good idea to allow for search engine traffic. All of these suggestions lead me to think about it some more. I definitely will need to understand t
106.
by EPendragon 1y ago
I believe that this might give birth to new tools, protocols, and tech which will enable the next evolution of the Open Web into something akin to Protected Open Web. I very much dislike the invasive scrapping approaches. If something were
107.
by EPendragon 1y ago
I can see how it could lead to more overhead when communicating with the requesters. That could be a lot in the event that lots of them might want to crawl your resource. I can see the argument that if I want to hide something, I should put
108.
by EPendragon 1y ago
Fair enough. It should be sufficient to say one way or the other.
109.
by EPendragon 1y ago
I had to discover the Post Inspector tool which was very helpful as it provided error messages. I didn't know that about the doctype tags. I must have had them right from the beginning. Didn't encounter those issues. It good to kn
110.
by EPendragon 1y ago
A brilliant analogy. Robots doesn't provide access controls. Authentication and authorization do.
111.
by EPendragon 1y ago
I believe that as search engines continue to move toward AI summaries and responses, it will reduce the traffic to the websites since most people will be ok accepting the answers that the AI gave them. My approach right now is to rely on so
112.
by EPendragon 1y ago
That was the first thing that I have learnt about the robots.txt file. Even RFC 9309 Robots Exclusion Protocol document: https://www.rfc-editor.org/rfc/rfc9309.html - mentions: > These rules are not a form of access
113.
by EPendragon 1y ago
Thank you for that note! I didn't know that. It is something I will need to figure out: either I'm ok with not being in the search engine OR I will update the robots.txt. Currently I'm relying on the social media traffic and
114.
by EPendragon 1y ago
One way to test if the model respects robots.txt could be to ask the model if it can scrape a given URL. One thing that it doesn't address though is the scrapping of the training data for the model. That area feels more like a wild wes
115.
by EPendragon 1y ago
There is an interesting article about AI tarpits that addresses a similar issue: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-... The argument is basically to have them scrape your website inde
116.
by EPendragon 1y ago
I think a paragraph could have been enough to describe the issue. My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a
117.
by EPendragon 1y ago
Hey, @jarofgreen! Thank you for the feedback! 1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your su
118.
by EPendragon 1y ago
Honestly, I am also surprised how this got on the frontpage. This was supposed to be a small post of what I have learnt in the process of fixing my LinkedIn previews. I don't know how we got here.
119.
by EPendragon 1y ago
That's exactly it. It was one of those unintended consequences of blocking everything that led me down the road of figuring it out. Like other commenters have indicated, I will likely need to go back and allow some other social media t
120.
by EPendragon 1y ago
I mean it was a realization for me, although I wouldn't call it profound. To your point, it was closer to obliviousness, which led me to learn more about Open Graph Protocol details and how Robots Exclusion Protocol works. I try to wri

More ›