EPendragon

9 ms·

1. 121.▲
   by EPendragon 1y ago

   I like the idea of using HEAD or OPTIONS methods and have all bots access that so that they get a high level idea of what's going on, without the access to actual content, if the owner decided to block it.
2. 122.▲
   by EPendragon 1y ago

   I do like your suggestion of creating some standard that categorizes using function or purpose like you mention. This could simplify things granted that there is a way to validate the function and for spoofing to be hard to achieve. And yes
3. 123.▲
   by EPendragon 1y ago

   I think a better solution would be to block all the traffic, but have a comment in robots.txt with a way to be added onto a whitelist to scrape the contents of the resource. This puts a burden of requesting the access on the owner of the bo
4. 124.▲
   by EPendragon 1y ago

   There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypas
5. 125.▲
   by EPendragon 1y ago

   That was my hunch. My initial post on robots.txt: https://evgeniipendragon.com/posts/i-am-disallowing-all-craw... - revolved around blocking AI models from doing that because I do not believe that it will bring more tr
6. 126.▲
   by EPendragon 1y ago

   For me (as I'm sure for plenty other people as well) limiting traffic to actual users matters a lot because I'm using a free tier for hosting in the time being. Bots could quickly exhaust it, and your website could be unavailable
7. 127.▲
   by EPendragon 1y ago

   It is awesome to see that there are efforts to improve upon the old standards with modern web security standards
8. 128.▲
   by EPendragon 1y ago

   I definitely agree here. My initial response was to block everything, however you realize that web is complex and interdependent. I still believe that everyone should have autonomy over their online resources if they desire. But that comes
9. 129.▲
   by EPendragon 1y ago

   Would robot traffic be considered tracking in light of GDPR standards? As far as I know there are no regulatory rules in relation to enforcing robots behaviors outside of robots.txt, which is more of an honor system.
10. 130.▲
    by EPendragon 1y ago

    I kind of agree. The rules for safety should be simple, straightforward, and protect you in the "edge cases", i.e. following while not paying 100% of attention, protect you with a malicious actor in mind aka reckless driver, etc.
11. 131.▲
    by EPendragon 1y ago

    I believe there is a difference between education and critical thinking. One may not have a certain level of education, but could exercise a great degree of critical thinking. I think that education can help you understand the context of th
12. 132.▲
    I was wrong about robots.txt(evgeniipendragon.com)

    171 points by EPendragon 1y ago|204 comments
13. 133.▲
    PocketBase – open-source back end in 1 file(pocketbase.io)

    3 points by EPendragon 1y ago|0 comments
14. 134.▲
    Uiverse – The Largest Library of Open-Source UI Elements(uiverse.io)

    1 points by EPendragon 1y ago|0 comments
15. 135.▲
    React Bits – Animated UI Components for React(reactbits.dev)

    2 points by EPendragon 1y ago|0 comments
16. 136.▲
    Reverse Engineering Midword.com – Evgenii Pendragon(evgeniipendragon.com)

    2 points by EPendragon 1y ago|0 comments
17. 137.▲
    by EPendragon 1y ago

    I played this game for a bit. Really loved it! Later, I realized that it is possible to get the secret word using JS in the console: https://evgeniipendragon.com/posts/reverse-engineering-midwo...