4 ms·

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah thi

by Bender 3mo ago

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.

deleted 3mo ago
[deleted]
AussieWog93 3mo ago
Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.
- Bender 3mo ago
  Thrill of the game is a great way to put it.
  - inigyou 3mo ago
    I wish it was framed as "silly ways to block bots" then so that people wouldn't take it as a recommendation
milkshakes 3mo ago
i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?
- Bender 3mo ago
  I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access. For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.
  - milkshakes 3mo ago
    i guess what i'm curious about is where you would draw the line, and why. how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer? are search bots desirable? the internet wayback machine? how do you feel about browser extensions and greasemonkey scripts?
    - Bender 3mo ago
      I do not define user-agents, they announce themselves and present a particular behavior. I let them define their behavior and I respond accordingly. I have no need for search bots personally. If I had a complex site I would build my own search feature. If it was a commercial site I would pay for advertising on popular sites that were of related categories. Wayback is fun to play with but it's just a toy to me. There is no concept of domain ownership. Most of the snapshots of the domain I am using were from when someone else rented it. They claim to respect robots.txt but that is not entirely true. They crawl and save content even if a site says not to and as soon as the site is offline and robots.txt is no longer accessible they will display all the archived content. This means a person has to park their domain on a server containing a robots.txt that matches their intentions. Browser extensions are mostly invisible. What is obvious to me is when people are using "reader" applications as web clients. They should be concerned more than me. They are reading random sites run by strangers using apps that may or may not have been battle hardened, reviewed by third party penetration testers and so on. Some of them are vibe coded in unsafe languages.
Bender 3mo ago
As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers. I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.
- Barbing 3mo ago
  Bender, even got yourself a great username. One of the most responsive authors in memory.
  - Bender 3mo ago
    Thankyou. It's a fun topic for me. Hopefully others will find it interesting to tinker and experiment with as well.
- Bender 3mo ago
  As of now 6622 real humans made it through and a small number of what look to be bots. Curious if any of these people will try their own experiments on this topic.

by Bender 3mo ago

deleted 3mo ago

[deleted]

AussieWog93 3mo ago

Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.

Bender 3mo ago
Thrill of the game is a great way to put it.
- inigyou 3mo ago
  I wish it was framed as "silly ways to block bots" then so that people wouldn't take it as a recommendation

milkshakes 3mo ago

i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows. is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?

Bender 3mo ago
I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access. For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.
- milkshakes 3mo ago
  i guess what i'm curious about is where you would draw the line, and why. how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer? are search bots desirable? the internet wayback machine? how do you feel about browser extensions and greasemonkey scripts?
  - Bender 3mo ago
    I do not define user-agents, they announce themselves and present a particular behavior. I let them define their behavior and I respond accordingly. I have no need for search bots personally. If I had a complex site I would build my own search feature. If it was a commercial site I would pay for advertising on popular sites that were of related categories. Wayback is fun to play with but it's just a toy to me. There is no concept of domain ownership. Most of the snapshots of the domain I am using were from when someone else rented it. They claim to respect robots.txt but that is not entirely true. They crawl and save content even if a site says not to and as soon as the site is offline and robots.txt is no longer accessible they will display all the archived content. This means a person has to park their domain on a server containing a robots.txt that matches their intentions. Browser extensions are mostly invisible. What is obvious to me is when people are using "reader" applications as web clients. They should be concerned more than me. They are reading random sites run by strangers using apps that may or may not have been battle hardened, reviewed by third party penetration testers and so on. Some of them are vibe coded in unsafe languages.

Bender 3mo ago

As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers. I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.

Barbing 3mo ago
Bender, even got yourself a great username. One of the most responsive authors in memory.
- Bender 3mo ago
  Thankyou. It's a fun topic for me. Hopefully others will find it interesting to tinker and experiment with as well.
Bender 3mo ago
As of now 6622 real humans made it through and a small number of what look to be bots. Curious if any of these people will try their own experiments on this topic.