4 ms·

Interesting how for a (junior) dinosaur, like me, this was never a problem: I still run my mail, web, nextcloud, xmpp, forgejo, etc server on a former thin clie

by pmlnr 2y ago

Interesting how for a (junior) dinosaur, like me, this was never a problem: I still run my mail, web, nextcloud, xmpp, forgejo, etc server on a former thin client under a cupboard. With a symmetrical fiber 1G connection it's been surprisingly reliable. FAQ: Yes, you can run self hosted mail. Yes, it's complex. No, it's not hard, but it takes time, as it is complex - if you want to understand it. If not, go for something like https://maddy.email/ https://maddy.email/ Yes, I have a fixed IP address with a reverse DNS entry configured. My ISP is zen.co.uk. Yes, I have reverse DNS, DKIM, DMARC, SPF, even mta-sts. No, I'm not switching from XMPP to Matrix - it's too server heavy, XMPP is more client oriented, which is my preference. I know about the UK's online safecty act, I used https://onlinesafetyact.co.uk/ra_my_self_hosted_single_user_mastodon_instance/ https://onlinesafetyact.co.uk/ra_my_self_hosted_single_user_... as a template to create mine. I think the act itself is basically useless and just paperwork, but something along it's ideas is actually needed. I block many AI crawlers from accessing code and photos, as eg. Claude is so aggressive that it's code crawlers makes my system sluggish, and I have no will to let anyone use my photos beyond printing them for their wall as decoration.

bsafta 2y ago
Can you please write more details about "I block many AI crawlers from accessing code and photos"? The bots are trying to access your nextcloud instance? I'm also self hosting a few services, including nextcloud.
- pmlnr 2y ago
  No, not nextcloud, it's the photos on my website. They are CC-BY-NC-ND-4.0 licensed, which genAI doesn't respect in any form. I added these in nginx.conf: map $http_user_agent $blocked_user_agent { default 0; "~*AI2Bot" 1; "~*AI2Bot-Dolma" 1; "~*Amazonbot" 1; "~*anthropic-ai" 1; "~*anthropic.com" 1; "~*Applebot" 1; "~*Applebot-Extended" 1; "~*AwarioBot" 1; "~*AwarioRssBot" 1; "~*AwarioSmartBot" 1; "~*Bytespider" 1; "~*CCBot" 1; "~*ChatGPT-User" 1; "~*ClaudeBot" 1; "~*Claude-Web" 1; "~*cohere-ai" 1; "~*cohere-training-data-crawler" 1; "~*DataForSeoBot" 1; "~*Diffbot" 1; "~*DuckAssistBot" 1; "~*FacebookBot" 1; "~*FriendlyCrawler" 1; "~*Googlebot-Extended" 1; "~*Google-CloudVertexBot" 1; "~*Google-Extended" 1; "~*GoogleOther" 1; "~*GoogleOther-Image" 1; "~*GoogleOther-Video" 1; "~*GPTBot" 1; "~*iaskspider/2.0" 1; "~*ICC-Crawler" 1; "~*ImagesiftBot" 1; "~*img2dataset" 1; "~*ISSCyberRiskCrawler" 1; "~*Kangaroo Bot" 1; "~*Meltwater" 1; "~*Meta-ExternalAgent" 1; "~*Meta-ExternalFetcher" 1; "~*OAI-SearchBot" 1; "~*Omgili" 1; "~*Omgilibot" 1; "~*openai.com" 1; "~*PanguBot" 1; "~*peer39_crawler" 1; "~*PerplexityBot" 1; "~*PetalBot" 1; "~*Scrapy" 1; "~*Seekr" 1; "~*SemrushBot" 1; "~*SemrushBot-OCOB" 1; "~*Sentibot" 1; "~*Sidetrade indexer bot" 1; "~*Timpibot" 1; "~*TurnitinBot" 1; "~*VelenPublicWebCrawler" 1; "~*webmeup-crawler.com" 1; "~*Webzio-Extended" 1; "~*YouBot" 1; } and then in each site's config: location / { if ($blocked_user_agent) { access_log /var/log/nginx/blockedbot.log ncsa; return 401; } But it's far from perfect. For better results, https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo... is probably better, but it was a tad too much for my needs.
  - deleted 2y ago
    [deleted]
  - bsafta 2y ago
    Thanks!
dracotomes 2y ago
I self host a mail server too. Static IPs are not available for a residential connection where I live which makes PTR and SPF records hard. What I've been doing is using Oracle's "always free" VM to host VyOS as an ingress/egress router. It's been pretty easy to set up the mail server itself using iRedMail. I've had some problems with spamhaus and outlook because that whole IP range is not trustworthy but otherwise it's just worked (Spamhaus was nice enough to put me on a whiltelist). I know hosting at Oracle is not moving away from US cloud services but I set this up two years ago and this could be achieved using any cheap or free VPS.
csomar 2y ago
If you have a fixed IP with a good reputation then 95% of your email problems are solved. If you use a client (like the iphone email client), your setup is really simple (as in set it up once and forget about it). Providers will also try re-sending email if it doesn't reach you, so availability is not that big of a deal. That being said, the major issue starting up is having that highly reputable ip/domain.

by pmlnr 2y ago

bsafta 2y ago

Can you please write more details about "I block many AI crawlers from accessing code and photos"? The bots are trying to access your nextcloud instance? I'm also self hosting a few services, including nextcloud.

pmlnr 2y ago
No, not nextcloud, it's the photos on my website. They are CC-BY-NC-ND-4.0 licensed, which genAI doesn't respect in any form. I added these in nginx.conf: map $http_user_agent $blocked_user_agent { default 0; "~*AI2Bot" 1; "~*AI2Bot-Dolma" 1; "~*Amazonbot" 1; "~*anthropic-ai" 1; "~*anthropic.com" 1; "~*Applebot" 1; "~*Applebot-Extended" 1; "~*AwarioBot" 1; "~*AwarioRssBot" 1; "~*AwarioSmartBot" 1; "~*Bytespider" 1; "~*CCBot" 1; "~*ChatGPT-User" 1; "~*ClaudeBot" 1; "~*Claude-Web" 1; "~*cohere-ai" 1; "~*cohere-training-data-crawler" 1; "~*DataForSeoBot" 1; "~*Diffbot" 1; "~*DuckAssistBot" 1; "~*FacebookBot" 1; "~*FriendlyCrawler" 1; "~*Googlebot-Extended" 1; "~*Google-CloudVertexBot" 1; "~*Google-Extended" 1; "~*GoogleOther" 1; "~*GoogleOther-Image" 1; "~*GoogleOther-Video" 1; "~*GPTBot" 1; "~*iaskspider/2.0" 1; "~*ICC-Crawler" 1; "~*ImagesiftBot" 1; "~*img2dataset" 1; "~*ISSCyberRiskCrawler" 1; "~*Kangaroo Bot" 1; "~*Meltwater" 1; "~*Meta-ExternalAgent" 1; "~*Meta-ExternalFetcher" 1; "~*OAI-SearchBot" 1; "~*Omgili" 1; "~*Omgilibot" 1; "~*openai.com" 1; "~*PanguBot" 1; "~*peer39_crawler" 1; "~*PerplexityBot" 1; "~*PetalBot" 1; "~*Scrapy" 1; "~*Seekr" 1; "~*SemrushBot" 1; "~*SemrushBot-OCOB" 1; "~*Sentibot" 1; "~*Sidetrade indexer bot" 1; "~*Timpibot" 1; "~*TurnitinBot" 1; "~*VelenPublicWebCrawler" 1; "~*webmeup-crawler.com" 1; "~*Webzio-Extended" 1; "~*YouBot" 1; } and then in each site's config: location / { if ($blocked_user_agent) { access_log /var/log/nginx/blockedbot.log ncsa; return 401; } But it's far from perfect. For better results, https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo... is probably better, but it was a tad too much for my needs.
- deleted 2y ago
  [deleted]
- bsafta 2y ago
  Thanks!

dracotomes 2y ago

I self host a mail server too. Static IPs are not available for a residential connection where I live which makes PTR and SPF records hard. What I've been doing is using Oracle's "always free" VM to host VyOS as an ingress/egress router. It's been pretty easy to set up the mail server itself using iRedMail. I've had some problems with spamhaus and outlook because that whole IP range is not trustworthy but otherwise it's just worked (Spamhaus was nice enough to put me on a whiltelist). I know hosting at Oracle is not moving away from US cloud services but I set this up two years ago and this could be achieved using any cheap or free VPS.

csomar 2y ago

If you have a fixed IP with a good reputation then 95% of your email problems are solved. If you use a client (like the iphone email client), your setup is really simple (as in set it up once and forget about it). Providers will also try re-sending email if it doesn't reach you, so availability is not that big of a deal. That being said, the major issue starting up is having that highly reputable ip/domain.