6 ms·
Magit manuals are available online again
- deleted 11mo ago[deleted]
- cratermoon 11mo ago"Thanks to LLM scrapers, hosting costs went up 5000% last month"
- ssivark 11mo agoUggghhhh! AI crawling is fast becoming a headache for self-hosted content. Is using a CDN the "lowest effort" solution? Or is there something better/simpler?
- embedding-shape 11mo agoNah, just add a rate limiter (which any public website should have anyways). Alternatively, add some honeypot URLs to robots.txt, then setup fail2ban to ban any IP accessing those URLs and you'll get rid of 99% of the crawling in half a day.
- LtWorf 11mo agoIf only it were that easy.
- macintux 11mo agoAnd for many people, "easy" is hardly the word to describe that. No wonder small businesses just put their information on Facebook instead of trying to manage a website.
- LtWorf 11mo agoI mean, people are held hostage by "professionals" that will set up some overcomplicated backend or vercel stuff instead of a static single html page with opening hours and the menu.
- nijave 11mo agoThe poison's also the cure! Just ask AI for a haproxy rate limit config
- LtWorf 11mo agoIt will give one. Will it work? No. You seem to not understand that AI crawlers mask as multiple clients to avoid rate limiting and are quite skilled at that.
- kstrauser 11mo agoI gave up after blocking 143,000 unique IPs hitting my personal Forgejo server one day. Rate limiting would have done literally nothing against the traffic patterns I saw.
- embedding-shape 11mo ago2 unique IPs or 200,000 shouldn't make a difference, ban the ones that make too many requests automatically and you basically don't have to do anything. Are people not using fail2ban and similar at all anymore? Used to be standard practice until I guess before people started using PaaS instead and "running web applications" became a different role than "developing web applications".
- nijave 11mo agoIt makes a difference if there's 143,000 unique IPs and 286,000 requests. I think that's what the parent post is saying (lots of requests but also not very many per IP since there's also lots of IPs) Even harder with IPv6 considering things like privacy extensions where the IPs intentionally and automatically rotate
- kstrauser 11mo agoYes, this is correct. I’d get at most 2 hits from an IP, spaced minutes apart. I went as far as blocking every AS that fetched a tripwire URL, but ended up blocking a huge chunk of the Internet, to the point that I asked myself whether it’d be easier to allowlist IPs, which is a horrid way to run a website. But I did block IPv6 addresses as /48 networks, figuring that was a reasonable prefixlen for an individual attacker.
- nijave 11mo agoDepending on the content and software stack, caching might be a fairly easy option. For instance, Wordpress W3 Total Cache used to be pretty easy to configure and could easily bring a small VPS from 6-10req/sec to 100-200req/sec. Also some solutions for generating static content sites instead of "dynamic" CMS where they store everything in a DB If it's new, I'd say the easiest option is start with a content hosting system that has built-in caching (assuming that exists for what you're trying to deploy)
- torben-friis 11mo ago>I immediately burned down my account with that hosting provider1, because they did not allow setting a spending limit. Is this true? He mentions the provider being AWS, surely some sort of threshold can be set?
- no_wizard 11mo agoAs far as I am aware, there is not. It’s been a long standing complaint about the platform.
- nijave 11mo agoIf it's AWS, yes it's true. All the billing is async and some as slow as daily (although it can be very granular/accurate). In addition, it's a pay-per-use platform
- forgotpwd16 11mo agoUnless something has changed recently, all you can do is set budget alerts on billing updates. Runaway costs for people simply testing AWS is common. (On the bright side, again unless something has changed recently, asking them in support to scrap them works.)
- electroly 11mo agoThere are two widely understood downsides of AWS: 1. High egress costs 2. No hard spending limits Both of these were problems for the author. I don't mean to "blame the victim" but the choice of AWS here had a predictable outcome. Static documentation is the easiest content to host and AWS is the most expensive way to host it.
- nijave 11mo agoReally high bandwidth costs in general. I've never worked anywhere large enough to hit them, but I've heard inter-AZ traffic in the same region can become quite expensive once you're big enough
- mystifyingpoi 11mo ago
- embedding-shape 11mo agoI don't know exactly what the website was, but if it's just HTML, CSS, some JS and some images, why would you ever host that on a "pay per visit/bandwidth" platform like AWS? Not only is AWS traffic extra expensive compared to pretty much any alternative, paying for bandwidth in that manner never made much sense to me. Even shared hosting like we did early 00s would have been a better solution for hosting a typical website than using AWS.
- anamexis 11mo agoAWS CloudFront pricing seems pretty competitive with other CDNs, at least for sites that are not very high traffic. https://aws.amazon.com/cloudfront/pricing/ https://aws.amazon.com/cloudfront/pricing/
- embedding-shape 11mo ago> AWS CloudFront pricing seems pretty competitive with other CDNs Sure, but it's unlikely you actually have to place a CDN in front of your manual, it's mostly text with few images. People default to using CDNs way too quickly today.
- anamexis 11mo agoThe whole point of CDNs is to host static assets, why wouldn't you use one? They are dead simple to use.
- DiabloD3 11mo agoFar more expensive than just having a dumb server somewhere at some normal host. People simply do not understand how expen$ive AWS is, and how little value it actually has for most people.
- anamexis 11mo agoMost people will never make it past the free tier on any of CloudFront, Cloudflare, Netlify, Render, etc. You can just drag and drop a folder and have a static site hosted in a few minutes.
- phplovesong 11mo agoI quit emacs 10 years ago. But i have fond menories from magit. Why was the manual taken offline?
- auslegung 11mo agohttps://github.com/magit/magit/issues/5472 https://github.com/magit/magit/issues/5472
- IceDane 11mo agoWhy didn't you just follow the link and find out?
- phplovesong 11mo agoLink did not show any data on the why. Im not reading a long blamegame on the why on some random issue.
- dbalatero 11mo agoit took me a minute to figure it out from the link. likely less time than you spent on this comment thread.
- phplovesong 11mo agoI cant get why people link to some random github issue instead of writing down a short TLDR and giving the option to read more. Guess its just lazyness.
- agumonkey 11mo agohe's waiting for the magit-transient shortcut :fff
- AceJohnny2 11mo agoI love Magit, it's the only git gui I can stomach (it helps that I use Emacs already). I donated a bit of money to help tarsius offset the cost of AWS LLM abuse, well deserved for the value I've gotten from his tools.
- tarsius 11mo agoThanks! Yesterday evening I saw that I had a few new sponsors and was wondering where they had come from. So in the end something good came of it. The one time donations covered the bill, and I also got a few new monthly sponsors. (Well, unless you also take the hours into account that it took me to move to new hosting, then its way way below minimal wage, but as a maintainer of free software, I am used to that by now.) Sooo... I guess I should take the opportunity and do a bit of marketing. I am still making a living maintaining Magit et al., so please consider sponsoring my day to day work too. Thanks!
- d4rkp4ttern 11mo agoMagit was my favorite terminal/TUI way to interact with Git, until I found GitUI.
- herewulf 11mo agoThis is a good reminder of the real financial costs incurred by maintainers of your favorite Emacs packages. Here's a nice repo with details on how to support them! https://github.com/tarsius/elisp-maintainers https://github.com/tarsius/elisp-maintainers Also worth pointing out that the author of Magit has made the unusual choice to make a living off developing Emacs packages. I've been happy to pitch in my own hard earned cash in return for the awesomeness that Magit is!