7 ms·
My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it do
by phito 1y ago
My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment
- dmesg 1y agoYes and it makes reading your logs needlessly harder. Sometimes I find an odd password being probed, search for it on the web and find an interesting story, that a new backdoor was discovered in a commercial appliance. In that regard reading my logs led me sometimes to interesting articles about cyber security. Also log flooding may result in your journaling service truncating the log and you miss something important.
- wvbdmp 1y agoYou log passwords?
- deleted 1y ago[deleted]
- zeta0134 1y agoJust about nobody logs passwords on purpose. But really stupid IoT devices accept credentials as like query strings, or part of the path or something, and it's common to log those. The attacker is sending you passwords meant for a much less secure system.
- SoftTalker 1y agoYou probably shouldn't log usernames then, or really any form fields, as users might accidentally enter a password into one of them. Kind of defeats the point of web forms, but safety is important!
- Dylan16807 1y agoAre you using a very weird definition of "logging" to make a joke? Web forms don't need any logging to work.
- SoftTalker 1y agoYou save them in a database. Probably in clear text. Six of one, half-dozen of the other.
- Dylan16807 1y agoA password being put into a normal text field in a properly submitted form is a lot less likely than getting into some query or path. And a database is more likely to be handled properly than some random log file. Six of one, .008 of a dozen of the other.
- hinkley 1y agoSo no access logs at all then? That sounds effective.
- stronglikedan 1y agoSure, why not. Log every secret you come across (or that comes across you). Just don't log your own secrets. Like OP said, it lead down some interesting trails.
- dpkirchner 1y agoI remember back before ssh was a thing folks would log login attempts -- it was easy to get some people's passwords because it was common for them to accidentally use them as the username (which are always safe to log, amirite?). All you had to do was watch for a failed login followed by a successful login from the same IP.
- deleted 1y ago[deleted]
- rollcat 1y ago> Sometimes I find an odd password being probed, search for it on the web and find an interesting story [...]. Yeah, this is beyond irresponsible. You know the moment you're pwned, __you__ become the new interesting story? For everyone else, use a password manager to pick a random password for everything.
- Thorrez 1y agoWhat is beyond irresponsible? Monitoring logs and researching odd things found there?
- JohnFen 1y agoHow are passwords ending up in your logs? Something is very, very wrong there.
- dmesg 1y agoDoes an attacking bot know your webserver is not a misconfigured router exposing its web interface to the net? I often am baffled what conclusions people come up with from half reading posts. I had bots attack me with SSH 2.0 login attempts on port 80 and 443. Some people underestimate how bad at computer science some skids are.
- socksy 1y agoAlso baffled that three separate people came to that conclusion. Do they not run web servers on the open web or something? Script kiddies are constantly probing urls, and urls come up in your logs. Sure it would be bad if that was how your app was architected. But it's not how it's architected, it's how the skids hope your app is architected. It's not like if someone sends me a request for /wp-login.php that my rails app suddenly becomes WordPress??
- JohnFen 1y ago> Do they not run web servers on the open web or something? Until AI crawlers chased me off of the web, I ran a couple of fairly popular websites. I just so rarely see anybody including passwords in the URLs anymore that I didn't really consider that as what the commenter was talking about.
- bob1029 1y agoThousands of requests per hour? So, something like 1-3 per second? If this is actually impacting perceived QoS then I think a gitea bug report would be justified. Clearly there's been some kind of a performance regression. Just looking at the logs seems to be an infohazard for many people. I don't see why you'd want to inspect the septic tanks of the internet unless absolutely necessary.
- zeta0134 1y agoOne of the most common issues we helped customers solve when I worked in web hosting was low disk alerts, usually because the log rotation had failed. Often the content of those logs was exactly this sort of nonsense and had spiked recently due to a scraper. The sheer size of the logs can absolutely be a problem on a smaller server, which is more and more common now that the inexpensive server is often a VM or a container.
- tedivm 1y agoDepending on what they're actually pulling down this can get pretty expensive. Bandwidth isn't free.
- dkiebd 1y agoI love the snark here. I work at a hosting company and the only customers who have issues with crawlers are those who have stupidly slow webpages. It’s hard to have any sympathy for them.
- egypturnash 1y agoIsn't it part of your job to help them fix that?
- 0x457 1y agoHow? They are hosting company, not a webshop.
- p3rls 1y agoi usually get 10 a second hitting the same content pages 10 times an hour, is that not what you guys are getting from google bot?
- wraptile 1y ago> thousounds of requests an hour from bots That's not much for any modern server so I genuinely don't understand the frustration. I'm pretty certain gitea should be able to handle thousands of read requests per minute (not per hour) without even breaking a sweat.
- q3k 1y agoServing file content/diff requests from gitea/forgejo is quite expensive computationally. And these bots tend to tarpit themselves when they come across eg. a Linux repo mirror. https://social.hackerspace.pl/@q3k/114358881508370524 https://social.hackerspace.pl/@q3k/114358881508370524
- rollcat 1y agoI think at this point every self-hosted forge should block diffs from anonymous users. Also: Anubis and go-away, but also: some people are on old browsers or underpowered computers.
- diggan 1y ago> Serving file content/diff requests from gitea/forgejo is quite expensive computationally One time, sure. But unauthenticated requests would surely be cached, authenticated ones skip the cache (just like HN works :) ), as most internet-facing websites end up using this pattern.
- q3k 1y agoYou can't feasibly cache large reposotories' diffs/content-at-version without reimplementing a significant part of git - this stuff is extremely high cardinality and you'd just constantly thrash the cache the moment someone does a BFS/DFS through available links (as these bots tend to do).
- Sesse__ 1y agoThere are _lots_ of objects in a large git repository. E.g., I happen to have a fork of VLC lying around. VLC has 70k+ commits (on that version). Each commit has about 10k files. The typical AI crawler wants, for every commit, to download every file (so 700M objects), every tarball (70k+ .tar.gz files), and the blame layer of every file (700M objects, where blame has to look back on average 35k commits). Plus some more. Saying “just cache this” is not sustainable. And this is only one repository; the only reasonable way to deal with this is some sort of traffic mitigation, you cannot just deal with the traffic as the happy path.
- ralferoo 1y agoWhat's worse is when you get bots blasting HTTP traffic at every open port, even well known services like SMTP. Seriously, it's a mail server. It identified itself as soon as the connection was opened, if they waited 100ms-300ms before spamming, they'd know that it wasn't HTTP because the other side wouldn't send anything at all if it was. There's literally no need to bombard a mail server on a well known port by continuing to send a load of junk that's just going to fill someone's log file.
- JdeBP 1y agoI remember putting dummy GET/PUT/HEAD/POST verbs into SMTP Relay softwares a quarter of a century ago. Attackers do not really save themselves time and money by being intelligent about this. So they aren't. There are attackers out there that send SIP/2.0 OPTIONS requests to the GOPHER port, over TCP.
- sidewndr46 1y agoIt's even funnier when you realize it is a request for a known exploit in WordPress. Does someone really run that on port 22?
- Sohcahtoa82 1y agoI HAVE heard of someone that runs SSH on port 443 and HTTPS on 22. It blocks a lot of bots, but I feel like just running on a high port number (10,000+) would likely do better.
- mjmas 1y agoI have a service running on a high port number on just a straight IPv4 and it does get a bit of bot traffic, but they are generally easy to filter out when looking at logs (well behaved ones have a domain in their User-Agent and bingbot takes my robots.txt into account. I dont think I've seen the Google crawler. Other bots can generally be worked out as anything that didn't request my manifest.json a few seconds after loading the main page)
- immibis 1y agoI have a small public gitea instance that got thousands of requests per hour from bots. I encountered exactly one actual problem: the temporary folder for zip snapshots filled up the disk since bots followed all snapshot links and it seems gitea doesn't delete generated snapshots. I made that directory read-only, deleted its contents, and the problem was solved, at the cost of only breaking zip snapshots. I experienced no other problems. I did put some user-agent checks in place a while later, but that was just for fun to see if AI would eventually ingest false information.
- kiitos 1y agoevery single IPv4 address in existence receives constant malicious traffic, from uncountably many malicious actors, on all common service ports (80, 443, 22, etc.) and, for HTTP specifically, to an enormous and growing number of common endpoints (mostly WordPress related, last I checked) if you put your server up on the public internet then this is just table stakes stuff that you always need to deal with, doesn't really matter whether the traffic is from botnets or crawlers or AI systems or anything else you're always gonna deal with this stuff well before the requests ever get to your application, with WAFs or reverse proxies or (idk) fail2ban or whatever else also 1000 req/hour is around 1 request every 4 seconds, which is statistically 0 rps for any endpoint that would ever be publicly accessible
- NegativeK 1y agoI've heard this point raised elsewhere, and I think it's underplaying the magnitude of the issue. Background scanner noise on the internet is incredibly common, but the AI scraping is not at the same level. Wikipedia has published that their infrastructure costs have notably shot up since LLMs started scraping them. I've seen similar idiotic behavior on a small wiki I run; a single AI company took the data usage from "who gives a crap" to "this is approaching the point where I'm not willing to pay to keep this site up." Businesses can "just" pass the costs onto the customers (which is pretty shit at the end of the day,) but a lot of privately run and open source sites are now having to deal with side crap that isn't relevant to their focus. The botnets and DDOS groups that are doing mass scanning and testing are targeted by law enforcement and eventually (hopefully) taken down, because what they're doing is acknowledged as bad. AI companies, however, are trying to make a profit off of this bad behavior and we're expected to be okay with it? At some point impacting my services with your business behavior goes from "it's just the internet being the internet" to willfully malicious.
- BlueTemplar 1y agoFrom your example (and many others), AI companies are engaging in DDoS too, so why wouldn't law enforcement target them too ?
- integralid 1y agoThousands per hour is 0.3-3 requests per second, which is... not a lot? I host a personal website and it got much more noise before LLMs were even a thing.