5 ms·

What are you doing that 50k daily hits is even worth investigating?

efitz 4y ago
Looking at his metrics and investigating an anomaly? Absolute magnitude is irrelevant. A standard deviation uptick vs baseline is always interesting.
- lfodofod 4y ago
  You see it’s a scraper, shrug, move on with your day. This is just standard background noise.
deleted 4y ago
[deleted]
tom_ 4y ago
Running his personal blog. I'm sure he's a lovely guy and that, and everybody likes him, and his writings are interesting, and so on - but it usually takes a bit more than this before 50,000 visitors per day stops being meaningful.
- costco 4y ago
  I have a few not particularly popular sites and like 60% of the nginx log entries are presumably botnets (unless someone legit is scanning from residential IPs in Colombia) doing automated vulnerability scans. I still don't get why people get so worked up about this stuff. It happens every day and there is very little you can do to stop it. I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space. Just use key based authentication and update your stuff. I don't really buy the load argument either - I hosted a site that was on the front page of HN for nearly a day on a $3/month Vultr VPS with 0 downtime.
  - masswerk 4y ago
    Well, we may still worry about green computing. 50K requests/day per site isn't nothing, just for scans. Is does sum up considerably.
  - lfodofod 4y ago
    >I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space There seems to be significant overlap between sysadmins who send handwritten abuse reports and “targeted individuals”/“gangstalking victims”.
deleted 4y ago
[deleted]
raggi 4y ago
Looks like it's DWiki, which is a python wiki. An uncached page load took 571ms, ICMP latency is 100ms. Frontend server is Apache, forcing a Connection:close on every serve (may actually be an advantage against the bot, unless it is also fetching the assets). Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.
- lfodofod 4y ago
  >Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top. No way a scraper running at less than 1rps is anywhere near a core-day without some horrible misconfiguration, it should be way easier to address that than to fight a war against scrapers. I have significantly more complicated flask sites handling 100+ rps on a single core, it doesn’t require any black magic.

Looking at his metrics and investigating an anomaly? Absolute magnitude is irrelevant. A standard deviation uptick vs baseline is always interesting.

lfodofod 4y ago
You see it’s a scraper, shrug, move on with your day. This is just standard background noise.

deleted 4y ago

[deleted]

tom_ 4y ago

Running his personal blog. I'm sure he's a lovely guy and that, and everybody likes him, and his writings are interesting, and so on - but it usually takes a bit more than this before 50,000 visitors per day stops being meaningful.

costco 4y ago
I have a few not particularly popular sites and like 60% of the nginx log entries are presumably botnets (unless someone legit is scanning from residential IPs in Colombia) doing automated vulnerability scans. I still don't get why people get so worked up about this stuff. It happens every day and there is very little you can do to stop it. I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space. Just use key based authentication and update your stuff. I don't really buy the load argument either - I hosted a site that was on the front page of HN for nearly a day on a $3/month Vultr VPS with 0 downtime.
- masswerk 4y ago
  Well, we may still worry about green computing. 50K requests/day per site isn't nothing, just for scans. Is does sum up considerably.
- lfodofod 4y ago
  >I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space There seems to be significant overlap between sysadmins who send handwritten abuse reports and “targeted individuals”/“gangstalking victims”.

deleted 4y ago

[deleted]

raggi 4y ago

Looks like it's DWiki, which is a python wiki. An uncached page load took 571ms, ICMP latency is 100ms. Frontend server is Apache, forcing a Connection:close on every serve (may actually be an advantage against the bot, unless it is also fetching the assets). Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.

lfodofod 4y ago
>Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top. No way a scraper running at less than 1rps is anywhere near a core-day without some horrible misconfiguration, it should be way easier to address that than to fight a war against scrapers. I have significantly more complicated flask sites handling 100+ rps on a single core, it doesn’t require any black magic.