5 ms·

> 2Gb of garbage I serve my robots.txt files with Transfer-Encoding: gzip and Content-Encoding: gzip, chunked and then just serve a two layered 10G gzip bomb (

> 2Gb of garbage I serve my robots.txt files with Transfer-Encoding: gzip and Content-Encoding: gzip, chunked and then just serve a two layered 10G gzip bomb (240kB file size) instead. Can recommend. Works pretty often.

pikseladam 5y ago
you are evil :)
samhw 5y ago
Doesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.
- cookiengineer 5y ago
  You can also insert a Disallow: /wp-admin/ in there and just serve the gzip bomb then :D
  - berkes 5y ago
    I have anyone requesting this automatic going to fail2ban¹, but serving a bomb at that url is actually much smarter. ¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.
    - hughrr 5y ago
      Actually now you mention that I’m going to see if I can set a trap for our pen testers :)
    - spookthesunset 5y ago
      Damn, that is a good idea. I wish I thought of that back when I ran a web forum.
      - berkes 5y ago
        There are fail2ban configs for most popular software ready to use. E.g. WP fail2ban¹ is a config that bans an IP after failed login attempts. The defaults it ships with are well-balanced between security and not-locking-yourself-out. Drupal, PHPBB, etc all have such generic config defaults online. For frameworks such as Rails, Django etc its a bit harder because the logging and url-schemes are not standard, but defined by the person developing in that framework. ¹https://wordpress.org/plugins/wp-fail2ban/ https://wordpress.org/plugins/wp-fail2ban/
    - Shadonototra 5y ago
      i hope you don't ban the IPs permanently nobody own their IP, and at some point a legitimate person will be assigned that IP and most people who do bad things on the internet use some proxies, usually residential ones it always makes me sad when people put such rules
      - nixgeek 5y ago
        Most crawlers aren’t going to come from ASN which are populated with eyeballs.
      - berkes 5y ago
        Fail2ban, by default, bans an IP for a limited time. For me that's 20mins. This means that commonly only about 30 IPs are on the banlist. The chance that anyone using the same IP as a bot who is visiting any of the apps and sites on that server is not 0%, but tiny. The only goal is to stop bots and worms from hammering my server. It hardly improves security; it shouldn't anyway.
  - alfiedotwtf 5y ago
    LOL that's actually really neat
- comonoid 5y ago
  Sane implementations do not read more than 500KiB of robots.txt contents.
durnygbur 5y ago
ur zo smart
lancesells 5y ago
Is this how a coordinated, large group of websites could "DDOS" Bing or Google?
Cthulhu_ 5y ago
I should try something like that for the endless bots that seem to try and access our site. Instead of outright banning them, put them in a special user group that gets a gzip bomb whenever they try to access the site again. Is there a way to make it so it doesn't outright make processes crash or lock up, but instead really slow them down / hang them up for a long time?
- cookiengineer 5y ago
  At some point I just got annoyed by all the 4chan script kiddies and went in active defense mode :D Also lookup "slowloris attack" -It's pretty efficient if the attacker has more network bandwidth than you have, and can even out the advantage. Won't be as effective against them having more networked machines though, so you have to use some other tricks against shodan, websnort, burp suite etc. There are some other tricks though, and you can also abuse HTTP smuggled headers (in the response) to specifically crash their proxies, for example :) Most common vulnerability scanners use some standard http libraries, so they are vulnerable to slowloris + chunked encodings, or they can't recover from a broken http trailing header, or from an http2/3 upgrade/downgrade loop that breaks their UDP packet parser. What also works is if you spoof their SOCKS5 proxy with UDP bind requests until they can't use it anymore. So for a lot of DDoS attack scenarios there are active defense options to mitigate the damage which the redteam can cause.
carapace 5y ago
That's delicious! BTW, how's your stealth browser project coming along? (I'd go look but I'm in a weird situation right now where I can't view most of the web. It's browser related so maybe this would amuse you: I'm running Firefox 91.5.1esr on FreeBSD from the official pkg. The vast majority of HTTPS sites give a "Your connection is not secure" error (Error code: NS_ERROR_NET_INADEQUATE_SECURITY) with no recourse (no button to let you make an exception or anything like that. It literally says, "The website administrator will need to fix the server first before you can visit the site.") The fix is to disable SPDY for HTTP2 connections. That works, but then FF crashes (with a core dump, hello FreeBSD) within ~10 seconds no matter what I do. There are a few clues, but I'm just not interested in shaving this yak. Frankly, I'm shopping for a new browser as I find the situation ridiculous. Your project seems way more interesting than what FF has become.)
- cookiengineer 5y ago
  Did you try to breakout using pwnat to get a public port? In enterprise grade environments it usually works :D I think I'd also try to run a UDP tunnel or try to use an amplification redirect on a UDP relaying proxy to get an open port that you can send requests through. Sometimes public DNS resolvers can help to punch a hole through the NAT because their IPs are allowlisted to unbreak secure browsing modes. > re: stealth I forked webkit into retrokit [1] these days and am gradually removing all APIs that could be abused for tracking purposes ... with the idea to create a minimalistic webview that is embeddable in a browser and still is able to parse/lex/render modern HTML5 and CSS4. This way I can later bundle nodejs with the retrokit webview that just displays the Browser UI that's served locally. [1] https://github.com/tholian-network/retrokit https://github.com/tholian-network/retrokit
  - carapace 5y ago
    I don't think it's a network issue (we're at home) because all the other browsers work. This version of FF is just weird about HTTP2/SPDY. I don't want to shave this yak. Retrokit looks great. :)

pikseladam 5y ago

you are evil :)

samhw 5y ago

Doesn't it mostly penalise the honest bots, though? Or do some bots use robots.txt files as a proxy sitemap? I'm not that familiar with this area.

cookiengineer 5y ago
You can also insert a Disallow: /wp-admin/ in there and just serve the gzip bomb then :D
- berkes 5y ago
  I have anyone requesting this automatic going to fail2ban¹, but serving a bomb at that url is actually much smarter. ¹anything .php,.asp and some other tech we never use, also phpmyadmin, cgi-bin, and some more, actually.
  - hughrr 5y ago
    Actually now you mention that I’m going to see if I can set a trap for our pen testers :)
  - spookthesunset 5y ago
    Damn, that is a good idea. I wish I thought of that back when I ran a web forum.
    - berkes 5y ago
      There are fail2ban configs for most popular software ready to use. E.g. WP fail2ban¹ is a config that bans an IP after failed login attempts. The defaults it ships with are well-balanced between security and not-locking-yourself-out. Drupal, PHPBB, etc all have such generic config defaults online. For frameworks such as Rails, Django etc its a bit harder because the logging and url-schemes are not standard, but defined by the person developing in that framework. ¹https://wordpress.org/plugins/wp-fail2ban/ https://wordpress.org/plugins/wp-fail2ban/
  - Shadonototra 5y ago
    i hope you don't ban the IPs permanently nobody own their IP, and at some point a legitimate person will be assigned that IP and most people who do bad things on the internet use some proxies, usually residential ones it always makes me sad when people put such rules
    - nixgeek 5y ago
      Most crawlers aren’t going to come from ASN which are populated with eyeballs.
    - berkes 5y ago
      Fail2ban, by default, bans an IP for a limited time. For me that's 20mins. This means that commonly only about 30 IPs are on the banlist. The chance that anyone using the same IP as a bot who is visiting any of the apps and sites on that server is not 0%, but tiny. The only goal is to stop bots and worms from hammering my server. It hardly improves security; it shouldn't anyway.
- alfiedotwtf 5y ago
  LOL that's actually really neat
comonoid 5y ago
Sane implementations do not read more than 500KiB of robots.txt contents.

durnygbur 5y ago

ur zo smart

lancesells 5y ago

Is this how a coordinated, large group of websites could "DDOS" Bing or Google?

Cthulhu_ 5y ago

I should try something like that for the endless bots that seem to try and access our site. Instead of outright banning them, put them in a special user group that gets a gzip bomb whenever they try to access the site again. Is there a way to make it so it doesn't outright make processes crash or lock up, but instead really slow them down / hang them up for a long time?

cookiengineer 5y ago
At some point I just got annoyed by all the 4chan script kiddies and went in active defense mode :D Also lookup "slowloris attack" -It's pretty efficient if the attacker has more network bandwidth than you have, and can even out the advantage. Won't be as effective against them having more networked machines though, so you have to use some other tricks against shodan, websnort, burp suite etc. There are some other tricks though, and you can also abuse HTTP smuggled headers (in the response) to specifically crash their proxies, for example :) Most common vulnerability scanners use some standard http libraries, so they are vulnerable to slowloris + chunked encodings, or they can't recover from a broken http trailing header, or from an http2/3 upgrade/downgrade loop that breaks their UDP packet parser. What also works is if you spoof their SOCKS5 proxy with UDP bind requests until they can't use it anymore. So for a lot of DDoS attack scenarios there are active defense options to mitigate the damage which the redteam can cause.

carapace 5y ago

That's delicious! BTW, how's your stealth browser project coming along? (I'd go look but I'm in a weird situation right now where I can't view most of the web. It's browser related so maybe this would amuse you: I'm running Firefox 91.5.1esr on FreeBSD from the official pkg. The vast majority of HTTPS sites give a "Your connection is not secure" error (Error code: NS_ERROR_NET_INADEQUATE_SECURITY) with no recourse (no button to let you make an exception or anything like that. It literally says, "The website administrator will need to fix the server first before you can visit the site.") The fix is to disable SPDY for HTTP2 connections. That works, but then FF crashes (with a core dump, hello FreeBSD) within ~10 seconds no matter what I do. There are a few clues, but I'm just not interested in shaving this yak. Frankly, I'm shopping for a new browser as I find the situation ridiculous. Your project seems way more interesting than what FF has become.)

cookiengineer 5y ago
Did you try to breakout using pwnat to get a public port? In enterprise grade environments it usually works :D I think I'd also try to run a UDP tunnel or try to use an amplification redirect on a UDP relaying proxy to get an open port that you can send requests through. Sometimes public DNS resolvers can help to punch a hole through the NAT because their IPs are allowlisted to unbreak secure browsing modes. > re: stealth I forked webkit into retrokit [1] these days and am gradually removing all APIs that could be abused for tracking purposes ... with the idea to create a minimalistic webview that is embeddable in a browser and still is able to parse/lex/render modern HTML5 and CSS4. This way I can later bundle nodejs with the retrokit webview that just displays the Browser UI that's served locally. [1] https://github.com/tholian-network/retrokit https://github.com/tholian-network/retrokit
- carapace 5y ago
  I don't think it's a network issue (we're at home) because all the other browsers work. This version of FF is just weird about HTTP2/SPDY. I don't want to shave this yak. Retrokit looks great. :)