4 ms·

Web servers should refuse requests for random, unnecessary URLs

2 points by petercooper 3y ago

petercooper 3y ago
This got me thinking about an idea that surely already exists, and I wondered if anyone here could tell me if it does. The idea is you'd publish some sort of manifest that HTTP servers and reverse proxies can use to determine valid URLs and then flat out to refuse to serve anything else. So you might end up with something for a host like: /.well-known/* /static/* /home /login /logout Basically every valid endpoint either specifically or wildcarded. I know apps are meant to deal with all of this themselves, but if you can even stop a reverse proxy sending through invalid URLs to a backend app, it seems like you could avoid a variety of problems. Or am I just being naive?
- LinuxBender 3y ago
  I like the idea. Like an enforced robots file. A manifest could be used with a script to write ACL's for load balancers, proxies and web servers. I have created the end-result manually in the past and with HAProxy one can drop requests for anything outside of the ACL's. There is even a silent drop that does not close the remote connection and just leaves the bots hanging until their timeout. I never actually created a script but wrote ACL's by hand. I could see someone creating a nifty script that would output ACL's for each server.
  - petercooper 3y ago
    Yeah, I guess it's a bit like a sitemap really, but more for servers (or even clients or intermediaries like Cloudflare) to validate requests before they so much get anywhere near an actual app process. It would also let you do things like remove files like WordPress's xml_rpc.php (but without actually removing the file) if you don't use that feature at all - no idea about nowadays, but it used to be a very common attack vector that I'd always rename.
    - LinuxBender 3y ago
      One of the nice side effects I had found doing this was that I could avoid logging anything dropped by my ACL's once I was confident that all of my legit URL's were accessible. It gets rid of all the bot noise.

2 points by petercooper 3y ago

This got me thinking about an idea that surely already exists, and I wondered if anyone here could tell me if it does. The idea is you'd publish some sort of manifest that HTTP servers and reverse proxies can use to determine valid URLs and then flat out to refuse to serve anything else. So you might end up with something for a host like: /.well-known/* /static/* /home /login /logout Basically every valid endpoint either specifically or wildcarded. I know apps are meant to deal with all of this themselves, but if you can even stop a reverse proxy sending through invalid URLs to a backend app, it seems like you could avoid a variety of problems. Or am I just being naive?

LinuxBender 3y ago
I like the idea. Like an enforced robots file. A manifest could be used with a script to write ACL's for load balancers, proxies and web servers. I have created the end-result manually in the past and with HAProxy one can drop requests for anything outside of the ACL's. There is even a silent drop that does not close the remote connection and just leaves the bots hanging until their timeout. I never actually created a script but wrote ACL's by hand. I could see someone creating a nifty script that would output ACL's for each server.
- petercooper 3y ago
  Yeah, I guess it's a bit like a sitemap really, but more for servers (or even clients or intermediaries like Cloudflare) to validate requests before they so much get anywhere near an actual app process. It would also let you do things like remove files like WordPress's xml_rpc.php (but without actually removing the file) if you don't use that feature at all - no idea about nowadays, but it used to be a very common attack vector that I'd always rename.
  - LinuxBender 3y ago
    One of the nice side effects I had found doing this was that I could avoid logging anything dropped by my ACL's once I was confident that all of my legit URL's were accessible. It gets rid of all the bot noise.