8 ms·

Web servers should refuse requests for random, unnecessary URLs

86 points by ingve 3y ago

deleted 3y ago
[deleted]
bradley13 3y ago
The requested URL did not exist, so the server must return some response other than 200. It could be not-found, it could be forwarding, or it could be something else. However, 200 (success) is simply wrong.
- wang_li 3y ago
  >However, 200 (success) is simply wrong. What makes it wrong? HTTP/HTTPS URLs/URIs don't mean files on the disk. You can return 200 as a default if you like.
  - shkkmo 3y ago
    Returning a 200 response for a resource that doesn't exist is wrong. The correct response code would be a 3XX or 4XX response code, depending on the client behavior that you want. In this specific case, a 303 response that redirects to the homepage would be most correct. Alternatively, you could return a 404 response alongside the default content.
    - phlakaton 3y ago
      And how do you know the resource doesn't exist in this case? The server says it does. The point is, "resource" is an abstraction that servers may implement as they wish. The only requirements, per HTTP, is that (1) it's something that can be identified with a URI (satisfied in this case), and (2) it has information associated with it that can be retrieved and/or managed via the HTTP protocol (satisfied in this case). We might imagine that the set of resources should be finite, or strictly mapped to extant data records in our system – but these are not requirements.
      - shkkmo 3y ago
        > We might imagine that the set of resources should be finite, or strictly mapped to extant data records in our system – but these are not requirements. Clearly they aren't requirements as I can think of many valid use cases that violate them. To me, creating an infinite number of identical aliases for a single resource is clearly wrong. The only reason to do it is either incompetence or laziness.
      - cellularmitosis 3y ago
        We can argue about the semantics of "wrong" all day, but the lesson here is that not following standards and conventions creates extra work for everyone downstream.
        moring 3y ago
        They followed the standard though. They returned 200 for a resource that exists. Many responses here seem to be based on the misunderstanding that this is about "not following a standard is sometimes okay-ish", when it's actually about the fact that the standard does not rigorously define what a "resource" is and what it means for the resource to "exist", other than a circular definition based on status codes.
        cellularmitosis 3y ago
        Serving an identical resource at all possible paths is no one's idea of following established conventions. That's what redirects are for.
        phlakaton 3y ago
        Is it an identical resource? How do you know? But let's say they are identical. Where in the HTTP spec does it say identical resources may not be served at different endpoints? Nowhere, of course. You talk about violating conventions and standards. The conventions are defined by the HTTP spec. The standard is the HTTP spec. Neither are, so far as I can tell, being violated here.
    - pravus 3y ago
      URLs refer to resources, not files.
    - eddythompson80 3y ago
      > depending on the client behavior that you want. What if you want to confuse the client?
    - tremon 3y ago
      Returning a 200 response for a resource that doesn't exist is wrong Isn't that just begging the question? What defines whether a resource exists? Is it the client's knowledge about the webserver backend configuration, or the server's? If a server returns a 200 response, the resource exists, by definition. It may or may not have data, but again, that's not up to the client to decide. If the server says the resource exists and has no data, then that's the authoritative answer.
    - toast0 3y ago
      > The correct response code would be a 3XX or 4XX response code, depending on the client behavior that you want. If the client behavior you want is to display your styled error page, that has historically meant returning a 200 status code, because some user-agents prefer their own error displays.
  - dekhn 3y ago
    I had to look this up recently, but Tim Berners Lee specifically said that URIs should be treated more or less identically to files on disk. That would mean that accessing a web server that is hosting a static directory should return not found for missing files. But this was just a guideline and it was never followed in practice.
    - lifeonlars 3y ago
      How the web works has changed quite a lot since the early 1990s though.
  - mock-possum 3y ago
    RFC 9110 - https://httpwg.org/specs/rfc9110.html#overview.of.status.codes https://httpwg.org/specs/rfc9110.html#overview.of.status.cod... It’s like saying “what makes it wrong to signal left but turn right?” It makes the component you’re operating behave less predictably, resulting in a less stable system overall. Follow the spec.
    - moring 3y ago
      They followed the spec though. They returned 200 for a resource that exists. > It’s like saying “what makes it wrong to signal left but turn right?” No, it's not like that. Signalling left but turning right would be like returning a 200 for a resource that does not exist.
- andrewaylett 3y ago
  And the penalty for being "wrong" is..? You can't write code relying on random third-party code doing what it's "mandated" to do by the standard, because malicious code exists and so do malicious services. So as a client, if you're connecting to services you don't control, you need to be able to handle misbehaving third-party code. And what it means for a request to be "successful" is entirely at the discretion of the author of the software. I think it's fairly clear that the software looked for a response, found one, and successfully returned it to the client. Which isn't to say that the code is optimal, necessarily, but unless you have a contract to say otherwise then I don't think you've any basis to claim that it must do anything in particular.
  - shkkmo 3y ago
    > I don't think you've any basis to claim that it must do anything in particular. The relevant part of the spec says that returning a 4XX code is a SHOULD [0] not a MUST. That means: > This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course [1] [0] https://www.rfc-editor.org/rfc/rfc9110.html#name-client-error-4xx https://www.rfc-editor.org/rfc/rfc9110.html#name-client-erro... [1] https://www.rfc-editor.org/rfc/rfc2119 https://www.rfc-editor.org/rfc/rfc2119 So yes, there is no requirement to return a 4XX message, but you should think carefully and understand the consequences of not doing so. One consequence is creating false positives in most vulnerability scanners. Another consequence is potentially messing up various crawlers, including those that help search engines index your content.
    - andrewaylett 3y ago
      In order to conform to the spec, yes. But almost by definition, people who don't care to conform to the spec aren't going to worry overmuch about whether what they're doing is conformant with the spec. As a client, you've got to be ready to deal with non-conformant implementations. As a service owner, there is no authority who will hunt you down for deploying a service that doesn't conform to the spec. Let me restate my claim slightly: acknowledging that impolite to claim to conform to a spec and then not do so, no-one has a right to claim that any arbitrary person must write code that conforms to any arbitrary RFC. > ["MUST",] "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification. (my emphasis) I may choose to return whatever response code I want in whatever circumstances I feel like, and there's absolutely nothing you can do to stop me. I probably won't, because that would be silly. But there's no must.
- pravus 3y ago
  How did the URL not exist? An empty resource is still a valid resource and 200 is an appropriate response. You can argue that Prometheus has a broken data model, but the request and response are actually valid in my book.
  - deleted 3y ago
    [deleted]
CipherThrowaway 3y ago
I would extend this to unused query parameters and trailing slashes or lack thereof.
- lifeonlars 3y ago
  This breaks the principle of being liberal in what you accept.
  - Omin 3y ago
    I think op goes too far by including trailing slashes, but being liberal in what you accept (Postel's law) is a bad idea as the last few decades of the web have shown. Once you accept things, you are locked into supporting them forever lest you break compatibility. When there are multiple different implementations of a standard, they grow incompatible over time.
    - cellularmitosis 3y ago
      Accepting and ignoring doesn't lock you into anything.
    - lifeonlars 3y ago
      Well, the web has been very successful in the last few decades. Arguably at least part of this success has been due to Postel's law since, for example, browsers interpreting HTML on a best effort basis has allowed for both diversity and innovation. Comparing the rapid adoption of the web to the failures of numerous closed and strongly specified protocols provides some empirical basis for this.
  - tzs 3y ago
    In retrospect I think we'd have been better off without that principle. Too often what happens is that the receiver is liberal, which results in senders not having to bother to fix violations of whatever spec the parties purport to be following. As the sending code continues to be developed occasionally new deviations from the actual spec and from whatever it is that current liberal receivers accept creep in, and the receivers get even more liberal to deal with them. A few years of this and writing a receiver for what should have been a fairly simple and easy to implement format requires huge parsers that handle a bazillion weird cases.
nubinetwork 3y ago
> What was happening instead is that the Prometheus host agent's HTTP server code will give you a HTTP 200 answer (with a generic front page) for any URL except the special URL for its metrics endpoint. > Neither party is exactly wrong here, but the result is not ideal. No, Prometheus has a bug if this is the case, and cks says it himself further on... > all it would need to do is only give a HTTP 200 response for '/' and then a 404 (with the same HTML) for everything else that it answers with the generic front page
- lifeonlars 3y ago
  ..or the vuln detection could be less stupid and try to decide whether the content it has retrieved is a bash history or not?
  - nubinetwork 3y ago
    I would agree in principle, but that would require an extensive parser for files it thinks it finds... it's easier to deal with this on the server side than the client side.
    - lifeonlars 3y ago
      No it's not. Dealing with it on the server side means that all tools and servers in the world have to go along with this decision. Posting on a blog saying that people 'SHOULD' do something and actually making it so that this is a reasonable expectation are two very different things. A vulnerability scanner having to implement hairy heuristics to decide what's a vuln and what's a common false positive is literally its whole job.
      - JoBrad 3y ago
        There is a standard for HTTP status codes that has dictated what web servers “should” do for quite a while, now. Most web servers respond properly with a 404 “out of the box”, when asked to serve up content that doesn’t exist.
colonelpopcorn 3y ago
We could extend this the other way to, security research tools shouldn't assert vulnerability based solely on HTTP status code. I think most SPAs require a setup like the one mentioned in the article.
- pixl97 3y ago
  "Security tools shouldn't" The bain of my existence here working with customers. See they like doing dumb things like having unified Nessus policies that alert if you have hyper threading on, so they disable HT on all their servers, including ones that don't run untrusted code. Then at the same time they complain that their expenses are nearly 50% higher than expected in execution costs on my highly multithreaded app. Reasonable policy and security don't really work well because there's not enough people trained in making this work properly across workloads in the enterprise .
28304283409234 3y ago
"an HTTP server", not "a HTTP server". On a university website, no less.
- stirfish 3y ago
  Both are correct
  - TacticalCoder 3y ago
    Oh!? Even though GP is being a bit pedantic I thought he was correct. Why is "a HTTP server" cromulent here? Do acronyms enjoy special rules? (FWIW english is not my native language)
    - nubinetwork 3y ago
      H is a consonant. "An" is used in this case if the next word starts with a vowel.
      - pm215 3y ago
        It's used if the next word starts with a vowel sound, so it depends on how you pronounce 'H' as a standalone letter. Personally I say it as "aitch", so for me "an" is correct here and what I would say.
        nubinetwork 3y ago
        What about L (ell)? You don't say "an LAMP server", it's "a LAMP server".
        stirfish 3y ago
        An ell ay em pee server A lamp server
        xnorswap 3y ago
ExoticPearTree 3y ago
Seems like a Prometheus “issue” on how I would guess node_exporter handles requests for unknown resources. The easy fix would be to answer with 404 for everytging except the configured /metrics endpoint. Any webserver, configured correctly, will return an appropriate code for non existing resources.
pacificpendant 3y ago
I had regular emails from a security testing tool telling me that internal IP addresses were being exposed on a webpage, in reality the page was a forum post where someone had pasted some console output including an IP address they were working with. In the end I blocked the emails from the tool because I wasn't allowed to mark things as false positives. If a tool wants to remain relevant it should try to minimise false positives, in some cases this might mean removing rules that are going to throw false positives significantly more often than true positives. Tools should also be run such that anyone that receive alerts should be able to flag false positives with minimal effort. The response to this false positive could be to fix Prometheus, but if you end up having to fix lots of things it's more of a sign of a bad rule that is making you concentrate on things with a low value to the goal of improving security.
- bombcar 3y ago
  192.168.0.1 10.10.10.10 172.16.31.5 I've exposed internal IPs!
  - Joker_vD 3y ago
    Oh no! You should contact your localhost's administrator ASAP and tell him to change those!
    - b112 3y ago
      You're joking, but jokes on him! I can ping some of those IPs, right now! A good example of how security through obscurity helps, they'd better get that fixed!
      - bombcar 3y ago
        Ah, fond memories of getting people to DOS themselves out of IRC by reporting that my IP was 127.63.78.41 or similar.
- tetha 3y ago
  Oh, you remind me of that day when our IDS went bonkers. Something was hammering us with SQL injections, it said. Like, 1-2 SQL injections per minute. And it gave successful HTTP responses, and actual JSON responses. The sky must be falling! We must be doomed! After a brief amount of panic, we figured out that we had a new customer for our knowledge base. This was an MSP and they were busy uploading their MSSQL and PostgreSQL runbooks into our knowledge base. Entirely beautiful documentation I have to say, clear steps, great instructions, smart queries to check, act and validate. We eventually had a good call about Postgres and such with those guys. But our IDS hated it.
gonzo41 3y ago
What's the ethics / law on handing back zip bombs as png's in gz files when you get scanned from rando's on the internet? Asking for a friend
- rwmj 3y ago
  The legal and ethical risk is approximately zero. I think the most likely problem is that by trying to be clever with your web server configuration you'll accidentally introduce an insecurity at your end. On my personal webserver, the vast majority of "rando" accesses are from search engine spiders that I've never heard of (as well as the ones you have heard of). These spiders really ought to be secure against anything a server can throw at them, and if they aren't, that's their problem.
- JoBrad 3y ago
  I don’t think there’s a rule that you have to respond with zip bombs, but it’s plainly impolite not to. This whole comment section has me wondering if an internet-connected tea pot should respond to all requests with a 418 status code.
- pravus 3y ago
  They asked for the resource. You are free to give it to them.
stevage 3y ago
I'm just surprised that anyone would implement the problematic behaviour of returning 200 by default, even if there's no such file to return. I mean, if you write a server in Express (NodeJS) or set up Nginx, it would be harder to get it wrong than to get it right.
- nailer 3y ago
  This is an odd article. I thought it must have been written in the nineties for the combination of obvious (see above) and bizarre (“web browsers and people mostly don't care about or notice the HTTP return code” advice). Your JS is absolutely likely to check response codes.
  - stevage 3y ago
    Yeah, I started doubting myself - I do often write code that checks response codes. I have to admit, I'm actually not sure what a browser does if a server is sending normal content but with 404 status codes. At the very least, I guess it messes up the caching?
    - nubinetwork 3y ago
      I know google treats it as a "soft 404"... There was a video on YouTube that talked about how browsers and bots handled different status codes, but I'm having a hard time finding it. Found it. https://youtu.be/4OztMJ4EL1s https://youtu.be/4OztMJ4EL1s
    - shkkmo 3y ago
      From the http spec about 4xx errors: > Except when responding to a HEAD request, the server SHOULD send a representation containing an explanation of the error situation, and whether it is a temporary or permanent condition. So sending content alongside a 4xx error should be the standard behavior and browsers should display that content. Not sending/displaying that content is the non-standard behavior.
paulnpace 3y ago
Depending on various factors, I tend to prefer any request that doesn't get a file or a redirect gets dropped (and without being logged). In reviewing my logs, more than 99.997% of 404 responses are garbage noise requests for /ty.php, or //xmlprc.php, or //wp-includes/wlwmanifest.php, and similar other bots scanning for vulnerabilities that can't exist on the server.
fastest963 3y ago
This is a consequence of the default ServeMux in Go. If you register "/" then effectively all URLs respond with that handler and in the case the handler is an index page.
ranting-moth 3y ago
If Prometheus says it's by design to accept everything incoming, I'd suggest they rename it from Prometheus to The Whore of Babylon.
gmuslera 3y ago
For Prometheus may be, or not. But taking that as as a generic rule for web servers and whatever they serve will break things too. Not everything is a file on disk or a full endpoint. That it may not the best for your particular use case doesn't mean that there are other use cases for which that policy would be wrong.
- 3np 3y ago
  Policies and generic rules can still have valid exceptions. Just keep the red tape under control.
danw1979 3y ago
> the Prometheus host agent's HTTP server code will give you a HTTP 200 answer (with a generic front page) for any URL except the special URL for its metrics endpoint. > Neither party is exactly wrong here Prometheus is in the wrong here, IMHO. I’m not sure why the article titled “Web servers should refuse requests for random, unnecessary URLs” would start out sitting on the fence.
vkaku 3y ago
What about /favicon.ico - or those pre-flight requests? The only real way to refuse requests is to just close the connection, no 404s or whatever should be sent. The deal with the way most people have designed the web from a client point of view, a lot of it works by convention - that's where this stuff starts to break. cgi-bin was a fairly well accepted convention a couple decades ago - and today those URIs are considered malicious. Unless people start depreciating conventions sooner, this problem will have false positives of its own.
- nubinetwork 3y ago
  In my tests, favicon gets loaded after the page, and 404ing it seems to go ignored by browsers.
tester756 3y ago
in this thread: way too many people being pissed off due to software's behaviour which messes with bots
- 3np 3y ago
  The web runs on bots. Your point being?
  - tester756 3y ago
    Not on those trying to access /admin.php and similar.
Sarkie 3y ago
Solving the wrong problem
bostik 3y ago
> Security scanners and other tools could adopt various heuristics to detect this sort of situation and reduce false positives [...] Wrong. Flat out wrong. Security scanners should adopt LESS heuristics and focus more on VALIDATING their perceived findings. Getting a 200 response for a missing file is a web server misconfiguration, but unless the returned data actually matches, it's a false positive. I may be repeating myself but any scanner (or researcher) who blindly raises a finding without validating it has as much to do with security as ripping wings off a fly has to do with bioengineering.
- dqv 3y ago
  That lack of validation is frustrating in the case of an automated scanner you can't control. "Your nginx version is exposed! Add `server_tokens off` to your config" Oh yeah, what version am I running then?
  - philsnow 3y ago
    This reminds me of the vigilante web security scanner sites who scan whatever infrastructure belongs to you, according to whatever sketchy mapping they use, and then publish a "report" with all their "findings". The findings are (nearly) all meritless, but it creates a lot of annoying work if a higher-up in the company (who doesn't know that it's a shakedown) says that the report needs to be addressed.
- tantalic 3y ago
  This is problem of misaligned incentives: if you are making a security scanner the last thing you want to do is miss a vulnerability. The result is many false positives.
  - fsociety 3y ago
    Alternatively, companies aren’t willing to pay for an automated security scanner that attempts to exploit potential vulnerabilities under fear of what it might do to their systems.
  - 3y ago
jonas-w 3y ago
Maybe the scanner should also look at the "resource-that-should-not-exist-whose-status-code-should-not-be-200 well-known URI" https://w3c.github.io/webappsec-change-password-url/response-code-reliability.html#the-resource-that-should-not-exist-whose-status-code-should-not-be-200-well-known-uri https://w3c.github.io/webappsec-change-password-url/response...
- tremon 3y ago
  Wow. If I were a webserver, I'd return 204 for that URL, just for shits and giggles.
  - wakamoleguy 3y ago
    Despite the URL using "200" and not "2xx", any status between 200-299 inclusive would flag the server as having unreliable status codes. https://w3c.github.io/webappsec-change-password-url/response-code-reliability.html#semantics https://w3c.github.io/webappsec-change-password-url/response...
  - deleted 3y ago
    [deleted]
  - superkuh 3y ago
    I have my webserver return the /index.html status 200 for any non-existent path in /. Since I didn't want bots to be able to detect I was doing this by this method I set nginx to return a 404 instead just for that path. location /.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200 { return 404; }
pravus 3y ago
I got tired of this sort of thing recently and now my web server returns 402 for everything that doesn't exist or has an invalid request. Most of what I do involves API and static data so I just serve everything from memory and avoid all disk access. It really bothered me that tons of stupid bots eat up resources asking the kernel for ENOENT.
gumby 3y ago
I feel like a 203 would work for this case.
jfoutz 3y ago
OK
jakub_g 3y ago
A followup blog also worth checking: https://utcc.utoronto.ca/~cks/space/blog/web/URLPresenceNotGoodSignal https://utcc.utoronto.ca/~cks/space/blog/web/URLPresenceNotG... > URLPresenceNotGoodSignal > There are a variety of situations where you (in the sense of programs and systems) want to know if a web server supports something or is under someone's control. One traditional way is to require the publication of specific URLs on the web server, often URLs with partially random names. The simplest way to implement this is to simply require the URL to exist and be accessible, which is to say that fetching it returns a HTTP 200 response. However, in light of web server implementations which will return HTTP 200 responses for any URL, or at least many of them, this simple check is clearly not sufficient in practice. The mere 'presence' of a URL on a web server proves very little. > If you need to implement this sort of protocol, you need to require the URL to contain some specific contents.
INTPenis 3y ago
Public prometheus endpoints without password protection? That's funny because our security scanners never ended up in this situation because we never allow public prometheus endpoints without password protection. Turns out we did a good job on that front at least. :)
paulddraper 3y ago
You've angered every modern web app. They're all configured to return 200.

86 points by ingve 3y ago

deleted 3y ago

[deleted]

The requested URL did not exist, so the server must return some response other than 200. It could be not-found, it could be forwarding, or it could be something else. However, 200 (success) is simply wrong.

wang_li 3y ago
>However, 200 (success) is simply wrong. What makes it wrong? HTTP/HTTPS URLs/URIs don't mean files on the disk. You can return 200 as a default if you like.
- shkkmo 3y ago
  Returning a 200 response for a resource that doesn't exist is wrong. The correct response code would be a 3XX or 4XX response code, depending on the client behavior that you want. In this specific case, a 303 response that redirects to the homepage would be most correct. Alternatively, you could return a 404 response alongside the default content.
  - phlakaton 3y ago
    And how do you know the resource doesn't exist in this case? The server says it does. The point is, "resource" is an abstraction that servers may implement as they wish. The only requirements, per HTTP, is that (1) it's something that can be identified with a URI (satisfied in this case), and (2) it has information associated with it that can be retrieved and/or managed via the HTTP protocol (satisfied in this case). We might imagine that the set of resources should be finite, or strictly mapped to extant data records in our system – but these are not requirements.
    - shkkmo 3y ago
      > We might imagine that the set of resources should be finite, or strictly mapped to extant data records in our system – but these are not requirements. Clearly they aren't requirements as I can think of many valid use cases that violate them. To me, creating an infinite number of identical aliases for a single resource is clearly wrong. The only reason to do it is either incompetence or laziness.
    - cellularmitosis 3y ago
      We can argue about the semantics of "wrong" all day, but the lesson here is that not following standards and conventions creates extra work for everyone downstream.
      - moring 3y ago
        They followed the standard though. They returned 200 for a resource that exists. Many responses here seem to be based on the misunderstanding that this is about "not following a standard is sometimes okay-ish", when it's actually about the fact that the standard does not rigorously define what a "resource" is and what it means for the resource to "exist", other than a circular definition based on status codes.
        cellularmitosis 3y ago
        Serving an identical resource at all possible paths is no one's idea of following established conventions. That's what redirects are for.
        phlakaton 3y ago
        Is it an identical resource? How do you know? But let's say they are identical. Where in the HTTP spec does it say identical resources may not be served at different endpoints? Nowhere, of course. You talk about violating conventions and standards. The conventions are defined by the HTTP spec. The standard is the HTTP spec. Neither are, so far as I can tell, being violated here.
  - pravus 3y ago
    URLs refer to resources, not files.
  - eddythompson80 3y ago
    > depending on the client behavior that you want. What if you want to confuse the client?
  - tremon 3y ago
    Returning a 200 response for a resource that doesn't exist is wrong Isn't that just begging the question? What defines whether a resource exists? Is it the client's knowledge about the webserver backend configuration, or the server's? If a server returns a 200 response, the resource exists, by definition. It may or may not have data, but again, that's not up to the client to decide. If the server says the resource exists and has no data, then that's the authoritative answer.
  - toast0 3y ago
    > The correct response code would be a 3XX or 4XX response code, depending on the client behavior that you want. If the client behavior you want is to display your styled error page, that has historically meant returning a 200 status code, because some user-agents prefer their own error displays.
- dekhn 3y ago
  I had to look this up recently, but Tim Berners Lee specifically said that URIs should be treated more or less identically to files on disk. That would mean that accessing a web server that is hosting a static directory should return not found for missing files. But this was just a guideline and it was never followed in practice.
  - lifeonlars 3y ago
    How the web works has changed quite a lot since the early 1990s though.
- mock-possum 3y ago
  RFC 9110 - https://httpwg.org/specs/rfc9110.html#overview.of.status.codes https://httpwg.org/specs/rfc9110.html#overview.of.status.cod... It’s like saying “what makes it wrong to signal left but turn right?” It makes the component you’re operating behave less predictably, resulting in a less stable system overall. Follow the spec.
  - moring 3y ago
    They followed the spec though. They returned 200 for a resource that exists. > It’s like saying “what makes it wrong to signal left but turn right?” No, it's not like that. Signalling left but turning right would be like returning a 200 for a resource that does not exist.
andrewaylett 3y ago
And the penalty for being "wrong" is..? You can't write code relying on random third-party code doing what it's "mandated" to do by the standard, because malicious code exists and so do malicious services. So as a client, if you're connecting to services you don't control, you need to be able to handle misbehaving third-party code. And what it means for a request to be "successful" is entirely at the discretion of the author of the software. I think it's fairly clear that the software looked for a response, found one, and successfully returned it to the client. Which isn't to say that the code is optimal, necessarily, but unless you have a contract to say otherwise then I don't think you've any basis to claim that it must do anything in particular.
- shkkmo 3y ago
  > I don't think you've any basis to claim that it must do anything in particular. The relevant part of the spec says that returning a 4XX code is a SHOULD [0] not a MUST. That means: > This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course [1] [0] https://www.rfc-editor.org/rfc/rfc9110.html#name-client-error-4xx https://www.rfc-editor.org/rfc/rfc9110.html#name-client-erro... [1] https://www.rfc-editor.org/rfc/rfc2119 https://www.rfc-editor.org/rfc/rfc2119 So yes, there is no requirement to return a 4XX message, but you should think carefully and understand the consequences of not doing so. One consequence is creating false positives in most vulnerability scanners. Another consequence is potentially messing up various crawlers, including those that help search engines index your content.
  - andrewaylett 3y ago
    In order to conform to the spec, yes. But almost by definition, people who don't care to conform to the spec aren't going to worry overmuch about whether what they're doing is conformant with the spec. As a client, you've got to be ready to deal with non-conformant implementations. As a service owner, there is no authority who will hunt you down for deploying a service that doesn't conform to the spec. Let me restate my claim slightly: acknowledging that impolite to claim to conform to a spec and then not do so, no-one has a right to claim that any arbitrary person must write code that conforms to any arbitrary RFC. > ["MUST",] "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification. (my emphasis) I may choose to return whatever response code I want in whatever circumstances I feel like, and there's absolutely nothing you can do to stop me. I probably won't, because that would be silly. But there's no must.
pravus 3y ago
How did the URL not exist? An empty resource is still a valid resource and 200 is an appropriate response. You can argue that Prometheus has a broken data model, but the request and response are actually valid in my book.
- deleted 3y ago
  [deleted]

CipherThrowaway 3y ago

I would extend this to unused query parameters and trailing slashes or lack thereof.

lifeonlars 3y ago
This breaks the principle of being liberal in what you accept.
- Omin 3y ago
  I think op goes too far by including trailing slashes, but being liberal in what you accept (Postel's law) is a bad idea as the last few decades of the web have shown. Once you accept things, you are locked into supporting them forever lest you break compatibility. When there are multiple different implementations of a standard, they grow incompatible over time.
  - cellularmitosis 3y ago
    Accepting and ignoring doesn't lock you into anything.
  - lifeonlars 3y ago
    Well, the web has been very successful in the last few decades. Arguably at least part of this success has been due to Postel's law since, for example, browsers interpreting HTML on a best effort basis has allowed for both diversity and innovation. Comparing the rapid adoption of the web to the failures of numerous closed and strongly specified protocols provides some empirical basis for this.
- tzs 3y ago
  In retrospect I think we'd have been better off without that principle. Too often what happens is that the receiver is liberal, which results in senders not having to bother to fix violations of whatever spec the parties purport to be following. As the sending code continues to be developed occasionally new deviations from the actual spec and from whatever it is that current liberal receivers accept creep in, and the receivers get even more liberal to deal with them. A few years of this and writing a receiver for what should have been a fairly simple and easy to implement format requires huge parsers that handle a bazillion weird cases.

nubinetwork 3y ago

> What was happening instead is that the Prometheus host agent's HTTP server code will give you a HTTP 200 answer (with a generic front page) for any URL except the special URL for its metrics endpoint. > Neither party is exactly wrong here, but the result is not ideal. No, Prometheus has a bug if this is the case, and cks says it himself further on... > all it would need to do is only give a HTTP 200 response for '/' and then a 404 (with the same HTML) for everything else that it answers with the generic front page

lifeonlars 3y ago
..or the vuln detection could be less stupid and try to decide whether the content it has retrieved is a bash history or not?
- nubinetwork 3y ago
  I would agree in principle, but that would require an extensive parser for files it thinks it finds... it's easier to deal with this on the server side than the client side.
  - lifeonlars 3y ago
    No it's not. Dealing with it on the server side means that all tools and servers in the world have to go along with this decision. Posting on a blog saying that people 'SHOULD' do something and actually making it so that this is a reasonable expectation are two very different things. A vulnerability scanner having to implement hairy heuristics to decide what's a vuln and what's a common false positive is literally its whole job.
    - JoBrad 3y ago
      There is a standard for HTTP status codes that has dictated what web servers “should” do for quite a while, now. Most web servers respond properly with a 404 “out of the box”, when asked to serve up content that doesn’t exist.

colonelpopcorn 3y ago

We could extend this the other way to, security research tools shouldn't assert vulnerability based solely on HTTP status code. I think most SPAs require a setup like the one mentioned in the article.

pixl97 3y ago
"Security tools shouldn't" The bain of my existence here working with customers. See they like doing dumb things like having unified Nessus policies that alert if you have hyper threading on, so they disable HT on all their servers, including ones that don't run untrusted code. Then at the same time they complain that their expenses are nearly 50% higher than expected in execution costs on my highly multithreaded app. Reasonable policy and security don't really work well because there's not enough people trained in making this work properly across workloads in the enterprise .

28304283409234 3y ago

"an HTTP server", not "a HTTP server". On a university website, no less.

stirfish 3y ago
Both are correct
- TacticalCoder 3y ago
  Oh!? Even though GP is being a bit pedantic I thought he was correct. Why is "a HTTP server" cromulent here? Do acronyms enjoy special rules? (FWIW english is not my native language)
  - nubinetwork 3y ago
    H is a consonant. "An" is used in this case if the next word starts with a vowel.
    - pm215 3y ago
      It's used if the next word starts with a vowel sound, so it depends on how you pronounce 'H' as a standalone letter. Personally I say it as "aitch", so for me "an" is correct here and what I would say.
      - nubinetwork 3y ago
        What about L (ell)? You don't say "an LAMP server", it's "a LAMP server".
        stirfish 3y ago
        An ell ay em pee server A lamp server
        xnorswap 3y ago

ExoticPearTree 3y ago

Seems like a Prometheus “issue” on how I would guess node_exporter handles requests for unknown resources. The easy fix would be to answer with 404 for everytging except the configured /metrics endpoint. Any webserver, configured correctly, will return an appropriate code for non existing resources.

pacificpendant 3y ago

I had regular emails from a security testing tool telling me that internal IP addresses were being exposed on a webpage, in reality the page was a forum post where someone had pasted some console output including an IP address they were working with. In the end I blocked the emails from the tool because I wasn't allowed to mark things as false positives. If a tool wants to remain relevant it should try to minimise false positives, in some cases this might mean removing rules that are going to throw false positives significantly more often than true positives. Tools should also be run such that anyone that receive alerts should be able to flag false positives with minimal effort. The response to this false positive could be to fix Prometheus, but if you end up having to fix lots of things it's more of a sign of a bad rule that is making you concentrate on things with a low value to the goal of improving security.

bombcar 3y ago
192.168.0.1 10.10.10.10 172.16.31.5 I've exposed internal IPs!
- Joker_vD 3y ago
  Oh no! You should contact your localhost's administrator ASAP and tell him to change those!
  - b112 3y ago
    You're joking, but jokes on him! I can ping some of those IPs, right now! A good example of how security through obscurity helps, they'd better get that fixed!
    - bombcar 3y ago
      Ah, fond memories of getting people to DOS themselves out of IRC by reporting that my IP was 127.63.78.41 or similar.
tetha 3y ago
Oh, you remind me of that day when our IDS went bonkers. Something was hammering us with SQL injections, it said. Like, 1-2 SQL injections per minute. And it gave successful HTTP responses, and actual JSON responses. The sky must be falling! We must be doomed! After a brief amount of panic, we figured out that we had a new customer for our knowledge base. This was an MSP and they were busy uploading their MSSQL and PostgreSQL runbooks into our knowledge base. Entirely beautiful documentation I have to say, clear steps, great instructions, smart queries to check, act and validate. We eventually had a good call about Postgres and such with those guys. But our IDS hated it.

gonzo41 3y ago

What's the ethics / law on handing back zip bombs as png's in gz files when you get scanned from rando's on the internet? Asking for a friend

rwmj 3y ago
The legal and ethical risk is approximately zero. I think the most likely problem is that by trying to be clever with your web server configuration you'll accidentally introduce an insecurity at your end. On my personal webserver, the vast majority of "rando" accesses are from search engine spiders that I've never heard of (as well as the ones you have heard of). These spiders really ought to be secure against anything a server can throw at them, and if they aren't, that's their problem.
JoBrad 3y ago
I don’t think there’s a rule that you have to respond with zip bombs, but it’s plainly impolite not to. This whole comment section has me wondering if an internet-connected tea pot should respond to all requests with a 418 status code.
pravus 3y ago
They asked for the resource. You are free to give it to them.

stevage 3y ago

I'm just surprised that anyone would implement the problematic behaviour of returning 200 by default, even if there's no such file to return. I mean, if you write a server in Express (NodeJS) or set up Nginx, it would be harder to get it wrong than to get it right.

nailer 3y ago
This is an odd article. I thought it must have been written in the nineties for the combination of obvious (see above) and bizarre (“web browsers and people mostly don't care about or notice the HTTP return code” advice). Your JS is absolutely likely to check response codes.
- stevage 3y ago
  Yeah, I started doubting myself - I do often write code that checks response codes. I have to admit, I'm actually not sure what a browser does if a server is sending normal content but with 404 status codes. At the very least, I guess it messes up the caching?
  - nubinetwork 3y ago
    I know google treats it as a "soft 404"... There was a video on YouTube that talked about how browsers and bots handled different status codes, but I'm having a hard time finding it. Found it. https://youtu.be/4OztMJ4EL1s https://youtu.be/4OztMJ4EL1s
  - shkkmo 3y ago
    From the http spec about 4xx errors: > Except when responding to a HEAD request, the server SHOULD send a representation containing an explanation of the error situation, and whether it is a temporary or permanent condition. So sending content alongside a 4xx error should be the standard behavior and browsers should display that content. Not sending/displaying that content is the non-standard behavior.

paulnpace 3y ago

Depending on various factors, I tend to prefer any request that doesn't get a file or a redirect gets dropped (and without being logged). In reviewing my logs, more than 99.997% of 404 responses are garbage noise requests for /ty.php, or //xmlprc.php, or //wp-includes/wlwmanifest.php, and similar other bots scanning for vulnerabilities that can't exist on the server.

fastest963 3y ago

This is a consequence of the default ServeMux in Go. If you register "/" then effectively all URLs respond with that handler and in the case the handler is an index page.

ranting-moth 3y ago

If Prometheus says it's by design to accept everything incoming, I'd suggest they rename it from Prometheus to The Whore of Babylon.

gmuslera 3y ago

For Prometheus may be, or not. But taking that as as a generic rule for web servers and whatever they serve will break things too. Not everything is a file on disk or a full endpoint. That it may not the best for your particular use case doesn't mean that there are other use cases for which that policy would be wrong.

3np 3y ago
Policies and generic rules can still have valid exceptions. Just keep the red tape under control.

danw1979 3y ago

> the Prometheus host agent's HTTP server code will give you a HTTP 200 answer (with a generic front page) for any URL except the special URL for its metrics endpoint. > Neither party is exactly wrong here Prometheus is in the wrong here, IMHO. I’m not sure why the article titled “Web servers should refuse requests for random, unnecessary URLs” would start out sitting on the fence.

vkaku 3y ago

What about /favicon.ico - or those pre-flight requests? The only real way to refuse requests is to just close the connection, no 404s or whatever should be sent. The deal with the way most people have designed the web from a client point of view, a lot of it works by convention - that's where this stuff starts to break. cgi-bin was a fairly well accepted convention a couple decades ago - and today those URIs are considered malicious. Unless people start depreciating conventions sooner, this problem will have false positives of its own.

nubinetwork 3y ago
In my tests, favicon gets loaded after the page, and 404ing it seems to go ignored by browsers.

tester756 3y ago

in this thread: way too many people being pissed off due to software's behaviour which messes with bots

3np 3y ago
The web runs on bots. Your point being?
- tester756 3y ago
  Not on those trying to access /admin.php and similar.

Sarkie 3y ago

Solving the wrong problem

bostik 3y ago

> Security scanners and other tools could adopt various heuristics to detect this sort of situation and reduce false positives [...] Wrong. Flat out wrong. Security scanners should adopt LESS heuristics and focus more on VALIDATING their perceived findings. Getting a 200 response for a missing file is a web server misconfiguration, but unless the returned data actually matches, it's a false positive. I may be repeating myself but any scanner (or researcher) who blindly raises a finding without validating it has as much to do with security as ripping wings off a fly has to do with bioengineering.

dqv 3y ago
That lack of validation is frustrating in the case of an automated scanner you can't control. "Your nginx version is exposed! Add `server_tokens off` to your config" Oh yeah, what version am I running then?
- philsnow 3y ago
  This reminds me of the vigilante web security scanner sites who scan whatever infrastructure belongs to you, according to whatever sketchy mapping they use, and then publish a "report" with all their "findings". The findings are (nearly) all meritless, but it creates a lot of annoying work if a higher-up in the company (who doesn't know that it's a shakedown) says that the report needs to be addressed.
tantalic 3y ago
This is problem of misaligned incentives: if you are making a security scanner the last thing you want to do is miss a vulnerability. The result is many false positives.
- fsociety 3y ago
  Alternatively, companies aren’t willing to pay for an automated security scanner that attempts to exploit potential vulnerabilities under fear of what it might do to their systems.
- 3y ago

jonas-w 3y ago

Maybe the scanner should also look at the "resource-that-should-not-exist-whose-status-code-should-not-be-200 well-known URI" https://w3c.github.io/webappsec-change-password-url/response-code-reliability.html#the-resource-that-should-not-exist-whose-status-code-should-not-be-200-well-known-uri https://w3c.github.io/webappsec-change-password-url/response...

tremon 3y ago
Wow. If I were a webserver, I'd return 204 for that URL, just for shits and giggles.
- wakamoleguy 3y ago
  Despite the URL using "200" and not "2xx", any status between 200-299 inclusive would flag the server as having unreliable status codes. https://w3c.github.io/webappsec-change-password-url/response-code-reliability.html#semantics https://w3c.github.io/webappsec-change-password-url/response...
- deleted 3y ago
  [deleted]
- superkuh 3y ago
  I have my webserver return the /index.html status 200 for any non-existent path in /. Since I didn't want bots to be able to detect I was doing this by this method I set nginx to return a 404 instead just for that path. location /.well-known/resource-that-should-not-exist-whose-status-code-should-not-be-200 { return 404; }

pravus 3y ago

I got tired of this sort of thing recently and now my web server returns 402 for everything that doesn't exist or has an invalid request. Most of what I do involves API and static data so I just serve everything from memory and avoid all disk access. It really bothered me that tons of stupid bots eat up resources asking the kernel for ENOENT.

gumby 3y ago

I feel like a 203 would work for this case.

jfoutz 3y ago

jakub_g 3y ago

A followup blog also worth checking: https://utcc.utoronto.ca/~cks/space/blog/web/URLPresenceNotGoodSignal https://utcc.utoronto.ca/~cks/space/blog/web/URLPresenceNotG... > URLPresenceNotGoodSignal > There are a variety of situations where you (in the sense of programs and systems) want to know if a web server supports something or is under someone's control. One traditional way is to require the publication of specific URLs on the web server, often URLs with partially random names. The simplest way to implement this is to simply require the URL to exist and be accessible, which is to say that fetching it returns a HTTP 200 response. However, in light of web server implementations which will return HTTP 200 responses for any URL, or at least many of them, this simple check is clearly not sufficient in practice. The mere 'presence' of a URL on a web server proves very little. > If you need to implement this sort of protocol, you need to require the URL to contain some specific contents.

INTPenis 3y ago

Public prometheus endpoints without password protection? That's funny because our security scanners never ended up in this situation because we never allow public prometheus endpoints without password protection. Turns out we did a good job on that front at least. :)

paulddraper 3y ago

You've angered every modern web app. They're all configured to return 200.