20 ms·

Any decent error message is a kind of oracle

29 points by bobbiechen 1y ago

Groxx 1y ago
For debugging purposes, because having users tell you what error they got is sometimes very useful: generate a random number (e.g. a uuid), log it with the error, and display that number. doesn't leak data because it's different every time, but you can uniquely pair it up with what they are seeing.
- ChrisMarshallNY 1y ago
  That's a good idea!
- spockz 1y ago
  Better yet. Just use the trace id for the message. That id is correlated with all log statements already and it will allow you to in one go follow the whole flow of the user without setting up another per-request flow lookup table.
  - RadiozRadioz 1y ago
    The prerequisite to this is setting up OTEL tracing and having already instrumented everything. Which is not trivial. You can't "just use" it. It's like you get to the middle of a recipe and they say to "just" add in the meat you've already been marinating for 48 hours.
    - Groxx 1y ago
      you really only need it at the entry point to have a trace ID... but yeah. tracing is hard to add on later, though I'm glad it's becoming more common for newer code.
    - spockz 1y ago
      Telemetry for me is table stakes. We have been doing forms of tracing (first home grown) since 2012. Strictly speaking only the edge gateway needs to know about it. Tracing is trivial to add with OTel. It is trivial with Spring Boot (since v3). At least on technical endpoint level. Adding tracing to business logic obviously requires work, but simple propagation is table stakes.
- kmoser 1y ago
  I like to have the error message include the line number that emitted it, e.g. "Unable to log in to your account (error code 123)". This comes in handy especially when you don't want (or are unable) to log messages, and/or several lines emit the same error message. So line 123 might be because no DB connection could be established, while line 456 might be because the connection timed out.
  - ChrisMarshallNY 1y ago
    I have found that having this in the user-facing message is not helpful. It seems that the parent suggestion is unpopular, but having a technically-available log, with a reference from an entry, seems to be a good compromise. Bit more work for the developer, but less for the user, which is really the goal.
  - deleted 1y ago
    [deleted]
deleted 1y ago
[deleted]
ChrisMarshallNY 1y ago
> So why aren’t these errors better? “Password is incorrect, try again.” or, “No account exists for this email.” Is that so hard? I can tell you exactly why I don't do this, for my app. I don't want to indicate which of the fields is an issue. Most folks use Sign up with Apple, though, which obviates this. The best error message is to avoid the error; either by effective design, or by good affordances. But this is what WFM. YMMV.
- deleted 1y ago
  [deleted]
- saagarjha 1y ago
  > I don't want to indicate which of the fields is an issue. Why not?
  - ChrisMarshallNY 1y ago
    Narrows down the possibilities, for a hacker by 50%.
    - saagarjha 1y ago
      Ideally you have enough entropy that you can spend a factor of two on making your software more pleasant to use
      - ChrisMarshallNY 1y ago
        That’s why I use Sign up with Apple. Fair bit of work. The nature of the target demographic demands that I don’t cut any corners, with security. But I’m also a big proponent of usability, so I would agree, for some applications.
    - sippeangelo 1y ago
      The answer should be that it's a privacy leak! Do you allow random actors to brute force your login?
louthy 1y ago
[EDIT: I’d only read up to: “So why aren’t these errors better? “Password is incorrect, try again.” or, “No account exists for this email.” Is that so hard?” — then I bailed, my bad, I should have kept reading. So my response below isn’t fair] This ignores the security risks from being too verbose and/or specific with error messages, especially if they’re coming from a server. You’ll usually fail security/pen-test audit. I agree that doing a better job of helping the user is laudable, but you need to know which battles to fight. Giving a unique error number that can be referenced by a support team (who could look up the event, look at stack traces, etc.) is the best way to deal with truly exceptional events. Otherwise, if it comes to authentication or authorisation, you have to extremely careful what information you share.
- username923409 1y ago
  You should try reading past the first page of the article before making a judgment like that.
  - louthy 1y ago
    Who has time? ;) But fair enough, I had stopped at the point where the advice was bad. My bad. I’ve clarified in my original comment.
upofadown 1y ago
When things go wrong there are three main entities to consider: The User They need a coherent explanation of what happened, based on the concepts they can be expected to understand. Ideally enough information/explanation should be provided to allow them to overcome the condition. The Maintenance This should be a super detailed description of the error and include any possible context. The user will not reliably preserve this sort of information. The Attracker Could they conceivably get access to the data? Could they get any use out of the data? Could they cause the error condition that generated the error.
kmoser 1y ago
> Many “Oopsie woopsie”-style errors are a fallback message that appears in unexpected errors - since the developer doesn’t expect it to happen, it could be dangerous to reveal application context as part of the error. In my experience most of those "oopsie woopsie" error messages are an attempt to seem less techie and/or lack of desire to provide meaningful information to the user. The main thing users really want to know: did they mess up, or did the system mess up? And what should they do next: try again? Contact support? That basic information is not only usually lacking, but providing it to the user would not threaten system security.
- recursivecaveat 1y ago
  It's kind of sad that every computer user has to learn that when they click a link to "example.com/news/2003-07-14/update/very-long-base64-encoded-string", the very patronizing error message accusing them of making a typo just means that the site deleted the content, but is too embarrassed to admit they do that kind of thing.
xg15 1y ago
I think the article conflates different scenarios: I don't think many people expect that a web service gives detailed technical error messages to external users who might be attackers. But the really annoying appearances of the "oopsie woopsie" messages are if they happen on a local system, where there is no other party - neither a potential attacker, nor any "administrator" who should be trusted instead of the user. So if there is no one else except the user, who should fix the error?

29 points by bobbiechen 1y ago

For debugging purposes, because having users tell you what error they got is sometimes very useful: generate a random number (e.g. a uuid), log it with the error, and display that number. doesn't leak data because it's different every time, but you can uniquely pair it up with what they are seeing.

ChrisMarshallNY 1y ago
That's a good idea!
spockz 1y ago
Better yet. Just use the trace id for the message. That id is correlated with all log statements already and it will allow you to in one go follow the whole flow of the user without setting up another per-request flow lookup table.
- RadiozRadioz 1y ago
  The prerequisite to this is setting up OTEL tracing and having already instrumented everything. Which is not trivial. You can't "just use" it. It's like you get to the middle of a recipe and they say to "just" add in the meat you've already been marinating for 48 hours.
  - Groxx 1y ago
    you really only need it at the entry point to have a trace ID... but yeah. tracing is hard to add on later, though I'm glad it's becoming more common for newer code.
  - spockz 1y ago
    Telemetry for me is table stakes. We have been doing forms of tracing (first home grown) since 2012. Strictly speaking only the edge gateway needs to know about it. Tracing is trivial to add with OTel. It is trivial with Spring Boot (since v3). At least on technical endpoint level. Adding tracing to business logic obviously requires work, but simple propagation is table stakes.
kmoser 1y ago
I like to have the error message include the line number that emitted it, e.g. "Unable to log in to your account (error code 123)". This comes in handy especially when you don't want (or are unable) to log messages, and/or several lines emit the same error message. So line 123 might be because no DB connection could be established, while line 456 might be because the connection timed out.
- ChrisMarshallNY 1y ago
  I have found that having this in the user-facing message is not helpful. It seems that the parent suggestion is unpopular, but having a technically-available log, with a reference from an entry, seems to be a good compromise. Bit more work for the developer, but less for the user, which is really the goal.
- deleted 1y ago
  [deleted]

deleted 1y ago

[deleted]

ChrisMarshallNY 1y ago

> So why aren’t these errors better? “Password is incorrect, try again.” or, “No account exists for this email.” Is that so hard? I can tell you exactly why I don't do this, for my app. I don't want to indicate which of the fields is an issue. Most folks use Sign up with Apple, though, which obviates this. The best error message is to avoid the error; either by effective design, or by good affordances. But this is what WFM. YMMV.

deleted 1y ago
[deleted]
saagarjha 1y ago
> I don't want to indicate which of the fields is an issue. Why not?
- ChrisMarshallNY 1y ago
  Narrows down the possibilities, for a hacker by 50%.
  - saagarjha 1y ago
    Ideally you have enough entropy that you can spend a factor of two on making your software more pleasant to use
    - ChrisMarshallNY 1y ago
      That’s why I use Sign up with Apple. Fair bit of work. The nature of the target demographic demands that I don’t cut any corners, with security. But I’m also a big proponent of usability, so I would agree, for some applications.
  - sippeangelo 1y ago
    The answer should be that it's a privacy leak! Do you allow random actors to brute force your login?

louthy 1y ago

[EDIT: I’d only read up to: “So why aren’t these errors better? “Password is incorrect, try again.” or, “No account exists for this email.” Is that so hard?” — then I bailed, my bad, I should have kept reading. So my response below isn’t fair] This ignores the security risks from being too verbose and/or specific with error messages, especially if they’re coming from a server. You’ll usually fail security/pen-test audit. I agree that doing a better job of helping the user is laudable, but you need to know which battles to fight. Giving a unique error number that can be referenced by a support team (who could look up the event, look at stack traces, etc.) is the best way to deal with truly exceptional events. Otherwise, if it comes to authentication or authorisation, you have to extremely careful what information you share.

username923409 1y ago
You should try reading past the first page of the article before making a judgment like that.
- louthy 1y ago
  Who has time? ;) But fair enough, I had stopped at the point where the advice was bad. My bad. I’ve clarified in my original comment.

upofadown 1y ago

When things go wrong there are three main entities to consider: The User They need a coherent explanation of what happened, based on the concepts they can be expected to understand. Ideally enough information/explanation should be provided to allow them to overcome the condition. The Maintenance This should be a super detailed description of the error and include any possible context. The user will not reliably preserve this sort of information. The Attracker Could they conceivably get access to the data? Could they get any use out of the data? Could they cause the error condition that generated the error.

kmoser 1y ago

> Many “Oopsie woopsie”-style errors are a fallback message that appears in unexpected errors - since the developer doesn’t expect it to happen, it could be dangerous to reveal application context as part of the error. In my experience most of those "oopsie woopsie" error messages are an attempt to seem less techie and/or lack of desire to provide meaningful information to the user. The main thing users really want to know: did they mess up, or did the system mess up? And what should they do next: try again? Contact support? That basic information is not only usually lacking, but providing it to the user would not threaten system security.

recursivecaveat 1y ago
It's kind of sad that every computer user has to learn that when they click a link to "example.com/news/2003-07-14/update/very-long-base64-encoded-string", the very patronizing error message accusing them of making a typo just means that the site deleted the content, but is too embarrassed to admit they do that kind of thing.

xg15 1y ago

I think the article conflates different scenarios: I don't think many people expect that a web service gives detailed technical error messages to external users who might be attackers. But the really annoying appearances of the "oopsie woopsie" messages are if they happen on a local system, where there is no other party - neither a potential attacker, nor any "administrator" who should be trusted instead of the user. So if there is no one else except the user, who should fix the error?