5 ms·

A bizarre “403 Forbidden” bug-hunting adventure

137 points by joshsharp 9y ago

exikyut 9y ago
This is both awesome and depressing. But it gave me a really interesting idea: a service/network/community/forum/etc for people to gather and discuss Really Confusing Bugs™ that they're trying to figure out. Not necessarily (?) for contemporary end users (maybe highly technical end users). This could actually be a really cool concept. Somewhere squarely between HackerOne and StackOverflow - not for exploits, and not for simple(r) stuff, but specifically for complicated and confusing bugs you've been staring at for days/weeks and nothing's making any sense. I can see a subscription model working for this, even - subscriptions would work both to allow people to provide extended assistance, and also because a contract makes NDAs easier. Hmm. Thinking about how the subscription model would work... you sign up, configure billing, that then allows you to request extended assistance. - One way that could work is that people offer you help in return for thanks, which would work like a configurable upvote; higher quality answers attract more rewards. Maybe anyone can reward answers (via the credit in their account) after the fact? - Another way would be setting a minimum or exact reward amount up front to attract more help. Regardless of how it worked, the site would have all discussion be public and open by default; you'd have to check a box to make the discussion private, and even after that you'd have the ability to go through and selectively un-redact parts of the conversations so everyone could be helped. And anyone could sign up and offer answers instantly, and the rewards credited to their account could be cashed out at any time. That would attract new users. I realize I've just described a weird kind of paid StackOverflow. I am very curious why SE hasn't pursued such an idea. As in, I am 1000% confident they've had this conversation at least once, and I'd really love to hear what the opinions were.
- hathym 9y ago
  Stackoverfliw ?
  - exikyut 9y ago
    ? What do you mean?
  - fapjacks 9y ago
    It would be great if this were possible, but unfortunately, even relatively novel and interesting bug posts will be indiscriminately closed on Stack Overflow. You can bet that it will be aggressively closed as a duplicate or as "unclear" or "too general". That is the fate of the vast majority of new posts there.
    - wolco 9y ago
      I never understood why they didn't allow duplicate questions. It provides a fresh prespective and a new updated way to solve a question.
      - mort96 9y ago
        Things I have experienced when finding an SO question from Google: * The question is closed as a duplicate, but the question it's a duplicate of is subtly different enough to be completely irrelevant to me. * The question is closed as a dupilcate, but there's no link to what it's a duplicate of. * The question is closed as a duplicate, but there are no good answers to the question it's a duplicate of.
        deleted 9y ago
        [deleted]
        fapjacks 9y ago
        At some point, there were an equal number of average and bad programmers on Stack Overflow as there were average and good programmers on Stack Overflow. But Stack Overflow does not reward people for being good programmers. SO rewards actions that game the system. And when SO points became a kind of currency for finding jobs ("I have five thousand points on SO!" on a resume), it skewed the userbase heavily in the direction of "game the system for more points". So you had average, bad, and awful programmers gaming the system and earning points, but because they are just terrible at what they do and the quality of all of their output is awful, this had the effect of driving away the average, good, and excellent programmers away from Stack Overflow. At some point, the whole of SO shifted, and what you had was a bunch of point-chasing, terrible "programmers" gaming the system for points. There are actually a substantial number of reasonable questions which are not even slightly related to the questions they're closed as duplicates of. But in the eyes of these shit-awful "programmers" they're similar enough to close. Stack Overflow is on its way out. It has served its purpose in life, and perhaps will remain somewhat relevant to the kinds of enormous enterprises that hire shit-awful programmers with their Stack Overflow points on their resumes. But all the reasons it was originally attractive are now gone, as are the majority of the excellent programmers that made it interesting. I'm not sure if Quora is its replacement, and I'm not sure what the next big place where excellent programmers go to share insights looks like. But it's not Stack Overflow.
- bigiain 9y ago
  " a service/network/community/forum/etc for people to gather and discuss Really Confusing Bugs" Should be universally open to all - at least until someone screws up badly enough to get temporarily or permanently barred, needs to be open outside regular work hours, should be free (or super a inexpensive nominal charge) to enter, would need to monetise itself via other means, perhaps selling beverages. We've just re-invented "the bar"... ;-)
- andreareina 9y ago
  ExpertsExchange was[1] a paid StackOverflow before the latter existed, SO came along and ate EE's lunch. Some of that was undoubtedly due to EE's (anecdotally, I never used it due to the money issue) bad UX, but free makes a big difference. [1] They still exist, and it apparently gets traffic/use, but never hear anyone talking about them, and it never comes up when searching to resolve or diagnose a problem.
  - pbhjpbhj 9y ago
    EE had a scheme where you could answer questions to get free membership; I used that for a while. I remember getting screwed over but don't remember the details: they locked up the site and I couldn't even access the solutions I'd written anymore. A bit later still they made it appear locked but you could still get to the answers if you knew how (I think that was to get their solutions on Google. I'd guess it was taken over and someone tried to squeeze the money out, hasn't happened to SO yet.
    - wolco 9y ago
      if I remember you had to scroll down past the ad content.
    - deleted 9y ago
      [deleted]
  - mort96 9y ago
    I still hear people talking about ExpertsExchange. Usually accompanied with a joke about their name; ExpertSexChange.
foota 9y ago
Are they using a header or s.t. for their authorization in app, then?
- joshsharp 9y ago
  Using the Authorization header.
  - foota 9y ago
    Thanks for following up! Glad y'all found the bug :)
luketheobscure 9y ago
"NSURLSession oh so helpfully auto-fucking-matically decided I would probably—no, definitely—want to send those cookies in every single request my app did after that. Forever and always." I imagine it honors the "expires" property of cookies, it just probably wasn't set. Also there's an "ephemeralSessionConfiguration" you can use if you don't want to store the cookies. I enjoyed the article, up until the author blamed the framework for their own ignorance.
- joshsharp 9y ago
  Sure, the cookies expire (in two weeks I think, in this case) but that doesn't invalidate the issue that the wkwebview silently shares cookies with nsurlsession, nor does it negate the confusing journey of tracking down the bug. Sheesh.
  - spullara 9y ago
    This is exactly what users want and you are basically breaking it by not honoring the cookies.
    - joshsharp 9y ago
      We do not need to honour cookies when making requests to the API as we use token authentication. Even if we wanted to keep users logged in within the web view, which we don't because the web view is just for static help pages, we wouldn't want to send those to the API.
      - eridius 9y ago
        Then you should disable cookies on your API calls by setting the `httpCookieStorage` property of your URLSessionConfiguration to nil. You could also use an ephemeral configuration (which will preserve cookies set by the API in memory but won't share them with webviews), which may or may not be a good idea depending on if you're using this session for anything that should be cached to disk (e.g. downloading images).
stanleydrew 9y ago
Not sure about the "fix", since now users can't persist a logged in browser session? If you aren't using cookies for API auth, is there not some way to configure your server framework to just ignore them if received?
- joshsharp 9y ago
  They shouldn't be logging in via the webview within the app anyway, so that's moot. We are using cookies, as the post says, as a fallback authentication so devs can browse the API from a browser if they're logged in to the site.
  - eridius 9y ago
    Why are devs using a browser to hit API endpoints? There are much better tools. My preferred one is called Paw (https://paw.cloud https://paw.cloud) but there are others.
    - grkvlt 9y ago
      Because Paw is GBP 39.99 and a browser is free?
      - eridius 9y ago
        There are other free tools too, like Postman (https://www.getpostman.com https://www.getpostman.com).
        grkvlt 9y ago
        True, I actually use Postman in Chrome, forgot to mention that.
udioron 9y ago
Another fix is to disable CSRF checks for all API calls which should not be served by a browser (assuming they use a different auth mechanism which does not involve cookies!). This makes sense since CSRF is a layer that relates to attacks involving browsers/cookie authentication.
- yeukhon 9y ago
  I always believe /login served by a browser and /login for api should never be the same, to echo your statement they require different handler. It should be two completely different view; but those views call the same login function internally. Perhaps /login is a bad example, but use that in views where authentication and authorization are needed.
- captn3m0 9y ago
  This is the right solution. CSRF checks make no sense for an API that is accessed by a non-browser clients.
Grollicus 9y ago
Despite using Functions he doesn't really understand and yada yada i think this is a good reason to have a dedicated api endpoint and not mix it with domains where the normal web access happens.
catshirt 9y ago
when you have a bizarre request issue... find a working request, find the failing request, compare the headers... problem solved.
- taormina 9y ago
  That was literally the first thing the author did.
  - catshirt 9y ago
    is that how you interpreted it? it's not until paragraph 13 (!) where author states: "Josh realised my app might be somehow getting cookies from visiting our site inside this web view, and then sharing those cookies with AFNetworking's underlying NSURLSession, which is handling my requests." which would have been clear if they'd checked the cookies of the NSURLSession request. then, in paragraph 17, the author starts concluding: > "And so the the CSRF check was failing every time, because I wasn't sending a Referer header, or any other related CSRF bits." which would have been clear if they tried comparing a working request to a failing request. and finally, > "And of course, I didn't even know about these cookies, so I wasn't deleting them on log out" which leads me believe the author never checked the request at all, let alone before anything else. i'm ignoring the fact the bug sat for months, and took a second person to fix, and got a write up on the blog... so i'm going to have to assume my advice has an audience (like anyone who would classify this bug as "bizarre"). (not trying to pass judgement on the author here at all... we have all been there :))
js2 9y ago
Oh hey, I know this. From a commit message earlier this year for an SDK I own: SDKSender: switch to ephemeral NSURLSession and disable cookie storage Switch to a private (ephemeral) session independent of the app and disables cookies. The SDK otherwise shares the cookie storage with the app, which is not desired. This was causing the SDK to send the $dayjob tracking cookie (and others) to $dayjob API endpoint.
oretoz 9y ago
Za
CodeWriter23 9y ago
Just popped in to say Wireshark would have made short work of this bug.

137 points by joshsharp 9y ago

This is both awesome and depressing. But it gave me a really interesting idea: a service/network/community/forum/etc for people to gather and discuss Really Confusing Bugs™ that they're trying to figure out. Not necessarily (?) for contemporary end users (maybe highly technical end users). This could actually be a really cool concept. Somewhere squarely between HackerOne and StackOverflow - not for exploits, and not for simple(r) stuff, but specifically for complicated and confusing bugs you've been staring at for days/weeks and nothing's making any sense. I can see a subscription model working for this, even - subscriptions would work both to allow people to provide extended assistance, and also because a contract makes NDAs easier. Hmm. Thinking about how the subscription model would work... you sign up, configure billing, that then allows you to request extended assistance. - One way that could work is that people offer you help in return for thanks, which would work like a configurable upvote; higher quality answers attract more rewards. Maybe anyone can reward answers (via the credit in their account) after the fact? - Another way would be setting a minimum or exact reward amount up front to attract more help. Regardless of how it worked, the site would have all discussion be public and open by default; you'd have to check a box to make the discussion private, and even after that you'd have the ability to go through and selectively un-redact parts of the conversations so everyone could be helped. And anyone could sign up and offer answers instantly, and the rewards credited to their account could be cashed out at any time. That would attract new users. I realize I've just described a weird kind of paid StackOverflow. I am very curious why SE hasn't pursued such an idea. As in, I am 1000% confident they've had this conversation at least once, and I'd really love to hear what the opinions were.

hathym 9y ago
Stackoverfliw ?
- exikyut 9y ago
  ? What do you mean?
- fapjacks 9y ago
  It would be great if this were possible, but unfortunately, even relatively novel and interesting bug posts will be indiscriminately closed on Stack Overflow. You can bet that it will be aggressively closed as a duplicate or as "unclear" or "too general". That is the fate of the vast majority of new posts there.
  - wolco 9y ago
    I never understood why they didn't allow duplicate questions. It provides a fresh prespective and a new updated way to solve a question.
    - mort96 9y ago
      Things I have experienced when finding an SO question from Google: * The question is closed as a duplicate, but the question it's a duplicate of is subtly different enough to be completely irrelevant to me. * The question is closed as a dupilcate, but there's no link to what it's a duplicate of. * The question is closed as a duplicate, but there are no good answers to the question it's a duplicate of.
      - deleted 9y ago
        [deleted]
      - fapjacks 9y ago
        At some point, there were an equal number of average and bad programmers on Stack Overflow as there were average and good programmers on Stack Overflow. But Stack Overflow does not reward people for being good programmers. SO rewards actions that game the system. And when SO points became a kind of currency for finding jobs ("I have five thousand points on SO!" on a resume), it skewed the userbase heavily in the direction of "game the system for more points". So you had average, bad, and awful programmers gaming the system and earning points, but because they are just terrible at what they do and the quality of all of their output is awful, this had the effect of driving away the average, good, and excellent programmers away from Stack Overflow. At some point, the whole of SO shifted, and what you had was a bunch of point-chasing, terrible "programmers" gaming the system for points. There are actually a substantial number of reasonable questions which are not even slightly related to the questions they're closed as duplicates of. But in the eyes of these shit-awful "programmers" they're similar enough to close. Stack Overflow is on its way out. It has served its purpose in life, and perhaps will remain somewhat relevant to the kinds of enormous enterprises that hire shit-awful programmers with their Stack Overflow points on their resumes. But all the reasons it was originally attractive are now gone, as are the majority of the excellent programmers that made it interesting. I'm not sure if Quora is its replacement, and I'm not sure what the next big place where excellent programmers go to share insights looks like. But it's not Stack Overflow.
bigiain 9y ago
" a service/network/community/forum/etc for people to gather and discuss Really Confusing Bugs" Should be universally open to all - at least until someone screws up badly enough to get temporarily or permanently barred, needs to be open outside regular work hours, should be free (or super a inexpensive nominal charge) to enter, would need to monetise itself via other means, perhaps selling beverages. We've just re-invented "the bar"... ;-)
andreareina 9y ago
ExpertsExchange was[1] a paid StackOverflow before the latter existed, SO came along and ate EE's lunch. Some of that was undoubtedly due to EE's (anecdotally, I never used it due to the money issue) bad UX, but free makes a big difference. [1] They still exist, and it apparently gets traffic/use, but never hear anyone talking about them, and it never comes up when searching to resolve or diagnose a problem.
- pbhjpbhj 9y ago
  EE had a scheme where you could answer questions to get free membership; I used that for a while. I remember getting screwed over but don't remember the details: they locked up the site and I couldn't even access the solutions I'd written anymore. A bit later still they made it appear locked but you could still get to the answers if you knew how (I think that was to get their solutions on Google. I'd guess it was taken over and someone tried to squeeze the money out, hasn't happened to SO yet.
  - wolco 9y ago
    if I remember you had to scroll down past the ad content.
  - deleted 9y ago
    [deleted]
- mort96 9y ago
  I still hear people talking about ExpertsExchange. Usually accompanied with a joke about their name; ExpertSexChange.

foota 9y ago

Are they using a header or s.t. for their authorization in app, then?

joshsharp 9y ago
Using the Authorization header.
- foota 9y ago
  Thanks for following up! Glad y'all found the bug :)

luketheobscure 9y ago

"NSURLSession oh so helpfully auto-fucking-matically decided I would probably—no, definitely—want to send those cookies in every single request my app did after that. Forever and always." I imagine it honors the "expires" property of cookies, it just probably wasn't set. Also there's an "ephemeralSessionConfiguration" you can use if you don't want to store the cookies. I enjoyed the article, up until the author blamed the framework for their own ignorance.

joshsharp 9y ago
Sure, the cookies expire (in two weeks I think, in this case) but that doesn't invalidate the issue that the wkwebview silently shares cookies with nsurlsession, nor does it negate the confusing journey of tracking down the bug. Sheesh.
- spullara 9y ago
  This is exactly what users want and you are basically breaking it by not honoring the cookies.
  - joshsharp 9y ago
    We do not need to honour cookies when making requests to the API as we use token authentication. Even if we wanted to keep users logged in within the web view, which we don't because the web view is just for static help pages, we wouldn't want to send those to the API.
    - eridius 9y ago
      Then you should disable cookies on your API calls by setting the `httpCookieStorage` property of your URLSessionConfiguration to nil. You could also use an ephemeral configuration (which will preserve cookies set by the API in memory but won't share them with webviews), which may or may not be a good idea depending on if you're using this session for anything that should be cached to disk (e.g. downloading images).

stanleydrew 9y ago

Not sure about the "fix", since now users can't persist a logged in browser session? If you aren't using cookies for API auth, is there not some way to configure your server framework to just ignore them if received?

joshsharp 9y ago
They shouldn't be logging in via the webview within the app anyway, so that's moot. We are using cookies, as the post says, as a fallback authentication so devs can browse the API from a browser if they're logged in to the site.
- eridius 9y ago
  Why are devs using a browser to hit API endpoints? There are much better tools. My preferred one is called Paw (https://paw.cloud https://paw.cloud) but there are others.
  - grkvlt 9y ago
    Because Paw is GBP 39.99 and a browser is free?
    - eridius 9y ago
      There are other free tools too, like Postman (https://www.getpostman.com https://www.getpostman.com).
      - grkvlt 9y ago
        True, I actually use Postman in Chrome, forgot to mention that.

udioron 9y ago

Another fix is to disable CSRF checks for all API calls which should not be served by a browser (assuming they use a different auth mechanism which does not involve cookies!). This makes sense since CSRF is a layer that relates to attacks involving browsers/cookie authentication.

yeukhon 9y ago
I always believe /login served by a browser and /login for api should never be the same, to echo your statement they require different handler. It should be two completely different view; but those views call the same login function internally. Perhaps /login is a bad example, but use that in views where authentication and authorization are needed.
captn3m0 9y ago
This is the right solution. CSRF checks make no sense for an API that is accessed by a non-browser clients.

Grollicus 9y ago

Despite using Functions he doesn't really understand and yada yada i think this is a good reason to have a dedicated api endpoint and not mix it with domains where the normal web access happens.

catshirt 9y ago

when you have a bizarre request issue... find a working request, find the failing request, compare the headers... problem solved.

taormina 9y ago
That was literally the first thing the author did.
- catshirt 9y ago
  is that how you interpreted it? it's not until paragraph 13 (!) where author states: "Josh realised my app might be somehow getting cookies from visiting our site inside this web view, and then sharing those cookies with AFNetworking's underlying NSURLSession, which is handling my requests." which would have been clear if they'd checked the cookies of the NSURLSession request. then, in paragraph 17, the author starts concluding: > "And so the the CSRF check was failing every time, because I wasn't sending a Referer header, or any other related CSRF bits." which would have been clear if they tried comparing a working request to a failing request. and finally, > "And of course, I didn't even know about these cookies, so I wasn't deleting them on log out" which leads me believe the author never checked the request at all, let alone before anything else. i'm ignoring the fact the bug sat for months, and took a second person to fix, and got a write up on the blog... so i'm going to have to assume my advice has an audience (like anyone who would classify this bug as "bizarre"). (not trying to pass judgement on the author here at all... we have all been there :))

js2 9y ago

Oh hey, I know this. From a commit message earlier this year for an SDK I own: SDKSender: switch to ephemeral NSURLSession and disable cookie storage Switch to a private (ephemeral) session independent of the app and disables cookies. The SDK otherwise shares the cookie storage with the app, which is not desired. This was causing the SDK to send the $dayjob tracking cookie (and others) to $dayjob API endpoint.

oretoz 9y ago

CodeWriter23 9y ago

Just popped in to say Wireshark would have made short work of this bug.