8 ms·

Post Mortem: Incorrect Cache Configuration Exposes Personal Information

79 points by dcminter 5y ago

tut-urut-utut 5y ago
It's funny that I haven't heard about it before, although I live in Germany and Klarna is probably the biggest payment processor in Germany. Luckily, I haven't used it. I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal? EDIT: I haven't heard about Klarna outage, Klarna is well known.
- Xylakant 5y ago
  Are you certain you haven’t used it? Some shopping sites use them as payment processor for regular CC transactions without much visibility. You have to look at the T&C to even notice.
  - tut-urut-utut 5y ago
    I usually don't leave CC info on some random site, only to major and well known. If the site is unknown, it's either PayPal, "pay by invoice" or no order at all. Either PayPal or CC issuer provide at least some buyers' protection, while given your bank credentials to Klarna is normally against ToC of your bank.
    - deleted 5y ago
      [deleted]
    - jokethrowaway 5y ago
      I try not to use PayPal in the hope they will become irrelevant as a business (I know, not holding my breath), mainly because their customer service and fees are terrible. CCs are not a great mechanism and I wished we had better bank services and safer integrations to process payments, but that's what we have. I check my bank statement regularly and I know I can get money back if fraud happens. If I don't trust a website I used a prepaid with the exact amount on.
    - zibzab 5y ago
      > I usually don't leave CC info on some random site, only to major and well known. That's the whole point of klarna (who also do CC processing so might have used them unknowingly): you don't need to trust Johnny's Little Online Store handling your CC, since the payment is handled by klarna.
      - tut-urut-utut 5y ago
        The irony is, as a buyer, I trust unknown Johnny's Little Online Store as much as Klarna. Maybe even more, since I can give a Little Johnny the benefit of the doubt. But in practice it's either invoice, PayPal or CC. Even if Klarna process that payment in the background, I can still object to my CC company and let them settle it, instead of having to deal with them.
        zibzab 5y ago
        Do you believe Johnnys has the resources to create a secure payment system and keep it secure? Also, Klarna is a major invoice and CC processor. You probably have already used them unknowingly.
- randomlurking 5y ago
  I agree, haven’t used it (knowingly) and regularly order from other sites if klarna is the only payment option.
- julianlam 5y ago
  > I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal? I am parroting other HN comments here, but the reason this is necessary is because banks do not provide a public (or private) API except between other banks. Therefore the only avenue for integration would be directly accessing the account.
  - ludamad 5y ago
    I wish this wasn't normal to me, but I've been fully conditioned to expect this from QuickBooksOnline
  - showsover 5y ago
    They do though. PDS2 requires all banks to have an accessible API.
    - samhw 5y ago
      This is different from banks actually _having_ an API. Many have stalled or successfully asked for exemptions, and even those who have gone ahead and, to some extent, complied, have still tried to make their APIs as difficult to use, feature-poor, and unreliable as possible. Source: worked at a bank which actually tried to do this well, and which also offered functionality which required us to integrate with other banks' APIs.
  - Tainnor 5y ago
    Not really true in Germany where FinTS and its precursor HBCI have been supported by many banks since the 90s or so. Ironically, it's typically the hip start-up banks like N26 that don't support this standard.
- Tainnor 5y ago
  > How is that even legal? It's worse than that. Several banks tried to enforce in their terms and conditions that sharing your bank login data would be a breach of contract or at least absolve the bank of any liability. This was deemed in violation of antitrust laws for some reason, so now banks have to actively allow this which is just brain-dead IMHO. The only upside of this is that we now have PSD2 now anyway, so the damage that can be done by someone who has your login credentials is limited.
  - mdaniel 5y ago
    For others wondering what PSD2 is: https://en.wikipedia.org/wiki/Payment_Services_Directive https://en.wikipedia.org/wiki/Payment_Services_Directive and there were also a lot of "what is it and how does it affect you" articles from name brand bank websites, too, but I figured start here
  - samhw 5y ago
    > This was deemed in violation of antitrust laws for some reason My best guess at what that "some reason" may be is that it breaks shoddily built integrations like [Teller](https://teller.io/ https://teller.io/), which, instead of using schemes like Open Banking where available, simply rely on credential sharing.
  - gmokki 5y ago
    In Finland it is illegal to give your bank login data to others, including scrapers. One couple both got sentenced when the other used the bank login data left on table to order something online. Fraud for the other and neglegence of leaving the login data easily accessible for the other.
willejs 5y ago
This sounds a bit like request collapsing on their CDN was enabled perhaps?
- terom 5y ago
  https://developer.fastly.com/learning/concepts/request-collapsing/ https://developer.fastly.com/learning/concepts/request-colla... fastly's VCL docs are interesting, particularly the bit about request queueing and `Cache-Control: private`. This sounds exactly like the kind of confusion that would lead to this kind of issue, while not being noticed in a low-traffic test environment with fewer simultaneous requests. > HINT: One of many reasons you should be using Cache-Control and not Expires for your caching configuration is the ability to use directives like private. The private directive tells Fastly that this response is good for only one user, the one that originally triggered the request. Anyone else that got tagged onto the queue must now be dequeued and their requests processed individually. > Responses not marked private, may be used to satisfy queued requests. Even a response with max-age=0 or the equivalent no-cache can be used to satisfy all the waiting clients, and the next request for the object will once again be a MISS. If your reason for giving a response a zero cache lifetime is that it contains content intended for a single user only, then ensure it is correctly marked private to avoid having this content sent to more than one user.
deleted 5y ago
[deleted]
irjustin 5y ago
We talked about this during the outage before but any API requests related to account information should not be going through a cacheable CDN. At the mild end, you could get stale data. Deposit some money and the cache doesn't reflect making you wonder whether the deposit happened or not. On the worst end, you get this - all users get exposed the most recently requested data. All this to save... a few kB?
- dijit 5y ago
  I agree. I've run some eCommerce sites before, we were PCI Tier 1, roughly 1% of web traffic: but nowhere _near_ the level of security Klarna should have; we had the CDN's activated only on subdomains... always. Usually static.<site>.com or whatever.. This has the added benefit of not sending cookies to the CDN, because that can affect latency and is also a bit less secure because a static image/css/js does not need the cookie. (and bandwidth, before we were charged per byte (and that's egress now anyway in most cases, god I'm old.)). If I had to venture a guess, they're probably scared of getting slammed with a DDoS so they "guard" everything behind the CDN, it's not about caching. We got slammed with DDoS's too so we had to have systems in place to push BGP routes via verisign to "scrub" the traffic... we didn't even have that many "credits" and it cost millions...
- johbjo 5y ago
  I suspect the background is the app is showing html-views rendered on the server, and they are sharing as much infrastructure as possible. To make it easier on the app team. If they separated app/cdn/web with different SSL certificates, something like this wouldn't be possible even when developers eff up.
- glenngillen 5y ago
  Whether a CDN is caching or not could itself just be a misconfiguration. Passing dynamic non-cached requests through a CDN isn’t about saving a few kB for the server (by definition it still has to generate it). It’s about improving end-user performance through things like having edge termination of TLS. So it’s not so easy to have such a blanket statement as no API requests should go through a CDN. While it’s all too easy to inadvertently turn on the caching of requests.
Deathmax 5y ago
This happened to Steam as well, where misconfigured caching leaked PII: https://store.steampowered.com/oldnews/19852 https://store.steampowered.com/oldnews/19852
- jokethrowaway 5y ago
  The same happened to the Italian government: https://moondo.info/en/Luciano-Vangone-suddenly-the-most-famous-citizen-of-Italy/ https://moondo.info/en/Luciano-Vangone-suddenly-the-most-fam...
- the-rc 5y ago
  It used to happen even with everything configured correctly, if your users connected over plain HTTP, from an ISP that forced their traffic to go through a transparent proxy with overaggressive caching. It's not fun to hear about users logged in with somebody else's account, but only in remote corners of the world instead of the more densely populated areas as you'd expect.
EE84M3i 5y ago
Seems to go out of their way to say which vendor they were using? Looks like now they're on Cloudfront.
gherkinnn 5y ago
Unrelated to the incident, but to Klarna. What is Klarna _for_? What does Klarna do that others don't? They brand themselves as "We enable you to buy hip shit you don't have the money for [and will become irrelevant before you payed off your debt]". Almost literally [0] Is it more of the same but dressed pretty or am I missing something? That said, sometimes a common thing done up well will be successful. Looking at you, Slack. 0 - https://www.klarna.com/us/pay-in-4/ https://www.klarna.com/us/pay-in-4/
- brtkdotse 5y ago
  There’s a convenience factor at play. A lot, if not all, e-commerce sites in Sweden have a nice big fat “pay with Klarna” button and it’s a one click purchase. No filling in addresses, getting your credit card out etc.
- jbwyme 5y ago
  I believe their value lies in their e-commerce partnerships. You can effectively check out (almost) as seamlessly as a using a credit card on partner sites. I used Klarna when I bought new golf clubs. They show up right on the checkout screen. I can afford to buy them outright but it was an interest free loan for two years so it made logical sense to me.
- tut-urut-utut 5y ago
  Well, they are all in one payment processor, loan institute and debt collector with terrible record, with some questionable business practices.
- __alexs 5y ago
  They are an elaborate pay day loans company.
  - samhw 5y ago
    Yeah, I would agree with that: it's payday loans with a friendly face. I once worked with someone who had previously been an exec at Klarna, and he had some 'interesting' stories about how they did credit decisioning. One of the examples that always stuck in my mind[0] was declining people with five or more vowels in their last name, because it was an effective proxy for West African people. (I remember that one because it also picks up my somewhat posh double-barrelled English surname, so I suppose I would have been collateral damage.) [0] I should say I can't remember for sure if this was from his time at Klarna or somewhere else, but it was at least illustrative of the kind of people and the culture there.
geofft 5y ago
This looks distressingly similar to https://blog.zulip.com/2021/03/20/zulip-cloud-security-incident/ https://blog.zulip.com/2021/03/20/zulip-cloud-security-incid... . Klarna: Our app is configured specifically to avoid inadvertent caching through the use of cache control directives found in the HTTP standard (no-cache, no-store, must-revalidate). However, the CDN configuration can at times override the app configuration. Zulip: Zulip separately marks its responses as non-cacheable, both to prevent private data from being sent to the wrong user, and also to prevent stale data from being sent even to the right user. Indicating that a response is not cacheable is the purpose of Zulip’s Cache-Control header: max-age=0 [...] no-cache [...] no-store [...] must-revalidate [...] private [...] CloudFront usually respects the Cache-Control and Expires headers, but with one critical exception when receiving multiple similar requests at roughly the same time It looks like Klarna's main website uses CloudFront, so I bet this was the exact same thing.
xbar 5y ago
Impressive success at burying the lead. Let me fix for it. "Detailed incident report: No One At Klarna Understands That Disabling CDN Cache Protections for Personal Information is Bad"
- brasetvik 5y ago
  Their initial post blamed the problem on “human error”, an anti-pattern in getting to the root cause of things. This post attempts to blame their CDN vendor. “All these reviewers couldn’t spot it.”
new_here 5y ago
The blog post doesn't say much more than the headline. I'm curious about the specifics of what could of actually happened here. In my limited experience working with CDNs wouldn't you just cache the responses of unique URLs and have some sort of cookie check at the edge before serving it. So my own app would request something like /api/account?id=123 with my own id in there. How would you end up getting other people's data in your app if your app only calls that unique URL?
- q3k 5y ago
  > How would you end up getting other people's data in your app if your app only calls that unique URL? From what I understand what happened with this outage, the CDN would still cache /api/account?id=123, and someone with account ID 234 could access it by altering the URL to retrieve the cached version, if account 123 has used the app recently. That's because a CDN has (usually) no concept of authorization/authentication and can't make decisions that /api/account?id=123 shouldn't be served to someone other than the owner of account 123. It would be less catastrophic (at least from a PR point of view) because people wouldn't get immediately served others' accounts, but you'd be vulnerable to attack.
  - Ndymium 5y ago
    In fact users did get other users' data just by using the app normally, so it likely didn't happen the way you describe it.
    - q3k 5y ago
      Right, what probably happened is that an `/accounts/me` endpoint got cached. But what GP seems to be asking about is: “Would having your app always encode a user ID into the endpoint have helped?”.
      - new_here 5y ago
        Yeah, I've just realised they've probably accidentally included a generic URL in the cache rules that they actually didn't intend to cache. I originally thought they were trying to cache account data responses and so wondered why they wouldn't just use unique query parameters in that case. Definitely risky business though. Out of interest, it looks like Cloudflare offers some sort of token authentication to authenticate at the edge: https://blog.cloudflare.com/token-authentication-for-cached-private-content-and-apis/ https://blog.cloudflare.com/token-authentication-for-cached-...
fuzzer37 5y ago
> This configuration change had been written by an authorized engineer in one team, consulting but misinterpreting vendor documentation. The change was then reviewed by three other authorized engineers in three separate teams, according to normal processes. None of them spotted the mistake in the configuration update. Is anyone else shocked by this? Are code reviews worthless? An engineer made a change reviewed by 3 other people. It seems like _someone_ would notice.
- peach 5y ago
  Once in a while a bunch of people will fail to notice something that might be cryptic if you don't know the details.
- coldcode 5y ago
  Reviewing someone's code you had not seen before, involving a third party vendor you know nothing about, that only happens in production environment and only randomly fails and otherwise works—does not shock me at all. No process no matter how thorough can still fail. Code reviews in my prior experience solved nothing as people were expected to work full time and yet still review code continuously; unless you give engineers sufficient time to do both without overtime effort, one or the other will be affected, and usually its the reviews.
  - mdaniel 5y ago
    I had always seen code review process as partially a knowledge sharing exercise, so fine, maybe none of the 3 other engineers had previously seen those vendor docs, but part of the process is to retrace the steps and see if (with the benefit of hindsight) the reviewer would reach the same conclusion > Code reviews in my prior experience solved nothing I find that assertion suspicious, unless the culture was "push the approval button without even reading because ship,ship,ship"

79 points by dcminter 5y ago

It's funny that I haven't heard about it before, although I live in Germany and Klarna is probably the biggest payment processor in Germany. Luckily, I haven't used it. I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal? EDIT: I haven't heard about Klarna outage, Klarna is well known.

Xylakant 5y ago
Are you certain you haven’t used it? Some shopping sites use them as payment processor for regular CC transactions without much visibility. You have to look at the T&C to even notice.
- tut-urut-utut 5y ago
  I usually don't leave CC info on some random site, only to major and well known. If the site is unknown, it's either PayPal, "pay by invoice" or no order at all. Either PayPal or CC issuer provide at least some buyers' protection, while given your bank credentials to Klarna is normally against ToC of your bank.
  - deleted 5y ago
    [deleted]
  - jokethrowaway 5y ago
    I try not to use PayPal in the hope they will become irrelevant as a business (I know, not holding my breath), mainly because their customer service and fees are terrible. CCs are not a great mechanism and I wished we had better bank services and safer integrations to process payments, but that's what we have. I check my bank statement regularly and I know I can get money back if fraud happens. If I don't trust a website I used a prepaid with the exact amount on.
  - zibzab 5y ago
    > I usually don't leave CC info on some random site, only to major and well known. That's the whole point of klarna (who also do CC processing so might have used them unknowingly): you don't need to trust Johnny's Little Online Store handling your CC, since the payment is handled by klarna.
    - tut-urut-utut 5y ago
      The irony is, as a buyer, I trust unknown Johnny's Little Online Store as much as Klarna. Maybe even more, since I can give a Little Johnny the benefit of the doubt. But in practice it's either invoice, PayPal or CC. Even if Klarna process that payment in the background, I can still object to my CC company and let them settle it, instead of having to deal with them.
      - zibzab 5y ago
        Do you believe Johnnys has the resources to create a secure payment system and keep it secure? Also, Klarna is a major invoice and CC processor. You probably have already used them unknowingly.
randomlurking 5y ago
I agree, haven’t used it (knowingly) and regularly order from other sites if klarna is the only payment option.
julianlam 5y ago
> I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal? I am parroting other HN comments here, but the reason this is necessary is because banks do not provide a public (or private) API except between other banks. Therefore the only avenue for integration would be directly accessing the account.
- ludamad 5y ago
  I wish this wasn't normal to me, but I've been fully conditioned to expect this from QuickBooksOnline
- showsover 5y ago
  They do though. PDS2 requires all banks to have an accessible API.
  - samhw 5y ago
    This is different from banks actually _having_ an API. Many have stalled or successfully asked for exemptions, and even those who have gone ahead and, to some extent, complied, have still tried to make their APIs as difficult to use, feature-poor, and unreliable as possible. Source: worked at a bank which actually tried to do this well, and which also offered functionality which required us to integrate with other banks' APIs.
- Tainnor 5y ago
  Not really true in Germany where FinTS and its precursor HBCI have been supported by many banks since the 90s or so. Ironically, it's typically the hip start-up banks like N26 that don't support this standard.
Tainnor 5y ago
> How is that even legal? It's worse than that. Several banks tried to enforce in their terms and conditions that sharing your bank login data would be a breach of contract or at least absolve the bank of any liability. This was deemed in violation of antitrust laws for some reason, so now banks have to actively allow this which is just brain-dead IMHO. The only upside of this is that we now have PSD2 now anyway, so the damage that can be done by someone who has your login credentials is limited.
- mdaniel 5y ago
  For others wondering what PSD2 is: https://en.wikipedia.org/wiki/Payment_Services_Directive https://en.wikipedia.org/wiki/Payment_Services_Directive and there were also a lot of "what is it and how does it affect you" articles from name brand bank websites, too, but I figured start here
- samhw 5y ago
  > This was deemed in violation of antitrust laws for some reason My best guess at what that "some reason" may be is that it breaks shoddily built integrations like [Teller](https://teller.io/ https://teller.io/), which, instead of using schemes like Open Banking where available, simply rely on credential sharing.
- gmokki 5y ago
  In Finland it is illegal to give your bank login data to others, including scrapers. One couple both got sentenced when the other used the bank login data left on table to order something online. Fraud for the other and neglegence of leaving the login data easily accessible for the other.

willejs 5y ago

This sounds a bit like request collapsing on their CDN was enabled perhaps?

terom 5y ago
https://developer.fastly.com/learning/concepts/request-collapsing/ https://developer.fastly.com/learning/concepts/request-colla... fastly's VCL docs are interesting, particularly the bit about request queueing and `Cache-Control: private`. This sounds exactly like the kind of confusion that would lead to this kind of issue, while not being noticed in a low-traffic test environment with fewer simultaneous requests. > HINT: One of many reasons you should be using Cache-Control and not Expires for your caching configuration is the ability to use directives like private. The private directive tells Fastly that this response is good for only one user, the one that originally triggered the request. Anyone else that got tagged onto the queue must now be dequeued and their requests processed individually. > Responses not marked private, may be used to satisfy queued requests. Even a response with max-age=0 or the equivalent no-cache can be used to satisfy all the waiting clients, and the next request for the object will once again be a MISS. If your reason for giving a response a zero cache lifetime is that it contains content intended for a single user only, then ensure it is correctly marked private to avoid having this content sent to more than one user.

deleted 5y ago

[deleted]

irjustin 5y ago

We talked about this during the outage before but any API requests related to account information should not be going through a cacheable CDN. At the mild end, you could get stale data. Deposit some money and the cache doesn't reflect making you wonder whether the deposit happened or not. On the worst end, you get this - all users get exposed the most recently requested data. All this to save... a few kB?

dijit 5y ago
I agree. I've run some eCommerce sites before, we were PCI Tier 1, roughly 1% of web traffic: but nowhere _near_ the level of security Klarna should have; we had the CDN's activated only on subdomains... always. Usually static.<site>.com or whatever.. This has the added benefit of not sending cookies to the CDN, because that can affect latency and is also a bit less secure because a static image/css/js does not need the cookie. (and bandwidth, before we were charged per byte (and that's egress now anyway in most cases, god I'm old.)). If I had to venture a guess, they're probably scared of getting slammed with a DDoS so they "guard" everything behind the CDN, it's not about caching. We got slammed with DDoS's too so we had to have systems in place to push BGP routes via verisign to "scrub" the traffic... we didn't even have that many "credits" and it cost millions...
johbjo 5y ago
I suspect the background is the app is showing html-views rendered on the server, and they are sharing as much infrastructure as possible. To make it easier on the app team. If they separated app/cdn/web with different SSL certificates, something like this wouldn't be possible even when developers eff up.
glenngillen 5y ago
Whether a CDN is caching or not could itself just be a misconfiguration. Passing dynamic non-cached requests through a CDN isn’t about saving a few kB for the server (by definition it still has to generate it). It’s about improving end-user performance through things like having edge termination of TLS. So it’s not so easy to have such a blanket statement as no API requests should go through a CDN. While it’s all too easy to inadvertently turn on the caching of requests.

Deathmax 5y ago

This happened to Steam as well, where misconfigured caching leaked PII: https://store.steampowered.com/oldnews/19852 https://store.steampowered.com/oldnews/19852

jokethrowaway 5y ago
The same happened to the Italian government: https://moondo.info/en/Luciano-Vangone-suddenly-the-most-famous-citizen-of-Italy/ https://moondo.info/en/Luciano-Vangone-suddenly-the-most-fam...
the-rc 5y ago
It used to happen even with everything configured correctly, if your users connected over plain HTTP, from an ISP that forced their traffic to go through a transparent proxy with overaggressive caching. It's not fun to hear about users logged in with somebody else's account, but only in remote corners of the world instead of the more densely populated areas as you'd expect.

EE84M3i 5y ago

Seems to go out of their way to say which vendor they were using? Looks like now they're on Cloudfront.

gherkinnn 5y ago

Unrelated to the incident, but to Klarna. What is Klarna _for_? What does Klarna do that others don't? They brand themselves as "We enable you to buy hip shit you don't have the money for [and will become irrelevant before you payed off your debt]". Almost literally [0] Is it more of the same but dressed pretty or am I missing something? That said, sometimes a common thing done up well will be successful. Looking at you, Slack. 0 - https://www.klarna.com/us/pay-in-4/ https://www.klarna.com/us/pay-in-4/

brtkdotse 5y ago
There’s a convenience factor at play. A lot, if not all, e-commerce sites in Sweden have a nice big fat “pay with Klarna” button and it’s a one click purchase. No filling in addresses, getting your credit card out etc.
jbwyme 5y ago
I believe their value lies in their e-commerce partnerships. You can effectively check out (almost) as seamlessly as a using a credit card on partner sites. I used Klarna when I bought new golf clubs. They show up right on the checkout screen. I can afford to buy them outright but it was an interest free loan for two years so it made logical sense to me.
tut-urut-utut 5y ago
Well, they are all in one payment processor, loan institute and debt collector with terrible record, with some questionable business practices.
__alexs 5y ago
They are an elaborate pay day loans company.
- samhw 5y ago
  Yeah, I would agree with that: it's payday loans with a friendly face. I once worked with someone who had previously been an exec at Klarna, and he had some 'interesting' stories about how they did credit decisioning. One of the examples that always stuck in my mind[0] was declining people with five or more vowels in their last name, because it was an effective proxy for West African people. (I remember that one because it also picks up my somewhat posh double-barrelled English surname, so I suppose I would have been collateral damage.) [0] I should say I can't remember for sure if this was from his time at Klarna or somewhere else, but it was at least illustrative of the kind of people and the culture there.

geofft 5y ago

This looks distressingly similar to https://blog.zulip.com/2021/03/20/zulip-cloud-security-incident/ https://blog.zulip.com/2021/03/20/zulip-cloud-security-incid... . Klarna: Our app is configured specifically to avoid inadvertent caching through the use of cache control directives found in the HTTP standard (no-cache, no-store, must-revalidate). However, the CDN configuration can at times override the app configuration. Zulip: Zulip separately marks its responses as non-cacheable, both to prevent private data from being sent to the wrong user, and also to prevent stale data from being sent even to the right user. Indicating that a response is not cacheable is the purpose of Zulip’s Cache-Control header: max-age=0 [...] no-cache [...] no-store [...] must-revalidate [...] private [...] CloudFront usually respects the Cache-Control and Expires headers, but with one critical exception when receiving multiple similar requests at roughly the same time It looks like Klarna's main website uses CloudFront, so I bet this was the exact same thing.

xbar 5y ago

Impressive success at burying the lead. Let me fix for it. "Detailed incident report: No One At Klarna Understands That Disabling CDN Cache Protections for Personal Information is Bad"

brasetvik 5y ago
Their initial post blamed the problem on “human error”, an anti-pattern in getting to the root cause of things. This post attempts to blame their CDN vendor. “All these reviewers couldn’t spot it.”

new_here 5y ago

The blog post doesn't say much more than the headline. I'm curious about the specifics of what could of actually happened here. In my limited experience working with CDNs wouldn't you just cache the responses of unique URLs and have some sort of cookie check at the edge before serving it. So my own app would request something like /api/account?id=123 with my own id in there. How would you end up getting other people's data in your app if your app only calls that unique URL?

q3k 5y ago
> How would you end up getting other people's data in your app if your app only calls that unique URL? From what I understand what happened with this outage, the CDN would still cache /api/account?id=123, and someone with account ID 234 could access it by altering the URL to retrieve the cached version, if account 123 has used the app recently. That's because a CDN has (usually) no concept of authorization/authentication and can't make decisions that /api/account?id=123 shouldn't be served to someone other than the owner of account 123. It would be less catastrophic (at least from a PR point of view) because people wouldn't get immediately served others' accounts, but you'd be vulnerable to attack.
- Ndymium 5y ago
  In fact users did get other users' data just by using the app normally, so it likely didn't happen the way you describe it.
  - q3k 5y ago
    Right, what probably happened is that an `/accounts/me` endpoint got cached. But what GP seems to be asking about is: “Would having your app always encode a user ID into the endpoint have helped?”.
    - new_here 5y ago
      Yeah, I've just realised they've probably accidentally included a generic URL in the cache rules that they actually didn't intend to cache. I originally thought they were trying to cache account data responses and so wondered why they wouldn't just use unique query parameters in that case. Definitely risky business though. Out of interest, it looks like Cloudflare offers some sort of token authentication to authenticate at the edge: https://blog.cloudflare.com/token-authentication-for-cached-private-content-and-apis/ https://blog.cloudflare.com/token-authentication-for-cached-...

fuzzer37 5y ago

> This configuration change had been written by an authorized engineer in one team, consulting but misinterpreting vendor documentation. The change was then reviewed by three other authorized engineers in three separate teams, according to normal processes. None of them spotted the mistake in the configuration update. Is anyone else shocked by this? Are code reviews worthless? An engineer made a change reviewed by 3 other people. It seems like _someone_ would notice.

peach 5y ago
Once in a while a bunch of people will fail to notice something that might be cryptic if you don't know the details.
coldcode 5y ago
Reviewing someone's code you had not seen before, involving a third party vendor you know nothing about, that only happens in production environment and only randomly fails and otherwise works—does not shock me at all. No process no matter how thorough can still fail. Code reviews in my prior experience solved nothing as people were expected to work full time and yet still review code continuously; unless you give engineers sufficient time to do both without overtime effort, one or the other will be affected, and usually its the reviews.
- mdaniel 5y ago
  I had always seen code review process as partially a knowledge sharing exercise, so fine, maybe none of the 3 other engineers had previously seen those vendor docs, but part of the process is to retrace the steps and see if (with the benefit of hindsight) the reviewer would reach the same conclusion > Code reviews in my prior experience solved nothing I find that assertion suspicious, unless the culture was "push the approval button without even reading because ship,ship,ship"