10 ms·

Why Monzo's bank transfers weren't working on the 30th of May

284 points by robinson-wall 7y ago

robinson-wall 7y ago
I just posted this semi-technical post-mortem on Monzo's about why we had an outage with Faster Payments (UK bank transfers) last Month. I'll hang around here to answer any more technical questions if anyone's interested.
- kennydude 7y ago
  Nice to see how Faster Payments actually work in a nice understandable way. That does sound weird how it happened by a corrupted date formatter. I'm assuming it's something like the formatter reset itself back to the langauge default
  - robinson-wall 7y ago
    Yeah, instead of `20190530` we were getting `['bert', 'time', 1559, 238096, 0]`. My understanding is a race condition put the code into a situation where it couldn't determine how to format that field, and we just got a default string representation of the underlying value.
    - tuukkah 7y ago
      Which platform gives you unformatted data if a format can't be determined? Or was the code similar to getDate(format) and format happened to be null? Was the race condition in your code or in the platform? (EDIT: I suppose these would be questions more for the developers of the third-party gateway.)
      - Gaelan 7y ago
        FTA: > The bug was in a computer program the [third party] Gateway uses to translate payment messages between two formats. When the program was operating under load, the system tried to clear memory it believed to be unused (a process known as garbage collection). > But because it was using an unsafe method to access memory, the code ended up reading memory that had already been cleared away, causing it not to know how to translate the date field in payment messages.
        btown 7y ago
        This is exactly the type of thing I would expect from old-school banking systems. Kudos to Monzo for even trying to bring sanity to this space.
      - robinson-wall 7y ago
        I haven't seen the code personally, so I'm not sure. The condition was inside the code of the Gateway provider.
      - mikekchar 7y ago
        IIUC, the problem seems to have been that the code was looking into freed memory and so the date format was essentially random data. I can imagine a case statement where you have a default case of "don't translate the date" with a comment over it saying, "This should never happen". I'm sure I've naively written similar code when I was sleepy and it tends to pass review because it's innocuous. It's easy to be hard on the programmer -- probably crashing is better than data in the wrong format, but then you are just pushing out the problem to a different layer. Error handling is completely non-trivial in complex systems. Maybe they should have thrown an exception in that case, but are you sure it's going to be handled? What is the downside in that case? It could easily be worse -- we have no way of knowing. Sometimes it gets down to, "Well, you need to make sure there are no mistakes in the code". If we're going to go down that route, then the incorrect timing of the memory freeing is the real cause (or if I'm being particularly nasty I might say, "You really shouldn't be using threads" ;-) ). I guess what I'm trying to say is that there is certainly a better way of doing defensive programming in this case, but I wouldn't be able to tell what it was without seeing the code. I also wouldn't expect any large codebase to be completely free of these kinds of problems because it's easy to make a mistake.
    - Dunedan 7y ago
      That reminds me of a race condition I encountered a while ago: There was this massive Java application processing hundreds of parallel requests per second. For each request it wrote a line with billing information into a log file. Those lines were fine from a quick glance, but when we tried processing them later we encountered invalid dates. Those log records contained future dates as well as invalid dates like 2019-02-30. Long story short: In the end we figured out that this was caused by the date formatting not being done thread safe (might have been SimpleDateFormat, but I don't remember the details anymore), causing the date components of multiple threads to get interleaved. Ouch, I guess somebody learned a lesson back then.
- hc91 7y ago
  You guys are using Form3 for FPS, is this correct?
  - robinson-wall 7y ago
    Ah "any more technical questions"... except this one. Sorry, but I can't name our partner.
  - wrboyce 7y ago
    We can take an educated guess... https://status.form3.tech/incidents/wyhyxydxgh30 https://status.form3.tech/incidents/wyhyxydxgh30
    - robinson-wall 7y ago
      I would note that this status page says "Our FPS Direct gateway provider".
      - wrboyce 7y ago
        Fair point. FWIW I thought your postmortem was excellent, and certainly puts Form3's to shame regardless of where the blame lies.
  - BillinghamJ 7y ago
    I believe it's PayPort, which is run by Vocalink - who also run FPS itself. As far as I understand, PayPort was/is the recommended options for all new "direct" connections. Though it seems that it is also possible to go more-direct into the FPS system itself.
  - jayelbe 7y ago
    You might be interested to know that Monzo's API refers to Faster Payments transactions as "payport_faster_payments".
- CornishPasty 7y ago
  Was the original BERT formatting from the Hub or from something inside the Gateway's network? I always associated BERT with Erlang, do you know if it was involved or if it was something else? Cheers for this blog post, by the way. It was really informative about the issue, and about how FPS works.
  - robinson-wall 7y ago
    It's used exclusively inside the Gateway's infrastructure. They're a mix of Java and Erlang but I'm not sure on the proportion. FPS uses ISO8583 for its messaging format, and I suspect at the edge the Gateway translates it to a BERT blob for passing around internally.
    - CornishPasty 7y ago
      Ah right that makes sense. It seems odd that it took them so long to identify and rectify the issue. Have they made you any assurances about what they'll do to reduce this time/prevent this from happening in future?
      - robinson-wall 7y ago
        Specifically this class of incident? Yes. They seem to be reasonably good at operational runbooks for known flaws, so I can imagine this class of incident being handed faster in future. What worries me is the next unknown unknown, which is why we are insourcing. One thing I think Monzo can be particularly proud of is our incident response, and debugging of our own systems at speed. That's why this article focuses more on what actions we are taking, and not what actions they are taking.
ablation 7y ago
Thank you for posting this. Great read, and nice to see the team at Monzo sharing this level of detail: consumable but still detailed.
peteretep 7y ago
> They later tell us they believed that datacentre was introducing the corruption What now? Their datacentre was ... rewriting (presumably) encrypted packets?
- robinson-wall 7y ago
  Sorry, perhaps this isn't very clear as I've tried to simplify the explanation to make it accessible to a wide audience. What I meant here is they could tell that the corruption was being introduced by some component in their infrastructure, and they were only observing it for messages passing through one of their two active-active sites.
  - noir_lord 7y ago
    I understood that as you intended. It's a fine line between understandable to laymen and people been pernickity sadly.
PhantomGremlin 7y ago
The software bug at the heart of the problem: The bug was in a computer program the Gateway uses to translate payment messages between two formats. When the program was operating under load, the system tried to clear memory it believed to be unused (a process known as garbage collection). But because it was using an unsafe method to access memory, the code ended up reading memory that had already been cleared away, causing it not to know how to translate the date field in payment messages. So apparently a dangling reference.
- seanmcdirmid 7y ago
  Is that really proper use of the term garbage collection? If you are doing memory management manually, it sounds more like the lack of garbage collection. Unless they were using an unsafe GC for C/C++?
  - jey 7y ago
    Sounds like they were hanging onto a pointer to an object allocated by GC. For example, in Python/C API if you use a borrowed reference PyObject* after it has gone out of scope and been GC'd.
- rahilb 7y ago
  I'm pretty sure they're a Java shop, and they're referring to sun.misc.unsafe and the goodies inside that let you manually allocate memory.
kjlfhg8 7y ago
Not related to the outage, but any plans to provide banking on pc's instead of just phones and any plans to provide small businesses accounts in the future?
- lol768 7y ago
  Yes, they've had job positions open for more web work. I think they will expand this. They already offer business accounts. I have one open for my Ltd company.
  - ownagefool 7y ago
    I registered my interest a while ago but they haven't given me one yet, so I wouldn't say they offer business accounts, so much as they're going to/are testing this. Starling does offer business accounts now, but you can only have one Person of Significant Control, i.e. over 25% owner. There is no monthly fee with their offering though, so it's probably the better offer.
    - lol768 7y ago
      Note that Starling will prevent you opening a business account if you've held a personal account in the past and closed it. That was the position I found myself in. For what it's worth, I've seen some suggestions regarding business account pricing on the Monzo Slack, and there are plans for separate tiers (including a free tier which lacks some of the more advanced accounting integrations).
      - ownagefool 7y ago
        I use a monzo personal account, would be more than happy using them for my ltd if and when they support me doing so.
- deleted 7y ago
  [deleted]
ziddoap 7y ago
Clear, detailed but accessible, plans in place moving forward, apology read sincerely, providing support to affected customers immediately, and answering follow up questions to technical users who are interested in more detail. A+ job on handling the unfortunate situation, Monzo. We can only hope more companies follow this great example.
- deleted 7y ago
  [deleted]
- ccrush 7y ago
  Every time I see a company say "we're sorry" I can't help but think about the South Park episode where the BP CEO says "we're sorry!" It's either that or the one with the Time Warner employees with the nursing flaps in their shirts.
  - _carl_jung 7y ago
    Using "we" in any public announcement, writeup, blog post, etc. is always in danger of sounding contrived.
    - ziddoap 7y ago
      Personally, I prefer something written in a relaxed style rather than a formal-voice only in most cases, especially for blog posts. Formal only generally comes across, to me, as cold and distant. Great for a persuasive essay or other mediums where you want to remove the topic from the author, not so great for communicating with your audience and wanting to come across as sincere. If anything, a strict formal-voice only blog post would come across, to me, as contrived. To each their own.
      - _carl_jung 7y ago
        I'm not advocating for formal over relaxed. I find blog posts work best in a relaxed style from the first person singular. This sort of apology should come from an individual at the top, and make reference to the whole team. You're probably right about personal differences though.
playpause 7y ago
This is a perfect post-mortem. Their communication and support has always been really good. I've been using Monzo as my primary bank account ever since they registered as a bank, and I've converted a lot of friends to it. But... over the last year, the iOS app has fallen in quality: long UI freezes, frequent sign-outs with no explanation, silly UI bugs. My non-technical friends have noticed the same issues. It's a real shame.
- Nextgrid 7y ago
  Agreed. This caused me to leave them for Starling Bank, though I’m considering switching back - I’d rather take a faulty app but good customer support than a good app but no support at all.
  - deleted 7y ago
    [deleted]
- kingofspain 7y ago
  Recently updated the iOS app and it's definitely got quite laggy, especially on the pots screen (I only have about 5 pots too). Used to be so nippy as well.
  - madeofpalk 7y ago
    I made the adventurous mistake of upgrading my main iPhone to iOS 13 and the Pots screen just refuses to load - tapping the icon freezes the app. As I keep most of my money in pots, I didn't have any money until I got out an old phone and installed Monzo on it.
    - andyrew 7y ago
      This is fixed for me in the latest Monzo update (came out yesterday)
    - _fzslm 7y ago
      If you still have problems after the latest App Store update, their TestFlight build fixed that issue.
yingw787 7y ago
@robinson-wall Nice writeup, definitely raises the standards in the banking industry! I have a few questions: 1. Was this post-mortem part of an official process or something of an individual initiative? I saw it published on the blog, but it might be helpful to have this information disambiguated from marketing material on a separate site: https://status.cloud.google.com/summary https://status.cloud.google.com/summary 2. I'm not sure how payment processors work, but would having multiple payment processors from Monzo's interface make sense from a cost/benefit perspective? 3. Any plans to expand to the U.S. anytime soon, or recommend any banks that follow Monzo's best practices? ;-)
- gr-eg 7y ago
  3. https://monzo.com/blog/2019/06/13/monzo-usa/ https://monzo.com/blog/2019/06/13/monzo-usa/
- adwww 7y ago
  The article links to a status page - https://monzo.statuspage.io/ https://monzo.statuspage.io/
- robinson-wall 7y ago
  1. A mix of both, we have a culture of being transparent by default - it's one of the first things that attracted me to come and work here. I was the incident lead for this on the day, and volunteered to write up this post-mortem. I did have help from colleagues in the marketing team to try and make this as accessible as possible. As another poster mentioned we already have a status page where we post about incidents as they happen (though obviously not in quite as much detail as here). Personally I think our main blog is a reasonable place to have this ️. 2. Multiple redundant payment processors would be great, but ultimately infeasible. As a settling FPS participant we have to have a single Bank of England settlement account, tied 1:1 to a "bank code". Multiple sort codes map to a single bank code, and migrating sort codes between bank codes is non-trivial. It'd be great if we could migrate sort codes easily between redundant connections, but as we build our own Gateway we'll have complete control over how our failover mechanisms work. Here's to much greater uptime in the future! 3. As another commenter mentioned - yes! We're just doing staff testing for now, but we've got a waiting list up. It'll be a prepaid product issued by another bank before we get a US banking license, just like we were in the UK a couple of years ago.
deleted 7y ago
[deleted]
retube 7y ago
What I still don't understand with bank transfers is: what control is there to ensure that debits and credits are offsetting. Doesn't this rely on the bank be being honest? Can't the sending bank just not debit the senders account?
- jessaustin 7y ago
  The sending bank is on the hook for that money when everything gets settled up, so it has a strong incentive to perform that debit.
  - retube 7y ago
    what is this settling up? how does that work?
    - jacobush 7y ago
      Search for bank clearing house
    - lukevp 7y ago
      I can speak to how it works for card processing and ACH is most likely similar. To participate in payment processing in the banking network, you have to have a Merchant ID that is tied to a bank account. The processor or gateway is holding a suspense/escrow account on your behalf throughout the day and when a batch of transactions settles, it will resolve the balance difference with your bank account. The amount of payments allowed into or out of your escrow account is set by the processors based on your company's financial health and a risk analysis since if you just debited say $10 million from the escrow account and you only had $5 in your account, the processor would need to collect that debt from you, and they do not have a guarantee that they'll be able to do so. This is how it works for debit cards and bank accounts since $ amounts are real. It's slightly different for credit cards because the $ amount is in a way fictional, so they don't do the escrow holding and just temporarily "allocate" part of the credit limit (this is called an authorization) and when it is settled this is "captured", which enqueues the authorization for future processing. A few days later it will process and be included in a lump sum of funding into your merchant account. This reply is my personal understanding and meant for educational reasons and doesn't represent opinions or viewpoints of any company, and should not be considered advice of any kind and it may be inaccurate.
osrec 7y ago
Tldr; unsafe memory management in a third party's software corrupted dates (under high load, due to garbage collection), causing transactions to fail or get reversed.
spiderfarmer 7y ago
Somewhat related question: How can Monzo offer 1.55% interest while the interest with most banks is around 0,3%?
- Jonnerz 7y ago
  Likely something to do with the fact Monzo are extremely lean and have minimal overheads compared to traditional banks.
- djhworld 7y ago
  The savings accounts are offered by third party banks, not Monzo. The 1.55% rate is fixed term for 12 months with no withdrawals
- justusthane 7y ago
  Maybe this comment was about UK banks which I can't speak to, but we have banks in the US (Ally as one that I'm familiar with and use, but there are many more) that offer high-interest savings accounts at above 2% interest.
- bradstewart 7y ago
  Could be keeping a good chunk of their deposited cash in money-market funds (or similar) which are currently paying around 2-2.5%, while providing customers with immediate access to funds with their remaining cash (insulating their customers from the delays associated with buying/selling those funds and so forth).
baby 7y ago
I've been using Monzo less and less since I moved to the US due to the cost of topping it up. It's really sad that there is no true equivalent to Monzo here :(
- Qasaur 7y ago
  TransferWise has a debit card and full banking facilities, I assume they are available in the United States?
  - xchaotic 7y ago
    Yes, I used Transfer Wise borderless card in the US in 2017
- zn44 7y ago
  https://monzo.com/blog/2019/06/13/monzo-usa/ https://monzo.com/blog/2019/06/13/monzo-usa/
- afarrell 7y ago
  Monzo just launched in the US. See their announcement at https://monzo.com/blog/2019/06/13/monzo-usa/ https://monzo.com/blog/2019/06/13/monzo-usa/ There is a waitlist to join though.
nvr219 7y ago
You can do anything at monzocom
sandGorgon 7y ago
Just curious - whats the stack you guys run ? I'm wondering what do you use to call these external processing APIs. I assume these are blocking calls.
- robinson-wall 7y ago
  There's a good writeup by Oliver, our head of engineering, about our tech stack on our blog[1] with an accompanying Kubecon talk[2]. TL;DR- Largely Go microservices running on k8s, with http-based RPC calls for synchronous communication, and kafka for asynchronous communication. As for sending and receiving of this kind of payment message, they are largely async but it does depend on the payment system we're talking about. When we build our own FPS gateway we're going to have to have something to manage "sessions" (TCP connections) which will block waiting for a response to an individual payment messages. Right now our communication with our third party Gateway is via a queue. [1]: https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/ https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... [2]: https://www.youtube.com/watch?v=YkOY7DgXKyws https://www.youtube.com/watch?v=YkOY7DgXKyws
  - sandGorgon 7y ago
    actually I kind of like this one - https://softwareengineeringdaily.com/wp-content/uploads/2018/06/SED616-Monzo-Bankbuilding.pdf https://softwareengineeringdaily.com/wp-content/uploads/2018... We have learnt a lot from you guys as we build out similar systems in India. Thank you for putting this stuff out! Quick question that I have always wondered about - would you have used something like Uber Cadence (https://github.com/uber/cadence https://github.com/uber/cadence) as the core of your infrastructure if it had been available back thhen ?
mwexler 7y ago
What I think is fascinating is not just that we applaud Monzo for this, but that we allow other important services that control our lives to get away with revealing nothing about what happened or what they've changed to prevent it. Can you imagine any large bank (for the US, say JP Morgan Chase, Citi, Bank of America, etc.) putting out a note with this level of transparency, accountability, and clear direction to change? For more about what makes a good apology, see https://withoutbullshit.com/?s=apology&submit=Search https://withoutbullshit.com/?s=apology&submit=Search by Josh Bernoff, a former Forrester editor and a very direct writer.
- FigmentEngine 7y ago
  In the UK they do have to: https://www.fca.org.uk/firms/fca-mandated-and-voluntary-information-current-account-services https://www.fca.org.uk/firms/fca-mandated-and-voluntary-info...
- dx7tnt 7y ago
  I can't imagine one of those giant banks having this kind of outage.
  - clankstar 7y ago
    Fast Payments break between EU banks all the time.
  - robinson-wall 7y ago
    It happens all the time, and they just don't tell you. We get automated notices when banks connect and disconnect from the Faster Payments network, something happens every few days. Not always this length, but occasionally. Just yesterday a major high street bank stopped sending payments for an hour, and was telling customers on Twitter that there were no problems. Hell, the central system (what I called the Hub in this article) had a 12 hour split brain meltdown last July which had banks emailing each other spreadsheets back and forth for two weeks afterwards.
    - 7ewis 7y ago
edraferi 7y ago
This is a very well-written postmortem. It’s clear enough that a non-technical customer effected by the outage could understand the explanation, at least at a high level. It’s also detailed enough that a technical person can trace the root cause to a buggy garbage collector in format transformation function. The whole thing uses clear language with a bare minimum of jargon. Nice work!
- aeorgnoieang 7y ago
  > the root cause to a buggy garbage collector Or, rather, unsafe access of memory managed by a garbage collector: > The bug was in a computer program the Gateway uses to translate payment messages between two formats. When the program was operating under load, the system tried to clear memory it believed to be unused (a process known as garbage collection). > > But because it was using an unsafe method to access memory, the code ended up reading memory that had already been cleared away, causing it not to know how to translate the date field in payment messages.
gregdoesit 7y ago
This is a well-written post-mortem for public reading. I encourage people to read through it. Being someone who also works in the payments space currently, relying on gateways, I have gone through several similar outages, where we detected a gateway issue causing an outage, notified the gateway who ack’d... and then we waited. More than one time, like Monzo, we built a workaround on our end, before the gateway provider could even mitigate the outage. Hats off to the Monzo team, who clearly have a solid oncall and incident mitigation strategy in-place. They determined an outage happening in 4 minutes, built a workaround as best they could and deployed it in 2 hours, while it took the gateway provider 9 hours only to mitigate their change that caused the issue the first place. Granted the issue seemed complex, this is still slow. Unfortunately, in cases like this, the best one can do is make sure there is a clear SLA in-place with the third party, with a contract stating financial liability in case the third party fails to meet this SLA. Monzo will not tell us much about this part, but I suspect the gateway will have to pay a hefty fee to Monzo, as their availability dropped to under 99% for this month, which should trigger payments/fee reductions from the third party with a well-written contract. It is good to see they are pushing the third party to do a proper post-mortem and prevention actions, as well as holding them accountable. Nice work!
GordonS 7y ago
The project I'm currently working on has a QA lag of 4-5 days for code to reach production. I'm seriously impressed they were able to deploy mitigations to product twice in the same few hours, especially given they are a bank (and a small one, at that), and the consequences of fucking up are enormous. It's been said here many times already, but I'll join those saying "well done" for handling this so well, and for the extraordinary level of transparency!

284 points by robinson-wall 7y ago

I just posted this semi-technical post-mortem on Monzo's about why we had an outage with Faster Payments (UK bank transfers) last Month. I'll hang around here to answer any more technical questions if anyone's interested.

kennydude 7y ago
Nice to see how Faster Payments actually work in a nice understandable way. That does sound weird how it happened by a corrupted date formatter. I'm assuming it's something like the formatter reset itself back to the langauge default
- robinson-wall 7y ago
  Yeah, instead of `20190530` we were getting `['bert', 'time', 1559, 238096, 0]`. My understanding is a race condition put the code into a situation where it couldn't determine how to format that field, and we just got a default string representation of the underlying value.
  - tuukkah 7y ago
    Which platform gives you unformatted data if a format can't be determined? Or was the code similar to getDate(format) and format happened to be null? Was the race condition in your code or in the platform? (EDIT: I suppose these would be questions more for the developers of the third-party gateway.)
    - Gaelan 7y ago
      FTA: > The bug was in a computer program the [third party] Gateway uses to translate payment messages between two formats. When the program was operating under load, the system tried to clear memory it believed to be unused (a process known as garbage collection). > But because it was using an unsafe method to access memory, the code ended up reading memory that had already been cleared away, causing it not to know how to translate the date field in payment messages.
      - btown 7y ago
        This is exactly the type of thing I would expect from old-school banking systems. Kudos to Monzo for even trying to bring sanity to this space.
    - robinson-wall 7y ago
      I haven't seen the code personally, so I'm not sure. The condition was inside the code of the Gateway provider.
    - mikekchar 7y ago
      IIUC, the problem seems to have been that the code was looking into freed memory and so the date format was essentially random data. I can imagine a case statement where you have a default case of "don't translate the date" with a comment over it saying, "This should never happen". I'm sure I've naively written similar code when I was sleepy and it tends to pass review because it's innocuous. It's easy to be hard on the programmer -- probably crashing is better than data in the wrong format, but then you are just pushing out the problem to a different layer. Error handling is completely non-trivial in complex systems. Maybe they should have thrown an exception in that case, but are you sure it's going to be handled? What is the downside in that case? It could easily be worse -- we have no way of knowing. Sometimes it gets down to, "Well, you need to make sure there are no mistakes in the code". If we're going to go down that route, then the incorrect timing of the memory freeing is the real cause (or if I'm being particularly nasty I might say, "You really shouldn't be using threads" ;-) ). I guess what I'm trying to say is that there is certainly a better way of doing defensive programming in this case, but I wouldn't be able to tell what it was without seeing the code. I also wouldn't expect any large codebase to be completely free of these kinds of problems because it's easy to make a mistake.
  - Dunedan 7y ago
    That reminds me of a race condition I encountered a while ago: There was this massive Java application processing hundreds of parallel requests per second. For each request it wrote a line with billing information into a log file. Those lines were fine from a quick glance, but when we tried processing them later we encountered invalid dates. Those log records contained future dates as well as invalid dates like 2019-02-30. Long story short: In the end we figured out that this was caused by the date formatting not being done thread safe (might have been SimpleDateFormat, but I don't remember the details anymore), causing the date components of multiple threads to get interleaved. Ouch, I guess somebody learned a lesson back then.
hc91 7y ago
You guys are using Form3 for FPS, is this correct?
- robinson-wall 7y ago
  Ah "any more technical questions"... except this one. Sorry, but I can't name our partner.
- wrboyce 7y ago
  We can take an educated guess... https://status.form3.tech/incidents/wyhyxydxgh30 https://status.form3.tech/incidents/wyhyxydxgh30
  - robinson-wall 7y ago
    I would note that this status page says "Our FPS Direct gateway provider".
    - wrboyce 7y ago
      Fair point. FWIW I thought your postmortem was excellent, and certainly puts Form3's to shame regardless of where the blame lies.
- BillinghamJ 7y ago
  I believe it's PayPort, which is run by Vocalink - who also run FPS itself. As far as I understand, PayPort was/is the recommended options for all new "direct" connections. Though it seems that it is also possible to go more-direct into the FPS system itself.
- jayelbe 7y ago
  You might be interested to know that Monzo's API refers to Faster Payments transactions as "payport_faster_payments".
CornishPasty 7y ago
Was the original BERT formatting from the Hub or from something inside the Gateway's network? I always associated BERT with Erlang, do you know if it was involved or if it was something else? Cheers for this blog post, by the way. It was really informative about the issue, and about how FPS works.
- robinson-wall 7y ago
  It's used exclusively inside the Gateway's infrastructure. They're a mix of Java and Erlang but I'm not sure on the proportion. FPS uses ISO8583 for its messaging format, and I suspect at the edge the Gateway translates it to a BERT blob for passing around internally.
  - CornishPasty 7y ago
    Ah right that makes sense. It seems odd that it took them so long to identify and rectify the issue. Have they made you any assurances about what they'll do to reduce this time/prevent this from happening in future?
    - robinson-wall 7y ago
      Specifically this class of incident? Yes. They seem to be reasonably good at operational runbooks for known flaws, so I can imagine this class of incident being handed faster in future. What worries me is the next unknown unknown, which is why we are insourcing. One thing I think Monzo can be particularly proud of is our incident response, and debugging of our own systems at speed. That's why this article focuses more on what actions we are taking, and not what actions they are taking.

ablation 7y ago

Thank you for posting this. Great read, and nice to see the team at Monzo sharing this level of detail: consumable but still detailed.

peteretep 7y ago

> They later tell us they believed that datacentre was introducing the corruption What now? Their datacentre was ... rewriting (presumably) encrypted packets?

robinson-wall 7y ago
Sorry, perhaps this isn't very clear as I've tried to simplify the explanation to make it accessible to a wide audience. What I meant here is they could tell that the corruption was being introduced by some component in their infrastructure, and they were only observing it for messages passing through one of their two active-active sites.
- noir_lord 7y ago
  I understood that as you intended. It's a fine line between understandable to laymen and people been pernickity sadly.

PhantomGremlin 7y ago

The software bug at the heart of the problem: The bug was in a computer program the Gateway uses to translate payment messages between two formats. When the program was operating under load, the system tried to clear memory it believed to be unused (a process known as garbage collection). But because it was using an unsafe method to access memory, the code ended up reading memory that had already been cleared away, causing it not to know how to translate the date field in payment messages. So apparently a dangling reference.

seanmcdirmid 7y ago
Is that really proper use of the term garbage collection? If you are doing memory management manually, it sounds more like the lack of garbage collection. Unless they were using an unsafe GC for C/C++?
- jey 7y ago
  Sounds like they were hanging onto a pointer to an object allocated by GC. For example, in Python/C API if you use a borrowed reference PyObject* after it has gone out of scope and been GC'd.
rahilb 7y ago
I'm pretty sure they're a Java shop, and they're referring to sun.misc.unsafe and the goodies inside that let you manually allocate memory.

kjlfhg8 7y ago

Not related to the outage, but any plans to provide banking on pc's instead of just phones and any plans to provide small businesses accounts in the future?

lol768 7y ago
Yes, they've had job positions open for more web work. I think they will expand this. They already offer business accounts. I have one open for my Ltd company.
- ownagefool 7y ago
  I registered my interest a while ago but they haven't given me one yet, so I wouldn't say they offer business accounts, so much as they're going to/are testing this. Starling does offer business accounts now, but you can only have one Person of Significant Control, i.e. over 25% owner. There is no monthly fee with their offering though, so it's probably the better offer.
  - lol768 7y ago
    Note that Starling will prevent you opening a business account if you've held a personal account in the past and closed it. That was the position I found myself in. For what it's worth, I've seen some suggestions regarding business account pricing on the Monzo Slack, and there are plans for separate tiers (including a free tier which lacks some of the more advanced accounting integrations).
    - ownagefool 7y ago
      I use a monzo personal account, would be more than happy using them for my ltd if and when they support me doing so.
deleted 7y ago
[deleted]

ziddoap 7y ago

Clear, detailed but accessible, plans in place moving forward, apology read sincerely, providing support to affected customers immediately, and answering follow up questions to technical users who are interested in more detail. A+ job on handling the unfortunate situation, Monzo. We can only hope more companies follow this great example.

deleted 7y ago
[deleted]
ccrush 7y ago
Every time I see a company say "we're sorry" I can't help but think about the South Park episode where the BP CEO says "we're sorry!" It's either that or the one with the Time Warner employees with the nursing flaps in their shirts.
- _carl_jung 7y ago
  Using "we" in any public announcement, writeup, blog post, etc. is always in danger of sounding contrived.
  - ziddoap 7y ago
    Personally, I prefer something written in a relaxed style rather than a formal-voice only in most cases, especially for blog posts. Formal only generally comes across, to me, as cold and distant. Great for a persuasive essay or other mediums where you want to remove the topic from the author, not so great for communicating with your audience and wanting to come across as sincere. If anything, a strict formal-voice only blog post would come across, to me, as contrived. To each their own.
    - _carl_jung 7y ago
      I'm not advocating for formal over relaxed. I find blog posts work best in a relaxed style from the first person singular. This sort of apology should come from an individual at the top, and make reference to the whole team. You're probably right about personal differences though.

playpause 7y ago

This is a perfect post-mortem. Their communication and support has always been really good. I've been using Monzo as my primary bank account ever since they registered as a bank, and I've converted a lot of friends to it. But... over the last year, the iOS app has fallen in quality: long UI freezes, frequent sign-outs with no explanation, silly UI bugs. My non-technical friends have noticed the same issues. It's a real shame.

Nextgrid 7y ago
Agreed. This caused me to leave them for Starling Bank, though I’m considering switching back - I’d rather take a faulty app but good customer support than a good app but no support at all.
- deleted 7y ago
  [deleted]
kingofspain 7y ago
Recently updated the iOS app and it's definitely got quite laggy, especially on the pots screen (I only have about 5 pots too). Used to be so nippy as well.
- madeofpalk 7y ago
  I made the adventurous mistake of upgrading my main iPhone to iOS 13 and the Pots screen just refuses to load - tapping the icon freezes the app. As I keep most of my money in pots, I didn't have any money until I got out an old phone and installed Monzo on it.
  - andyrew 7y ago
    This is fixed for me in the latest Monzo update (came out yesterday)
  - _fzslm 7y ago
    If you still have problems after the latest App Store update, their TestFlight build fixed that issue.

yingw787 7y ago

@robinson-wall Nice writeup, definitely raises the standards in the banking industry! I have a few questions: 1. Was this post-mortem part of an official process or something of an individual initiative? I saw it published on the blog, but it might be helpful to have this information disambiguated from marketing material on a separate site: https://status.cloud.google.com/summary https://status.cloud.google.com/summary 2. I'm not sure how payment processors work, but would having multiple payment processors from Monzo's interface make sense from a cost/benefit perspective? 3. Any plans to expand to the U.S. anytime soon, or recommend any banks that follow Monzo's best practices? ;-)

gr-eg 7y ago
3. https://monzo.com/blog/2019/06/13/monzo-usa/ https://monzo.com/blog/2019/06/13/monzo-usa/
adwww 7y ago
The article links to a status page - https://monzo.statuspage.io/ https://monzo.statuspage.io/
robinson-wall 7y ago
1. A mix of both, we have a culture of being transparent by default - it's one of the first things that attracted me to come and work here. I was the incident lead for this on the day, and volunteered to write up this post-mortem. I did have help from colleagues in the marketing team to try and make this as accessible as possible. As another poster mentioned we already have a status page where we post about incidents as they happen (though obviously not in quite as much detail as here). Personally I think our main blog is a reasonable place to have this ️. 2. Multiple redundant payment processors would be great, but ultimately infeasible. As a settling FPS participant we have to have a single Bank of England settlement account, tied 1:1 to a "bank code". Multiple sort codes map to a single bank code, and migrating sort codes between bank codes is non-trivial. It'd be great if we could migrate sort codes easily between redundant connections, but as we build our own Gateway we'll have complete control over how our failover mechanisms work. Here's to much greater uptime in the future! 3. As another commenter mentioned - yes! We're just doing staff testing for now, but we've got a waiting list up. It'll be a prepaid product issued by another bank before we get a US banking license, just like we were in the UK a couple of years ago.

deleted 7y ago

[deleted]

retube 7y ago

What I still don't understand with bank transfers is: what control is there to ensure that debits and credits are offsetting. Doesn't this rely on the bank be being honest? Can't the sending bank just not debit the senders account?

jessaustin 7y ago
The sending bank is on the hook for that money when everything gets settled up, so it has a strong incentive to perform that debit.
- retube 7y ago
  what is this settling up? how does that work?
  - jacobush 7y ago
    Search for bank clearing house
  - lukevp 7y ago
    I can speak to how it works for card processing and ACH is most likely similar. To participate in payment processing in the banking network, you have to have a Merchant ID that is tied to a bank account. The processor or gateway is holding a suspense/escrow account on your behalf throughout the day and when a batch of transactions settles, it will resolve the balance difference with your bank account. The amount of payments allowed into or out of your escrow account is set by the processors based on your company's financial health and a risk analysis since if you just debited say $10 million from the escrow account and you only had $5 in your account, the processor would need to collect that debt from you, and they do not have a guarantee that they'll be able to do so. This is how it works for debit cards and bank accounts since $ amounts are real. It's slightly different for credit cards because the $ amount is in a way fictional, so they don't do the escrow holding and just temporarily "allocate" part of the credit limit (this is called an authorization) and when it is settled this is "captured", which enqueues the authorization for future processing. A few days later it will process and be included in a lump sum of funding into your merchant account. This reply is my personal understanding and meant for educational reasons and doesn't represent opinions or viewpoints of any company, and should not be considered advice of any kind and it may be inaccurate.

osrec 7y ago

Tldr; unsafe memory management in a third party's software corrupted dates (under high load, due to garbage collection), causing transactions to fail or get reversed.

spiderfarmer 7y ago

Somewhat related question: How can Monzo offer 1.55% interest while the interest with most banks is around 0,3%?

Jonnerz 7y ago
Likely something to do with the fact Monzo are extremely lean and have minimal overheads compared to traditional banks.
djhworld 7y ago
The savings accounts are offered by third party banks, not Monzo. The 1.55% rate is fixed term for 12 months with no withdrawals
justusthane 7y ago
Maybe this comment was about UK banks which I can't speak to, but we have banks in the US (Ally as one that I'm familiar with and use, but there are many more) that offer high-interest savings accounts at above 2% interest.
bradstewart 7y ago
Could be keeping a good chunk of their deposited cash in money-market funds (or similar) which are currently paying around 2-2.5%, while providing customers with immediate access to funds with their remaining cash (insulating their customers from the delays associated with buying/selling those funds and so forth).

baby 7y ago

I've been using Monzo less and less since I moved to the US due to the cost of topping it up. It's really sad that there is no true equivalent to Monzo here :(

Qasaur 7y ago
TransferWise has a debit card and full banking facilities, I assume they are available in the United States?
- xchaotic 7y ago
  Yes, I used Transfer Wise borderless card in the US in 2017
zn44 7y ago
https://monzo.com/blog/2019/06/13/monzo-usa/ https://monzo.com/blog/2019/06/13/monzo-usa/
afarrell 7y ago
Monzo just launched in the US. See their announcement at https://monzo.com/blog/2019/06/13/monzo-usa/ https://monzo.com/blog/2019/06/13/monzo-usa/ There is a waitlist to join though.

nvr219 7y ago

You can do anything at monzocom

sandGorgon 7y ago

Just curious - whats the stack you guys run ? I'm wondering what do you use to call these external processing APIs. I assume these are blocking calls.

robinson-wall 7y ago
There's a good writeup by Oliver, our head of engineering, about our tech stack on our blog[1] with an accompanying Kubecon talk[2]. TL;DR- Largely Go microservices running on k8s, with http-based RPC calls for synchronous communication, and kafka for asynchronous communication. As for sending and receiving of this kind of payment message, they are largely async but it does depend on the payment system we're talking about. When we build our own FPS gateway we're going to have to have something to manage "sessions" (TCP connections) which will block waiting for a response to an individual payment messages. Right now our communication with our third party Gateway is via a queue. [1]: https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/ https://monzo.com/blog/2016/09/19/building-a-modern-bank-bac... [2]: https://www.youtube.com/watch?v=YkOY7DgXKyws https://www.youtube.com/watch?v=YkOY7DgXKyws
- sandGorgon 7y ago
  actually I kind of like this one - https://softwareengineeringdaily.com/wp-content/uploads/2018/06/SED616-Monzo-Bankbuilding.pdf https://softwareengineeringdaily.com/wp-content/uploads/2018... We have learnt a lot from you guys as we build out similar systems in India. Thank you for putting this stuff out! Quick question that I have always wondered about - would you have used something like Uber Cadence (https://github.com/uber/cadence https://github.com/uber/cadence) as the core of your infrastructure if it had been available back thhen ?

mwexler 7y ago

What I think is fascinating is not just that we applaud Monzo for this, but that we allow other important services that control our lives to get away with revealing nothing about what happened or what they've changed to prevent it. Can you imagine any large bank (for the US, say JP Morgan Chase, Citi, Bank of America, etc.) putting out a note with this level of transparency, accountability, and clear direction to change? For more about what makes a good apology, see https://withoutbullshit.com/?s=apology&submit=Search https://withoutbullshit.com/?s=apology&submit=Search by Josh Bernoff, a former Forrester editor and a very direct writer.

FigmentEngine 7y ago
In the UK they do have to: https://www.fca.org.uk/firms/fca-mandated-and-voluntary-information-current-account-services https://www.fca.org.uk/firms/fca-mandated-and-voluntary-info...
dx7tnt 7y ago
I can't imagine one of those giant banks having this kind of outage.
- clankstar 7y ago
  Fast Payments break between EU banks all the time.
- robinson-wall 7y ago
  It happens all the time, and they just don't tell you. We get automated notices when banks connect and disconnect from the Faster Payments network, something happens every few days. Not always this length, but occasionally. Just yesterday a major high street bank stopped sending payments for an hour, and was telling customers on Twitter that there were no problems. Hell, the central system (what I called the Hub in this article) had a 12 hour split brain meltdown last July which had banks emailing each other spreadsheets back and forth for two weeks afterwards.
  - 7ewis 7y ago

edraferi 7y ago

This is a very well-written postmortem. It’s clear enough that a non-technical customer effected by the outage could understand the explanation, at least at a high level. It’s also detailed enough that a technical person can trace the root cause to a buggy garbage collector in format transformation function. The whole thing uses clear language with a bare minimum of jargon. Nice work!

aeorgnoieang 7y ago
> the root cause to a buggy garbage collector Or, rather, unsafe access of memory managed by a garbage collector: > The bug was in a computer program the Gateway uses to translate payment messages between two formats. When the program was operating under load, the system tried to clear memory it believed to be unused (a process known as garbage collection). > > But because it was using an unsafe method to access memory, the code ended up reading memory that had already been cleared away, causing it not to know how to translate the date field in payment messages.

gregdoesit 7y ago

This is a well-written post-mortem for public reading. I encourage people to read through it. Being someone who also works in the payments space currently, relying on gateways, I have gone through several similar outages, where we detected a gateway issue causing an outage, notified the gateway who ack’d... and then we waited. More than one time, like Monzo, we built a workaround on our end, before the gateway provider could even mitigate the outage. Hats off to the Monzo team, who clearly have a solid oncall and incident mitigation strategy in-place. They determined an outage happening in 4 minutes, built a workaround as best they could and deployed it in 2 hours, while it took the gateway provider 9 hours only to mitigate their change that caused the issue the first place. Granted the issue seemed complex, this is still slow. Unfortunately, in cases like this, the best one can do is make sure there is a clear SLA in-place with the third party, with a contract stating financial liability in case the third party fails to meet this SLA. Monzo will not tell us much about this part, but I suspect the gateway will have to pay a hefty fee to Monzo, as their availability dropped to under 99% for this month, which should trigger payments/fee reductions from the third party with a well-written contract. It is good to see they are pushing the third party to do a proper post-mortem and prevention actions, as well as holding them accountable. Nice work!

GordonS 7y ago

The project I'm currently working on has a QA lag of 4-5 days for code to reach production. I'm seriously impressed they were able to deploy mitigations to product twice in the same few hours, especially given they are a bank (and a small one, at that), and the consequences of fucking up are enormous. It's been said here many times already, but I'll join those saying "well done" for handling this so well, and for the extraordinary level of transparency!