7 ms·

DBS Bank outage well into its second day with no meaningful update from DBD

83 points by audiometry 5y ago

lordnacho 5y ago
It's well worth your time to have a second bank account with a small overdraft, just in case this sort of thing happens. It also helps in case something happens to the bank card you're given. Even on an institutional level, I've connected my hedge funds to secondary prime brokers just because there might be some outage, either IT or business level. Sometimes there's a PB team at some bank that's keen to get business this way.
qsi 5y ago
For context: DBS is Singapore's biggest bank, and this outage affects most (all?) electronic banking services made through the app or website, including the popular PayNow transfers (usually small transactions made by scanning a QR code, or sending it to the recipient by phone number). I am also not really surprised as my experience of DBS IT as a customer is mediocre at best. Trying to apply for a credit card online was a comedy of poor UX. Errors preventing the submission, but not marked, files to upload could have a max of 20 characters in the file name, trying to re-upload failed until you deleted the failed previous upload (not explained), and best of all, their "We value your feedback" webform does not accept "special characters" such as apostrophe. "Your website doesn't work" cannot be submitted because of "doesn't." Really. This was a few months ago. Oh, did I mention the DBS app takes forever to load? There's other stuff I am trying to forget. As they're the biggest bank here they can offer some attractive packages, which is why I have put up with it...
- dustintrex 5y ago
  Their customer service UX is also hilariously bad. You can send a message to the bank from the app, which still makes sense, but to read the response, you: - need to wait 3-5 business days - eventually get an email notifying you that there is a response and directing you to the chat to read it (!?) - to access chat, you need to complete a separate authentication process that requires SMS (can't use the digital token used everywhere else) - and then you finally get a chat bubble containing the precious response. - The kicker? If you want to respond, you need to do all of this again and get routed to another random agent with no access to your previous messages, since there is no concept of threading at all!
  - harha 5y ago
    Sounds like most customer interactions in Singapore that don't perfectly fit into the standard workflow to me.
  - pydry 5y ago
    I had to do that to unsubscribe from their marketing email lists. They initially asked me to provide my bank account number to do it, too. Eventually a person intervened and took me off their mailing list manually. IDK why Singapore didnt make having an unsubscribe link mandatory yet.
    - dustintrex 5y ago
      It does: http://www.imda.gov.sg/for-community/Infocomm-regulation-and-guides/unsolicited-communications/best-practices-for-organisations http://www.imda.gov.sg/for-community/Infocomm-regulation-and... Unfortunately the Act is aimed at out and out spam, not marketing from companies where you actually are a customer.
      - pydry 5y ago
        Ought to have applied to me then since I was an ex-customer.
- tsechin 5y ago
  The DBS UX is absolute garbage. Their android app is extremely slow, and is full of (probably?) iOS patterns. It's almost like they decided to implement their own cross-platform framework, but poorly. (I apologize in advance if they are in fact using someone else's cross-platform framework, but poorly.) Oh, and their web internet banking doesn't allow pasting passwords, because that's how we do security in 2021. I've written software for them in the past. I... wouldn't say they have high standards. At least when I was working on systems like this (many years ago), a worrying amount of code was written for them by overworked contractors like myself.
dustintrex 5y ago
Context: DBS https://www.dbs.com.sg/ https://www.dbs.com.sg/ is the largest bank in Southeast Asia and very, very proud of being "The World's Best Digital Bank": https://knowledge.insead.edu/blog/insead-blog/how-dbs-became-the-worlds-best-bank-17671 https://knowledge.insead.edu/blog/insead-blog/how-dbs-became... Note their own "GANDALF" acronym, composed of Google, Amazon, Netflix, Apple, LinkedIn, Facebook and, you guessed it, DBS. Most companies in Singapore also pay out wages at the end of the month, so it's an extra bad time to be down. On the other hand, their slogan is "Live More, Bank Less", so at least they're living up to the second half.
- rich_sasha 5y ago
  If they put themselves in the same league as LinkedIn, that explains a lot. Shittiest website I ever need to use from a technical point of view, never mind “content” and dodgy hard-sell techniques.
  - virgilp 5y ago
    They had to get the acronym to work, and presumably "LinkedIn's-owner" sounded a bit forced.
  - blntechie 5y ago
    It might sound repetitive that recently only I wrote a similar comment. Sometimes it makes me wonder why are people in FB or LinkedIn put on a pedestal when the output (we can see) is bad quality websites and apps. Maybe I’m trivialising the problem considering the scale involved but still makes one wonder.
    - rich_sasha 5y ago
      Purely from a technology point of view, I find Facebook and Twitter far superior. They "just work" (not that I'm a heavy user, and certainly not endorsing the business they serve). LI is slow, buggy, crashy, search works poorly etc. The only thing they have going for them is the size of the network.
latch 5y ago
Adding my $0.02. I've had a non-trivial # of bank accounts in numerous countries, and generally speaking, DBS is good. I think people constantly complaining about DBS have no clue how awful retail banking usually is (especially internet-banking), and that most definitely includes other Singapore banks. Apps are generally modern, usable and responsive, and I find their fees and transparency reasonable. But what sets the bank apart is their customer service, the people on the other side of the phone (or counter, those few times) are incredibly friendly and helpful (again, compared to, say HSBC). (I didn't need to interact with any other bank during COVID, so I can't compare, but I was particularly pleased with how well DBS let me do even complex banking remotely, and I hope they adopt it as a permanent option). Things do seem to be getting _slightly_ worse though. One issue I have with the bank is that they use push notifications to advertise products and also use push notifications for security purposes. It makes me wish apple/google would require push notifications to be categorized. Also, over the last year, push notification spam seems to be linked directly to my purchases, e.g. "Got a new gadget, consider DBS payment plans..." shortly after buying new phones. It's like they've confused digitization with social network.
- qsi 5y ago
  As an addendum to my previous reply on the DBS's abysmal UX, I will echo the point made here that once you get to talk to someone, it's generally fine. I've had some great service when I could get through to a human, but they're making it increasingly difficult to do so as they're going all digital. Many branches have closed, and even the higher-end Treasures locations are now down to just two I think (a major mistake IMO especially as you target the wealthier). Navigating their phone support. takes. a. lot. of. patience. I also agree that other banks can be a lot worse, and HSBC in particularly has been abysmal. (edited for grammar)
  - ValentineC 5y ago
    > I also agree that other banks can be a lot worse, and HSBC in particularly has been abysmal. At least HSBC got something right with their app UX: they give me an option to log in using my PIN instead of automatically trying Face ID multiple times, in this era when masks are still mandatory.
jcims 5y ago
I’m not super familiar with the full content of their regulations but in my experience working with banks that maintain a presence there, Singapore control requirements tend to be fairly specific, opinionated and far reaching. For example, in developing OS security standards, I found myself having to actually incorporate requirements specifically for the Monetary Authority of Singapore that bled into operations in North America. These requirements extend into resiliency as well. I’m curious how well DBS has been fairing in audits.
- qsi 5y ago
  A few months from now the headline will be that MAS (Monetary Authority of Singapore) slaps DBS with a hefty fine.
- cameronh90 5y ago
  MAS requirements are incredibly specific. However, I wonder if that opinionatedness could sometimes make things worse. We had to do a lot of work to convince MAS that AWS was safe enough for our tiny operation out there, for example. Whereas, if we'd gone for a local colo, it would have been much easier - despite having worse resilience and likely worse security.
  - jcims 5y ago
    In my case it 100% made things worse because there is was no room for contextual interpretation of the rules. The MAS requirements went too far in one direction and prevented us from developing more granular and observable control strategies. We essentially wound up stuck with security paradigms from the 90's. I was able to compartmentalize most of it to in-country workloads but some of it ultimately leaked back into our global builds because it was going to be too costly to maintain a total snowflake.
sohkamyung 5y ago
Original headline says, "Singapore bank DBS says services disrupted for second day", so I'm not sure where the "with no meaningful update from DBD" in the HN headline came from, nor am I sure of the meaning of DBD.
- qsi 5y ago
  More details, but not much more news yet: https://www.channelnewsasia.com/singapore/dbs-posb-down-not-available-digital-banking-2334661 https://www.channelnewsasia.com/singapore/dbs-posb-down-not-... From that article: >DBS Singapore country head Shee Tse Koon said the bank identified a problem with its access control servers on Tuesday and has been working "round the clock" with third-party engineering providers to fix the issue.
  - Animats 5y ago
    A little more detail from the bank's own web site. Apparently they got the access control servers up again, but then they went down a few hours later, at least partially. Which, after 24 hours of downtime, probably means that most of the people who got the systems back up were trying to get some sleep. [1] https://www.dbs.com/newsroom/DBS_Singapore_Country_head_provides_update_on_disruption_in_digital_banking_services https://www.dbs.com/newsroom/DBS_Singapore_Country_head_prov...
Havoc 5y ago
Ransomware?
TheSpiceIsLife 5y ago
No discussion related to internet banking is complete without someone mentioning Commonwealth Bank here in Australia. Commonwealth Bank CIO says any code older than three months is like “food in the fridge”, past its ‘best before’ date. https://www.itnews.com.au/news/cba-to-treat-its-software-like-food-in-the-fridge-570930 https://www.itnews.com.au/news/cba-to-treat-its-software-lik... Which explains why their online banking services have become very brittle the past couple years.
- asah 5y ago
  Three months?! I'm guessing he's never written any.
  - TheSpiceIsLife 5y ago
    I’d hazard a guess you’re probably right. Still worrying that someone who’s prepared to spout that opinion made it that far up the organisation.
- ValentineC 5y ago
  I don't know. As a Singapore resident with way too many bank accounts, it seems like the most unstable experiences for me tend to be the banks that try to reinvent features, like the aforementioned DBS. I'd speculate that shipping new features, not stability, is somewhat linked to career progression. It could also be because most Singapore banking apps end up using the same mobile security framework to pass their audits [1] — except that same security framework prefers security over user experience. [1] https://en.wikipedia.org/wiki/V-Key https://en.wikipedia.org/wiki/V-Key
- dustintrex 5y ago
  The interview is a bit ambiguous, but to me it looks like he's referring to prod binaries over 3 months old being stale, which is considerably more reasonable.
- cgio 5y ago
  At a high level, CBA’s technology strategy is “quite simple to think about but quite difficult to execute” That’s the problem right there, it sounds like it is too simplistic to support execution.
latch 5y ago
Small update from their country head: https://www.youtube.com/watch?v=EuXtrtH0uKY https://www.youtube.com/watch?v=EuXtrtH0uKY 1 - problem identified with their access control services (or did he say servers?). (doesn't sound good) 2 - working with their 3rd party engineering providers (how much is outsourced?)
- tsechin 5y ago
  Story time! Years ago, I was a contractor writing (a small piece of) software for DBS. It was probably a couple of hundred lines of Java, and I probably delivered it by emailing a zip file to someone. I don't recall their code review standards being particularly ... existent. Mine was a relatively small project with them. There were other teams at my firm working on larger projects at DBS. Another story! A new employee joined my firm, and - on his first day - was sent to join one of these teams onsite at DBS. His day ended at 9pm /on his first day of work/. He gave notice on day two.
  - _tik_ 5y ago
    My experience as an outsource partner with them 9 years ago. DBS only employed Project Manager and outsourced all their technical stuff. Yes the working hours is very long.
dav43 5y ago
The DBS - worlds best digital bank - is an absolute joke. Their local app is terrible in Singapore and you can't even download transaction history more than 6 months!
deleted 5y ago
[deleted]
crossroadsguy 5y ago
I see many people dissing it and it might be valid for their location. But DBS is in India as well. Compared to local banks, among which most are shitty in a way that it feels they present “remaining shitty” as a non-negotiable goal to their tech team, and some foreign banks, some of those are shitty as well, DBS India’s app and Internet banking is really polished and functional. They bought a local bank recently so they have got a major physical presence push. I used to have bad experiences with their iOS app but since last 7-8 months it’s been really been good. I had complained around a year ago. 7-8 months back, an individual reached out on email thanking me for my feedback and saying they’ve done a major overhaul and they’d be glad to hear my feedback on it. This shit doesn’t happen with banks here. Be it private or govt banks, you’re lucky if you’re not treated like shit. They don’t spam (you’d not at all understand what it actually means if you don’t live in India!!!). One short email to my RM and even those rare marketing emails dropped to zero (one of them I had actually benefited from; another banking rarity here). PS: In last ~decade and a half I’ve tried too many banks and I keep ~10 bank accounts active at a time (due to insurance from Govt limit when banks fail; I know, I know)
- korginator 5y ago
  Glad to hear your positive experiences. Could I share your feedback with a few colleagues in DBS here? As you say, it's about context. Here we keep pushing the message that Singapore is an ideal place to do business, that the regulations are business-friendly, and the financial systems are efficient, well run and reliable, and this is largely true. A failure of this nature and size may be a blip in the Japanese or European markets, but it's also about perception. That takes a hit. It will affect the investment appetite, and our economy depends a lot on international business and money moving through our financial institutions. That's the first impact. More practically, and looking locally, about three-quarters of our GDP is in the services sector [1]. Pretty much everyone is banked, and payments are increasingly cashless. A lot of these payments (including salaries, payment for services like air-con repairs, etc.) go through our local equivalent of your UPI called PayNow, and DBS' PayNow systems were failing. I remember carrying cash in my wallet just a handful of times over the past couple of years, and I pay for pretty much everything electronically. So... these systems which failed are far more central to a well functioning economy here than in some other countries. That's the second impact. MAS (our regulator) is rightly concerned about the long-term implications. You can be sure they will have a thorough root-cause analysis. Some heads may roll. DBS will definitely be slapped with a multi-million dollar fine and be told to present a comprehensive plan to ensure this doesn't happen again. MAS will appoint people to oversee & manage how this is done, and knowing MAS they may start other audits to identify potential problems in other systems. They'll definitely be watching DBS closely, and they will probably look at other banks too. References: [1] https://www.singstat.gov.sg/modules/infographics/economy https://www.singstat.gov.sg/modules/infographics/economy
nreilly 5y ago
https://twitter.com/dbsbank/status/1463409021411594245 https://twitter.com/dbsbank/status/1463409021411594245
pydry 5y ago
This sounds very similar to the TSB meltdown playbook. 1) Have upper level management who fundamentally do not understand technology. 2) Outsource systems development to cheapest possible bodyshop and give them strict deadlines. 3) Explosion of technical debt that is entirely invisible to middle management and above. 4) IT meltdown as a result. CEO (hopefully) resigns in disgrace.
korginator 5y ago
The regulator MAS is already starting to make a noise about "appropriate actions" [1] once the dust has settled. It's also a matter of national pride, DBS is very iconic and there's been a lot of publicity about the bank as a regional success story, specially as they were publicising their "successful" transition to become a tech company going head to head with global giants, while also offering banking services [2]. I hope they don't forget their roots and end up treating banking as an afterthought and failures of this magnitude as par for the course. A sizeable portion of the population still goes into the bank to transact physically. There's the historical angle and another point of national pride [3]. Since the post office savings bank (now part of DBS) was established by the British in 1877 it catered to low income groups, and provided a very high quality of service while keeping operating costs low. They were also early regional adopters of technology, starting to computerise their accounts in 1972. For a long time POSB was run as a statutory board with directors that were directly appointed by a government ministry. All this is to say that people have a slightly different relationship with POSB/DBS compared to other banks. There's a lot of trust that's been carefully cultivated over decades. People believe and trust in this bank, they trust that the bank has their best interests at heart, and this has been generally true. Failures of this magnitude will start to break this trust. That in itself is not a big concern, there are other banks and we can jump ship. There's a social angle with POSB/DBS, specially with the older generations. When POSB was handed to DBS [4] on a silver platter in 1998 a lot of people voiced concerns that the bank would become yet another money-grubbing profit-seeking megacorp, and about how this would affect their earlier objective of serving the weaker strata of society. It's hard to predict what may happen if this trust starts to break. References: [1] https://www.channelnewsasia.com/singapore/dbs-bank-service-disruption-mas-appropriate-supervisory-actions-2335931 https://www.channelnewsasia.com/singapore/dbs-bank-service-d... [2] https://innovationinbanking.efma.com/content/articles/transforming-dbs-bank-tech-company https://innovationinbanking.efma.com/content/articles/transf... [3] https://eresources.nlb.gov.sg/infopedia/articles/SIP_2014-01-07_112742.html https://eresources.nlb.gov.sg/infopedia/articles/SIP_2014-01... [4] https://www.dbs.com/about-us/who-we-are/our-heritage https://www.dbs.com/about-us/who-we-are/our-heritage

83 points by audiometry 5y ago

It's well worth your time to have a second bank account with a small overdraft, just in case this sort of thing happens. It also helps in case something happens to the bank card you're given. Even on an institutional level, I've connected my hedge funds to secondary prime brokers just because there might be some outage, either IT or business level. Sometimes there's a PB team at some bank that's keen to get business this way.

qsi 5y ago

For context: DBS is Singapore's biggest bank, and this outage affects most (all?) electronic banking services made through the app or website, including the popular PayNow transfers (usually small transactions made by scanning a QR code, or sending it to the recipient by phone number). I am also not really surprised as my experience of DBS IT as a customer is mediocre at best. Trying to apply for a credit card online was a comedy of poor UX. Errors preventing the submission, but not marked, files to upload could have a max of 20 characters in the file name, trying to re-upload failed until you deleted the failed previous upload (not explained), and best of all, their "We value your feedback" webform does not accept "special characters" such as apostrophe. "Your website doesn't work" cannot be submitted because of "doesn't." Really. This was a few months ago. Oh, did I mention the DBS app takes forever to load? There's other stuff I am trying to forget. As they're the biggest bank here they can offer some attractive packages, which is why I have put up with it...

dustintrex 5y ago
Their customer service UX is also hilariously bad. You can send a message to the bank from the app, which still makes sense, but to read the response, you: - need to wait 3-5 business days - eventually get an email notifying you that there is a response and directing you to the chat to read it (!?) - to access chat, you need to complete a separate authentication process that requires SMS (can't use the digital token used everywhere else) - and then you finally get a chat bubble containing the precious response. - The kicker? If you want to respond, you need to do all of this again and get routed to another random agent with no access to your previous messages, since there is no concept of threading at all!
- harha 5y ago
  Sounds like most customer interactions in Singapore that don't perfectly fit into the standard workflow to me.
- pydry 5y ago
  I had to do that to unsubscribe from their marketing email lists. They initially asked me to provide my bank account number to do it, too. Eventually a person intervened and took me off their mailing list manually. IDK why Singapore didnt make having an unsubscribe link mandatory yet.
  - dustintrex 5y ago
    It does: http://www.imda.gov.sg/for-community/Infocomm-regulation-and-guides/unsolicited-communications/best-practices-for-organisations http://www.imda.gov.sg/for-community/Infocomm-regulation-and... Unfortunately the Act is aimed at out and out spam, not marketing from companies where you actually are a customer.
    - pydry 5y ago
      Ought to have applied to me then since I was an ex-customer.
tsechin 5y ago
The DBS UX is absolute garbage. Their android app is extremely slow, and is full of (probably?) iOS patterns. It's almost like they decided to implement their own cross-platform framework, but poorly. (I apologize in advance if they are in fact using someone else's cross-platform framework, but poorly.) Oh, and their web internet banking doesn't allow pasting passwords, because that's how we do security in 2021. I've written software for them in the past. I... wouldn't say they have high standards. At least when I was working on systems like this (many years ago), a worrying amount of code was written for them by overworked contractors like myself.

dustintrex 5y ago

Context: DBS https://www.dbs.com.sg/ https://www.dbs.com.sg/ is the largest bank in Southeast Asia and very, very proud of being "The World's Best Digital Bank": https://knowledge.insead.edu/blog/insead-blog/how-dbs-became-the-worlds-best-bank-17671 https://knowledge.insead.edu/blog/insead-blog/how-dbs-became... Note their own "GANDALF" acronym, composed of Google, Amazon, Netflix, Apple, LinkedIn, Facebook and, you guessed it, DBS. Most companies in Singapore also pay out wages at the end of the month, so it's an extra bad time to be down. On the other hand, their slogan is "Live More, Bank Less", so at least they're living up to the second half.

rich_sasha 5y ago
If they put themselves in the same league as LinkedIn, that explains a lot. Shittiest website I ever need to use from a technical point of view, never mind “content” and dodgy hard-sell techniques.
- virgilp 5y ago
  They had to get the acronym to work, and presumably "LinkedIn's-owner" sounded a bit forced.
- blntechie 5y ago
  It might sound repetitive that recently only I wrote a similar comment. Sometimes it makes me wonder why are people in FB or LinkedIn put on a pedestal when the output (we can see) is bad quality websites and apps. Maybe I’m trivialising the problem considering the scale involved but still makes one wonder.
  - rich_sasha 5y ago
    Purely from a technology point of view, I find Facebook and Twitter far superior. They "just work" (not that I'm a heavy user, and certainly not endorsing the business they serve). LI is slow, buggy, crashy, search works poorly etc. The only thing they have going for them is the size of the network.

latch 5y ago

Adding my $0.02. I've had a non-trivial # of bank accounts in numerous countries, and generally speaking, DBS is good. I think people constantly complaining about DBS have no clue how awful retail banking usually is (especially internet-banking), and that most definitely includes other Singapore banks. Apps are generally modern, usable and responsive, and I find their fees and transparency reasonable. But what sets the bank apart is their customer service, the people on the other side of the phone (or counter, those few times) are incredibly friendly and helpful (again, compared to, say HSBC). (I didn't need to interact with any other bank during COVID, so I can't compare, but I was particularly pleased with how well DBS let me do even complex banking remotely, and I hope they adopt it as a permanent option). Things do seem to be getting _slightly_ worse though. One issue I have with the bank is that they use push notifications to advertise products and also use push notifications for security purposes. It makes me wish apple/google would require push notifications to be categorized. Also, over the last year, push notification spam seems to be linked directly to my purchases, e.g. "Got a new gadget, consider DBS payment plans..." shortly after buying new phones. It's like they've confused digitization with social network.

qsi 5y ago
As an addendum to my previous reply on the DBS's abysmal UX, I will echo the point made here that once you get to talk to someone, it's generally fine. I've had some great service when I could get through to a human, but they're making it increasingly difficult to do so as they're going all digital. Many branches have closed, and even the higher-end Treasures locations are now down to just two I think (a major mistake IMO especially as you target the wealthier). Navigating their phone support. takes. a. lot. of. patience. I also agree that other banks can be a lot worse, and HSBC in particularly has been abysmal. (edited for grammar)
- ValentineC 5y ago
  > I also agree that other banks can be a lot worse, and HSBC in particularly has been abysmal. At least HSBC got something right with their app UX: they give me an option to log in using my PIN instead of automatically trying Face ID multiple times, in this era when masks are still mandatory.

jcims 5y ago

I’m not super familiar with the full content of their regulations but in my experience working with banks that maintain a presence there, Singapore control requirements tend to be fairly specific, opinionated and far reaching. For example, in developing OS security standards, I found myself having to actually incorporate requirements specifically for the Monetary Authority of Singapore that bled into operations in North America. These requirements extend into resiliency as well. I’m curious how well DBS has been fairing in audits.

qsi 5y ago
A few months from now the headline will be that MAS (Monetary Authority of Singapore) slaps DBS with a hefty fine.
cameronh90 5y ago
MAS requirements are incredibly specific. However, I wonder if that opinionatedness could sometimes make things worse. We had to do a lot of work to convince MAS that AWS was safe enough for our tiny operation out there, for example. Whereas, if we'd gone for a local colo, it would have been much easier - despite having worse resilience and likely worse security.
- jcims 5y ago
  In my case it 100% made things worse because there is was no room for contextual interpretation of the rules. The MAS requirements went too far in one direction and prevented us from developing more granular and observable control strategies. We essentially wound up stuck with security paradigms from the 90's. I was able to compartmentalize most of it to in-country workloads but some of it ultimately leaked back into our global builds because it was going to be too costly to maintain a total snowflake.

sohkamyung 5y ago

Original headline says, "Singapore bank DBS says services disrupted for second day", so I'm not sure where the "with no meaningful update from DBD" in the HN headline came from, nor am I sure of the meaning of DBD.

qsi 5y ago
More details, but not much more news yet: https://www.channelnewsasia.com/singapore/dbs-posb-down-not-available-digital-banking-2334661 https://www.channelnewsasia.com/singapore/dbs-posb-down-not-... From that article: >DBS Singapore country head Shee Tse Koon said the bank identified a problem with its access control servers on Tuesday and has been working "round the clock" with third-party engineering providers to fix the issue.
- Animats 5y ago
  A little more detail from the bank's own web site. Apparently they got the access control servers up again, but then they went down a few hours later, at least partially. Which, after 24 hours of downtime, probably means that most of the people who got the systems back up were trying to get some sleep. [1] https://www.dbs.com/newsroom/DBS_Singapore_Country_head_provides_update_on_disruption_in_digital_banking_services https://www.dbs.com/newsroom/DBS_Singapore_Country_head_prov...

Havoc 5y ago

Ransomware?

TheSpiceIsLife 5y ago

No discussion related to internet banking is complete without someone mentioning Commonwealth Bank here in Australia. Commonwealth Bank CIO says any code older than three months is like “food in the fridge”, past its ‘best before’ date. https://www.itnews.com.au/news/cba-to-treat-its-software-like-food-in-the-fridge-570930 https://www.itnews.com.au/news/cba-to-treat-its-software-lik... Which explains why their online banking services have become very brittle the past couple years.

asah 5y ago
Three months?! I'm guessing he's never written any.
- TheSpiceIsLife 5y ago
  I’d hazard a guess you’re probably right. Still worrying that someone who’s prepared to spout that opinion made it that far up the organisation.
ValentineC 5y ago
I don't know. As a Singapore resident with way too many bank accounts, it seems like the most unstable experiences for me tend to be the banks that try to reinvent features, like the aforementioned DBS. I'd speculate that shipping new features, not stability, is somewhat linked to career progression. It could also be because most Singapore banking apps end up using the same mobile security framework to pass their audits [1] — except that same security framework prefers security over user experience. [1] https://en.wikipedia.org/wiki/V-Key https://en.wikipedia.org/wiki/V-Key
dustintrex 5y ago
The interview is a bit ambiguous, but to me it looks like he's referring to prod binaries over 3 months old being stale, which is considerably more reasonable.
cgio 5y ago
At a high level, CBA’s technology strategy is “quite simple to think about but quite difficult to execute” That’s the problem right there, it sounds like it is too simplistic to support execution.

latch 5y ago

Small update from their country head: https://www.youtube.com/watch?v=EuXtrtH0uKY https://www.youtube.com/watch?v=EuXtrtH0uKY 1 - problem identified with their access control services (or did he say servers?). (doesn't sound good) 2 - working with their 3rd party engineering providers (how much is outsourced?)

tsechin 5y ago
Story time! Years ago, I was a contractor writing (a small piece of) software for DBS. It was probably a couple of hundred lines of Java, and I probably delivered it by emailing a zip file to someone. I don't recall their code review standards being particularly ... existent. Mine was a relatively small project with them. There were other teams at my firm working on larger projects at DBS. Another story! A new employee joined my firm, and - on his first day - was sent to join one of these teams onsite at DBS. His day ended at 9pm /on his first day of work/. He gave notice on day two.
- _tik_ 5y ago
  My experience as an outsource partner with them 9 years ago. DBS only employed Project Manager and outsourced all their technical stuff. Yes the working hours is very long.

dav43 5y ago

The DBS - worlds best digital bank - is an absolute joke. Their local app is terrible in Singapore and you can't even download transaction history more than 6 months!

deleted 5y ago

[deleted]

crossroadsguy 5y ago

I see many people dissing it and it might be valid for their location. But DBS is in India as well. Compared to local banks, among which most are shitty in a way that it feels they present “remaining shitty” as a non-negotiable goal to their tech team, and some foreign banks, some of those are shitty as well, DBS India’s app and Internet banking is really polished and functional. They bought a local bank recently so they have got a major physical presence push. I used to have bad experiences with their iOS app but since last 7-8 months it’s been really been good. I had complained around a year ago. 7-8 months back, an individual reached out on email thanking me for my feedback and saying they’ve done a major overhaul and they’d be glad to hear my feedback on it. This shit doesn’t happen with banks here. Be it private or govt banks, you’re lucky if you’re not treated like shit. They don’t spam (you’d not at all understand what it actually means if you don’t live in India!!!). One short email to my RM and even those rare marketing emails dropped to zero (one of them I had actually benefited from; another banking rarity here). PS: In last ~decade and a half I’ve tried too many banks and I keep ~10 bank accounts active at a time (due to insurance from Govt limit when banks fail; I know, I know)

korginator 5y ago
Glad to hear your positive experiences. Could I share your feedback with a few colleagues in DBS here? As you say, it's about context. Here we keep pushing the message that Singapore is an ideal place to do business, that the regulations are business-friendly, and the financial systems are efficient, well run and reliable, and this is largely true. A failure of this nature and size may be a blip in the Japanese or European markets, but it's also about perception. That takes a hit. It will affect the investment appetite, and our economy depends a lot on international business and money moving through our financial institutions. That's the first impact. More practically, and looking locally, about three-quarters of our GDP is in the services sector [1]. Pretty much everyone is banked, and payments are increasingly cashless. A lot of these payments (including salaries, payment for services like air-con repairs, etc.) go through our local equivalent of your UPI called PayNow, and DBS' PayNow systems were failing. I remember carrying cash in my wallet just a handful of times over the past couple of years, and I pay for pretty much everything electronically. So... these systems which failed are far more central to a well functioning economy here than in some other countries. That's the second impact. MAS (our regulator) is rightly concerned about the long-term implications. You can be sure they will have a thorough root-cause analysis. Some heads may roll. DBS will definitely be slapped with a multi-million dollar fine and be told to present a comprehensive plan to ensure this doesn't happen again. MAS will appoint people to oversee & manage how this is done, and knowing MAS they may start other audits to identify potential problems in other systems. They'll definitely be watching DBS closely, and they will probably look at other banks too. References: [1] https://www.singstat.gov.sg/modules/infographics/economy https://www.singstat.gov.sg/modules/infographics/economy

nreilly 5y ago

https://twitter.com/dbsbank/status/1463409021411594245 https://twitter.com/dbsbank/status/1463409021411594245

pydry 5y ago

This sounds very similar to the TSB meltdown playbook. 1) Have upper level management who fundamentally do not understand technology. 2) Outsource systems development to cheapest possible bodyshop and give them strict deadlines. 3) Explosion of technical debt that is entirely invisible to middle management and above. 4) IT meltdown as a result. CEO (hopefully) resigns in disgrace.

korginator 5y ago

The regulator MAS is already starting to make a noise about "appropriate actions" [1] once the dust has settled. It's also a matter of national pride, DBS is very iconic and there's been a lot of publicity about the bank as a regional success story, specially as they were publicising their "successful" transition to become a tech company going head to head with global giants, while also offering banking services [2]. I hope they don't forget their roots and end up treating banking as an afterthought and failures of this magnitude as par for the course. A sizeable portion of the population still goes into the bank to transact physically. There's the historical angle and another point of national pride [3]. Since the post office savings bank (now part of DBS) was established by the British in 1877 it catered to low income groups, and provided a very high quality of service while keeping operating costs low. They were also early regional adopters of technology, starting to computerise their accounts in 1972. For a long time POSB was run as a statutory board with directors that were directly appointed by a government ministry. All this is to say that people have a slightly different relationship with POSB/DBS compared to other banks. There's a lot of trust that's been carefully cultivated over decades. People believe and trust in this bank, they trust that the bank has their best interests at heart, and this has been generally true. Failures of this magnitude will start to break this trust. That in itself is not a big concern, there are other banks and we can jump ship. There's a social angle with POSB/DBS, specially with the older generations. When POSB was handed to DBS [4] on a silver platter in 1998 a lot of people voiced concerns that the bank would become yet another money-grubbing profit-seeking megacorp, and about how this would affect their earlier objective of serving the weaker strata of society. It's hard to predict what may happen if this trust starts to break. References: [1] https://www.channelnewsasia.com/singapore/dbs-bank-service-disruption-mas-appropriate-supervisory-actions-2335931 https://www.channelnewsasia.com/singapore/dbs-bank-service-d... [2] https://innovationinbanking.efma.com/content/articles/transforming-dbs-bank-tech-company https://innovationinbanking.efma.com/content/articles/transf... [3] https://eresources.nlb.gov.sg/infopedia/articles/SIP_2014-01-07_112742.html https://eresources.nlb.gov.sg/infopedia/articles/SIP_2014-01... [4] https://www.dbs.com/about-us/who-we-are/our-heritage https://www.dbs.com/about-us/who-we-are/our-heritage