5 ms·

What is the typical root cause of these issues ? Is it improper architecture / transaction management that only gets exposed when certain things that usually wo

by pylua 3y ago

What is the typical root cause of these issues ? Is it improper architecture / transaction management that only gets exposed when certain things that usually work become unreliable ?

pavpanchekha 3y ago
Every large bank is the merger of thousands of banks, and the it systems are just strapped together.
- pylua 3y ago
  I get that, but Wells Fargo didn’t just have a major merger did it? The real problem with the mergers is that the system are too complex for the analysts and architects to understand, and the software mirrors that. The knowledge is also distributed across a huge array of people that have no reason to change the status quo to keep their knowledge valuable to the bank.
baz00 3y ago
Functionally it's usually an edge case somewhere or someone doing something stupid. Recently where I reside there was an overflow scenario of a registration number and the powers that be decided to add another digit. Of course to maintain "compatibility" this number is a string and they added "0" to the start of it. Some organisations treated it as a string, some as a number. Ones who treated it as a string ended up with two orgs defined as "01234" and "1234". Of course org "1234", now known as "01234" wondered where the hell all the integration data went. Even worse someone registered an org "045" when "45" already existed which means that the wrong org got the integration data. This manifests itself as stuff being missing usually. At that point, the outsourcing managers adopt the spiderman pointing pose ( https://i.imgur.com/ypYY6yg.png https://i.imgur.com/ypYY6yg.png ) and try and establish blame. This takes forever to resolve due to a Mexican-standoff scenario, usually broken by one of them getting hungry which can take a few hours. Eventually blame is assigned and a team is set upon it. Because staff turnover is so high due to the low salary, no one at the outsourcing outfit knows anything about some rancid bit of COBOL/REXX/RPL or if you're really lucky Java which is holding all this together. In a panic, only initiated after someone informs them that they are going to have to pay a hefty compliance fine, the management team start running around like headless chickens. Eventually they bump into someone they forgot to fire who can remember who the original engineers are who wrote this. Delving into an ancient HR database, they find the mobile number of the last guy who touched and call him. His widow answers and informs them that Bob died 3 years ago but his old colleague Terry is still alive and you might want to call him. They call up Terry who is on a golf course somewhere. Terry is concerned but not that concerned about the situation because he knew what a steaming shit show the org was and moved his money well away from it. Eventually the manager starts waving daily rates around meeting the "I can be bothered" level of interest. Terry finishes his game, throws the clubs in his car and heads into the office to look at it. After 3 hours of unsuccessfully trying to work out how to get Terry an account on the mainframe again without having to involve the outsourced helpdesk, which is currently down due to a huge mudslide, he sits at someone else's terminal and stares intently at the problem. He utters a few words, edits a few characters in one file and resubmits the job. Several minutes later, the phone lines stop ringing. All is good again, Terry made $1500 and the org only lost 2% of its customers compared to the 0% it should have lost if they hadn't outsourced everything. Now imagine this in 20 years with Kubernetes!
- pylua 3y ago
  Sounds like business as usual, which is very sad. This was very entertaining, however.

by pylua 3y ago

What is the typical root cause of these issues ? Is it improper architecture / transaction management that only gets exposed when certain things that usually work become unreliable ?

pavpanchekha 3y ago

Every large bank is the merger of thousands of banks, and the it systems are just strapped together.

pylua 3y ago
I get that, but Wells Fargo didn’t just have a major merger did it? The real problem with the mergers is that the system are too complex for the analysts and architects to understand, and the software mirrors that. The knowledge is also distributed across a huge array of people that have no reason to change the status quo to keep their knowledge valuable to the bank.

baz00 3y ago

Functionally it's usually an edge case somewhere or someone doing something stupid. Recently where I reside there was an overflow scenario of a registration number and the powers that be decided to add another digit. Of course to maintain "compatibility" this number is a string and they added "0" to the start of it. Some organisations treated it as a string, some as a number. Ones who treated it as a string ended up with two orgs defined as "01234" and "1234". Of course org "1234", now known as "01234" wondered where the hell all the integration data went. Even worse someone registered an org "045" when "45" already existed which means that the wrong org got the integration data. This manifests itself as stuff being missing usually. At that point, the outsourcing managers adopt the spiderman pointing pose ( https://i.imgur.com/ypYY6yg.png https://i.imgur.com/ypYY6yg.png ) and try and establish blame. This takes forever to resolve due to a Mexican-standoff scenario, usually broken by one of them getting hungry which can take a few hours. Eventually blame is assigned and a team is set upon it. Because staff turnover is so high due to the low salary, no one at the outsourcing outfit knows anything about some rancid bit of COBOL/REXX/RPL or if you're really lucky Java which is holding all this together. In a panic, only initiated after someone informs them that they are going to have to pay a hefty compliance fine, the management team start running around like headless chickens. Eventually they bump into someone they forgot to fire who can remember who the original engineers are who wrote this. Delving into an ancient HR database, they find the mobile number of the last guy who touched and call him. His widow answers and informs them that Bob died 3 years ago but his old colleague Terry is still alive and you might want to call him. They call up Terry who is on a golf course somewhere. Terry is concerned but not that concerned about the situation because he knew what a steaming shit show the org was and moved his money well away from it. Eventually the manager starts waving daily rates around meeting the "I can be bothered" level of interest. Terry finishes his game, throws the clubs in his car and heads into the office to look at it. After 3 hours of unsuccessfully trying to work out how to get Terry an account on the mainframe again without having to involve the outsourced helpdesk, which is currently down due to a huge mudslide, he sits at someone else's terminal and stares intently at the problem. He utters a few words, edits a few characters in one file and resubmits the job. Several minutes later, the phone lines stop ringing. All is good again, Terry made $1500 and the org only lost 2% of its customers compared to the 0% it should have lost if they hadn't outsourced everything. Now imagine this in 20 years with Kubernetes!

pylua 3y ago
Sounds like business as usual, which is very sad. This was very entertaining, however.