4 ms·

I think it’s mostly true. The complaint with this kind of “industrial global Python code base”, be it at banks or elsewhere, is that often they are hastily cobb

I think it’s mostly true. The complaint with this kind of “industrial global Python code base”, be it at banks or elsewhere, is that often they are hastily cobbled together and depend on extreme user care to not flop over all the time. I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. When something does break in the end, someone senior just shouts at the poor devs until it works again - usually fixed with a hasty patch again. Similarly, documentation? Access control? Sanity? These seem to be left behind.

lmm 5y ago
Hmm, I found the opposite - the fact that there was this global framework that managed all the data and code meant that access control was actually pretty good, better than most tech companies I've worked for. You had a single source of truth for what your access rights were, there was integrated Kerberos any time you needed to access a system outside Minerva. And having all the code in a managed place meant good deprecation cycles - not instant deprecation like the Google monorepo, but tracking and policy for which old versions of libraries were in use and how much that was tolerated. Documentation was at least attempted, and while platform stability/enhancement work did have to be coupled to business initiatives to a certain extent (e.g. "we're doing this performance work to enable us to run risk estimation more often to meet MIFID requirements at low cost") there was leadership that put a value on maintaining high quality code and this paid dividends.
- stevesimmons 5y ago
  This was 100% my experience too. The biggest productivity gains were: - having a single source of truth for both data and code (in a closely coupled environment) - strong, battle-tested libraries to take care of all infrastructure concerns. - enforced code dev/test/review/deployment workflows This let the front-office devs be highly productive on adding real business value for their trading desks. Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. The infra we all take for granted today at AWS/GCP/Azure simply did not exist back then, and banks' data security policies at the time did not allow cloud processing.
  - FabHK 5y ago
    > Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. GS had „these systems“ well before 2000 (via J Aron). I think around the time you mentioned they spread to other firms (in their Python reincarnation).
simonh 5y ago
You could not be more wrong. Quartz has first class documentation, solid tooling, very well thought out and rigorous code review and access controls. Banks are regulated up to the eyeballs, so everything has to be audited and justified in detail. It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but the people building these things are top notch technologists and know what they're doing.
- rich_sasha 5y ago
  Well, I don’t know about Quartz. I worked with DB systems and they were awful in that regard. They worked, but largely because people stuck to convention. For example, changing scheduler jobs required submitting a change in Excel and having it approved (twice...) by someone. Except the table was world-writable and changes not logged. So in principle only your appropriate superior could approve change, in practice anyone could, and you’d never even know.
  - simonh 5y ago
    Youch, that's nasty. I can completely believe it though, banks are huge unwieldy organisations. I spent a fair bit of time working with auditors though, so I know a huge amount of effort goes into rooting out things like that. The thing is just because a team in a bank did this thing, that doesn't mean "The Bank" thinks that's a good idea. Like any company, banks are communities. I'm not making excuses, the fact this system wasn't properly architected is a failure of governance, but I've been on the other side of this trying to get teams to fix their problems and adopt resilient processes and procedures. Every offender thinks their service is special and their violation of the standards is justified.
- kwertyoowiyop 5y ago
  For some reason people think everything is as good as it can be at FAANG and other big name tech companies, and everyone else just walks around with their pants down at their ankles bumping into walls until 5:00. It’s just not true.
misja111 5y ago
> I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. It depends which department you are in, but in general: absolutely not. Actually the reverse is true. Banks have huge risks to manage: just imagine for instance what damage a hack of their account system could cause. Or a crash of their payment system. Therefore it is of the utmost importance that systems are stable and bug free. In most departments, feature development is only in second place: stability and reliability have priority one. This concern is so important that it is not just left to the responsibility of the banks themselves: for many systems, banks have to comply with external standards and are audited for that by external agencies.
- dolmen 5y ago
  And that's why many old banks still use COBOL.
  - secondaryacct 5y ago
    Well it s repeated a lot but I work in a big bank and I have yet to see tech older than Java 6 :(

lmm 5y ago

Hmm, I found the opposite - the fact that there was this global framework that managed all the data and code meant that access control was actually pretty good, better than most tech companies I've worked for. You had a single source of truth for what your access rights were, there was integrated Kerberos any time you needed to access a system outside Minerva. And having all the code in a managed place meant good deprecation cycles - not instant deprecation like the Google monorepo, but tracking and policy for which old versions of libraries were in use and how much that was tolerated. Documentation was at least attempted, and while platform stability/enhancement work did have to be coupled to business initiatives to a certain extent (e.g. "we're doing this performance work to enable us to run risk estimation more often to meet MIFID requirements at low cost") there was leadership that put a value on maintaining high quality code and this paid dividends.

stevesimmons 5y ago
This was 100% my experience too. The biggest productivity gains were: - having a single source of truth for both data and code (in a closely coupled environment) - strong, battle-tested libraries to take care of all infrastructure concerns. - enforced code dev/test/review/deployment workflows This let the front-office devs be highly productive on adding real business value for their trading desks. Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. The infra we all take for granted today at AWS/GCP/Azure simply did not exist back then, and banks' data security policies at the time did not allow cloud processing.
- FabHK 5y ago
  > Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. GS had „these systems“ well before 2000 (via J Aron). I think around the time you mentioned they spread to other firms (in their Python reincarnation).

simonh 5y ago

You could not be more wrong. Quartz has first class documentation, solid tooling, very well thought out and rigorous code review and access controls. Banks are regulated up to the eyeballs, so everything has to be audited and justified in detail. It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but the people building these things are top notch technologists and know what they're doing.

rich_sasha 5y ago
Well, I don’t know about Quartz. I worked with DB systems and they were awful in that regard. They worked, but largely because people stuck to convention. For example, changing scheduler jobs required submitting a change in Excel and having it approved (twice...) by someone. Except the table was world-writable and changes not logged. So in principle only your appropriate superior could approve change, in practice anyone could, and you’d never even know.
- simonh 5y ago
  Youch, that's nasty. I can completely believe it though, banks are huge unwieldy organisations. I spent a fair bit of time working with auditors though, so I know a huge amount of effort goes into rooting out things like that. The thing is just because a team in a bank did this thing, that doesn't mean "The Bank" thinks that's a good idea. Like any company, banks are communities. I'm not making excuses, the fact this system wasn't properly architected is a failure of governance, but I've been on the other side of this trying to get teams to fix their problems and adopt resilient processes and procedures. Every offender thinks their service is special and their violation of the standards is justified.
kwertyoowiyop 5y ago
For some reason people think everything is as good as it can be at FAANG and other big name tech companies, and everyone else just walks around with their pants down at their ankles bumping into walls until 5:00. It’s just not true.

misja111 5y ago

> I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. It depends which department you are in, but in general: absolutely not. Actually the reverse is true. Banks have huge risks to manage: just imagine for instance what damage a hack of their account system could cause. Or a crash of their payment system. Therefore it is of the utmost importance that systems are stable and bug free. In most departments, feature development is only in second place: stability and reliability have priority one. This concern is so important that it is not just left to the responsibility of the banks themselves: for many systems, banks have to comply with external standards and are audited for that by external agencies.

dolmen 5y ago
And that's why many old banks still use COBOL.
- secondaryacct 5y ago
  Well it s repeated a lot but I work in a big bank and I have yet to see tech older than Java 6 :(