5 ms·
I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as im
by lmm 5y ago
I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based).
Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their more nimble competitors. Being able to calculate the risk of the whole bank isn't just a cool feature, it's the core value proposition of this platform.
Being able to just upload your code and run it is really cool, and if you squint it looks a bit like what the outside world is trying to set up with serverless/lambda-style platforms - just write a function, submit it, and there, it's running. (But it's worth remembering that Python is not a typical programming language; python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages). Obviously there's a tension between this and having good version control, diffs, easy rollbacks etc. - but because Minevera is already designed to do all that for data (because you need that kind of functionality for modifications to your bonds or whatever), doing it this way strikes a much better compromise than something like editing PHP files directly on your live server.
What this article calls data-first design has a lot in common with functional programming. I hope that as the outside world adopts more functional programming and non-relational datastores, Minerva-style programming will get more popular. It really is a much better way to write code in many ways. The difficulty of integrating with outside libraries is a shame though.
- rich_sasha 5y agoI think it’s mostly true. The complaint with this kind of “industrial global Python code base”, be it at banks or elsewhere, is that often they are hastily cobbled together and depend on extreme user care to not flop over all the time. I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. When something does break in the end, someone senior just shouts at the poor devs until it works again - usually fixed with a hasty patch again. Similarly, documentation? Access control? Sanity? These seem to be left behind.
- lmm 5y agoHmm, I found the opposite - the fact that there was this global framework that managed all the data and code meant that access control was actually pretty good, better than most tech companies I've worked for. You had a single source of truth for what your access rights were, there was integrated Kerberos any time you needed to access a system outside Minerva. And having all the code in a managed place meant good deprecation cycles - not instant deprecation like the Google monorepo, but tracking and policy for which old versions of libraries were in use and how much that was tolerated. Documentation was at least attempted, and while platform stability/enhancement work did have to be coupled to business initiatives to a certain extent (e.g. "we're doing this performance work to enable us to run risk estimation more often to meet MIFID requirements at low cost") there was leadership that put a value on maintaining high quality code and this paid dividends.
- stevesimmons 5y agoThis was 100% my experience too. The biggest productivity gains were: - having a single source of truth for both data and code (in a closely coupled environment) - strong, battle-tested libraries to take care of all infrastructure concerns. - enforced code dev/test/review/deployment workflows This let the front-office devs be highly productive on adding real business value for their trading desks. Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. The infra we all take for granted today at AWS/GCP/Azure simply did not exist back then, and banks' data security policies at the time did not allow cloud processing.
- FabHK 5y ago> Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. GS had „these systems“ well before 2000 (via J Aron). I think around the time you mentioned they spread to other firms (in their Python reincarnation).
- simonh 5y agoYou could not be more wrong. Quartz has first class documentation, solid tooling, very well thought out and rigorous code review and access controls. Banks are regulated up to the eyeballs, so everything has to be audited and justified in detail. It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but the people building these things are top notch technologists and know what they're doing.
- andrewshadura 5y ago> python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages I'm not sure how to react to that, but these features in Python are miles ahead of what many other languages have (or actually don't have).
- danmur 5y agoI think there are too many options, or not enough direction for busy people. Once you understand how it all works and pick / build the right tools it all works pretty well.
- Icathian 5y agoI would agree for all but deployment. I know my way around python reasonably well, but pyinstaller and friends still make me have bad days pretty regularly.
- pmontra 5y agoFour Python projects, same customer, five different deployment systems. Docker, a Capistano look-a-like I coded in bash, git pull (their former standard), git format-patch plus scp, zip archives. Yes, python file.zip works if it contains the right files. Probably the latter is the easiest way, except it doesn't address the dependencies.
- kristjansson 5y ago> except it doesn't address the dependencies It does if you put them all in the zip :) (and build for exactly the platform your customer is going to deploy on)
- infecto 5y agoAm I the only one who has never had enough headache over the years to say its awful? I do think dependency management is a somewhat difficult problem to solve and a lot of systems have pros/cons but I never have had huge issues with Python's.
- sandGorgon 5y agoDoes everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ? I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL). What does the devops there look like ? Snapshot every minute ?
- joshuamorton 5y agoThis works until... Specifically, until you realize that pickle changes based on python version, so updating from py3.x to py3.x+1 will prevent your application from reading previously stored data.
- detaro 5y agoThis is wrong. pickle can read old files just fine and lets you generate files in old pickle format versions if you require backwards compatibility further than when the current protocol was introduced (it does not get increased with each python version).
- qorrect 5y agoI use Pickle quite a lot for caching, a file read is almost always faster than a DB query. For long-term persistent data ? Seems very dangerous to me, even reading a pickle from say PyPy vs a Cython intepreter corrupts the damn thing.
- miohtama 5y agoZODB is the object oriented database as a giant pickle dump. Surprisingly, it works and scales wuite well. The downside is that non-Python tools cannot access it all. https://zodb.org/en/latest/ https://zodb.org/en/latest/
- rbanffy 5y agoI remember explaining our tech stack (Python and Zope) to clients. “Where is the code for that page?” “It’s in the database” “Oh… Like MySQL?” “No. It’s an object database” “???” I called it “Martian Technology Syndrome”. But it worked. At later stages we paid the price and had to serialize the datastore for migrations, but that’s what you get for relying on pickles.
- joshu 5y agoit's been 15 years and i'm still a bit traumatized by aurora