9 ms·

I was the person who first deployed Python at Goldman Sachs. At the time it was an "unapproved technology" but the partner in charge of my division called me an

• lanevorockz 5y ago
  Python Quants for the win ! I was lazy enough to stay with R but it was nice of Wes of building pandas, made adoption super easy.
• tikkabhuna 5y ago
  I joined the BAML grad scheme 10 or so years ago. We had a presentation from one of the Quartz guys and someone asked how they’d manage upgrading the version of Python. They were using something like 2.6.5. The whole move to 3.x was a thing. The Quartz guy just flat out said they wouldn’t upgrade. Seemed crazy to a new grad back then, but now I wouldn’t want to consider it either. Thanks for your contribution! It was amazing that even in my role where I didn’t use Quartz, I could see and search all the code. Felt quite novel back then.
  • lozenge 5y ago
    BAML(Quartz) and JPM(Athena) both had Python 3 migrations well underway as of PyCon UK 2019. It took me more than half way down the article to identify I have not worked at the same bank as the author...
    • deleted 5y ago
      [deleted]
  • keithalewis 5y ago
    A friend I've known since I first started on Wall Street now rides herd on the BofA Quartz libraries. One component of his job is to make developers aware of existing libraries they can use to solve business problems instead of reinventing the wheel. His theory of why they always have excuses not to do that is that they have no training in software development. They are still at the point in their learning where the are just excited that they can press keys on a computer and get it to do things they are barely able to understand.
• simonh 5y ago
  I worked on Quartz for 3 years and loved it. Some devs grumbled about various aspects of it, but I come from an application support background and taught myself python, so I suppose I had fewer developer habits to un-learn. From what I understand, all this started with SecDB at Goldman, which was a the prototype for all these systems but wasn't Python based. The lore is that SecDB was instrumental in Goldman being able to rapidly figure out what their exposure was during the 2008 crisis. Some of that team, lead by Kirat Singh went on to start Athena at JP Morgan and then Quartz. I met Kirat once, he was considered a bit of a rock star in the bank tech community. He now runs Beacon, which is basically Bank Python as a service.
  • Paladiamors 5y ago
    I work at Beacon.io and it's an awesome place to be. Kirat is indeed a rockstar and it's awesome to work with an CEO that knows great code. We also landed a Series C last month and we're growing :) https://www.crunchbase.com/funding_round/beacon-series-c--60705dbc https://www.crunchbase.com/funding_round/beacon-series-c--60... We've also got a bunch of positions open too for those that are interested in joining! https://www.beacon.io/careers/ https://www.beacon.io/careers/
    • Icathian 5y ago
      I took a quick look, it seems like all the postings are London or New York. What's the feeling internally about remote hires? I'm assuming that's still out of fashion in finance and Beacon feels the same?
      • Paladiamors 5y ago
        Because of Covid I've been remote since last year. It's still an evolving situation. But things have worked out pretty well. Our CEO was also remote during that time too and here's him giving a webinar from a cabin :) https://www.youtube.com/watch?v=fXPDXbrPdxI&ab_channel=BeaconPlatform%2CInc https://www.youtube.com/watch?v=fXPDXbrPdxI&ab_channel=Beaco....
    • alecco 5y ago
• stevesimmons 5y ago
  I worked on Athena at JPMorgan for 8 years, and loved it. Seeing Python at the core of trading, risk and post-trade processing for Commodities, FX, Credit etc was such a great developer experience. By the time I left JPM, there were 4500 devs globally making 20k commits weekly into the Athena codebase. (I did a PyData presentation on this [1] for more details). The one downside was the delayed transition from Py2.7 to 3; I left just as that was getting underway. [1] https://www.youtube.com/watch?v=ZYD9yyMh9Hk https://www.youtube.com/watch?v=ZYD9yyMh9Hk The
  • oogetyboogety 5y ago
    That's funny they mentioned replayable financial message queues. Those are a hit here
• victor106 5y ago
  If you don’t mind me asking which year was this?
• JoshTriplett 5y ago
  There are a number of workplaces where I'd have been willing to rely on "probably wouldn't be fired", but a bank is definitely not one of them. Congratulations on shipping something useful in the face of that risk and uncertainty.
  • op00to 5y ago
    Investment banks at that time were a little bit different.
  • simonh 5y ago
    The trading community walk on a knife edge all the time, it's not a place for the faint of heart. I used to support derivatives trading systems and a few times there were issues that meant they'd lost control of orders on an exchange. Scary stuff. It requires a crazy mixture of careful, deliberate, calculated risk control on the one hand; but once you commit to something you jump in with both feet and throw everything into it. You need to be both meticulously risk averse, and also willing to do whatever needs to be done when it needs doing, and accept responsibility. It was great!
    • gjvc 5y ago
      An excellent description.
    • ravibala1 5y ago
      I worked at UBS on the G10 FX options desks in Stamford and Singapore. Remember being very surprised by how my interview and training were incredibly stressful. It was very intentional as well, where my trainer knew the exact reactions he was creating with his behavior. It was only a couple of weeks in, when I had to react within 60 seconds (USDJPY option expiry, NYC cut) on a position our front book would have lost MM on; with senior sales MDs screaming at me as well. Lo and behold, I was just used to it and could focus and execute based on my training. My wife calls my thinking on the training - Stockholm syndrome. I still believe those skills were incredibly valuable for me, just perhaps delivered in a more 2021 acceptable approach.
  • 5y ago
• odiroot 5y ago
  Very interesting story. I'm actually impressed, you're allowed to give us such "internals" about GS.
• vba 5y ago
  What year was this out of interest?
  • seanhunter 5y ago
    I think 2002 or 3. I worked there from 2001 to about 2010 and it was pretty early in my time there.
• govg 5y ago
  For some context (as an ex-Goldman employee myself), "Armen" in the quote is most probably https://www.goldmansachs.com/insights/outlook/bios/armen-avanessians-bio.pdf https://www.goldmansachs.com/insights/outlook/bios/armen-ava... , who has quite a legendary reputation within the firm for the work he's done. He was also one of the first to be hired as a "strat", which used to be how Goldman referred to its quants who sat between front office and tech systems and worked with both sides.
  • deleted 5y ago
    [deleted]
  • FabHK 5y ago
    SecDB/Slang originated around 1992 at the commodity trading shop J Aron, which GS had bought 1981. Later (end of the 90s) there was a push to extend it to the rest of the firm, first fixed income, then equities. Armen flew the whole world-wide strat team to NY and gave a presentation, and to drive the point home, he played a clip from StarTrek: "You will be assimilated. Resistance is futile!"
  • collinmanderson 5y ago
    > Prior to joining the firm, Armen was a member of the technical staff at Bell Laboratories in Murray Hill. Wow. Bell Labs too.
• arthurcolle 5y ago
  I worked for GSAM for a bit as my first NAPA project - I guess you're referring to Armen Avanessians? Haha haven't heard references to the "train" in so long. Did you ever do any Slang/SecDB dev? I was mostly in FICC Tech so was pretty much slangin' slang most of my time there. JSI (Java Slang Integration) was just getting off the ground but there wasn't too much for the front office tech teams to do there until it was to mature in the coming years. Good times, thanks for sharing the history ;)
  • seanhunter 5y ago
    Yes, that's who I was referring to. I did absolutely tons of slang and a fair amount of TSecdb as well as a lot of work on the C++ infra and the build and distribution code.
  • helsinki 5y ago
    My former team, previously known as AIM, wrote JSI :)
• gjvc 5y ago
  what version of Python was it?
• tomrod 5y ago
  Beautiful! I worked a bit with Athena and was in the group that got Anaconda into the banking sector. Small world we live in. I think I remember your name from JPMC days, but it has been awhile.
• anentropic 5y ago
  Ah, now I finally know what it means when I get job ads for Python+Quartz ...! I could tell from the context it was nothing to do with QuickTime...
• helsinki 5y ago
  Ha, cool. I learned Python on your distribution ca. 2014.
• sizzle 5y ago
  Curious why would this high-powered person go to bat for a technology decision they didn’t seem to have done any risk assessment of? Wouldn’t he be liable if something was exploited and hurt and company, like his head would be on the chopping block for giving the go ahead when they traced it up the chain of command?
  • seanhunter 5y ago
    That's precisely why he did it the way he did. He had total deniability. Here's how the conversation would go if it went wrong somehow: "Armen, did you tell Sean to install python?" "No". "Sean, did Armen tell you to install python?" "Err... no. He said I probably wouldn't be fired." "Well it turns out he's not right about everything. Here's a cardboard box for your things."
    • simonh 5y ago
      This is only my opinion, but I think the reason Armen said it like he did was because by not making it an order he's giving Sean the option of not doing it, if he's not up for accepting the risk. However the risk was both of them could have got fired. Armen must have known people would know Python had been put on these machines and that he authorised it, in fact what's the point of putting it on them if nobody knows and nobody uses it? I can guarantee you that within 24 hours someone was asking Armen why he'd authorised this and was justifying it. There cannot have been any possibility of dodging responsibility for this decision. If anyone got fired it would have been Armen, with a possibility of Sean going as collateral damage. This is the big league. You make your decisions and you accept responsibility for them.
      • sizzle 5y ago
        Exactly, so what is this Armen character getting out of this other than a potentially big amount of liability and unarticulated risk. The OP said he told people openly that Armen told him he could do it when asked. This makes no sense to me, what’s the upside to Armen? If he is business savvy, he needs to be gaining something in exchange for having his name thrown around by OP as signing off on this.
• mirko22 5y ago
  Oh wow, I remember Quartz at BAML… Though this was several years after initial deployment and when core devs left. One day I will sit down and write a small poem about the insanity of software development based on my experience with Quartz. It will be an intriguing story of love and hate being told through a sensual dance between sales and engineering. The battles will not be epic but the consequences of one’s actions will be far reaching. It was indeed and experience worth having.
  • Maven911 5y ago
    Tell me more about what worked and didn't..I recall the pain of watching QzDesktop load and Bob/HUGS jobs failing..but what else, and what did you enjoy
    • mirko22 5y ago
      All great stories use mundane circumstances in life to convey much deeper and abstract ideas. One such story, I humbly hope, would be my poem. Yes the b00bs were all over the place as they are in real life, and in my opinion how they got there is as interesting as real life. Original people solving the original problem had a good understanding of what they are dealing with. However with subsequent generations it became a monkey problem https://m.youtube.com/watch?v=5QuwPeH9P7Y https://m.youtube.com/watch?v=5QuwPeH9P7Y When all you have is a DAG all solutions end up with sales person coming at your door problem. But years down the line we didn’t have traveling salesman problem but out of date libraries problem. The monkeys tried to make their risk taking more safe by introducing all kinda of random constraints not realising that what gave power to the whole idea was actually risk taking. And no, they didn’t try to manage risk but have completely fail to understand that risk is what made product good in the first place. And this is how every great idea in human history fell apart, the original people had different view of a current problem however the following generations have only understood the simplified problem and down the line the the solution to original problem had became the actual problem. I still believe the poem would make greater justice to the whole idea, but what I tried to explain here is that the simple things we have all witnessed actually hide much deeper truths about life in general. And from there I assume QzDesktop loading times where not the issue at the beginning of this deterministic chaotic system, but simply one of the possible generational products. I am yet to understand how to solve the monkey issue.
• smallnamespace 5y ago
  Armen was very passionate about the value of "strats" (GS's own term for "quants", and later broadened to include software engineers and data scientists). A favorite quip of his: At GS, I'm like an arms dealer. When a desk has a problem, I send in the strats, and they blow away all the competition! Also, SecDB's core idea is not just tight integration between the backend and development environment, but that all objects ("Security" in SecDB lingo) were functionally reactive. For example, you would write a pure function that defines: Price(Security) = F(stock price, interest rate, ...) When the market inputs changed, Price(Security) would automatically update (the framework handled the fiddly bits of caching intermediate values for you, so even an expensive Price function is not problematic). This is loosely the same idea that drives React, ObservableHQ, Kafka, and other event-streaming architectures, but I first encountered this ~15 years ago at a bank.
  • barrkel 5y ago
    It's as old as VisiCalc, it's how spreadsheets work. I built a similarly reactive system for web UI binding back in 2004, running binding expressions on the back end with cached UI state to compute the minimal update to return to the front end, in the form of attributes to set on named elements.
    • AmericanBlarney 5y ago
      Yes, although doing it in distributed fashion at the scale of SecDB or Athena introduces quite a few more complexities.
  • jb_s 5y ago
    This immediately struck me when I was reading this article. To be honest, this whole paradigm seems absurdly fucking efficient for the developers. But I wonder about stuff like * What happens if the data model needs to change? If you need to move something from db["some/path"]? * How is it coordinated at a larger scale, how does everyone know what is running and how it interacts with everything else - can you figure out what depends on an object? What if the data used by your Price(Security) object changes and breaks it?
• shrike21 5y ago
  Hello Sean, I remember Slang when I first saw the code, a parse tree based evaluator in 1997. Come on folks. Separate parsing from evaluation. Opaque types with message passing. Inference anyone? Clearly no one read hindly milner. Add parse time optimizations, add locals, hey globals and locals weren't handled properly. Python in the 90s anyone? Shitty KV store with 32K object limits related to some random OS2 btree limits. Add huge pages. Deal with random rubbish related to inconsistent non transactional indices. Figure out you should layer nodes in a dag. A dag is topologically 2 dimensions, fairly limiting. Figure out that's somewhat similar to layering databases, it's just another dimension. Hmm bottom up evaluators, you actually need top down as well, create a turing complete specification. well, limit it a bit. Ah KLL points out that layers on top of dimensions end up being combinatorial, but you can actually cache the meta traversal and it's small n. Lots of people point out category theory parallels. Haskell is pretty but completely unusable. I'm a math guy, and it's still unusable, I don't like feeling smart when I do simple things. But interestingly creating imperative functions with pure input/outputs with implied function calls is pretty interesting. You can create an OOP paradigm with self and args as known roots aka linda tuple spaces. Ah and each tuple space point can be scheduled independently, some issues with serialization and locality... Go to another bank and choose to use python, foolishly decide to rely on pickle. Do that twice. Bad idea. But write a much better petabyte scale multi-master optimistic locking database with 4 operations. Insert, Upsert, Update, delete. WAL as a first class API. Finally decide that writing a coding scheme to convert python objects to json is not really hard. And of course cloud native and cloud agnostic is really the only way to go nowadays. I'm always confused why people complain about Athena/Quartz, hell we wrote it all, fix it if you don't like it. Open source it if you want other people to contribute. If we made stupid decisions on pickling data, well there's a version id, add json serialization, it's not hard, don't take things as given.
  • rbanffy 5y ago
    > foolishly decide to rely on pickle. This scared me a bit TBH. It’s one of those decisions that come back to bite us repeatedly.
• DrBazza 5y ago
  There was no single person that introduced python at Citigroup that I am aware of. It came in via a variety of teams mostly because of the fact that the alternative was perl, and no one wanted to write perl (yet somehow kdb was acceptable a few years later).