7 ms·

As the comment by timsworkaccount says, this is definitely Athena at JPMorgan. For those interested, I gave a talk on Athena at PyData UK 2018 called "Python a

As the comment by timsworkaccount says, this is definitely Athena at JPMorgan. For those interested, I gave a talk on Athena at PyData UK 2018 called "Python at Massive Scale". 4500 developers making 20k commits a week. Codebase with 35m LOC. The video is here: https://www.youtube.com/watch?v=ZYD9yyMh9Hk https://www.youtube.com/watch?v=ZYD9yyMh9Hk It covers Athena's origins, what it is used for, application architecture, infrastructure, dev tooling and culture.

silverlake 6y ago
IIRC the folks who worked on SecDB at GS bounced around Wall St. recreating this abomination in Python.
cdavid 6y ago
I worked on that project 10 years ago as a consultant, and it was certainly strategic at that time. It was fairly well known that one of the main stakeholder was pushing for those system and making a career out of it by jumping bank every few years. I understand the value of it, but as an experienced scientific developer and python dev, the culture shock was huge. I don't have fond memory working on it. It is very different from traditional programming environment, closer to a kind of reimplementation of smalltalk env w/ python. I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. One of the fancy thing was integrating reactive programming, which worked through ugly hacks, at least at that time, by parsing code to detect dependencies. IIRC, it could manage list comprehension but not loops. They also had their own python binary w/ both python2 and python3 in one process.
- stevesimmons 6y ago
  The culture shock you mention was very real. I joined in 2010 (when Athena was 3 years old) and left 8 years later in 2018. I remember my first few months unlearning normal Python and figuring out how to build the 'pixie graph', a lazily-evaluated Python DAG suited for calculating financial instruments. It took a while to get your head around this, but when you did it was a very powerful and productive way to building trading and risk management applications. To get some sense in how this worked, here are two public projects on github with good introductions: * https://github.com/timkpaine/tributary https://github.com/timkpaine/tributary * https://github.com/janushendersonassetallocation/loman https://github.com/janushendersonassetallocation/loman
  - doublesCs 6y ago
    Hi Steve, thanks for your talk and for the links. Are those repos pixie dependencies? Or are you saying that pixie works like those repos?
    - stevesimmons 6y ago
      They are "toy" projects illustrative of the general concepts. ("toy" as in smaller in scope; one of the two is by a former colleague). JPM's actual pixie code is proprietary, extremely performant after 12 years of pushing into bigger and bigger scale problems, and is definitely not on GitHub!
  - timsworkaccount 6y ago
    I really like this video for explaining the DAG - https://www.youtube.com/watch?v=lTOP_shhVBQ https://www.youtube.com/watch?v=lTOP_shhVBQ
  - FabHK 6y ago
    Sounds like a descendant of Goldman Sachs‘s SecDB/Slang with automatic dependency graph building. Did it have purple children (nodes whose value influences the graph structure) and twiddle scopes (modified copies of subtrees)? :-)
    - stevesimmons 6y ago
      Yes indeed. Quite a few of Athena's early developers were from GS. I've not heard the term "twiddle scopes" before; in Athena it is "tweaking".
      - FabHK 6y ago
        Come to think of it, it might have been a “diddle scope” (I’m not a native speaker). For finance applications (risk) the whole concept works quite well, doesn’t it - combining the natural advantages of code and Excel.
- james412 6y ago
  > python2 and python3 in one process This would have had me running screaming from the building
  - exikyut 6y ago
    I stared at my screen with deer-headlight eyes for half a second while hearing screaming in my head too. AAAAaaaaaa
- deleted 6y ago
  [deleted]
- zeveb 6y ago
  > I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. Imagine the benefit to the business had they just stuck with Smalltalk the whole time! It is remarkable how many businesses could have been quietly making money for years using a high-quality dynamic language such as Smalltalk or Lisp rather than going through the whole C/C++/Java/Python/Ruby/JavaScript/Go treadmill. But I guess if they had done that then their programmers might not have gotten to type as many {}! Nor gotten to reinvent the wheel so often.
  - lucozade 6y ago
    Kapital (the Smalltalk system) wasn't much of an influence on Athena. Athena was very much a SecDB/Slang derived effort. And Kapital was an interesting experiment but rightfully died. Smalltalk just wasn't the right environment for something that complex and scale. I'd expect most lisps to suffer in the same way. And TBH it was more the image based model rather than the dynamic language per se though that was an issue. There's a point where dynamism becomes a liability and Kapital went way beyond that point. Python fares a bit better but still, a huge effort is needed to control its failings at scale. This isn't helped at all by sticking a DAG in the middle of it as was done with Athena and its ilk. This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. Source: I suffered through it.
    - FabHK 6y ago
      > This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. I had to LOL. You have very nicely put into words an experience I bet many have had.
    - JackFr 6y ago
      Kapital was also rewritten in Java and survived in a few successor places ('Derivatives Studio' at CSFB). There it was my introduction to 'thunks'.
    - eru 6y ago
      > I'd expect most lisps to suffer in the same way. Why? Eg Scheme or even the uglier Common Lisp are quite disciplined. And Clojure ain't too bad either. As far as I can tell, Lispers don't usually go around monkey-patching things.
- goatinaboat 6y ago
  I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. That system you're thinking of was Kapital. http://www.cincomsmalltalk.com/main/successes/financial-services/jpmorgan/ http://www.cincomsmalltalk.com/main/successes/financial-serv... It's the only Smalltalk usage at JPM I believe, they could have doubled-down on it but instead went to Python and as you say, tried reinventing the wheel in it.
  - kbenson 6y ago
    > tried reinventing the wheel in it. That's not reinventing the wheel, that's building a new wheel you hope performs better/different with different materials and maybe some different techniques, and has a long and storied history of both successes and failures. Reinventing the wheel would be if they published this system and some other bank recreated it without using what they published, thus "inventing" something that already existed and was available.
  - CraigJPerry 6y ago
    I never realised Kapital was written in smalltalk. Interesting
    - scroot 6y ago
      Apparently when it booted in the terminal, there was a nice ASCII portrait of Marx. I also have it on good authority that the program that launched the Smalltalk image from the terminal was called "das," so you literally would type "das kapital" to get the thing going.
- FabHK 6y ago
  > IIRC, it could manage list comprehension but not loops. If it was like the predecessor system I worked with, it could handle either. But in a list comprehension, it could figure out the individual dependencies of the individual list items. So, suppose you had a a function that took in a list, and returned a list with some computations performed on each element. If done with list comprehension, and one input element changed, only the corresponding output element would need to be recomputed, and only that would be recomputed. If done with a loop, and one input element changed, the entire output array would be recomputed (even if all but one element remained unchanged - it would be hard to guarantee that property by code inspection at the time the dependency graph is built).
drglitch 6y ago
Hi Steve, nice to see you on here and doing well! :) My first thought was Athena as well, but it could be Quartz (it started in 08-09 at BaML, vs Athena starting in 2006). Both Citi and Barcap also tried (albeit to less success than JPM/BaML) to do similar things. With that said, For those unfamiliar - at least two major banks run insanely large production python installs across thousands of developers. Core skills (nowadays) are python and react - and yes, we’re all actively hiring :) All of the large banks also have pretty good open source initiatives - just check out github repos for JPMorgan, BaML, or GS. We have also been active participants in Pycon (key sponsors and doing sessions, like Steve’s) since at least 2009.
- stevesimmons 6y ago
  This thread is a real trip down memory lane! I joined JPMorgan in 2010, as the first bank in London to offer me a Python role. I ended up staying for 8 years, almost all of it working in Athena: Commodities and FX trading; a bit of Equities; then with the Athena Core team working on the machine learning environment. I learned a ton, worked with many hugely impressive people - on both the trading and technology sides of the business - and left with lots of good memories.
  - tomrod 6y ago
    I remember interacting with you at one point. Time flies!
- 4gotunameagain 6y ago
  A serious question, how do you feel about working at such a powerful bank that has been involved in so many controversies, to put it lightly? It is not a far-fetched argument that institutes like that are only a burden to society, existing only to widen socio-economical gaps. The point of view of an insider would be very interesting I think.
  - Infinitesimus 6y ago
    Do you believe banks are a net negative to society? Or more about institutions that partake in trading and the like?
eru 6y ago
And wasn't Athena JPMorgan's attempt at re-creating something like Slang and SecDB?
- decafninja 6y ago
  Yes. If I'm not mistaken it was from the same guy (and team?) that created Slang/SecDB at Goldman? Same team subsequently moved onto Bank of America to create Quartz, which is BoA's version of the same concept. Then subsequently moved on to found Washington Square Technologies, where I think they're doing something similar, but not attached to a specific bank? Not 100% sure. I've heard the concept was shopped around to a few other banks as well, but did not take root.
fancyfredbot 6y ago
This is really quite unlikely to be Athena, which isn't scheduled to complete python 3 migration until Q4 2020 according to: https://www.techrepublic.com/article/jpmorgans-athena-has-35-million-lines-of-python-code-and-wont-be-updated-to-python-3-in-time/ https://www.techrepublic.com/article/jpmorgans-athena-has-35... The update isn't a job for one person, and I don't think it is finished yet, unless it is ahead of this alleged schedule...
- testnet 6y ago
  This is talking about "Athena Server Pages" which is a web framework within Athena.
  - user5994461 6y ago
    touché
  - fancyfredbot 6y ago
    Ah! Not sure how you figured that out, but it would make sense and explain the "legacy" comments (Athena is not legacy as far as I know but a component like that could be).
    - decafninja 6y ago
      As someone who has worked with these Python-based bank frameworks on the UI side, the UI frameworks leaves a lot to be desired. Lots of reinventing the wheel (resulting in an inferior wheel). Thankfully, both JPM and BAML seem to have started to ditch the UI frameworks, at least on the web side (in favor of React mostly). I don't know what the situation is on the desktop UI side. But I know what they had before was loathed by many. Developers hated it because it was like working with one hand tied behind your back. Traders hated it because developers couldn't deliver the best product because they were working with one hand tied behind their back. I presume the core teams responsible for building the framework hated it because trying to get support for the proprietary UI tools was painful. I imagine hiring managers and HR hated it because they would blatantly lie about the job - that the role would be WinForms/WPF (desktop) or JavaScript (web) UI development, without mentioning that all of that was really wrapped up by the proprietary Python framework, and very rarely would you get to touch the underlying industry standard tech stacks.
beagle3 6y ago
From the discussion here, sounds like this is Python version of K's (built in) trigger + dependency mechanism, which traces its origins to A+ and A ; A and A+ were originally developed inside Morgan Stanley by Arthur Whitney, and K was developed for a bank client IIRC, (UBS? or MS?)
- appleflaxen 6y ago
  This sounds really interesting; can you elaborate on it at all?
  - beagle3 6y ago
    K4 uses : for assignment, and :: for dependency, so: c::a+b means c is defined to be the sum of a and b and dependent on both. If either a or b changes, c will be marked stale, and the next time you use it, it will be recomputed. It's essentially as simple as that, and is unified with database views - e.g. total::select sum(balance) from accounts means that any insert/update to the "accounts" table will mark "total" as stale, and the next use will recompute it, but otherwise it behaves like any other variable. I'm not sure if the K4 engine is smart enough to avoid recompute on updates to other columns. If you want more fine grained control, you can set up triggers - e.g. "execute this code on update to variable a" - where your code gets the changed indices of a list/vector (or changed tuples of a table) as arguments; this is useful if a plain complete recompute is too costly, or you need to do something at the time of change, rather than at the next evaluation. K2 and A+ had a similar system, each with different syntax.
retrobox 6y ago
I thought it sounded like Athena. It’s the best CI/CD I’ve ever used despite its quirks. I talk fondly of it at future jobs I’ve gone to
dehrmann 6y ago
I've lost count of the number of internal projects I've seen named after mythological figures.
- goatinaboat 6y ago
  I've lost count of the number of internal projects I've seen named after mythological figures. The Athena team ran out of names and one major component of it is called Bob.
  - euroclear 6y ago
    Is it the job scheduler?

silverlake 6y ago

IIRC the folks who worked on SecDB at GS bounced around Wall St. recreating this abomination in Python.

cdavid 6y ago

I worked on that project 10 years ago as a consultant, and it was certainly strategic at that time. It was fairly well known that one of the main stakeholder was pushing for those system and making a career out of it by jumping bank every few years. I understand the value of it, but as an experienced scientific developer and python dev, the culture shock was huge. I don't have fond memory working on it. It is very different from traditional programming environment, closer to a kind of reimplementation of smalltalk env w/ python. I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. One of the fancy thing was integrating reactive programming, which worked through ugly hacks, at least at that time, by parsing code to detect dependencies. IIRC, it could manage list comprehension but not loops. They also had their own python binary w/ both python2 and python3 in one process.

stevesimmons 6y ago
The culture shock you mention was very real. I joined in 2010 (when Athena was 3 years old) and left 8 years later in 2018. I remember my first few months unlearning normal Python and figuring out how to build the 'pixie graph', a lazily-evaluated Python DAG suited for calculating financial instruments. It took a while to get your head around this, but when you did it was a very powerful and productive way to building trading and risk management applications. To get some sense in how this worked, here are two public projects on github with good introductions: * https://github.com/timkpaine/tributary https://github.com/timkpaine/tributary * https://github.com/janushendersonassetallocation/loman https://github.com/janushendersonassetallocation/loman
- doublesCs 6y ago
  Hi Steve, thanks for your talk and for the links. Are those repos pixie dependencies? Or are you saying that pixie works like those repos?
  - stevesimmons 6y ago
    They are "toy" projects illustrative of the general concepts. ("toy" as in smaller in scope; one of the two is by a former colleague). JPM's actual pixie code is proprietary, extremely performant after 12 years of pushing into bigger and bigger scale problems, and is definitely not on GitHub!
- timsworkaccount 6y ago
  I really like this video for explaining the DAG - https://www.youtube.com/watch?v=lTOP_shhVBQ https://www.youtube.com/watch?v=lTOP_shhVBQ
- FabHK 6y ago
  Sounds like a descendant of Goldman Sachs‘s SecDB/Slang with automatic dependency graph building. Did it have purple children (nodes whose value influences the graph structure) and twiddle scopes (modified copies of subtrees)? :-)
  - stevesimmons 6y ago
    Yes indeed. Quite a few of Athena's early developers were from GS. I've not heard the term "twiddle scopes" before; in Athena it is "tweaking".
    - FabHK 6y ago
      Come to think of it, it might have been a “diddle scope” (I’m not a native speaker). For finance applications (risk) the whole concept works quite well, doesn’t it - combining the natural advantages of code and Excel.
james412 6y ago
> python2 and python3 in one process This would have had me running screaming from the building
- exikyut 6y ago
  I stared at my screen with deer-headlight eyes for half a second while hearing screaming in my head too. AAAAaaaaaa
deleted 6y ago
[deleted]
zeveb 6y ago
> I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. Imagine the benefit to the business had they just stuck with Smalltalk the whole time! It is remarkable how many businesses could have been quietly making money for years using a high-quality dynamic language such as Smalltalk or Lisp rather than going through the whole C/C++/Java/Python/Ruby/JavaScript/Go treadmill. But I guess if they had done that then their programmers might not have gotten to type as many {}! Nor gotten to reinvent the wheel so often.
- lucozade 6y ago
  Kapital (the Smalltalk system) wasn't much of an influence on Athena. Athena was very much a SecDB/Slang derived effort. And Kapital was an interesting experiment but rightfully died. Smalltalk just wasn't the right environment for something that complex and scale. I'd expect most lisps to suffer in the same way. And TBH it was more the image based model rather than the dynamic language per se though that was an issue. There's a point where dynamism becomes a liability and Kapital went way beyond that point. Python fares a bit better but still, a huge effort is needed to control its failings at scale. This isn't helped at all by sticking a DAG in the middle of it as was done with Athena and its ilk. This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. Source: I suffered through it.
  - FabHK 6y ago
    > This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. I had to LOL. You have very nicely put into words an experience I bet many have had.
  - JackFr 6y ago
    Kapital was also rewritten in Java and survived in a few successor places ('Derivatives Studio' at CSFB). There it was my introduction to 'thunks'.
  - eru 6y ago
    > I'd expect most lisps to suffer in the same way. Why? Eg Scheme or even the uglier Common Lisp are quite disciplined. And Clojure ain't too bad either. As far as I can tell, Lispers don't usually go around monkey-patching things.
goatinaboat 6y ago
I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. That system you're thinking of was Kapital. http://www.cincomsmalltalk.com/main/successes/financial-services/jpmorgan/ http://www.cincomsmalltalk.com/main/successes/financial-serv... It's the only Smalltalk usage at JPM I believe, they could have doubled-down on it but instead went to Python and as you say, tried reinventing the wheel in it.
- kbenson 6y ago
  > tried reinventing the wheel in it. That's not reinventing the wheel, that's building a new wheel you hope performs better/different with different materials and maybe some different techniques, and has a long and storied history of both successes and failures. Reinventing the wheel would be if they published this system and some other bank recreated it without using what they published, thus "inventing" something that already existed and was available.
- CraigJPerry 6y ago
  I never realised Kapital was written in smalltalk. Interesting
  - scroot 6y ago
    Apparently when it booted in the terminal, there was a nice ASCII portrait of Marx. I also have it on good authority that the program that launched the Smalltalk image from the terminal was called "das," so you literally would type "das kapital" to get the thing going.
FabHK 6y ago
> IIRC, it could manage list comprehension but not loops. If it was like the predecessor system I worked with, it could handle either. But in a list comprehension, it could figure out the individual dependencies of the individual list items. So, suppose you had a a function that took in a list, and returned a list with some computations performed on each element. If done with list comprehension, and one input element changed, only the corresponding output element would need to be recomputed, and only that would be recomputed. If done with a loop, and one input element changed, the entire output array would be recomputed (even if all but one element remained unchanged - it would be hard to guarantee that property by code inspection at the time the dependency graph is built).

drglitch 6y ago

Hi Steve, nice to see you on here and doing well! :) My first thought was Athena as well, but it could be Quartz (it started in 08-09 at BaML, vs Athena starting in 2006). Both Citi and Barcap also tried (albeit to less success than JPM/BaML) to do similar things. With that said, For those unfamiliar - at least two major banks run insanely large production python installs across thousands of developers. Core skills (nowadays) are python and react - and yes, we’re all actively hiring :) All of the large banks also have pretty good open source initiatives - just check out github repos for JPMorgan, BaML, or GS. We have also been active participants in Pycon (key sponsors and doing sessions, like Steve’s) since at least 2009.

stevesimmons 6y ago
This thread is a real trip down memory lane! I joined JPMorgan in 2010, as the first bank in London to offer me a Python role. I ended up staying for 8 years, almost all of it working in Athena: Commodities and FX trading; a bit of Equities; then with the Athena Core team working on the machine learning environment. I learned a ton, worked with many hugely impressive people - on both the trading and technology sides of the business - and left with lots of good memories.
- tomrod 6y ago
  I remember interacting with you at one point. Time flies!
4gotunameagain 6y ago
A serious question, how do you feel about working at such a powerful bank that has been involved in so many controversies, to put it lightly? It is not a far-fetched argument that institutes like that are only a burden to society, existing only to widen socio-economical gaps. The point of view of an insider would be very interesting I think.
- Infinitesimus 6y ago
  Do you believe banks are a net negative to society? Or more about institutions that partake in trading and the like?

eru 6y ago

And wasn't Athena JPMorgan's attempt at re-creating something like Slang and SecDB?

decafninja 6y ago
Yes. If I'm not mistaken it was from the same guy (and team?) that created Slang/SecDB at Goldman? Same team subsequently moved onto Bank of America to create Quartz, which is BoA's version of the same concept. Then subsequently moved on to found Washington Square Technologies, where I think they're doing something similar, but not attached to a specific bank? Not 100% sure. I've heard the concept was shopped around to a few other banks as well, but did not take root.

fancyfredbot 6y ago

This is really quite unlikely to be Athena, which isn't scheduled to complete python 3 migration until Q4 2020 according to: https://www.techrepublic.com/article/jpmorgans-athena-has-35-million-lines-of-python-code-and-wont-be-updated-to-python-3-in-time/ https://www.techrepublic.com/article/jpmorgans-athena-has-35... The update isn't a job for one person, and I don't think it is finished yet, unless it is ahead of this alleged schedule...

testnet 6y ago
This is talking about "Athena Server Pages" which is a web framework within Athena.
- user5994461 6y ago
  touché
- fancyfredbot 6y ago
  Ah! Not sure how you figured that out, but it would make sense and explain the "legacy" comments (Athena is not legacy as far as I know but a component like that could be).
  - decafninja 6y ago
    As someone who has worked with these Python-based bank frameworks on the UI side, the UI frameworks leaves a lot to be desired. Lots of reinventing the wheel (resulting in an inferior wheel). Thankfully, both JPM and BAML seem to have started to ditch the UI frameworks, at least on the web side (in favor of React mostly). I don't know what the situation is on the desktop UI side. But I know what they had before was loathed by many. Developers hated it because it was like working with one hand tied behind your back. Traders hated it because developers couldn't deliver the best product because they were working with one hand tied behind their back. I presume the core teams responsible for building the framework hated it because trying to get support for the proprietary UI tools was painful. I imagine hiring managers and HR hated it because they would blatantly lie about the job - that the role would be WinForms/WPF (desktop) or JavaScript (web) UI development, without mentioning that all of that was really wrapped up by the proprietary Python framework, and very rarely would you get to touch the underlying industry standard tech stacks.

beagle3 6y ago

From the discussion here, sounds like this is Python version of K's (built in) trigger + dependency mechanism, which traces its origins to A+ and A ; A and A+ were originally developed inside Morgan Stanley by Arthur Whitney, and K was developed for a bank client IIRC, (UBS? or MS?)

appleflaxen 6y ago
This sounds really interesting; can you elaborate on it at all?
- beagle3 6y ago
  K4 uses : for assignment, and :: for dependency, so: c::a+b means c is defined to be the sum of a and b and dependent on both. If either a or b changes, c will be marked stale, and the next time you use it, it will be recomputed. It's essentially as simple as that, and is unified with database views - e.g. total::select sum(balance) from accounts means that any insert/update to the "accounts" table will mark "total" as stale, and the next use will recompute it, but otherwise it behaves like any other variable. I'm not sure if the K4 engine is smart enough to avoid recompute on updates to other columns. If you want more fine grained control, you can set up triggers - e.g. "execute this code on update to variable a" - where your code gets the changed indices of a list/vector (or changed tuples of a table) as arguments; this is useful if a plain complete recompute is too costly, or you need to do something at the time of change, rather than at the next evaluation. K2 and A+ had a similar system, each with different syntax.

retrobox 6y ago

I thought it sounded like Athena. It’s the best CI/CD I’ve ever used despite its quirks. I talk fondly of it at future jobs I’ve gone to

dehrmann 6y ago

I've lost count of the number of internal projects I've seen named after mythological figures.

goatinaboat 6y ago
I've lost count of the number of internal projects I've seen named after mythological figures. The Athena team ran out of names and one major component of it is called Bob.
- euroclear 6y ago
  Is it the job scheduler?