7 ms·

I worked on that project 10 years ago as a consultant, and it was certainly strategic at that time. It was fairly well known that one of the main stakeholder wa

by cdavid 6y ago

I worked on that project 10 years ago as a consultant, and it was certainly strategic at that time. It was fairly well known that one of the main stakeholder was pushing for those system and making a career out of it by jumping bank every few years. I understand the value of it, but as an experienced scientific developer and python dev, the culture shock was huge. I don't have fond memory working on it. It is very different from traditional programming environment, closer to a kind of reimplementation of smalltalk env w/ python. I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. One of the fancy thing was integrating reactive programming, which worked through ugly hacks, at least at that time, by parsing code to detect dependencies. IIRC, it could manage list comprehension but not loops. They also had their own python binary w/ both python2 and python3 in one process.

stevesimmons 6y ago
The culture shock you mention was very real. I joined in 2010 (when Athena was 3 years old) and left 8 years later in 2018. I remember my first few months unlearning normal Python and figuring out how to build the 'pixie graph', a lazily-evaluated Python DAG suited for calculating financial instruments. It took a while to get your head around this, but when you did it was a very powerful and productive way to building trading and risk management applications. To get some sense in how this worked, here are two public projects on github with good introductions: * https://github.com/timkpaine/tributary https://github.com/timkpaine/tributary * https://github.com/janushendersonassetallocation/loman https://github.com/janushendersonassetallocation/loman
- doublesCs 6y ago
  Hi Steve, thanks for your talk and for the links. Are those repos pixie dependencies? Or are you saying that pixie works like those repos?
  - stevesimmons 6y ago
    They are "toy" projects illustrative of the general concepts. ("toy" as in smaller in scope; one of the two is by a former colleague). JPM's actual pixie code is proprietary, extremely performant after 12 years of pushing into bigger and bigger scale problems, and is definitely not on GitHub!
- timsworkaccount 6y ago
  I really like this video for explaining the DAG - https://www.youtube.com/watch?v=lTOP_shhVBQ https://www.youtube.com/watch?v=lTOP_shhVBQ
- FabHK 6y ago
  Sounds like a descendant of Goldman Sachs‘s SecDB/Slang with automatic dependency graph building. Did it have purple children (nodes whose value influences the graph structure) and twiddle scopes (modified copies of subtrees)? :-)
  - stevesimmons 6y ago
    Yes indeed. Quite a few of Athena's early developers were from GS. I've not heard the term "twiddle scopes" before; in Athena it is "tweaking".
    - FabHK 6y ago
      Come to think of it, it might have been a “diddle scope” (I’m not a native speaker). For finance applications (risk) the whole concept works quite well, doesn’t it - combining the natural advantages of code and Excel.
      - adamdonahue 6y ago
        It was indeed "diddle," not "twiddle." At BAML, it's a "tweak." Beacon Platform is another implementation by the same team (but with support for the public cloud, and many more advanced features, I believe, including tighter web integration). I think it uses the terminology "bind." [Disclosure: I was part of the Quartz Core team in 2011/2012.]
  - eru 6y ago
    I was positively surprised that Slang was actually quite usable. I had expected much worse. I suspect the Common Lisp influence was beneficial. The weirdest thing, language-wise, that I noticed was scoping. It's neither dynamic nor static scoping, but weird scoping. (But there are work-around to get something like static scoping.) Outside of the language, the whole CVS-based version control and review process was weird. But understandable as a product of the late 1990s, when review-before-going-into-permanent-history must have been way ahead of its time.
    - 6y ago
- infinite8s 6y ago
  Man AHL also had a version that worked on top of nodes representing timeseries: https://github.com/man-group/mdf https://github.com/man-group/mdf
- edparcell 6y ago
  Loman author here. Thank you very much for the mention. Amazed that I never heard of Athena or pixie graphs. Our intention with Loman was to create a library scoped for a single process - we looked at the possibility of creating a system responsible for executing much larger graphs on a real-time ongoing basis, but it felt like a larger project than we'd be able to execute well. It sounds like Athena was that, and it worked well, subject to being a culture shock for people coming into it?
  - infinite8s 6y ago
    A similar library from another asset manager - https://github.com/man-group/mdf https://github.com/man-group/mdf. Although MDF seemed to work at the level of timeseries instead of scalar values.
james412 6y ago
> python2 and python3 in one process This would have had me running screaming from the building
- exikyut 6y ago
  I stared at my screen with deer-headlight eyes for half a second while hearing screaming in my head too. AAAAaaaaaa
deleted 6y ago
[deleted]
zeveb 6y ago
> I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. Imagine the benefit to the business had they just stuck with Smalltalk the whole time! It is remarkable how many businesses could have been quietly making money for years using a high-quality dynamic language such as Smalltalk or Lisp rather than going through the whole C/C++/Java/Python/Ruby/JavaScript/Go treadmill. But I guess if they had done that then their programmers might not have gotten to type as many {}! Nor gotten to reinvent the wheel so often.
- lucozade 6y ago
  Kapital (the Smalltalk system) wasn't much of an influence on Athena. Athena was very much a SecDB/Slang derived effort. And Kapital was an interesting experiment but rightfully died. Smalltalk just wasn't the right environment for something that complex and scale. I'd expect most lisps to suffer in the same way. And TBH it was more the image based model rather than the dynamic language per se though that was an issue. There's a point where dynamism becomes a liability and Kapital went way beyond that point. Python fares a bit better but still, a huge effort is needed to control its failings at scale. This isn't helped at all by sticking a DAG in the middle of it as was done with Athena and its ilk. This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. Source: I suffered through it.
  - FabHK 6y ago
    > This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. I had to LOL. You have very nicely put into words an experience I bet many have had.
  - JackFr 6y ago
    Kapital was also rewritten in Java and survived in a few successor places ('Derivatives Studio' at CSFB). There it was my introduction to 'thunks'.
  - eru 6y ago
    > I'd expect most lisps to suffer in the same way. Why? Eg Scheme or even the uglier Common Lisp are quite disciplined. And Clojure ain't too bad either. As far as I can tell, Lispers don't usually go around monkey-patching things.
goatinaboat 6y ago
I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. That system you're thinking of was Kapital. http://www.cincomsmalltalk.com/main/successes/financial-services/jpmorgan/ http://www.cincomsmalltalk.com/main/successes/financial-serv... It's the only Smalltalk usage at JPM I believe, they could have doubled-down on it but instead went to Python and as you say, tried reinventing the wheel in it.
- kbenson 6y ago
  > tried reinventing the wheel in it. That's not reinventing the wheel, that's building a new wheel you hope performs better/different with different materials and maybe some different techniques, and has a long and storied history of both successes and failures. Reinventing the wheel would be if they published this system and some other bank recreated it without using what they published, thus "inventing" something that already existed and was available.
- CraigJPerry 6y ago
  I never realised Kapital was written in smalltalk. Interesting
  - scroot 6y ago
    Apparently when it booted in the terminal, there was a nice ASCII portrait of Marx. I also have it on good authority that the program that launched the Smalltalk image from the terminal was called "das," so you literally would type "das kapital" to get the thing going.
FabHK 6y ago
> IIRC, it could manage list comprehension but not loops. If it was like the predecessor system I worked with, it could handle either. But in a list comprehension, it could figure out the individual dependencies of the individual list items. So, suppose you had a a function that took in a list, and returned a list with some computations performed on each element. If done with list comprehension, and one input element changed, only the corresponding output element would need to be recomputed, and only that would be recomputed. If done with a loop, and one input element changed, the entire output array would be recomputed (even if all but one element remained unchanged - it would be hard to guarantee that property by code inspection at the time the dependency graph is built).

by cdavid 6y ago

stevesimmons 6y ago

The culture shock you mention was very real. I joined in 2010 (when Athena was 3 years old) and left 8 years later in 2018. I remember my first few months unlearning normal Python and figuring out how to build the 'pixie graph', a lazily-evaluated Python DAG suited for calculating financial instruments. It took a while to get your head around this, but when you did it was a very powerful and productive way to building trading and risk management applications. To get some sense in how this worked, here are two public projects on github with good introductions: * https://github.com/timkpaine/tributary https://github.com/timkpaine/tributary * https://github.com/janushendersonassetallocation/loman https://github.com/janushendersonassetallocation/loman

doublesCs 6y ago
Hi Steve, thanks for your talk and for the links. Are those repos pixie dependencies? Or are you saying that pixie works like those repos?
- stevesimmons 6y ago
  They are "toy" projects illustrative of the general concepts. ("toy" as in smaller in scope; one of the two is by a former colleague). JPM's actual pixie code is proprietary, extremely performant after 12 years of pushing into bigger and bigger scale problems, and is definitely not on GitHub!
timsworkaccount 6y ago
I really like this video for explaining the DAG - https://www.youtube.com/watch?v=lTOP_shhVBQ https://www.youtube.com/watch?v=lTOP_shhVBQ
FabHK 6y ago
Sounds like a descendant of Goldman Sachs‘s SecDB/Slang with automatic dependency graph building. Did it have purple children (nodes whose value influences the graph structure) and twiddle scopes (modified copies of subtrees)? :-)
- stevesimmons 6y ago
  Yes indeed. Quite a few of Athena's early developers were from GS. I've not heard the term "twiddle scopes" before; in Athena it is "tweaking".
  - FabHK 6y ago
    Come to think of it, it might have been a “diddle scope” (I’m not a native speaker). For finance applications (risk) the whole concept works quite well, doesn’t it - combining the natural advantages of code and Excel.
    - adamdonahue 6y ago
      It was indeed "diddle," not "twiddle." At BAML, it's a "tweak." Beacon Platform is another implementation by the same team (but with support for the public cloud, and many more advanced features, I believe, including tighter web integration). I think it uses the terminology "bind." [Disclosure: I was part of the Quartz Core team in 2011/2012.]
- eru 6y ago
  I was positively surprised that Slang was actually quite usable. I had expected much worse. I suspect the Common Lisp influence was beneficial. The weirdest thing, language-wise, that I noticed was scoping. It's neither dynamic nor static scoping, but weird scoping. (But there are work-around to get something like static scoping.) Outside of the language, the whole CVS-based version control and review process was weird. But understandable as a product of the late 1990s, when review-before-going-into-permanent-history must have been way ahead of its time.
  - 6y ago
infinite8s 6y ago
Man AHL also had a version that worked on top of nodes representing timeseries: https://github.com/man-group/mdf https://github.com/man-group/mdf
edparcell 6y ago
Loman author here. Thank you very much for the mention. Amazed that I never heard of Athena or pixie graphs. Our intention with Loman was to create a library scoped for a single process - we looked at the possibility of creating a system responsible for executing much larger graphs on a real-time ongoing basis, but it felt like a larger project than we'd be able to execute well. It sounds like Athena was that, and it worked well, subject to being a culture shock for people coming into it?
- infinite8s 6y ago
  A similar library from another asset manager - https://github.com/man-group/mdf https://github.com/man-group/mdf. Although MDF seemed to work at the level of timeseries instead of scalar values.

james412 6y ago

> python2 and python3 in one process This would have had me running screaming from the building

exikyut 6y ago
I stared at my screen with deer-headlight eyes for half a second while hearing screaming in my head too. AAAAaaaaaa

deleted 6y ago

[deleted]

zeveb 6y ago

> I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. Imagine the benefit to the business had they just stuck with Smalltalk the whole time! It is remarkable how many businesses could have been quietly making money for years using a high-quality dynamic language such as Smalltalk or Lisp rather than going through the whole C/C++/Java/Python/Ruby/JavaScript/Go treadmill. But I guess if they had done that then their programmers might not have gotten to type as many {}! Nor gotten to reinvent the wheel so often.

lucozade 6y ago
Kapital (the Smalltalk system) wasn't much of an influence on Athena. Athena was very much a SecDB/Slang derived effort. And Kapital was an interesting experiment but rightfully died. Smalltalk just wasn't the right environment for something that complex and scale. I'd expect most lisps to suffer in the same way. And TBH it was more the image based model rather than the dynamic language per se though that was an issue. There's a point where dynamism becomes a liability and Kapital went way beyond that point. Python fares a bit better but still, a huge effort is needed to control its failings at scale. This isn't helped at all by sticking a DAG in the middle of it as was done with Athena and its ilk. This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. Source: I suffered through it.
- FabHK 6y ago
  > This all can be made to work but in the "cos its Turing complete" sense rather than the language helping in any meaningful way. I had to LOL. You have very nicely put into words an experience I bet many have had.
- JackFr 6y ago
  Kapital was also rewritten in Java and survived in a few successor places ('Derivatives Studio' at CSFB). There it was my introduction to 'thunks'.
- eru 6y ago
  > I'd expect most lisps to suffer in the same way. Why? Eg Scheme or even the uglier Common Lisp are quite disciplined. And Clojure ain't too bad either. As far as I can tell, Lispers don't usually go around monkey-patching things.

goatinaboat 6y ago

I believe an influence was actually an old system that used smalltalk in the 90ies at JPM. That system you're thinking of was Kapital. http://www.cincomsmalltalk.com/main/successes/financial-services/jpmorgan/ http://www.cincomsmalltalk.com/main/successes/financial-serv... It's the only Smalltalk usage at JPM I believe, they could have doubled-down on it but instead went to Python and as you say, tried reinventing the wheel in it.

kbenson 6y ago
> tried reinventing the wheel in it. That's not reinventing the wheel, that's building a new wheel you hope performs better/different with different materials and maybe some different techniques, and has a long and storied history of both successes and failures. Reinventing the wheel would be if they published this system and some other bank recreated it without using what they published, thus "inventing" something that already existed and was available.
CraigJPerry 6y ago
I never realised Kapital was written in smalltalk. Interesting
- scroot 6y ago
  Apparently when it booted in the terminal, there was a nice ASCII portrait of Marx. I also have it on good authority that the program that launched the Smalltalk image from the terminal was called "das," so you literally would type "das kapital" to get the thing going.

FabHK 6y ago

> IIRC, it could manage list comprehension but not loops. If it was like the predecessor system I worked with, it could handle either. But in a list comprehension, it could figure out the individual dependencies of the individual list items. So, suppose you had a a function that took in a list, and returned a list with some computations performed on each element. If done with list comprehension, and one input element changed, only the corresponding output element would need to be recomputed, and only that would be recomputed. If done with a loop, and one input element changed, the entire output array would be recomputed (even if all but one element remained unchanged - it would be hard to guarantee that property by code inspection at the time the dependency graph is built).