7 ms·

I run a hedge fund. On any given day I hear a large number of complaints from the technologists that complex python systems are difficult to look after and we s

I run a hedge fund. On any given day I hear a large number of complaints from the technologists that complex python systems are difficult to look after and we should use something else instead. There's some Rust being used, but there's little chance to get a quant to use Rust to do research because research is an exploratory process and the last thing one wants is a language that requires a lot of thought about lifetimes etc. How is the python-ocaml interop story? To be clear, any language that does not have first-class interop with python is basically dead in the water (at least for our case).

SquishyPanda23 5y ago
Some of the Python FFI tools are listed here: https://ocamlverse.github.io/content/ffi.html https://ocamlverse.github.io/content/ffi.html. But clicking through to GitHub, the repos haven't been updated in a while.
- thingification 5y ago
  https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml -- 2 days ago
  - SquishyPanda23 5y ago
    I wasn't counting changes to project metadata like gitignore
jstimpfle 5y ago
> last thing one wants is a language that requires a lot of thought about lifetimes etc. I challenge you: A lack of understanding about the data lifetimes in a program means lack of understanding about the data. Not saying you can't have a lot of short-lived data items that you don't want to manage one-by-one. I'm saying that for the vast majority of data items, one should be able to give a reasonably well defined lifetime upper bound. So a good solution is to make a few boxes that group items by lifetime. And from time to time, throw the outdated boxes away. And of the few items that don't have such an upper bound at creation time, many can be created in a special box that allows migrating boxes later when required.
- chrisseaton 5y ago
  > A lack of understanding about the data lifetimes in a program means lack of understanding about the data. But this argument can extend forever. Is your program precisely dependently typed? If not is that a lack of understanding about the nature of the data as well and should you challenge yourself to fix that? You have to trade-off how much you specify things with how valuable it is to get the result more quickly.
  - jstimpfle 5y ago
    What you say is true. I only brought up "boxes" because the concept is still not widely known.
  - deleted 5y ago
    [deleted]
- jenny91 5y ago
  You don't have to challenge the person you're responding to. You have to challenge their quants. And they're not going to want to add that into the million other things they're thinking about while doing research in a Jupyter notebook or something. You're just not going to get this buy-in from people who want to use a tool to get their work done.
- short_sells_poo 5y ago
  Thanks, but I think we may be talking cross purpose here. 99% of the research code ends up being thrown away (well, archived). Not because it's bad code necessarily, but because the idea that was being prototyped is a dead end. This means it's paramount that the language you use has to be as low friction and interactive as possible. Imagine you are trying to establish whether there's a relationship between timeseries X and timeseries Y. You just want a tool that allows you to quickly calculate some summary statistics of these timeseries, clean them, convince yourself that they behave according to your expectations and then run some form of regression. Nowhere in this process do you care about lifetimes. It's literally irrelevant. In fact, as long as all your work fits into memory, you don't even care about memory management. Your objective is to answer the primary question, everything else is a costly distraction. The 1% of ideas that ends up being worthwhile is what gets productionized and needs to be robust. But obviously rewriting everything from language A to radically different language B adds it's own headaches.
bhy 5y ago
Python type checking (type annotation, mypy) should at least partially solve the problem of maintaining complex Python systems. Though it doesn't help with performance.
- pharmakom 5y ago
  The larger problem in my view is that big Python systems tend to follow OOP design since functional programming patterns do not work well in Python. So you start with something minimal and simple inside a script or notebook, but quickly it evolves into something more like a Java code-base. Typing does help, agreed.
  - nerdponx 5y ago
    I strongly suggest the Attrs library for cutting down the boilerplate of making small "data classes": https://attrs.org/ https://attrs.org/ With type annotations, you can move away from "inheritance OO" to logicless "data classes" and functions that operate on them.
  - awild 5y ago
    Java has good to great FP, making most objects immutable is also trivial. Mypy is good to get strict typing enforced but, but I still prefer the native and slightly wonky typesystem to a tacked on one that builds on trust.
    - dunefox 5y ago
      > Java has good to great FP Java does absolutely not have "good to great FP" support. It's an imperative and OO programming language that recently got lambdas, no more, no less.
      - Zababa 5y ago
        There's sealed classes, records, pattern matching, optionals. It's getting there.
deleted 5y ago
[deleted]
hajile 5y ago
If your language doesn't worry about lifetimes, they don't go away. It just means you have to worry about them yourself instead. Sometimes that is great. Other times, that will be very hard and error-prone.
- typon 5y ago
  An extremely simple thing like having two objects stored in a struct where one object has a reference to the other is a Herculean task in Rust. This is not a language designed for prototyping...
  - carlmr 5y ago
    You can easily avoid such constructs in most cases though.
    - typon 5y ago
      I've allocated both these things on the heap and Rust simply won't let me store them both together. I don't know about you but this is an extremely common pattern in almost all languages, you just don't think about it in the gc languages and in C++ storing pointers is no issue. The popularity of crates like rental also shows that it's not as easily avoidable as you suggest.
      - jjnoakes 5y ago
        Can you post a simple code example? It is hard to imagine what the difficulty is.
        typon 5y ago
        You can read through this SO question and the top response does a good job explaining the possible solutions: https://stackoverflow.com/questions/32300132/why-cant-i-store-a-value-and-a-reference-to-that-value-in-the-same-struct https://stackoverflow.com/questions/32300132/why-cant-i-stor...
nerdponx 5y ago
Have you looked into Julia, Nim, Clojure, or even Common Lisp? I'm not sure bout Python interop with CL, but Nim and Clojure seems to have some kind of beta-grade interop, and there's a solid interop story in Julia. And all of those languages have some of their own "native" data analysis and scientific computing toolkits (Julia having more than "some", of course). That said, complicated Python systems can be improved a lot by adding type annotations. That's more of a solution for web servers and other "easily type-able" applications. Typing support for scientific computing isn't quite there yet. So it depends on what kinds of systems are the complicated ones.
- lgessler 5y ago
  Link to the interop lib for Clojure you're referring to for people who don't know it: https://github.com/clj-python/libpython-clj https://github.com/clj-python/libpython-clj Really a remarkable feat of engineering. Here's its author giving a talk: https://www.youtube.com/watch?v=vQPW16_jixs https://www.youtube.com/watch?v=vQPW16_jixs
  - adenozine 5y ago
    I was thinking of linking this as well. Clj-python is just such a fascinating junction. I don't care so much for clojure, though I'm continuously impressed by the ecosystem and the productivity of it's experts. Very cool stuff. Core.logic and the like opened a huge door and similar ideas have exhausted my free time for several years now.
- dunefox 5y ago
  Julias interop with Python is excellent: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl (also with R, see RCall.jl). It's just not statically typed, so the original problem is not solved - albeit Julia being the better language for scientific purposes. CL could also be great language-wise (https://digikar99.github.io/py4cl2/ https://digikar99.github.io/py4cl2/, https://github.com/snmsts/burgled-batteries3 https://github.com/snmsts/burgled-batteries3) but I don't know how good the interop is in reality since I haven't tried it.
- 5y ago
ajoseps 5y ago
I personally haven't used it but Jane Street heavily uses OCaml and has written a blog post on this: https://blog.janestreet.com/using-python-and-ocaml-in-the-same-jupyter-notebook/ https://blog.janestreet.com/using-python-and-ocaml-in-the-sa...
mtoner23 5y ago
Its not the language its the people, instagram is almost entirely run on python, if they can so can you. https://instagram-engineering.com/tagged/python https://instagram-engineering.com/tagged/python
- nv-vn 5y ago
  This is a terribly misinformed take. If you throw enough resources at Python then sure, you can probably get adequate throughput. The problem is that in finance a lot of problems require you to think about latency, which is a total non-starter for Python
  - mtoner23 5y ago
    If it were a total non starter than why would their entire company be using it?
    - short_sells_poo 5y ago
      I have no doubts that we are able to handle our requirements with python, but if there's a better way, does it not make sense to investigate? A skilled carpenter can undoubtedly use a hammer instead of a screwdriver. This doesn't mean that I should insist they use a hammer when a screwdriver would do a better job.
    - nv-vn 5y ago
      Sounds like they're already using Rust where performance matters and are looking to switch
    - upbeat_general 5y ago
      Instagram has much lower latency requirements…
      - carnitine 5y ago
        I believe they’re referring to the hedge fund. Lower latency requirement is a confusing way of phrasing it as lower latency is a higher bar. But generally most hedge funds do not need to operate at the speed Instagram does.
typon 5y ago
I would write principled Python with strict coding standards. Make type annotations mandatory and turn up pylint or flake8 to maximum warnings. It really helps avoid a bunch of silly mistakes, while still providing a way out for doing crazy stuff that Python is good at.
rkangel 5y ago
I think Elixir would be interesting for your usecase. It's a dynamic, garbage collected language. It's easy to pick up and get going with. As a functional programming language there isn't a lot to learn in the way of language constructs, and you don't even have to do the 'wrestling with the type system' thing that you have to do in compiled functional languages like OCaml or Haskell (like you do in Rust). Its processing 'horsepower' is probably comparable to Python, but it's much better for building low latency things if you want to run something in a bit more of a production use case. This is also improving due to the recent addition of a JIT. The addition of NX is making Elixir an increasingly interesting place to do ML - write Elixir, have it run on GPU etc. See https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... Python integration is probably best done using the Erlang 'port' system - running Python as a managed process and communicating with it using messages over stdin/stdout. I use it for C interop and it works well (and fits well with the Elixir/Erlang process model). It's not difficult to roll your own in Python e.g. https://github.com/fujimisakari/erlang-port-with-python/blob/master/erlang_port/protocol.py/ https://github.com/fujimisakari/erlang-port-with-python/blob... or look at something like http://erlport.org/ http://erlport.org/
- short_sells_poo 5y ago
  Thank you! So this looks interesting but it seems like there's no easy way to share numpy arrays? The main use case for a language other than python is a more robust codebase but also performance. We need to be able to efficiently ship lots of large arrays between the languages and the Rust-Python interop supports zero copy arrays for example.
  - pdimitar 5y ago
    Elixir and Rust are very good friends, so to speak. Writing a library in Rust that you can use from Elixir is only slightly worse than trivially easy. But I agree somebody has to put the work. I've made a good career with Elixir but I still don't think it's a good fit for a hedge fund. IMO invest in Rust.
  - rkangel 5y ago
philzook 5y ago
There is an actively developed python to ocaml interop library for purposes quite similar to yours. I have seen demos where ocaml and python are used within the same jupyter notebook https://signalsandthreads.com/python-ocaml-and-machine-learning/ https://signalsandthreads.com/python-ocaml-and-machine-learn... https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml
- short_sells_poo 5y ago
  Thank you, I'll pass this on. An important feature is zero copy arrays, which seem to be supported.

SquishyPanda23 5y ago

Some of the Python FFI tools are listed here: https://ocamlverse.github.io/content/ffi.html https://ocamlverse.github.io/content/ffi.html. But clicking through to GitHub, the repos haven't been updated in a while.

thingification 5y ago
https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml -- 2 days ago
- SquishyPanda23 5y ago
  I wasn't counting changes to project metadata like gitignore

jstimpfle 5y ago

> last thing one wants is a language that requires a lot of thought about lifetimes etc. I challenge you: A lack of understanding about the data lifetimes in a program means lack of understanding about the data. Not saying you can't have a lot of short-lived data items that you don't want to manage one-by-one. I'm saying that for the vast majority of data items, one should be able to give a reasonably well defined lifetime upper bound. So a good solution is to make a few boxes that group items by lifetime. And from time to time, throw the outdated boxes away. And of the few items that don't have such an upper bound at creation time, many can be created in a special box that allows migrating boxes later when required.

chrisseaton 5y ago
> A lack of understanding about the data lifetimes in a program means lack of understanding about the data. But this argument can extend forever. Is your program precisely dependently typed? If not is that a lack of understanding about the nature of the data as well and should you challenge yourself to fix that? You have to trade-off how much you specify things with how valuable it is to get the result more quickly.
- jstimpfle 5y ago
  What you say is true. I only brought up "boxes" because the concept is still not widely known.
- deleted 5y ago
  [deleted]
jenny91 5y ago
You don't have to challenge the person you're responding to. You have to challenge their quants. And they're not going to want to add that into the million other things they're thinking about while doing research in a Jupyter notebook or something. You're just not going to get this buy-in from people who want to use a tool to get their work done.
short_sells_poo 5y ago
Thanks, but I think we may be talking cross purpose here. 99% of the research code ends up being thrown away (well, archived). Not because it's bad code necessarily, but because the idea that was being prototyped is a dead end. This means it's paramount that the language you use has to be as low friction and interactive as possible. Imagine you are trying to establish whether there's a relationship between timeseries X and timeseries Y. You just want a tool that allows you to quickly calculate some summary statistics of these timeseries, clean them, convince yourself that they behave according to your expectations and then run some form of regression. Nowhere in this process do you care about lifetimes. It's literally irrelevant. In fact, as long as all your work fits into memory, you don't even care about memory management. Your objective is to answer the primary question, everything else is a costly distraction. The 1% of ideas that ends up being worthwhile is what gets productionized and needs to be robust. But obviously rewriting everything from language A to radically different language B adds it's own headaches.

bhy 5y ago

Python type checking (type annotation, mypy) should at least partially solve the problem of maintaining complex Python systems. Though it doesn't help with performance.

pharmakom 5y ago
The larger problem in my view is that big Python systems tend to follow OOP design since functional programming patterns do not work well in Python. So you start with something minimal and simple inside a script or notebook, but quickly it evolves into something more like a Java code-base. Typing does help, agreed.
- nerdponx 5y ago
  I strongly suggest the Attrs library for cutting down the boilerplate of making small "data classes": https://attrs.org/ https://attrs.org/ With type annotations, you can move away from "inheritance OO" to logicless "data classes" and functions that operate on them.
- awild 5y ago
  Java has good to great FP, making most objects immutable is also trivial. Mypy is good to get strict typing enforced but, but I still prefer the native and slightly wonky typesystem to a tacked on one that builds on trust.
  - dunefox 5y ago
    > Java has good to great FP Java does absolutely not have "good to great FP" support. It's an imperative and OO programming language that recently got lambdas, no more, no less.
    - Zababa 5y ago
      There's sealed classes, records, pattern matching, optionals. It's getting there.

deleted 5y ago

[deleted]

hajile 5y ago

If your language doesn't worry about lifetimes, they don't go away. It just means you have to worry about them yourself instead. Sometimes that is great. Other times, that will be very hard and error-prone.

typon 5y ago
An extremely simple thing like having two objects stored in a struct where one object has a reference to the other is a Herculean task in Rust. This is not a language designed for prototyping...
- carlmr 5y ago
  You can easily avoid such constructs in most cases though.
  - typon 5y ago
    I've allocated both these things on the heap and Rust simply won't let me store them both together. I don't know about you but this is an extremely common pattern in almost all languages, you just don't think about it in the gc languages and in C++ storing pointers is no issue. The popularity of crates like rental also shows that it's not as easily avoidable as you suggest.
    - jjnoakes 5y ago
      Can you post a simple code example? It is hard to imagine what the difficulty is.
      - typon 5y ago
        You can read through this SO question and the top response does a good job explaining the possible solutions: https://stackoverflow.com/questions/32300132/why-cant-i-store-a-value-and-a-reference-to-that-value-in-the-same-struct https://stackoverflow.com/questions/32300132/why-cant-i-stor...

nerdponx 5y ago

Have you looked into Julia, Nim, Clojure, or even Common Lisp? I'm not sure bout Python interop with CL, but Nim and Clojure seems to have some kind of beta-grade interop, and there's a solid interop story in Julia. And all of those languages have some of their own "native" data analysis and scientific computing toolkits (Julia having more than "some", of course). That said, complicated Python systems can be improved a lot by adding type annotations. That's more of a solution for web servers and other "easily type-able" applications. Typing support for scientific computing isn't quite there yet. So it depends on what kinds of systems are the complicated ones.

lgessler 5y ago
Link to the interop lib for Clojure you're referring to for people who don't know it: https://github.com/clj-python/libpython-clj https://github.com/clj-python/libpython-clj Really a remarkable feat of engineering. Here's its author giving a talk: https://www.youtube.com/watch?v=vQPW16_jixs https://www.youtube.com/watch?v=vQPW16_jixs
- adenozine 5y ago
  I was thinking of linking this as well. Clj-python is just such a fascinating junction. I don't care so much for clojure, though I'm continuously impressed by the ecosystem and the productivity of it's experts. Very cool stuff. Core.logic and the like opened a huge door and similar ideas have exhausted my free time for several years now.
dunefox 5y ago
Julias interop with Python is excellent: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl (also with R, see RCall.jl). It's just not statically typed, so the original problem is not solved - albeit Julia being the better language for scientific purposes. CL could also be great language-wise (https://digikar99.github.io/py4cl2/ https://digikar99.github.io/py4cl2/, https://github.com/snmsts/burgled-batteries3 https://github.com/snmsts/burgled-batteries3) but I don't know how good the interop is in reality since I haven't tried it.
5y ago

ajoseps 5y ago

I personally haven't used it but Jane Street heavily uses OCaml and has written a blog post on this: https://blog.janestreet.com/using-python-and-ocaml-in-the-same-jupyter-notebook/ https://blog.janestreet.com/using-python-and-ocaml-in-the-sa...

mtoner23 5y ago

Its not the language its the people, instagram is almost entirely run on python, if they can so can you. https://instagram-engineering.com/tagged/python https://instagram-engineering.com/tagged/python

nv-vn 5y ago
This is a terribly misinformed take. If you throw enough resources at Python then sure, you can probably get adequate throughput. The problem is that in finance a lot of problems require you to think about latency, which is a total non-starter for Python
- mtoner23 5y ago
  If it were a total non starter than why would their entire company be using it?
  - short_sells_poo 5y ago
    I have no doubts that we are able to handle our requirements with python, but if there's a better way, does it not make sense to investigate? A skilled carpenter can undoubtedly use a hammer instead of a screwdriver. This doesn't mean that I should insist they use a hammer when a screwdriver would do a better job.
  - nv-vn 5y ago
    Sounds like they're already using Rust where performance matters and are looking to switch
  - upbeat_general 5y ago
    Instagram has much lower latency requirements…
    - carnitine 5y ago
      I believe they’re referring to the hedge fund. Lower latency requirement is a confusing way of phrasing it as lower latency is a higher bar. But generally most hedge funds do not need to operate at the speed Instagram does.

typon 5y ago

I would write principled Python with strict coding standards. Make type annotations mandatory and turn up pylint or flake8 to maximum warnings. It really helps avoid a bunch of silly mistakes, while still providing a way out for doing crazy stuff that Python is good at.

rkangel 5y ago

I think Elixir would be interesting for your usecase. It's a dynamic, garbage collected language. It's easy to pick up and get going with. As a functional programming language there isn't a lot to learn in the way of language constructs, and you don't even have to do the 'wrestling with the type system' thing that you have to do in compiled functional languages like OCaml or Haskell (like you do in Rust). Its processing 'horsepower' is probably comparable to Python, but it's much better for building low latency things if you want to run something in a bit more of a production use case. This is also improving due to the recent addition of a JIT. The addition of NX is making Elixir an increasingly interesting place to do ML - write Elixir, have it run on GPU etc. See https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... Python integration is probably best done using the Erlang 'port' system - running Python as a managed process and communicating with it using messages over stdin/stdout. I use it for C interop and it works well (and fits well with the Elixir/Erlang process model). It's not difficult to roll your own in Python e.g. https://github.com/fujimisakari/erlang-port-with-python/blob/master/erlang_port/protocol.py/ https://github.com/fujimisakari/erlang-port-with-python/blob... or look at something like http://erlport.org/ http://erlport.org/

short_sells_poo 5y ago
Thank you! So this looks interesting but it seems like there's no easy way to share numpy arrays? The main use case for a language other than python is a more robust codebase but also performance. We need to be able to efficiently ship lots of large arrays between the languages and the Rust-Python interop supports zero copy arrays for example.
- pdimitar 5y ago
  Elixir and Rust are very good friends, so to speak. Writing a library in Rust that you can use from Elixir is only slightly worse than trivially easy. But I agree somebody has to put the work. I've made a good career with Elixir but I still don't think it's a good fit for a hedge fund. IMO invest in Rust.
- rkangel 5y ago

philzook 5y ago

There is an actively developed python to ocaml interop library for purposes quite similar to yours. I have seen demos where ocaml and python are used within the same jupyter notebook https://signalsandthreads.com/python-ocaml-and-machine-learning/ https://signalsandthreads.com/python-ocaml-and-machine-learn... https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml

short_sells_poo 5y ago
Thank you, I'll pass this on. An important feature is zero copy arrays, which seem to be supported.