6 ms·

Have you considered python for the research/algo side of things? I work in a hedge fund in London at the moment and there's a massive shift away from R towards

Have you considered python for the research/algo side of things? I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1] [1] https://github.com/quantopian/zipline https://github.com/quantopian/zipline

zygomega 14y ago
Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes. The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.
- boothead 14y ago
  Are you planning to run R in production, or is it purely for research that then gets translated to haskell?
  - zygomega 14y ago
    I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
    - boothead 14y ago
      I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?
      - zygomega 14y ago
        definitely. I intend to move all of that code into haskell once I get a handle on whether the algo design is on the right track.
  - boothead 14y ago
    Replying here as I can't add on another reply below The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
    - zygomega 14y ago
      Sounds great, but way beyond either my paygrade or budget.
    - jmandzik 14y ago
      Thatsvery similar to what I/we use to build our realtime network monitoring tools, swapping out python for PHP (regrettably). It's validating to see other shops in different industries zeroing in on the similar stacks.
      - boothead 14y ago
        Yes realtime over websocket to a data-binding style library like knockout (I'd probably choose angular if was starting now) saves you a ton of work.
      - tmzt 14y ago
        Are you using ReactPHP for the ZMQ/webSocket parts or something else?
- fawce 14y ago
  zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT. The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language. I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.
  - boothead 14y ago
    I would see haskell very much as the plumbing side of things. The tools available for handling and reasoning about streams are streets ahead of anything I've seen elsewhere. With zeromq and protocol buffers (that's what we use in our stack) you could very nicely separate the plumbing of the data from its consumption. I'd love to see something like this as well! How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
  - zygomega 14y ago
    Hi fawce, I'm a big fan of quantopian but didn't realize that zipline was a separated project. Will have a good scout around the project and see what you guys are up to :)
  - skebanga 13y ago
    I too would be interested in collaborating if you switched to python. In my ideal world it would be C++ backend, python frontend
- pygy_ 14y ago
  Did you look at Julia[0]? It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end. The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic). The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2]. I looks like a good fit for your project. [0] http://julialang.org http://julialang.org [1] http://docs.julialang.org/en/release-0.1/packages/packagelist.html http://docs.julialang.org/en/release-0.1/packages/packagelis... [2] https://groups.google.com/forum/?fromgroups=#!forum/julia-stats https://groups.google.com/forum/?fromgroups=#!forum/julia-st...
  - zygomega 14y ago
    yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.
    - pygy_ 14y ago
      Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
      - zygomega 14y ago
        It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use. mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with. I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
  - carterschonwald 14y ago
wikiburner 14y ago
Hi boothead, just curious what you do when you run into the inevitable performance issues that can't be ameliorated with libraries like NumPy? Is Cython or PyPy useful, or do you use Python more for prototyping, and rewrite some portions in C?
- aidos 14y ago
  I haven't worked in anything as high performance as this (though I've been doing lots of number crunching in Python recently) but there are a couple of great libraries that you can use before you jump into Cython. Check out Numba [1] and numexpr [2] [1] https://github.com/numba/numba https://github.com/numba/numba [2] https://code.google.com/p/numexpr/ https://code.google.com/p/numexpr/
- boothead 14y ago
  Python is definitely preferred on the research rather than the production side generally. Some organisations do use it on the infrastructure side too, but these guys are algorithmic traders, and not HFTs so latency isn't that big a deal. WRT to the problem of deploying prototyping/research code, there are the following unsolved problems that I'm aware of: * Going from matrix operations over the whole timeseries (taking care to avoid problems with your algo looking ahead) for speed in a research setting to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure. * Concurrency. The options in python all suck to some degree - especially if you have to interact with C libraries or extensions. I don't think it will ever make sense to build your real time market date in python. Again here haskell has an advantage. However the python ecosystem seems to be almost perfect for researchers: * Excellent and flexible data slurping/munging/transforming. * numpy, scipy, pandas, theano, scikits... 'nuff said * ipython * Cross platform HTH
  - zygomega 14y ago
    > Going from matrix operations over the whole timeseries ... to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure. exactly my thought. Algo guys get stuck in matrix land because that's where their tools take them. Whereas this came out in R last week: http://cran.r-project.org/web/packages/stream/index.html http://cran.r-project.org/web/packages/stream/index.html python is a better toolset than R maybe, but the R problem domain seems broader in the last few months anyway. My interest is in monitoring thousands of algorithms in real time directly within the messaging environment, and before the data hits a database. That type of concurrency is where haskell can muscle up and do the job.
chicagobob 14y ago
Are you aware of zipline ports (or similar libraries) in Ruby or ObjectiveC (or plain C)?

zygomega 14y ago

Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes. The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.

boothead 14y ago
Are you planning to run R in production, or is it purely for research that then gets translated to haskell?
- zygomega 14y ago
  I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
  - boothead 14y ago
    I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?
    - zygomega 14y ago
      definitely. I intend to move all of that code into haskell once I get a handle on whether the algo design is on the right track.
- boothead 14y ago
  Replying here as I can't add on another reply below The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
  - zygomega 14y ago
    Sounds great, but way beyond either my paygrade or budget.
  - jmandzik 14y ago
    Thatsvery similar to what I/we use to build our realtime network monitoring tools, swapping out python for PHP (regrettably). It's validating to see other shops in different industries zeroing in on the similar stacks.
    - boothead 14y ago
      Yes realtime over websocket to a data-binding style library like knockout (I'd probably choose angular if was starting now) saves you a ton of work.
    - tmzt 14y ago
      Are you using ReactPHP for the ZMQ/webSocket parts or something else?
fawce 14y ago
zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT. The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language. I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.
- boothead 14y ago
  I would see haskell very much as the plumbing side of things. The tools available for handling and reasoning about streams are streets ahead of anything I've seen elsewhere. With zeromq and protocol buffers (that's what we use in our stack) you could very nicely separate the plumbing of the data from its consumption. I'd love to see something like this as well! How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
- zygomega 14y ago
  Hi fawce, I'm a big fan of quantopian but didn't realize that zipline was a separated project. Will have a good scout around the project and see what you guys are up to :)
- skebanga 13y ago
  I too would be interested in collaborating if you switched to python. In my ideal world it would be C++ backend, python frontend
pygy_ 14y ago
Did you look at Julia[0]? It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end. The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic). The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2]. I looks like a good fit for your project. [0] http://julialang.org http://julialang.org [1] http://docs.julialang.org/en/release-0.1/packages/packagelist.html http://docs.julialang.org/en/release-0.1/packages/packagelis... [2] https://groups.google.com/forum/?fromgroups=#!forum/julia-stats https://groups.google.com/forum/?fromgroups=#!forum/julia-st...
- zygomega 14y ago
  yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.
  - pygy_ 14y ago
    Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
    - zygomega 14y ago
      It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use. mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with. I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
- carterschonwald 14y ago

wikiburner 14y ago

Hi boothead, just curious what you do when you run into the inevitable performance issues that can't be ameliorated with libraries like NumPy? Is Cython or PyPy useful, or do you use Python more for prototyping, and rewrite some portions in C?

aidos 14y ago
I haven't worked in anything as high performance as this (though I've been doing lots of number crunching in Python recently) but there are a couple of great libraries that you can use before you jump into Cython. Check out Numba [1] and numexpr [2] [1] https://github.com/numba/numba https://github.com/numba/numba [2] https://code.google.com/p/numexpr/ https://code.google.com/p/numexpr/
boothead 14y ago
Python is definitely preferred on the research rather than the production side generally. Some organisations do use it on the infrastructure side too, but these guys are algorithmic traders, and not HFTs so latency isn't that big a deal. WRT to the problem of deploying prototyping/research code, there are the following unsolved problems that I'm aware of: * Going from matrix operations over the whole timeseries (taking care to avoid problems with your algo looking ahead) for speed in a research setting to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure. * Concurrency. The options in python all suck to some degree - especially if you have to interact with C libraries or extensions. I don't think it will ever make sense to build your real time market date in python. Again here haskell has an advantage. However the python ecosystem seems to be almost perfect for researchers: * Excellent and flexible data slurping/munging/transforming. * numpy, scipy, pandas, theano, scikits... 'nuff said * ipython * Cross platform HTH
- zygomega 14y ago
  > Going from matrix operations over the whole timeseries ... to deploying to an environment that streams updates to the timeseries one at a time. I think that this is an area that haskell has the potential to excel at, given it's strong guarantees on structure. exactly my thought. Algo guys get stuck in matrix land because that's where their tools take them. Whereas this came out in R last week: http://cran.r-project.org/web/packages/stream/index.html http://cran.r-project.org/web/packages/stream/index.html python is a better toolset than R maybe, but the R problem domain seems broader in the last few months anyway. My interest is in monitoring thousands of algorithms in real time directly within the messaging environment, and before the data hits a database. That type of concurrency is where haskell can muscle up and do the job.

chicagobob 14y ago

Are you aware of zipline ports (or similar libraries) in Ruby or ObjectiveC (or plain C)?