6 ms·

Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes

Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes. The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.

boothead 14y ago
Are you planning to run R in production, or is it purely for research that then gets translated to haskell?
- zygomega 14y ago
  I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
  - boothead 14y ago
    I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?
    - zygomega 14y ago
      definitely. I intend to move all of that code into haskell once I get a handle on whether the algo design is on the right track.
- boothead 14y ago
  Replying here as I can't add on another reply below The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
  - zygomega 14y ago
    Sounds great, but way beyond either my paygrade or budget.
  - jmandzik 14y ago
    Thatsvery similar to what I/we use to build our realtime network monitoring tools, swapping out python for PHP (regrettably). It's validating to see other shops in different industries zeroing in on the similar stacks.
    - boothead 14y ago
      Yes realtime over websocket to a data-binding style library like knockout (I'd probably choose angular if was starting now) saves you a ton of work.
    - tmzt 14y ago
      Are you using ReactPHP for the ZMQ/webSocket parts or something else?
fawce 14y ago
zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT. The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language. I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.
- boothead 14y ago
  I would see haskell very much as the plumbing side of things. The tools available for handling and reasoning about streams are streets ahead of anything I've seen elsewhere. With zeromq and protocol buffers (that's what we use in our stack) you could very nicely separate the plumbing of the data from its consumption. I'd love to see something like this as well! How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
- zygomega 14y ago
  Hi fawce, I'm a big fan of quantopian but didn't realize that zipline was a separated project. Will have a good scout around the project and see what you guys are up to :)
- skebanga 14y ago
  I too would be interested in collaborating if you switched to python. In my ideal world it would be C++ backend, python frontend
pygy_ 14y ago
Did you look at Julia[0]? It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end. The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic). The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2]. I looks like a good fit for your project. [0] http://julialang.org http://julialang.org [1] http://docs.julialang.org/en/release-0.1/packages/packagelist.html http://docs.julialang.org/en/release-0.1/packages/packagelis... [2] https://groups.google.com/forum/?fromgroups=#!forum/julia-stats https://groups.google.com/forum/?fromgroups=#!forum/julia-st...
- zygomega 14y ago
  yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.
  - pygy_ 14y ago
    Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
    - zygomega 14y ago
      It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use. mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with. I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
- carterschonwald 14y ago

boothead 14y ago

Are you planning to run R in production, or is it purely for research that then gets translated to haskell?

zygomega 14y ago
I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
- boothead 14y ago
  I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?
  - zygomega 14y ago
    definitely. I intend to move all of that code into haskell once I get a handle on whether the algo design is on the right track.
boothead 14y ago
Replying here as I can't add on another reply below The way we do our monitoring/UI is with zeromq/protocol buffers as an external surface to the trading system (basically a PUB socket) and a very thin bridging and translation layer to WebSockets and JSON in python. That way you get to use d3 or anything else on the front end. We use backbone, knockback and d3 with coffeescript.
- zygomega 14y ago
  Sounds great, but way beyond either my paygrade or budget.
- jmandzik 14y ago
  Thatsvery similar to what I/we use to build our realtime network monitoring tools, swapping out python for PHP (regrettably). It's validating to see other shops in different industries zeroing in on the similar stacks.
  - boothead 14y ago
    Yes realtime over websocket to a data-binding style library like knockout (I'd probably choose angular if was starting now) saves you a ton of work.
  - tmzt 14y ago
    Are you using ReactPHP for the ZMQ/webSocket parts or something else?

fawce 14y ago

zygomega, I'm one of the zipline maintainers and we'd love to collaborate with you. There are a few people doing HFT research with zipline, and there's a lot of work to do. At quantopian (my day job), we focus on longer hold periods, so there is room in the zipline ecosystem for you to do HFT. The main benefit we've found with python as the algo language is that it allows for stat programming with pandas, but also OO or functional programming for the algo logic. This smoothes the transition from research to production, just as you're describing with R -> haskell, but you can stay in one language. I think one of the biggest potential wins with parallelization is if you can assume all positions are closed overnight, most often true for HFT. That way, you can simulate all the trading days in a test range in parallel. This is quite similar to the parallel processing we do to handle the large number of concurrent backtests running at quantopian. We did all of that with python, but I'd be fascinated to see it done with haskell.

boothead 14y ago
I would see haskell very much as the plumbing side of things. The tools available for handling and reasoning about streams are streets ahead of anything I've seen elsewhere. With zeromq and protocol buffers (that's what we use in our stack) you could very nicely separate the plumbing of the data from its consumption. I'd love to see something like this as well! How would you handle the position sizing part of the algo if you're testing all days in parallel? Wouldn't the trade size depend on the all of the previous day's PNL?
zygomega 14y ago
Hi fawce, I'm a big fan of quantopian but didn't realize that zipline was a separated project. Will have a good scout around the project and see what you guys are up to :)
skebanga 14y ago
I too would be interested in collaborating if you switched to python. In my ideal world it would be C++ backend, python frontend

pygy_ 14y ago

Did you look at Julia[0]? It is a Lisp, semantically, with a MATLAB-like syntax (including all the linear algebra sugar), and a LLVM back end. The stated goal of the language is to give to scienctific programmers the convenience of high level languages for prototyping, yet the speed of low level ones. It has multiple dispatch, based on a powerful type system, homoiconicity and macros (optionally hygienic). The package collection is growing steadily [1]. There's also a mailing list dedicated to doing statistics with Julia[2]. I looks like a good fit for your project. [0] http://julialang.org http://julialang.org [1] http://docs.julialang.org/en/release-0.1/packages/packagelist.html http://docs.julialang.org/en/release-0.1/packages/packagelis... [2] https://groups.google.com/forum/?fromgroups=#!forum/julia-stats https://groups.google.com/forum/?fromgroups=#!forum/julia-st...

zygomega 14y ago
yes. Julia looks like a quality language but suffers the usual new kid on teh block hassles - very few packages so you have to spend too long rolling your own.
- pygy_ 14y ago
  Are there specific packages that you would currently need, but aren't there (yet), or is it a more general concern about the relative package scarcity and unforseen changes to / future needs of your code?
  - zygomega 14y ago
    It's not specific, it's just my recent experience. Hadley released his bigvis package a few weeks ago, and it contains the beginnings of a weighted average routine I sure can use. mmap is a package that let me create a fast db from scratch in a day. ggplot, lattice and all the charting routines are a joy to work with. I'm interested in random forests today, and I bet there's a package that links to the standard libraries whatever they are. This is also probably true for python, but Julia you'd have to build the API.
carterschonwald 14y ago