15 ms·

Show HN: I'm building an open-source, high-frequency trading system

301 points by zygomega 14y ago

white_devil 14y ago
Are you trying to get hired by the financial industry, or is there some other reason for doing this? As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.
- cmdkeen 14y ago
  HFT isn't necessarily a scourge - if anything it discourages people from short term investing, which is a Good Thing. Long term investing isn't affected by HFT, except in providing more liquidity at the point you want to exit any position. If you're planning on holding for 5+ years you're competing against relatively few people and can do good research. Any shorter timeframe and you're competing against millions, against computers and against unknowables that will distort the price in the short term.
  - white_devil 14y ago
    It's weird how this "providing liquidity" distraction never seems to go out of fashion. HFT provides liquidity if you happen to be another HFT algorithm. Otherwise, not so much. Why would short term investing be bad, and how long do you think HFT algorithms hold their "investments"? I hope you're not just trying to blow smoke up my ass here.
    - cmdkeen 14y ago
      It isn't just about liquidity, though it does. But it also reduces the bid/ask spread, i.e. price discovery. The HFT trader may make a tiny profit when it processes a trade from me, but I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case. In terms of liquidity those trades are available to all, they love buying from non-HFT trades because they can make a small profit from their less accurate pricing. You benefit from the lower spreads and that liquidity. Before HFT you'd have paid a greater cost due to the spread on the price. Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. To the extent to which that is ever knowable it is unlikely you have the skill, experience and data to be able to beat the large number of professionals doing it.
      - white_devil 14y ago
        It sounds like you work for the financial industry too. > The HFT trader You say it like it's a person making trades. > I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case. What's a "quoted exchange price" if it's not the price you actually pay? > Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. Isn't this what HFT is all about? Trading on insider information is bad, of course. But if you're on a level playing field, I don't see anything wrong with profiting from something you correctly predicted short-term.
        cmdkeen 14y ago
        I do - but a 'proper' long term investor that cares about the price in 20 years time. The quoted exchange price is an average of the buy/sell offers available, there is no guarantee if you try and trade that you will pay that amount. Especially if you are shifting large blocks of stock. There is nothing wrong with "short term predicting" but it isn't investing. It is often a very easy way to lose money, "bad" doesn't just mean morally so. It comes down to time horizons, how much you want to make, and how many people are also trying to predict that event. You can correctly predict that event and still not make money as it is already priced in.
        TheCoelacanth 14y ago
        > What's a "quoted exchange price" if it's not the price you actually pay? The actual price you have to pay is based on how much you want to buy and how much other people are willing to sell at what price. People who want to buy give a "bid" price and people who want to sell give an "ask" price. Whenever there is an ask price that is lower than a bid price, a sale takes place. This results in the lowest ask price always being higher than the highest bid price. The quoted exchange price will be somewhere in between the two prices. If you want to sell right away, the most you can get is the highest bid price on the books. If you want to buy right away, the cheapest you can get it is the lowest bid price on the books. This means that you have to pay more than the quoted exchange price to buy and receive less than the quoted exchange price to sell.
  - zygomega 14y ago
    Long-term investors trade every day. You wake up and the market has gone down a few percent, you're going to rebalance at least. You decide to buy 30 year treasuries you're not going to hold them for thirty years - you're going to sell 29.5 year bonds and buy 30 year bonds in a few months time. And managers turn over long-term investor portfolios quickly. The average holding period for an SP500 company shareholder is 100 days. And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.
    - cmdkeen 14y ago
      But long term investors don't care about the "clip" that HFTs take, they care about the 80% rise in price over the next 5 years. Rebalancing portfolios again isn't something where getting a few basis points off the HFT best execution price is an issue. I work for a proper long term investor, and we trade over a period of days for the big trades. HFTs aren't even an issue apart from their occasional spectacular blowups that cause "could this happen to us?" memos from upstairs. Long term investors don't need to evolve from HFTs, they're more than happy making money not involving computers or trying to compete with them.
    - ycombobreaker 14y ago
      > And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming. I think it is natural, but fallacious, to apply line-of-sight properties to trading. If you plan to trade, then there are two ways that another participant can "see you coming": 1. If you don't have direct market access, your order gets routed through a broker. The broker sees your order before it hits the market, and if he jumps in line ahead of you then that is front-running, and a Bad Thing. Your broker can get in a lot of trouble for this sort of thing. 2. If you are trying to move a large position by sending multiple orders to the market (one after another), then all market participants have the chance to react to the first order. It's really a game to try to move a lot of inventory at once, without tipping your hand to anyone else in the room. Thems the breaks. Nobody else gets to see anyone's order before the matching engine has already processed it, so there's no way to jump "ahead" of it. OTOH, maybe your long-term investor is trying to time the market: wait for a signal intraday, and pick that moment to send an order. In that case, if it is a good intraday signal then it is likely that someone else will compete. It is unlikely for a long-term trader to have spent as much on infrastructure as a HF trader, so the juicy signals will result in missed executions that _look_ like front-running.
- zygomega 14y ago
  No, I'm already in the finance industry. There are two main reasons I'm doing this: - I think the finance industry is very closed when it comes to intellectual property development, and an open source approach can be seriously competitive. An open approach may well be the future when it comes to being 'connected' - HFT is an interesting multi-disciplinary problem and the shear breadth of expertise required - modern chip design, low-latency software/hardware interaction, lock-free concurrency, fault-tolerant system design, adaptive learning algorithms, k-means clustering - means I'm learning heaps every day. I just don't agree that HFT is a scourge. It's an ecological shift (neither good or evil) and longer-horizon investors need to evolve.
  - nezza-_- 14y ago
    No, long-horizon investors are normally overall not interested in HFT because they won't 'suffer' from it.
  - zamalek 14y ago
    "HFT is an interesting multi-disciplinary problem" This makes this a very worthwhile project: whether or not anything comes of it (I considered writing technical analysis trading software for the exact same reason). Some documentation about the patterns and techniques you used would likely benefit the community a tiny bit more. Not everyone understands Haskell (or, forbid, has the time to peruse large codebases). Best of luck to you!
    - xradionut 14y ago
      There are many other interesting multi-disciplinary problems that are not as contentious as HFT and probably more publically acceptable and worthwhile. But if HFT is the domain you care about, go for it.
  - white_devil 14y ago
    > an open source approach can be seriously competitive Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone? Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a fraction of what the sociopaths running the show get, of course, but good money nevertheless. > I just don't agree that HFT is a scourge. Good thing you're not at all biased.
- topbanana 14y ago
  > As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from. This is the line peddled by the popular press but it isn't true. HFT works to reduce spreads, which benefits both buyers and sellers.
  - fnordfnordfnord 14y ago
    Latency arbing is a scourge, and makes limit orders practically useless. Quote stuffing, would be illegal if trading were still done on little slips of paper. Imagine dumping 10,000 slips of paper on the trading desk, and then shouting "just kidding!" In the words of Lawrence from Office Space, "You'd get your ass kicked" Or even worse, 10,000 empty bids. That's why people hate HFT.
    - kasey_junk 14y ago
      I'm not trying to be argumentative, but I am curious why you find latency arb so problematic, and what you think it's impact is on limit orders? I for one think latency arb is one of the bigger net wins for hft. As a market participant, each venue I have to maintain a presence at is a cost to me. I'm willing to pay the latency arb shops their cut to provide me price consistency because for my models it is much cheaper to do so than to continually reevaluate and redeploy to every possible venue. It frees me to shop for venues that provide the best features and fees. As for quote stuffing, you are absolutely right it is awful. That's why almost every venue out there has taken or is taking steps to curtail it. They did so because their customers agree with you.
      - dchichkov 14y ago
        'Quote stuffing' is there, because of absolutely ridiculous SEC rules. If you'd put in fractional prices and remove NBBO rules, there wouldn't be any 'quote stuffing'. And latency arb. What's wrong with latency arb? How else can you move information from one exchange to another fairly? Think of HFT's as of part of the financial network infrastructure. Where exchanges play role of nodes and HFT companies role of links/queues/buffers.
- fnordfnordfnord 14y ago
  >HFT is a scourge on the world's economy I'd point out that a lot of the things that are going on in HFT, are rehashes of old trading scams, and either are or would be illegal if there was any adult supervision. I was tempted to say something about the SEC being left behind the technology, and doesn't understand it. But for that to be true, and have this crap go on for so long, they either need to be complete fools, or they benefit from the status quo. >and it's a game only the biggest and best-connected players benefit from. That's definitely true in the case of latency arbitrage. But here again. It's the exchanges themselves that have decided that some players get an advantage over others. As an example, latency arbitrage, and a whole host of other problems could be sorted out by putting traders on exchange-hosted virtual machines. http://www.dailyfinance.com/2010/06/05/rigged-market-latency-arbitrage-3-billion/ http://www.dailyfinance.com/2010/06/05/rigged-market-latency... If not virtual machines, there are other ways to fix these problems, but first you need to get the players to acknowledge that there is a problem. http://www.zerohedge.com/article/do-it-yourself-latency-arbitrage-how-hfts-can-manipulate-nbbo-whim-courtesy-nyse-quote-stuff http://www.zerohedge.com/article/do-it-yourself-latency-arbi... Nanex has done a lot of good analysis of HFT, some of which is published here: nanex.net/FlashCrash/OngoingResearch.html
  - ycombobreaker 14y ago
    Nanex is wonderful to read, but IMO they frequently misinterpret the data. Off the top of my head: - They only use consolidated feeds for US equities, never direct market-data feeds. The consolidated feed necessarily contains less information that direct feeds (to satisfy more stringent bandwidth requirements), which masks some "interesting" effects of how the exchanges publish their data. - They disregard that the CME feed publishes a fixed depth-of-book, and whenever they look at total liquidity in the book it can appear to flicker when deep levels fall "out" of the back of the book, even if liquidity is actually improving with the presence of a new inside level. - They make a big deal about wholesaler matching only occurring when the consolidated book is not locked. Their rationale is that subpenny prices are always wholesalers, and (erronously) therefore a lack of subpenny-priced trades must mean a lack of wholesale matching. These mistakes sound believable, but they do not hold up to any of scrutiny. Use their site to find interesting events, but be very careful about taking their conclusions at face value. Edit: bullet-list formatting
cmdkeen 14y ago
I'm curious about the "best trading platform" claim - what is going to make it better than any other? Especially as the real High Frequency Traders are busy spending fortunes on placing their hardware as close to the exchange as physically possible.
- zygomega 14y ago
  Speed is only one issue with autonomous algorithm design. Yes, the speed thing grabs the headlines but the boiler-plate objective is to front-run the slower players. There's a wealth of opportunity in processing the event stream in a more robust way than others and faster too. Think semi-HFT, semi-autonomous. Most trading platforms are primarily loss leaders for the 'professional' version and otherwise attached to a non-open source business plan.
  - cmdkeen 14y ago
    But doesn't that mean you're at the mercy of the professional HFTs who can "front run" anyone running on your platform?
    - kasey_junk 14y ago
      You've hit a pet peeve of mine so sorry about the tirade. People use the term "front run" incorrectly and it is important to point this out. Front running is a very specific activity and is illegal. If there is evidence of front running people need to be prosecuted. That said, I've worked on several systems and never seen any of them that would allow a faster market participant to see your order flow without your permission. This is what front running is. People who don't understand the term seem to think seeing market data before other participants is front running it's not. There have always been and will always be some traders that see (and can react to) market data faster than everyone else. In the era of electronic trading this information latency cost is going down, not going up.
nezza-_- 14y ago
You need to do instead of icon-iconname... The links all have icon names in them. Edit: Ah. It now works only with JS on when putting the icon-name into and not in the class. Sorry for that, seems to be new in Bootstrap.
- Alexx 14y ago
 It's Twitter bootstrap. http://twitter.github.io/bootstrap/base-css.html#icons http://twitter.github.io/bootstrap/base-css.html#icons Bootstrap V3 has tweaked the way icons are implemented slightly.
 - deleted 14y ago
 [deleted]
 - deleted 14y ago
 [deleted]
 - zygomega 14y ago
 thanks heaps for that!
deleted 14y ago
[deleted]
cr1t1calh1t 14y ago
Check out IBrokers: http://cran.r-project.org/web/packages/IBrokers/index.html http://cran.r-project.org/web/packages/IBrokers/index.html It'll get you up and running with IB in no time.
- zygomega 14y ago
  Yeah, IBrokers is great. R is even better! Everytime I need a specialist piece of code it seems to already exist as an R package and to have been released in the last few weeks.
- brotchie 14y ago
  On a similar note, a while back I wrote a bi-directional, fan-out adapter between the Interactive Brokers API and ZeroMQ. https://github.com/brotchie/ib-zmq https://github.com/brotchie/ib-zmq The annoying thing about the IB API is that there's no framing; that is, you can't simply consume the message types you are interested in without parsing the entirety of every variable length messages. ib-zmq resolves this annoyance by parsing incoming messages and placing them individually into ZeroMQ message frames. I also wrote an alternative to the IBrokers R package with a much nicer interface using this ZeroMQ adapter. It parses most IB API messages, but hasn't been used in production yet. https://github.com/brotchie/r-zerotws https://github.com/brotchie/r-zerotws
deleted 14y ago
[deleted]
nazka 14y ago
I am curious about this project because I love this subject. However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast trading system. The other reason is you can't build a perfect HFT trading system because it's a moving target. It is the war in the micro seconde market. The sharks will send you wrong orders just to disrupt your system or they will manipulate the market. And finally there is a hardware problem. To do a HFT trading system you need to rent your own severs inside the market, use FPGA cards, etc. In conclusion, I only say there is a wrong goal to say "I will build the best open source HFT trading system" rather than "I will build the best open source fast trading system". This project is interesting (and there is Haskell cool!) so I will stay connected to see how it evolves.
- olalonde 14y ago
  Couldn't the infrastructure be open source while the specific trading algorithms kept closed source?
  - conformal 14y ago
    i believe that is how it works: open source infrastructure, private algos
    - zygomega 14y ago
      I intend to publish and collaborate on private algos. Even if others have the code they still don't know where I choose to operate and deploy capital. And if they get rich on the code good luck to them - maybe they'll sponsor a dedicated server for 1.0
  - nazka 14y ago
    Yes you send your code on the servers of a company and you use their API to hit the market with their servers, or you rent your own racks. It will cost you a lot but in the HFT world it is the only solution. Any nano second counts. You are the fastest or you are not.
jschulenklopper 14y ago
This could be an interesting course for people wanting to know more about 'computational investing' and the algorithms behind stock (market) analysis and trading: https://www.coursera.org/course/compinvesting1 https://www.coursera.org/course/compinvesting1. Not specially targeted at HFT, but interesting basic background information nevertheless.
conformal 14y ago
you're doing it wrong: haskell is too slow.
- zygomega 14y ago
  I thought so too for a long while, then I tried to do a touch of concurrent code in c++ and had to gouge my eyes out. I'm excited about the speed up haskell brings to development. You can plan things in haskell you can't imagine in other languages.
  - conformal 14y ago
    if concurrency with C/C++ is bad for you, go makes it very easy. however, it is not very fast either. doing it in go and then using cgo where necessary will get you pretty close to C speed.
  - NateDad 14y ago
    +1 for doing it in Go. If concurrency is your objective, Go makes it easy... it's also very fast both to program in and to run.
    - cakebread 14y ago
      > +1 for doing it in Go Conspicuously you don't address Haskell at all. Then again, who cares what might actually be best for OP? There's an advocacy bingo card in play!
- zura 14y ago
  But how elegant it is, umm... :)
- z3phyr 14y ago
  How much experiance do you have in programming, for claiming that?
  - sseveran 14y ago
fmstephe 14y ago
I'm building an open source matching engine. I would love to pair up the two systems for a stress test. I'll keep tracking your project and ping you again when I have a system up a running if you're interested. https://github.com/fmstephe/matching_engine https://github.com/fmstephe/matching_engine
- zygomega 14y ago
  We have some plans on the drawing board for market maker simulators that could do with some fast matching so I'll keep an ear open for that ping.
  - fmstephe 14y ago
    What sort of performance baselines would you require?
    - zygomega 14y ago
      I have no idea yet. I see you're geeking out in go and there's been a lot of advocacy coming through to look at go as an alternative.
      - fmstephe 14y ago
        Yeah, Go is a great language. However, I am trying to develop some low level queues that should be faster, and less generic, than the ones that Go provides. But I am having some difficulty putting memory fences into my code. But if I can get a reasonable level of performance out of the standard Go queues then I will just use those in the meantime.
- ycombobreaker 14y ago
  I see a comment about ITCH in there... Are you modeling your interface after NASDAQ's ITCH/OUCH interfaces? If so, I think it would be interesting to look at the protocol specifications for all of the other major US equity exchanges. I think they are all freely-downloadable.
niggler 14y ago
Have you actually used this in a production setting?
- zygomega 14y ago
  No, and I am making no claims. I have a market event feed coming in, a good idea of what the event processing looks like and a rough idea of how to send an order to a broker. I think the project needs to get to production fast though.
boothead 14y ago
Have you considered python for the research/algo side of things? I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1] [1] https://github.com/quantopian/zipline https://github.com/quantopian/zipline
- zygomega 14y ago
  Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes. The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.
  - boothead 14y ago
    Are you planning to run R in production, or is it purely for research that then gets translated to haskell?
    - zygomega 14y ago
      I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
      - boothead 14y ago
        I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?
guiomie 14y ago
I thought HFT was now based on FPGAs and ASIC circuits ... how could this compete in terms of latency/processing ?
- zygomega 14y ago
  By putting it on FPGAs maybe. Stage one is to get a robust process for measuring and researching latency - I'm in Australia and US data is bouncing into my mac via a lousy connection and a tcp port in 250 millsecs on average. There's a long way to go. I suspect that if you can keep within a bulls roar of the low-latency crowd the big gains will be on the algorithm processing side. Algorithms still add up a few million numbers every time they recalculate a moving average - there's better ways to do it.
- consz 14y ago
  FPGAs and ASICs are rare -- at least the HFT firm I work for, they're basically non-existant.
ablut 14y ago
Isn't the barrier to HFT the fact that you need enough capital so that your profits cover the cost of co-located servers and FPGAs in the exchange datacenter, without which you have a latency handicap? (in addition to, of course, coming up with good algorithms)
- zygomega 14y ago
  Yes, if the only edge you have is speed. But a good algorithm may well be the next battleground as the benefits of ultra-low latency reduce.
qompiler 14y ago
Sorry to burst your bubble but high-frequency trading is only viable and necessary for market makers. The end result of any type of algorithm you think of is always a curve-fitting function of existing data.
- zygomega 14y ago
  No need to apologise - curve fitting is a big issue. But I would think that everything that happens on kaggle would be curve fitting by your definition right? And how is the stuff that happens in brains something other than curve fitting? It's how you connect the data dots and what you choose to focus on that makes the difference.
  - qompiler 14y ago
    You are asking how the human brain works, that's a whole different issue and I don't know what Kaggle is/does.
iTrollFreely 14y ago
i don't think you know what high frequency in hft means. haskell, r, emacs org-mode (lmao wtf?)... yeah good luck with that.
tolitius 14y ago
1. Timestamps in your seed data might benefit from nanoseconds if you really talking about "high" frequency. 2. I agree with you comment that it is easier to think about concurrency in Haskell than in something like C++, however you can't really compete with C/C++ in Haskell. Not even with cgo (Go packages that call C code), not with OCaml or any other higher level beasts that promise the speed. Fortran would be the only one faster for the "algo" part of your initiative. But again, if this is just an exercise, Haskell and others (I prefer Clojure for example :) will do just fine. 3. Would make sense to split the "platform" in two (very different) parts: "Quantitative Analysis" (a collection of tools and rules) and "Technical Glue to Read and Stream". Each can/should be divided further of course, but the two above are essential yet very different for a true "HTF Platform".
- zygomega 14y ago
  I wish I could get down to nano units. iqfeed (a good value feed) just got millisecs in so will settle for that. I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts. The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.
  - mayank 14y ago
    Keep in mind that if you're not an experienced C/C++ coder, you're going to be (largely) benchmarking your relative ability in either language rather than the intrinsic speed of each language.
    - kasey_junk 14y ago
      This is true of any language/coder. It's what makes cross platform benchmarking so difficult for non-trivial problems.
  - deleted 14y ago
    [deleted]
otikik 14y ago
I would be much more interested in an open source solution to fight against HFT, or at least make it more difficult. I see the financial world as a necessary evil, and I think some of its parts have benefits for the rest of the world. HTF is not one of those parts; I see it as a pure burden. You could have made an open source Patent Trolling system and I would feel the same.
- zygomega 14y ago
  Wow, that's a pretty low ranking being shoved into the patent troll category of evilness. When someone goes and patents an obvious algorithm at some point in the future and my little project shows up as prior art would I be redeemed :)
rags_123 14y ago
+1 for using Haskell.
dhosek 14y ago
Aside from all the other points raised against this, there's also the fact that the timing on HFT is so close that it requires close colocation of the physical hardware to the trading systems. Only big boys get to have access to those racks.
alexforster 14y ago
I have no experience in the financial industry, but HFT has always fascinated me as a potential source for very high rates of "events". Could you share your general insight about the sheer volume of data that commonly gets pushed through an HFT system? I'd also be terribly interested in a multi-megabyte/gigabyte "recording" of HFT trade data. http://www.nyxdata.com/capacity http://www.nyxdata.com/capacity
- sseveran 14y ago
  This project is not HFT. It does not use a full book data feed. Really even the feeds that come from the exchanges are not precisely timestamped enough and HFT firms stamp their own . The data is not huge although trading system need to be able to deal with large spikes in the rate of events. Usually now people are using 10G/40G/Infinibad as a connect to the matching engine.
- zygomega 14y ago
  So, yes, on my toy wannabe hft feed, I'm currently clocking about 10 million ticks per day on about 8 futures tickers I'm tracking. The raw feed is between 1 and 1.5 gig per day. I can imagine the petabyteness of the bigger guys.
z3phyr 14y ago
Good to see Haskell. I must give you a complement for being part of pushing Haskell to the mainstream! BTW, many trading firms do use haskell in Trading, but thats all privately held.
kasey_junk 14y ago
This is a nice little research project, and I hope you learn a lot from it. Having written algorithmic trading systems I think you are missing on a couple of central points: 1. There is no such thing as the "best" algorithmic trading platform, because algorithmic trading is such a broad term. Architectures that make sense for one class of trades does not make sense for other class of trades. 2. Contrary to popular opinion on this forum, algorithmic trading is not necessarily only low-latency and high-frequency. Frequency, latency, and algorithmic levels are really dials on the specification system and along with cost dictate how you will build the system. Do some work figuring out the trades you are targeting before setting those dials. 3. The actual execution system part of an algo trading system is usually the easiest part. If you've written one before, writing new ones is usually trivial. Finding the trade, building appropriate risk systems/practices, building back testing frameworks and exchange reference data systems are all much more challenging and take the majority of the time in these systems. 4. Finally, if you are truly targeting low-latency and high frequency events concurrency is your enemy, not your friend. To everyone saying that the cost in these systems is the cost of co-locating servers or fpga cards, you're wrong. You can get hosting deals/leases on those kinds of things for the same cost as high-end web hosting. The cost of running these systems are 2 fold: 1 paying the employees (because your competitors can pay them a lot) and 2 having deep enough pockets to survive the bad days/weeks/months. These are the same costs that have always existed in the trading space and have nothing to do with electronic trading. In fact, electronic trading has lowered the information asymmetry and made it easier for new participants to be involved in the markets.
- bink-lynch 14y ago
  > ...truly targeting low-latency and high frequency events concurrency is your enemy, not your friend. This reminds me of the LMAX financial trading platform where they started with a concurrent model but ended up using a single-thread "...that will process 6 million orders per second..." http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html
  - windust 14y ago
    Yes, it is interesting, when you are becoming highly concurrent the penalty for context switching start taking its toll, but that's only when the rest of your trading system is already optimized.
mikevm 14y ago
How can someone who has absolutely no idea about the financial industry and HFT learn about it? Can anyone recommend any resources (books, tutorials, etc..)?
- relaxitup 14y ago
  mikevm, check out Dark Pools by Scott Patterson. Great book about the evolution of HFT: http://www.amazon.com/Dark-Pools-High-Speed-I-Financial/dp/0307887170 http://www.amazon.com/Dark-Pools-High-Speed-I-Financial/dp/0...
rbc 14y ago
I worked on a framework for managing stock and options positions based on feedback methods developed by Stafford Beer. It doesn't do the algorithmic/transaction part, but rather focuses on providing reporting and alerting for large hierarchies of institutions, accounts, stock and options positions. The proprietary algorithmic code could be added by subclassing the code that I wrote. It's called the Viable System Agent. It's written in Smalltalk, tested under both Squeak and Pharo. It's licensed under the BSD license. It can be found at: http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ Look at the RBC-VSA-Portfolio category. That's where all the stock/option code is.
carterschonwald 14y ago
In another month or two the analytical tooling for doing stats and numerics in Haskell land are going to have a huge leap forward in capabilities. Might be worth considering going full Haskell then :-)
- sseveran 14y ago
  Whats happening then?
  - carterschonwald 14y ago
    you know exactly what i'm up to slowly with wellposed :) (building numerics/ data analysis tools that dont' fill me with rage and ire over terrible engineering and usability. A matrix / linear algebra kernel of tools are on track to be ready for hackage release + paid pro versions in 1-2 months)
    - zygomega 14y ago
      Classical landing page had me hooked anyways :-)
      - carterschonwald 14y ago
        thankee good sir. I'm glad some people appreciate my vague semblance of prioritization skills. Hopefully that pans out to being able to have any early customers be of the sophisticated sort that I'm excited to work with / help!
        nazka 14y ago
        What is the name of your library? Or where do you post news?
        carterschonwald 14y ago
        It's not out yet. If you really want to hear about things as they happen, signup for the announce list linked to from www.wellposed.com , I've yet to fire off any emails to that list, but I anticipate 2-3 emails over the next 1-2 months (after a year of hard work and focused thinking). What's also kinda awesome is I think the alpha release with all the Lin alg functionality should be under 2k loc. a lot of the work has been figuring out how to make the design composable and extensible enough that I can write a first working version with good performance just on my own. Ironically, that's also a compelling way to Validate that my tech delivers.
bdunbar 14y ago
> using haskell,R and emacs org-mode. That last is so very, very, cool. I'll have to dive in, take a look, see if I can find out _how_ you're using it. Been wanting to do a project with org-mode for a while, now.
- zygomega 14y ago
  It is - I can't imagine working without it. But I'm not sure I'm the poster child for how to use org-mode properly. Every piece of code in the repo actually resides in the org file which is a big monolithic journal of where I'm at. o-blog is a great emacs package that lets you publish sites straight from org.
indigo_daddy 14y ago
On a possibly related note, Josh Levine, creator of the Island trading engine, released the Foxpro source a few years ago.. Interesting stuff.. http://josh.com/notes/island-ecn-10th-birthday/default.htm http://josh.com/notes/island-ecn-10th-birthday/default.htm http://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.TXT http://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.TX...
SeanDav 14y ago
The industry is doing the vast majority of its HFT development in C++ or Java. Data analysis is mostly done using Python although I still see a lot of R. If you are doing this to break into the industry, I suspect the languages you used should have been the above. Also the above languages would probably have been better to attract open source developers who are also hoping to use their code and experience from this project to break into the industry.
- zygomega 14y ago
  I've been in the industry for too long (though not in low-latency) - I'm doing this to break out of the industry! And I think the finance game is long overdue for a good disruptive technologic event (not implying my humble project is it). R is a personal choice for data analytics (I love ggplot2). But the haskell thing is more than that. It's been done before (http://www.starling-software.com/misc/icfp-2009-cjs.pdf http://www.starling-software.com/misc/icfp-2009-cjs.pdf) and even goldman's has a thriving erlang/OCaml hacker ethic (http://www.zerohedge.com/article/aleynikov-code-dump-uncovered http://www.zerohedge.com/article/aleynikov-code-dump-uncover...).
- fnordfnordfnord 14y ago
  He didn't really mean HFT. At least I don't think he did.
fnordfnordfnord 14y ago
FYI: This isn't an HFT bot but is a decent, recent Python + ncurses trading client framework for Bitcoin on MtGox. http://prof7bit.github.io/goxtool/ http://prof7bit.github.io/goxtool/ Even if you aren't interested in Bitcoin, it might be useful as a real-ish and cheap place to test with low barriers to entry. I've found that backtesting on historical data is usually not realistic enough since most people fail to consider liquidity. I'm interested because I like to trade, would like to learn more, and I have had some small success. But I have trouble sticking to my plan. I let emotion get the best of me and wind up losing my gains.
- zygomega 14y ago
  Isn't MtGox over as a bitcoin trading platform? I agree more generally that bitcoin could be a useful market to test out. And your story fits exactly with a hands-off automated approach to trading. We wont ever be charting a price series in the production environment because computers don't get charts.
  - fnordfnordfnord 14y ago
    >Isn't MtGox over as a bitcoin trading platform? I wish. I keep waiting for some more interesting news from Coinlab [1][2], but so far, Mtgox are still the largest exchange by volume. Here is a chart showing volume by (selected) exchanges. http://bitcoincharts.com/markets/currency/USD.html http://bitcoincharts.com/markets/currency/USD.html The recent events seem to only have got them more customers (though it remains to be seen how many of them will stick). In any case, some of the other exchanges have API's that are similar to mtgox. [1] https://mtgox.com/press_release_20130228.html https://mtgox.com/press_release_20130228.html [2] http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab-attracts-500000-in-venture-capital-for-bitcoin-projects/ http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab-at...
mangrish 14y ago
Cool! Very interesting. I think given the languages you have selected you are coming more from a quant background? These languages are great for heuristics and analysis but you would really want all 'static' components such as connectivity built in assembly/C/C++. For 'algo' components I like Java as you can still pull microsecond order latencies when crunching numbers but more importantly it gives you a huge time to market advantage than the C/C++ for almost the same speeds. I'm also not clear on if you are connecting directly to the market for market data or using aggregation (like reuters). The latter would be too slow. I'm also not clear on what middleware you are using which is probably the biggest decision you will have to make. Most either use inhouse tech or 29west LBM (everyone still calls it that even though they were bought out). An overlooked part of HFT in my experience OS optimisation and even things like TCP bypass (for some components) which can lead to huge speed advantages and end to end latency reductions. I agree with those about FPGA.. in my experience they really don't come into the equation except for components that rarely change. Anyway a few guys including Martin Thompson have felt similarly to you and initiated the lodestone project (http://lodestonefoundation.wordpress.com/ http://lodestonefoundation.wordpress.com/). If you are keen to learn more their architectures for low latency, distributed and componentisation then I feel that you could join forces and contribute to that initiative. FYI: Martin built most of the technology behind the disruptor (http://lmax-exchange.github.io/disruptor/ http://lmax-exchange.github.io/disruptor/). Great to see interest in making this knowledge more widely available :)
- zygomega 14y ago
  thanks mangrish, quant. How could you tell? I'm conecting with an aggregator cause the direct market feeds are monopolistic price gougers. It's an easy switch if we make it up that curve. I'm not sure if I even understand what middleware is (I'm a quant), but I think the answer is the disruptor!
  - mangrish 14y ago
    From the languages.. I've worked with lots of quants and they all rave about R! Yeah..the aggregator is where you get done in both cost and latency. The prop shops and hedge funds pay through the nose for that stuff so unless you come packing a little capital, true HFT is an issue. On the middleware, not really.. so you would have a market data component that will be pushing stuff to various components (real time risk, the pricer, and the trading engine). The disruptor sits on the 'in' queue to those components.. the middleware is what pushes messages between instances running on the same/different machines. Hope that helps :)
zopf 14y ago
I assume you've checked out Marketcetera ( http://www.marketcetera.com http://www.marketcetera.com ). What will be your platform's primary differentiators?
- zygomega 14y ago
  Not having to register and login is one immediate differentiator. Marketcetera is a big, big code base - I think our ambitions are more focused.
relaxitup 14y ago
On a possibly related note, Josh Levine, creator of the Island trading engine, released the Foxpro source a few years ago.. Interesting stuff.. http://josh.com/notes/island-ecn-10th-birthday/default.htm http://josh.com/notes/island-ecn-10th-birthday/default.htm http://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.TXT http://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.TX...
- ycombobreaker 14y ago
  Jerk boy!
dasickis 14y ago
I have access to servers in all the major trading rooms if you need to start deploying this to a production environment to start trading. Contact me at dasickis [at] gmail.com (I'll reply back from my non-filtered e-mail address)
ttty 14y ago
Which software you used to build the diagram on the main page?
- zygomega 14y ago
  The graphviz dot language: http://www.graphviz.org/Documentation.php http://www.graphviz.org/Documentation.php It has good support in org-mode and a there's a nice haskell package that takes the dot code and turns it into an internal graph representation.

301 points by zygomega 14y ago

Are you trying to get hired by the financial industry, or is there some other reason for doing this? As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from.

cmdkeen 14y ago
HFT isn't necessarily a scourge - if anything it discourages people from short term investing, which is a Good Thing. Long term investing isn't affected by HFT, except in providing more liquidity at the point you want to exit any position. If you're planning on holding for 5+ years you're competing against relatively few people and can do good research. Any shorter timeframe and you're competing against millions, against computers and against unknowables that will distort the price in the short term.
- white_devil 14y ago
  It's weird how this "providing liquidity" distraction never seems to go out of fashion. HFT provides liquidity if you happen to be another HFT algorithm. Otherwise, not so much. Why would short term investing be bad, and how long do you think HFT algorithms hold their "investments"? I hope you're not just trying to blow smoke up my ass here.
  - cmdkeen 14y ago
    It isn't just about liquidity, though it does. But it also reduces the bid/ask spread, i.e. price discovery. The HFT trader may make a tiny profit when it processes a trade from me, but I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case. In terms of liquidity those trades are available to all, they love buying from non-HFT trades because they can make a small profit from their less accurate pricing. You benefit from the lower spreads and that liquidity. Before HFT you'd have paid a greater cost due to the spread on the price. Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. To the extent to which that is ever knowable it is unlikely you have the skill, experience and data to be able to beat the large number of professionals doing it.
    - white_devil 14y ago
      It sounds like you work for the financial industry too. > The HFT trader You say it like it's a person making trades. > I've benefited because I can buy/sell at a price close to the quoted exchange price, which didn't use to be the case. What's a "quoted exchange price" if it's not the price you actually pay? > Short term investing is "bad" because you're not investing in the future success of the company you're trading on the basis of what you think the market is going to do. Isn't this what HFT is all about? Trading on insider information is bad, of course. But if you're on a level playing field, I don't see anything wrong with profiting from something you correctly predicted short-term.
      - cmdkeen 14y ago
        I do - but a 'proper' long term investor that cares about the price in 20 years time. The quoted exchange price is an average of the buy/sell offers available, there is no guarantee if you try and trade that you will pay that amount. Especially if you are shifting large blocks of stock. There is nothing wrong with "short term predicting" but it isn't investing. It is often a very easy way to lose money, "bad" doesn't just mean morally so. It comes down to time horizons, how much you want to make, and how many people are also trying to predict that event. You can correctly predict that event and still not make money as it is already priced in.
      - TheCoelacanth 14y ago
        > What's a "quoted exchange price" if it's not the price you actually pay? The actual price you have to pay is based on how much you want to buy and how much other people are willing to sell at what price. People who want to buy give a "bid" price and people who want to sell give an "ask" price. Whenever there is an ask price that is lower than a bid price, a sale takes place. This results in the lowest ask price always being higher than the highest bid price. The quoted exchange price will be somewhere in between the two prices. If you want to sell right away, the most you can get is the highest bid price on the books. If you want to buy right away, the cheapest you can get it is the lowest bid price on the books. This means that you have to pay more than the quoted exchange price to buy and receive less than the quoted exchange price to sell.
- zygomega 14y ago
  Long-term investors trade every day. You wake up and the market has gone down a few percent, you're going to rebalance at least. You decide to buy 30 year treasuries you're not going to hold them for thirty years - you're going to sell 29.5 year bonds and buy 30 year bonds in a few months time. And managers turn over long-term investor portfolios quickly. The average holding period for an SP500 company shareholder is 100 days. And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming.
  - cmdkeen 14y ago
    But long term investors don't care about the "clip" that HFTs take, they care about the 80% rise in price over the next 5 years. Rebalancing portfolios again isn't something where getting a few basis points off the HFT best execution price is an issue. I work for a proper long term investor, and we trade over a period of days for the big trades. HFTs aren't even an issue apart from their occasional spectacular blowups that cause "could this happen to us?" memos from upstairs. Long term investors don't need to evolve from HFTs, they're more than happy making money not involving computers or trying to compete with them.
  - ycombobreaker 14y ago
    > And every time a trade happens you run the risk of getting clipped by the faster guys who see you coming. I think it is natural, but fallacious, to apply line-of-sight properties to trading. If you plan to trade, then there are two ways that another participant can "see you coming": 1. If you don't have direct market access, your order gets routed through a broker. The broker sees your order before it hits the market, and if he jumps in line ahead of you then that is front-running, and a Bad Thing. Your broker can get in a lot of trouble for this sort of thing. 2. If you are trying to move a large position by sending multiple orders to the market (one after another), then all market participants have the chance to react to the first order. It's really a game to try to move a lot of inventory at once, without tipping your hand to anyone else in the room. Thems the breaks. Nobody else gets to see anyone's order before the matching engine has already processed it, so there's no way to jump "ahead" of it. OTOH, maybe your long-term investor is trying to time the market: wait for a signal intraday, and pick that moment to send an order. In that case, if it is a good intraday signal then it is likely that someone else will compete. It is unlikely for a long-term trader to have spent as much on infrastructure as a HF trader, so the juicy signals will result in missed executions that _look_ like front-running.
zygomega 14y ago
No, I'm already in the finance industry. There are two main reasons I'm doing this: - I think the finance industry is very closed when it comes to intellectual property development, and an open source approach can be seriously competitive. An open approach may well be the future when it comes to being 'connected' - HFT is an interesting multi-disciplinary problem and the shear breadth of expertise required - modern chip design, low-latency software/hardware interaction, lock-free concurrency, fault-tolerant system design, adaptive learning algorithms, k-means clustering - means I'm learning heaps every day. I just don't agree that HFT is a scourge. It's an ecological shift (neither good or evil) and longer-horizon investors need to evolve.
- nezza-_- 14y ago
  No, long-horizon investors are normally overall not interested in HFT because they won't 'suffer' from it.
- zamalek 14y ago
  "HFT is an interesting multi-disciplinary problem" This makes this a very worthwhile project: whether or not anything comes of it (I considered writing technical analysis trading software for the exact same reason). Some documentation about the patterns and techniques you used would likely benefit the community a tiny bit more. Not everyone understands Haskell (or, forbid, has the time to peruse large codebases). Best of luck to you!
  - xradionut 14y ago
    There are many other interesting multi-disciplinary problems that are not as contentious as HFT and probably more publically acceptable and worthwhile. But if HFT is the domain you care about, go for it.
- white_devil 14y ago
  > an open source approach can be seriously competitive Do you really think any Joe Schmoe off the street could just grab your open source HFT and start making money with it? If not, how does your project benefit anyone? Are you working in the industry, but not in HFT? Maybe this is project is just practise for getting a job in HFT? I imagine that's where programmers get the fattest paychecks in the world. Only a fraction of what the sociopaths running the show get, of course, but good money nevertheless. > I just don't agree that HFT is a scourge. Good thing you're not at all biased.
topbanana 14y ago
> As you may well be aware, HFT is a scourge on the world's economy, and it's a game only the biggest and best-connected players benefit from. This is the line peddled by the popular press but it isn't true. HFT works to reduce spreads, which benefits both buyers and sellers.
- fnordfnordfnord 14y ago
  Latency arbing is a scourge, and makes limit orders practically useless. Quote stuffing, would be illegal if trading were still done on little slips of paper. Imagine dumping 10,000 slips of paper on the trading desk, and then shouting "just kidding!" In the words of Lawrence from Office Space, "You'd get your ass kicked" Or even worse, 10,000 empty bids. That's why people hate HFT.
  - kasey_junk 14y ago
    I'm not trying to be argumentative, but I am curious why you find latency arb so problematic, and what you think it's impact is on limit orders? I for one think latency arb is one of the bigger net wins for hft. As a market participant, each venue I have to maintain a presence at is a cost to me. I'm willing to pay the latency arb shops their cut to provide me price consistency because for my models it is much cheaper to do so than to continually reevaluate and redeploy to every possible venue. It frees me to shop for venues that provide the best features and fees. As for quote stuffing, you are absolutely right it is awful. That's why almost every venue out there has taken or is taking steps to curtail it. They did so because their customers agree with you.
    - dchichkov 14y ago
      'Quote stuffing' is there, because of absolutely ridiculous SEC rules. If you'd put in fractional prices and remove NBBO rules, there wouldn't be any 'quote stuffing'. And latency arb. What's wrong with latency arb? How else can you move information from one exchange to another fairly? Think of HFT's as of part of the financial network infrastructure. Where exchanges play role of nodes and HFT companies role of links/queues/buffers.
fnordfnordfnord 14y ago
>HFT is a scourge on the world's economy I'd point out that a lot of the things that are going on in HFT, are rehashes of old trading scams, and either are or would be illegal if there was any adult supervision. I was tempted to say something about the SEC being left behind the technology, and doesn't understand it. But for that to be true, and have this crap go on for so long, they either need to be complete fools, or they benefit from the status quo. >and it's a game only the biggest and best-connected players benefit from. That's definitely true in the case of latency arbitrage. But here again. It's the exchanges themselves that have decided that some players get an advantage over others. As an example, latency arbitrage, and a whole host of other problems could be sorted out by putting traders on exchange-hosted virtual machines. http://www.dailyfinance.com/2010/06/05/rigged-market-latency-arbitrage-3-billion/ http://www.dailyfinance.com/2010/06/05/rigged-market-latency... If not virtual machines, there are other ways to fix these problems, but first you need to get the players to acknowledge that there is a problem. http://www.zerohedge.com/article/do-it-yourself-latency-arbitrage-how-hfts-can-manipulate-nbbo-whim-courtesy-nyse-quote-stuff http://www.zerohedge.com/article/do-it-yourself-latency-arbi... Nanex has done a lot of good analysis of HFT, some of which is published here: nanex.net/FlashCrash/OngoingResearch.html
- ycombobreaker 14y ago
  Nanex is wonderful to read, but IMO they frequently misinterpret the data. Off the top of my head: - They only use consolidated feeds for US equities, never direct market-data feeds. The consolidated feed necessarily contains less information that direct feeds (to satisfy more stringent bandwidth requirements), which masks some "interesting" effects of how the exchanges publish their data. - They disregard that the CME feed publishes a fixed depth-of-book, and whenever they look at total liquidity in the book it can appear to flicker when deep levels fall "out" of the back of the book, even if liquidity is actually improving with the presence of a new inside level. - They make a big deal about wholesaler matching only occurring when the consolidated book is not locked. Their rationale is that subpenny prices are always wholesalers, and (erronously) therefore a lack of subpenny-priced trades must mean a lack of wholesale matching. These mistakes sound believable, but they do not hold up to any of scrutiny. Use their site to find interesting events, but be very careful about taking their conclusions at face value. Edit: bullet-list formatting

cmdkeen 14y ago

I'm curious about the "best trading platform" claim - what is going to make it better than any other? Especially as the real High Frequency Traders are busy spending fortunes on placing their hardware as close to the exchange as physically possible.

zygomega 14y ago
Speed is only one issue with autonomous algorithm design. Yes, the speed thing grabs the headlines but the boiler-plate objective is to front-run the slower players. There's a wealth of opportunity in processing the event stream in a more robust way than others and faster too. Think semi-HFT, semi-autonomous. Most trading platforms are primarily loss leaders for the 'professional' version and otherwise attached to a non-open source business plan.
- cmdkeen 14y ago
  But doesn't that mean you're at the mercy of the professional HFTs who can "front run" anyone running on your platform?
  - kasey_junk 14y ago
    You've hit a pet peeve of mine so sorry about the tirade. People use the term "front run" incorrectly and it is important to point this out. Front running is a very specific activity and is illegal. If there is evidence of front running people need to be prosecuted. That said, I've worked on several systems and never seen any of them that would allow a faster market participant to see your order flow without your permission. This is what front running is. People who don't understand the term seem to think seeing market data before other participants is front running it's not. There have always been and will always be some traders that see (and can react to) market data faster than everyone else. In the era of electronic trading this information latency cost is going down, not going up.

nezza-_- 14y ago

You need to do instead of icon-iconname... The links all have icon names in them. Edit: Ah. It now works only with JS on when putting the icon-name into and not in the class. Sorry for that, seems to be new in Bootstrap.

Alexx 14y ago
It's Twitter bootstrap. http://twitter.github.io/bootstrap/base-css.html#icons http://twitter.github.io/bootstrap/base-css.html#icons Bootstrap V3 has tweaked the way icons are implemented slightly.
- deleted 14y ago
  [deleted]
  - deleted 14y ago
    [deleted]
- zygomega 14y ago
  thanks heaps for that!

deleted 14y ago

[deleted]

cr1t1calh1t 14y ago

Check out IBrokers: http://cran.r-project.org/web/packages/IBrokers/index.html http://cran.r-project.org/web/packages/IBrokers/index.html It'll get you up and running with IB in no time.

zygomega 14y ago
Yeah, IBrokers is great. R is even better! Everytime I need a specialist piece of code it seems to already exist as an R package and to have been released in the last few weeks.
brotchie 14y ago
On a similar note, a while back I wrote a bi-directional, fan-out adapter between the Interactive Brokers API and ZeroMQ. https://github.com/brotchie/ib-zmq https://github.com/brotchie/ib-zmq The annoying thing about the IB API is that there's no framing; that is, you can't simply consume the message types you are interested in without parsing the entirety of every variable length messages. ib-zmq resolves this annoyance by parsing incoming messages and placing them individually into ZeroMQ message frames. I also wrote an alternative to the IBrokers R package with a much nicer interface using this ZeroMQ adapter. It parses most IB API messages, but hasn't been used in production yet. https://github.com/brotchie/r-zerotws https://github.com/brotchie/r-zerotws

deleted 14y ago

[deleted]

nazka 14y ago

I am curious about this project because I love this subject. However, it is impossible to build an open source HFT system especially because it will be open source. The other sharks will know how you trade and they will only need to play against you by sending opposite orders every time. That is what happens when the source code of a big US bank were stolen. So it should be more appropriate to say it will be a fast trading system. The other reason is you can't build a perfect HFT trading system because it's a moving target. It is the war in the micro seconde market. The sharks will send you wrong orders just to disrupt your system or they will manipulate the market. And finally there is a hardware problem. To do a HFT trading system you need to rent your own severs inside the market, use FPGA cards, etc. In conclusion, I only say there is a wrong goal to say "I will build the best open source HFT trading system" rather than "I will build the best open source fast trading system". This project is interesting (and there is Haskell cool!) so I will stay connected to see how it evolves.

olalonde 14y ago
Couldn't the infrastructure be open source while the specific trading algorithms kept closed source?
- conformal 14y ago
  i believe that is how it works: open source infrastructure, private algos
  - zygomega 14y ago
    I intend to publish and collaborate on private algos. Even if others have the code they still don't know where I choose to operate and deploy capital. And if they get rich on the code good luck to them - maybe they'll sponsor a dedicated server for 1.0
- nazka 14y ago
  Yes you send your code on the servers of a company and you use their API to hit the market with their servers, or you rent your own racks. It will cost you a lot but in the HFT world it is the only solution. Any nano second counts. You are the fastest or you are not.

jschulenklopper 14y ago

This could be an interesting course for people wanting to know more about 'computational investing' and the algorithms behind stock (market) analysis and trading: https://www.coursera.org/course/compinvesting1 https://www.coursera.org/course/compinvesting1. Not specially targeted at HFT, but interesting basic background information nevertheless.

conformal 14y ago

you're doing it wrong: haskell is too slow.

zygomega 14y ago
I thought so too for a long while, then I tried to do a touch of concurrent code in c++ and had to gouge my eyes out. I'm excited about the speed up haskell brings to development. You can plan things in haskell you can't imagine in other languages.
- conformal 14y ago
  if concurrency with C/C++ is bad for you, go makes it very easy. however, it is not very fast either. doing it in go and then using cgo where necessary will get you pretty close to C speed.
- NateDad 14y ago
  +1 for doing it in Go. If concurrency is your objective, Go makes it easy... it's also very fast both to program in and to run.
  - cakebread 14y ago
    > +1 for doing it in Go Conspicuously you don't address Haskell at all. Then again, who cares what might actually be best for OP? There's an advocacy bingo card in play!
zura 14y ago
But how elegant it is, umm... :)
z3phyr 14y ago
How much experiance do you have in programming, for claiming that?
- sseveran 14y ago

fmstephe 14y ago

I'm building an open source matching engine. I would love to pair up the two systems for a stress test. I'll keep tracking your project and ping you again when I have a system up a running if you're interested. https://github.com/fmstephe/matching_engine https://github.com/fmstephe/matching_engine

zygomega 14y ago
We have some plans on the drawing board for market maker simulators that could do with some fast matching so I'll keep an ear open for that ping.
- fmstephe 14y ago
  What sort of performance baselines would you require?
  - zygomega 14y ago
    I have no idea yet. I see you're geeking out in go and there's been a lot of advocacy coming through to look at go as an alternative.
    - fmstephe 14y ago
      Yeah, Go is a great language. However, I am trying to develop some low level queues that should be faster, and less generic, than the ones that Go provides. But I am having some difficulty putting memory fences into my code. But if I can get a reasonable level of performance out of the standard Go queues then I will just use those in the meantime.
ycombobreaker 14y ago
I see a comment about ITCH in there... Are you modeling your interface after NASDAQ's ITCH/OUCH interfaces? If so, I think it would be interesting to look at the protocol specifications for all of the other major US equity exchanges. I think they are all freely-downloadable.

niggler 14y ago

Have you actually used this in a production setting?

zygomega 14y ago
No, and I am making no claims. I have a market event feed coming in, a good idea of what the event processing looks like and a rough idea of how to send an order to a broker. I think the project needs to get to production fast though.

boothead 14y ago

Have you considered python for the research/algo side of things? I work in a hedge fund in London at the moment and there's a massive shift away from R towards python in the algorithmic shops that I know about here. There's also the great work that quantopian are doing on their backtesting framework zipline [1] [1] https://github.com/quantopian/zipline https://github.com/quantopian/zipline

zygomega 14y ago
Yes, I am aware of that trend and considering a switch. I'm personally more used to R is all, but if I found collaborators I'd jump to where the code base goes. The only other factor is I find R pretty aligned with haskell having a somewhat functional pedigree, so that code translates pretty nicely between a rapid hack at the problem to the more robust approach.
- boothead 14y ago
  Are you planning to run R in production, or is it purely for research that then gets translated to haskell?
  - zygomega 14y ago
    I see the research as production. There's an autonomous system processing the event stream and deciding on trades that's hands free. I see that as a haskell codebase with C speedups for critical stuff, with no R anywhere to be seen. But then I see a wrapper piece that reports to humans about what's going on within the system. I see R as a great visualization tool for that. I've had good success so far with using R as a fast column database, solving big data hassles, but there may be better solutions in that space.
    - boothead 14y ago
      I've had a quick read through the .org file now and it appears that you're using R for things like moving averages over the stream of prices. Have you considered using one of Haskell's FRP (or pipes/conduit) libraries for this side of things?

guiomie 14y ago

I thought HFT was now based on FPGAs and ASIC circuits ... how could this compete in terms of latency/processing ?

zygomega 14y ago
By putting it on FPGAs maybe. Stage one is to get a robust process for measuring and researching latency - I'm in Australia and US data is bouncing into my mac via a lousy connection and a tcp port in 250 millsecs on average. There's a long way to go. I suspect that if you can keep within a bulls roar of the low-latency crowd the big gains will be on the algorithm processing side. Algorithms still add up a few million numbers every time they recalculate a moving average - there's better ways to do it.
consz 14y ago
FPGAs and ASICs are rare -- at least the HFT firm I work for, they're basically non-existant.

ablut 14y ago

Isn't the barrier to HFT the fact that you need enough capital so that your profits cover the cost of co-located servers and FPGAs in the exchange datacenter, without which you have a latency handicap? (in addition to, of course, coming up with good algorithms)

zygomega 14y ago
Yes, if the only edge you have is speed. But a good algorithm may well be the next battleground as the benefits of ultra-low latency reduce.

qompiler 14y ago

Sorry to burst your bubble but high-frequency trading is only viable and necessary for market makers. The end result of any type of algorithm you think of is always a curve-fitting function of existing data.

zygomega 14y ago
No need to apologise - curve fitting is a big issue. But I would think that everything that happens on kaggle would be curve fitting by your definition right? And how is the stuff that happens in brains something other than curve fitting? It's how you connect the data dots and what you choose to focus on that makes the difference.
- qompiler 14y ago
  You are asking how the human brain works, that's a whole different issue and I don't know what Kaggle is/does.

iTrollFreely 14y ago

i don't think you know what high frequency in hft means. haskell, r, emacs org-mode (lmao wtf?)... yeah good luck with that.

tolitius 14y ago

1. Timestamps in your seed data might benefit from nanoseconds if you really talking about "high" frequency. 2. I agree with you comment that it is easier to think about concurrency in Haskell than in something like C++, however you can't really compete with C/C++ in Haskell. Not even with cgo (Go packages that call C code), not with OCaml or any other higher level beasts that promise the speed. Fortran would be the only one faster for the "algo" part of your initiative. But again, if this is just an exercise, Haskell and others (I prefer Clojure for example :) will do just fine. 3. Would make sense to split the "platform" in two (very different) parts: "Quantitative Analysis" (a collection of tools and rules) and "Technical Glue to Read and Stream". Each can/should be divided further of course, but the two above are essential yet very different for a true "HTF Platform".

zygomega 14y ago
I wish I could get down to nano units. iqfeed (a good value feed) just got millisecs in so will settle for that. I'm preparing some speed tests between C++ and haskell on an identical block of processing so stay tuned! You might be surprised - haskell is way ahead of clojure on compiler smarts. The split you suggest is exactly what I think is wrong with the way things get done right now. I'd like to integrate the quant inside the read and stream - now that's potentially a large speed up that might compensate a tight budget.
- mayank 14y ago
  Keep in mind that if you're not an experienced C/C++ coder, you're going to be (largely) benchmarking your relative ability in either language rather than the intrinsic speed of each language.
  - kasey_junk 14y ago
    This is true of any language/coder. It's what makes cross platform benchmarking so difficult for non-trivial problems.
- deleted 14y ago
  [deleted]

otikik 14y ago

I would be much more interested in an open source solution to fight against HFT, or at least make it more difficult. I see the financial world as a necessary evil, and I think some of its parts have benefits for the rest of the world. HTF is not one of those parts; I see it as a pure burden. You could have made an open source Patent Trolling system and I would feel the same.

zygomega 14y ago
Wow, that's a pretty low ranking being shoved into the patent troll category of evilness. When someone goes and patents an obvious algorithm at some point in the future and my little project shows up as prior art would I be redeemed :)

rags_123 14y ago

+1 for using Haskell.

dhosek 14y ago

Aside from all the other points raised against this, there's also the fact that the timing on HFT is so close that it requires close colocation of the physical hardware to the trading systems. Only big boys get to have access to those racks.

alexforster 14y ago

I have no experience in the financial industry, but HFT has always fascinated me as a potential source for very high rates of "events". Could you share your general insight about the sheer volume of data that commonly gets pushed through an HFT system? I'd also be terribly interested in a multi-megabyte/gigabyte "recording" of HFT trade data. http://www.nyxdata.com/capacity http://www.nyxdata.com/capacity

sseveran 14y ago
This project is not HFT. It does not use a full book data feed. Really even the feeds that come from the exchanges are not precisely timestamped enough and HFT firms stamp their own . The data is not huge although trading system need to be able to deal with large spikes in the rate of events. Usually now people are using 10G/40G/Infinibad as a connect to the matching engine.
zygomega 14y ago
So, yes, on my toy wannabe hft feed, I'm currently clocking about 10 million ticks per day on about 8 futures tickers I'm tracking. The raw feed is between 1 and 1.5 gig per day. I can imagine the petabyteness of the bigger guys.

z3phyr 14y ago

Good to see Haskell. I must give you a complement for being part of pushing Haskell to the mainstream! BTW, many trading firms do use haskell in Trading, but thats all privately held.

kasey_junk 14y ago

This is a nice little research project, and I hope you learn a lot from it. Having written algorithmic trading systems I think you are missing on a couple of central points: 1. There is no such thing as the "best" algorithmic trading platform, because algorithmic trading is such a broad term. Architectures that make sense for one class of trades does not make sense for other class of trades. 2. Contrary to popular opinion on this forum, algorithmic trading is not necessarily only low-latency and high-frequency. Frequency, latency, and algorithmic levels are really dials on the specification system and along with cost dictate how you will build the system. Do some work figuring out the trades you are targeting before setting those dials. 3. The actual execution system part of an algo trading system is usually the easiest part. If you've written one before, writing new ones is usually trivial. Finding the trade, building appropriate risk systems/practices, building back testing frameworks and exchange reference data systems are all much more challenging and take the majority of the time in these systems. 4. Finally, if you are truly targeting low-latency and high frequency events concurrency is your enemy, not your friend. To everyone saying that the cost in these systems is the cost of co-locating servers or fpga cards, you're wrong. You can get hosting deals/leases on those kinds of things for the same cost as high-end web hosting. The cost of running these systems are 2 fold: 1 paying the employees (because your competitors can pay them a lot) and 2 having deep enough pockets to survive the bad days/weeks/months. These are the same costs that have always existed in the trading space and have nothing to do with electronic trading. In fact, electronic trading has lowered the information asymmetry and made it easier for new participants to be involved in the markets.

bink-lynch 14y ago
> ...truly targeting low-latency and high frequency events concurrency is your enemy, not your friend. This reminds me of the LMAX financial trading platform where they started with a concurrent model but ended up using a single-thread "...that will process 6 million orders per second..." http://martinfowler.com/articles/lmax.html http://martinfowler.com/articles/lmax.html
- windust 14y ago
  Yes, it is interesting, when you are becoming highly concurrent the penalty for context switching start taking its toll, but that's only when the rest of your trading system is already optimized.

mikevm 14y ago

How can someone who has absolutely no idea about the financial industry and HFT learn about it? Can anyone recommend any resources (books, tutorials, etc..)?

relaxitup 14y ago
mikevm, check out Dark Pools by Scott Patterson. Great book about the evolution of HFT: http://www.amazon.com/Dark-Pools-High-Speed-I-Financial/dp/0307887170 http://www.amazon.com/Dark-Pools-High-Speed-I-Financial/dp/0...

rbc 14y ago

I worked on a framework for managing stock and options positions based on feedback methods developed by Stafford Beer. It doesn't do the algorithmic/transaction part, but rather focuses on providing reporting and alerting for large hierarchies of institutions, accounts, stock and options positions. The proprietary algorithmic code could be added by subclassing the code that I wrote. It's called the Viable System Agent. It's written in Smalltalk, tested under both Squeak and Pharo. It's licensed under the BSD license. It can be found at: http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ Look at the RBC-VSA-Portfolio category. That's where all the stock/option code is.

carterschonwald 14y ago

In another month or two the analytical tooling for doing stats and numerics in Haskell land are going to have a huge leap forward in capabilities. Might be worth considering going full Haskell then :-)

sseveran 14y ago
Whats happening then?
- carterschonwald 14y ago
  you know exactly what i'm up to slowly with wellposed :) (building numerics/ data analysis tools that dont' fill me with rage and ire over terrible engineering and usability. A matrix / linear algebra kernel of tools are on track to be ready for hackage release + paid pro versions in 1-2 months)
  - zygomega 14y ago
    Classical landing page had me hooked anyways :-)
    - carterschonwald 14y ago
      thankee good sir. I'm glad some people appreciate my vague semblance of prioritization skills. Hopefully that pans out to being able to have any early customers be of the sophisticated sort that I'm excited to work with / help!
      - nazka 14y ago
        What is the name of your library? Or where do you post news?
        carterschonwald 14y ago
        It's not out yet. If you really want to hear about things as they happen, signup for the announce list linked to from www.wellposed.com , I've yet to fire off any emails to that list, but I anticipate 2-3 emails over the next 1-2 months (after a year of hard work and focused thinking). What's also kinda awesome is I think the alpha release with all the Lin alg functionality should be under 2k loc. a lot of the work has been figuring out how to make the design composable and extensible enough that I can write a first working version with good performance just on my own. Ironically, that's also a compelling way to Validate that my tech delivers.

bdunbar 14y ago

> using haskell,R and emacs org-mode. That last is so very, very, cool. I'll have to dive in, take a look, see if I can find out _how_ you're using it. Been wanting to do a project with org-mode for a while, now.

zygomega 14y ago
It is - I can't imagine working without it. But I'm not sure I'm the poster child for how to use org-mode properly. Every piece of code in the repo actually resides in the org file which is a big monolithic journal of where I'm at. o-blog is a great emacs package that lets you publish sites straight from org.

indigo_daddy 14y ago

On a possibly related note, Josh Levine, creator of the Island trading engine, released the Foxpro source a few years ago.. Interesting stuff.. http://josh.com/notes/island-ecn-10th-birthday/default.htm http://josh.com/notes/island-ecn-10th-birthday/default.htm http://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.TXT http://josh.com/notes/island-ecn-10th-birthday/ISLAND.PRG.TX...

SeanDav 14y ago

The industry is doing the vast majority of its HFT development in C++ or Java. Data analysis is mostly done using Python although I still see a lot of R. If you are doing this to break into the industry, I suspect the languages you used should have been the above. Also the above languages would probably have been better to attract open source developers who are also hoping to use their code and experience from this project to break into the industry.

zygomega 14y ago
I've been in the industry for too long (though not in low-latency) - I'm doing this to break out of the industry! And I think the finance game is long overdue for a good disruptive technologic event (not implying my humble project is it). R is a personal choice for data analytics (I love ggplot2). But the haskell thing is more than that. It's been done before (http://www.starling-software.com/misc/icfp-2009-cjs.pdf http://www.starling-software.com/misc/icfp-2009-cjs.pdf) and even goldman's has a thriving erlang/OCaml hacker ethic (http://www.zerohedge.com/article/aleynikov-code-dump-uncovered http://www.zerohedge.com/article/aleynikov-code-dump-uncover...).
fnordfnordfnord 14y ago
He didn't really mean HFT. At least I don't think he did.

fnordfnordfnord 14y ago

FYI: This isn't an HFT bot but is a decent, recent Python + ncurses trading client framework for Bitcoin on MtGox. http://prof7bit.github.io/goxtool/ http://prof7bit.github.io/goxtool/ Even if you aren't interested in Bitcoin, it might be useful as a real-ish and cheap place to test with low barriers to entry. I've found that backtesting on historical data is usually not realistic enough since most people fail to consider liquidity. I'm interested because I like to trade, would like to learn more, and I have had some small success. But I have trouble sticking to my plan. I let emotion get the best of me and wind up losing my gains.

zygomega 14y ago
Isn't MtGox over as a bitcoin trading platform? I agree more generally that bitcoin could be a useful market to test out. And your story fits exactly with a hands-off automated approach to trading. We wont ever be charting a price series in the production environment because computers don't get charts.
- fnordfnordfnord 14y ago
  >Isn't MtGox over as a bitcoin trading platform? I wish. I keep waiting for some more interesting news from Coinlab [1][2], but so far, Mtgox are still the largest exchange by volume. Here is a chart showing volume by (selected) exchanges. http://bitcoincharts.com/markets/currency/USD.html http://bitcoincharts.com/markets/currency/USD.html The recent events seem to only have got them more customers (though it remains to be seen how many of them will stick). In any case, some of the other exchanges have API's that are similar to mtgox. [1] https://mtgox.com/press_release_20130228.html https://mtgox.com/press_release_20130228.html [2] http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab-attracts-500000-in-venture-capital-for-bitcoin-projects/ http://www.forbes.com/sites/jonmatonis/2012/04/24/coinlab-at...

mangrish 14y ago

Cool! Very interesting. I think given the languages you have selected you are coming more from a quant background? These languages are great for heuristics and analysis but you would really want all 'static' components such as connectivity built in assembly/C/C++. For 'algo' components I like Java as you can still pull microsecond order latencies when crunching numbers but more importantly it gives you a huge time to market advantage than the C/C++ for almost the same speeds. I'm also not clear on if you are connecting directly to the market for market data or using aggregation (like reuters). The latter would be too slow. I'm also not clear on what middleware you are using which is probably the biggest decision you will have to make. Most either use inhouse tech or 29west LBM (everyone still calls it that even though they were bought out). An overlooked part of HFT in my experience OS optimisation and even things like TCP bypass (for some components) which can lead to huge speed advantages and end to end latency reductions. I agree with those about FPGA.. in my experience they really don't come into the equation except for components that rarely change. Anyway a few guys including Martin Thompson have felt similarly to you and initiated the lodestone project (http://lodestonefoundation.wordpress.com/ http://lodestonefoundation.wordpress.com/). If you are keen to learn more their architectures for low latency, distributed and componentisation then I feel that you could join forces and contribute to that initiative. FYI: Martin built most of the technology behind the disruptor (http://lmax-exchange.github.io/disruptor/ http://lmax-exchange.github.io/disruptor/). Great to see interest in making this knowledge more widely available :)

zygomega 14y ago
thanks mangrish, quant. How could you tell? I'm conecting with an aggregator cause the direct market feeds are monopolistic price gougers. It's an easy switch if we make it up that curve. I'm not sure if I even understand what middleware is (I'm a quant), but I think the answer is the disruptor!
- mangrish 14y ago
  From the languages.. I've worked with lots of quants and they all rave about R! Yeah..the aggregator is where you get done in both cost and latency. The prop shops and hedge funds pay through the nose for that stuff so unless you come packing a little capital, true HFT is an issue. On the middleware, not really.. so you would have a market data component that will be pushing stuff to various components (real time risk, the pricer, and the trading engine). The disruptor sits on the 'in' queue to those components.. the middleware is what pushes messages between instances running on the same/different machines. Hope that helps :)

zopf 14y ago

I assume you've checked out Marketcetera ( http://www.marketcetera.com http://www.marketcetera.com ). What will be your platform's primary differentiators?

zygomega 14y ago
Not having to register and login is one immediate differentiator. Marketcetera is a big, big code base - I think our ambitions are more focused.

relaxitup 14y ago

ycombobreaker 14y ago
Jerk boy!

dasickis 14y ago

I have access to servers in all the major trading rooms if you need to start deploying this to a production environment to start trading. Contact me at dasickis [at] gmail.com (I'll reply back from my non-filtered e-mail address)

ttty 14y ago

Which software you used to build the diagram on the main page?

zygomega 14y ago
The graphviz dot language: http://www.graphviz.org/Documentation.php http://www.graphviz.org/Documentation.php It has good support in org-mode and a there's a nice haskell package that takes the dot code and turns it into an internal graph representation.