5 ms·

Arms Race in Quantitative Trading?

34 points by cellis 17y ago

Maro 17y ago
"An easy example illustrating this point is maybe the following. Consider the sequence 'MDMD' and suppose that you can only store, say these 4 letters. A 4-letter-strategy might predict something like 'MD' for the next two letters. If those letters though represent the initial of the weekdays, the next 3 letters will be 'FSS'. It is impossible though to predict this sequence solely using information about the past on the last 4 letters. The situation changes if we can store up to 7 letters 'FSSMDMD'. Then a prediction is possible." I was confused for a while, then I realized the guy is german. Montag (Monday) Dienstag Mittwoch Donnerstag Freitag Samstag Sonntag (Sunday)
w1ntermute 17y ago
Is there anyone on HN who works as a programmer in this field who would be willing to share some of their experiences?
- ig1 17y ago
  What sort of stuff do you want to know ?
  - w1ntermute 17y ago
    Just any notable stories (anonymized, of course). I wouldn't really know where to start asking questions, but here are a few: What sets the field apart from other areas of programming, in terms of your problem-solving approach? What kind of special qualifications do recruiters look for? Also, what kind of platform (OS, language, etc.) do you use? Do you hand-optimize the code?
    - ig1 17y ago
      Generally C++ is the dominant language (although Java is growing), unix the platform (Solaris and Linux mainly). I know at least a couple of firms that are embedding code onto programmable network cards for equities trading. If you go to the eFinancialCareers website and search for "low latency" you can see the kind of thing recruiters look for. In terms of problem solving it's not that different from other areas. Concurrency is obviously very big (lock free algorithms, etc) as is being comfortable with low-level stuff. You don't need it all the time but sometimes you do need to be able to read through kernel code and get an intimate understanding of things like tcp/ip.
      - w1ntermute 17y ago
        Generally C++ is the dominant language (although Java is growing) Do you typically hand-optimize the assembled code?
        ig1 17y ago
        No. Some places do hand-optimize particularly cpu intensive parts of code , but it's not the norm for most code. I presume the guys who embed code on network cards probably do it more often, but I don't know.
    - gaius 17y ago
      Yaron Minsky talks a a bit about this: http://janestcapital.com/?q=node/61 http://janestcapital.com/?q=node/61
  - rw 17y ago
    I want to know what the ratio of "junior developers" to "MIT PhDs" is. As an economics+machine-learning+software-engineering autodidact, I firmly believe that I am qualified to provide insights in high-frequency trading; but, the impression I get is that, since it's such a cool and profitable field, the firms have their pick of the elite-of-the-elite.
    - falsestprophet 17y ago
      They do. And they probably won't give you a second look.
    - obiefernandez 17y ago
      I know programmers at one of these kinds of firms, without degrees and making $200-500k per year, including bonuses. It's all about talent.
    - eru 17y ago
      Try Jane Street.
    - gaius 17y ago
      When I worked in the field (not HFT, but automated trading nonetheless), I was the only one in my cluster of desks with a lowly MSc. Even the QA girl was a maths PhD. It was funny, all the alpha-PhD male programmers were always competing over who was the second-smartest... The smartest guy was light years ahead of anyone else (and the nicest bloke you could hope to meet). I always used to say, the rest of them should pick up his laundry and wash his car so he could concentrate more (they really liked that, HAHA). I didn't do anything beyond the simplest algorithms, most of my work was on interfaces between our code and other stuff (like databases, data feeds, etc). Just grunt work really. Someone's got to do it tho'...
ig1 17y ago
I disagree, those milliseconds are important as the market is continuously moving. From the time a trade enters your system until it's booked and executed you're carrying the risk that a deal you think you've done might not have been done. When you're processing thousands of trades a second that's not a small risk.
- ggruschow 17y ago
  Although that's true, it's an unnecessary problem they've created for themselves through their now over-simplified trade matching and naive insistence on time-priority right down to time-frames so small that nobody could possibly care outside of the lame system. Think about it - even if you got the news the president was shot, or even if the unemployment numbers are significantly off, you'll only be able to take small bets initially due to the lack of confidence in the news until further confirmation.. And confirmation can only come from reading or hearing words, which even if a computer does that, somebody has to type or say them... which means seconds, not milliseconds, much less microseconds.
  - andrew1 17y ago
    The 'news' events they're reacting to are financial events at the level of price changes, order book depths etc. They don't react to things like the president being shot, and I'd imagine that if that did happen then at least some of those traders would be switching their models off - the markets could well go crazy at that point and a model won't have been back-tested in those conditions, hence you have no idea what will happen.
    - ggruschow 17y ago
      I think we agree on what you said, but I take it further and claim it's evidence that the common trade matching rules are badly flawed. As a simple example, consider that common limit order matching rules often let time take priority over anything else (price, quantity, market maker status). That fails their stated goals. They've got concurrency bugs. It initially came from naive implementations of pit trading rules, but it's now often touted as a feature. As a result, customers often get screwed into trades they didn't want, or get worse prices than what's fair. PLUS as a result of all the jockeying, they pay higher fees to support the escalating infrastructure costs.
anonymous 17y ago
I remember talking to a guy working in this field when I was job hunting earlier. For all the talk of CPU speed, there seemed to be little interest in nontraditional architectures (e.g., FPGAs, etc), for the reason that their incoming data rate was the primary bottleneck. Then again, I may have been talking to the wrong one. E.g., maybe a few've gotten some of those fpga ethernet boards? Still, I'd expect a little more VHDL demand than I'm seeing.
known 17y ago
HFT = http://en.wikipedia.org/wiki/Algorithmic_trading http://en.wikipedia.org/wiki/Algorithmic_trading + http://en.wikipedia.org/wiki/Complex_Event_Processing http://en.wikipedia.org/wiki/Complex_Event_Processing
1gor 17y ago
I wonder why the term 'market efficiency' crops up over and over again in the discussions like this? In 1986 Andy Lo and Craig MacKinlay performed a rigorous statistical test of Efficient Market Hypothesis and decisively rejected it, simultaneously supporting previous findings by Benoit Mandelbrot. Unlike Mandelbrot’s their work has been broadly discussed and accepted by the scientific community.
SandB0x 17y ago
A touch off topic, but is anyone else dismayed at seeing so many bright friends disappear into banking as soon as they graduate? Nothing wrong with making money, but I just wonder how many inventions and discoveries are lost this way.
- andrew1 17y ago
  I suppose maybe the counter-argument to that is that the technological advancements that come about as a result of finance pushing for them, propagate out to the wider world and can help speed up the rate of discoveries as new technologies become available. I also have the feeling that quite often discoveries and advances are not really to do with the discoverer but to do with the fact that an idea's 'time has come'. Take Google for example, they revolutionized search, but supposing the founders had headed off to Goldman Sachs rather than doing their PhDs, what would have filled the gap? I think that the stage was set for new things to happen in the search area and that other researchers would have come up with similar ideas. I wouldn't want to say that the inventor is completely incidental to the discovery; I appreciate that there are brilliant people out there, and moments of inspiration, I just think that the world will steam-roller on with or without a particular person and we'll often end up at roughly the same place.
known 17y ago
I think SEC must confine HFT between 9am - 10am
speek 17y ago
Anybody have any pointers/books about into getting into Quantitative analysis/trading?
- SandB0x 17y ago
  All these people hang out at: http://www.wilmott.com/ http://www.wilmott.com/

34 points by cellis 17y ago

Maro 17y ago

"An easy example illustrating this point is maybe the following. Consider the sequence 'MDMD' and suppose that you can only store, say these 4 letters. A 4-letter-strategy might predict something like 'MD' for the next two letters. If those letters though represent the initial of the weekdays, the next 3 letters will be 'FSS'. It is impossible though to predict this sequence solely using information about the past on the last 4 letters. The situation changes if we can store up to 7 letters 'FSSMDMD'. Then a prediction is possible." I was confused for a while, then I realized the guy is german. Montag (Monday) Dienstag Mittwoch Donnerstag Freitag Samstag Sonntag (Sunday)

w1ntermute 17y ago

Is there anyone on HN who works as a programmer in this field who would be willing to share some of their experiences?

ig1 17y ago
What sort of stuff do you want to know ?
- w1ntermute 17y ago
  Just any notable stories (anonymized, of course). I wouldn't really know where to start asking questions, but here are a few: What sets the field apart from other areas of programming, in terms of your problem-solving approach? What kind of special qualifications do recruiters look for? Also, what kind of platform (OS, language, etc.) do you use? Do you hand-optimize the code?
  - ig1 17y ago
    Generally C++ is the dominant language (although Java is growing), unix the platform (Solaris and Linux mainly). I know at least a couple of firms that are embedding code onto programmable network cards for equities trading. If you go to the eFinancialCareers website and search for "low latency" you can see the kind of thing recruiters look for. In terms of problem solving it's not that different from other areas. Concurrency is obviously very big (lock free algorithms, etc) as is being comfortable with low-level stuff. You don't need it all the time but sometimes you do need to be able to read through kernel code and get an intimate understanding of things like tcp/ip.
    - w1ntermute 17y ago
      Generally C++ is the dominant language (although Java is growing) Do you typically hand-optimize the assembled code?
      - ig1 17y ago
        No. Some places do hand-optimize particularly cpu intensive parts of code , but it's not the norm for most code. I presume the guys who embed code on network cards probably do it more often, but I don't know.
  - gaius 17y ago
    Yaron Minsky talks a a bit about this: http://janestcapital.com/?q=node/61 http://janestcapital.com/?q=node/61
- rw 17y ago
  I want to know what the ratio of "junior developers" to "MIT PhDs" is. As an economics+machine-learning+software-engineering autodidact, I firmly believe that I am qualified to provide insights in high-frequency trading; but, the impression I get is that, since it's such a cool and profitable field, the firms have their pick of the elite-of-the-elite.
  - falsestprophet 17y ago
    They do. And they probably won't give you a second look.
  - obiefernandez 17y ago
    I know programmers at one of these kinds of firms, without degrees and making $200-500k per year, including bonuses. It's all about talent.
  - eru 17y ago
    Try Jane Street.
  - gaius 17y ago
    When I worked in the field (not HFT, but automated trading nonetheless), I was the only one in my cluster of desks with a lowly MSc. Even the QA girl was a maths PhD. It was funny, all the alpha-PhD male programmers were always competing over who was the second-smartest... The smartest guy was light years ahead of anyone else (and the nicest bloke you could hope to meet). I always used to say, the rest of them should pick up his laundry and wash his car so he could concentrate more (they really liked that, HAHA). I didn't do anything beyond the simplest algorithms, most of my work was on interfaces between our code and other stuff (like databases, data feeds, etc). Just grunt work really. Someone's got to do it tho'...

ig1 17y ago

I disagree, those milliseconds are important as the market is continuously moving. From the time a trade enters your system until it's booked and executed you're carrying the risk that a deal you think you've done might not have been done. When you're processing thousands of trades a second that's not a small risk.

ggruschow 17y ago
Although that's true, it's an unnecessary problem they've created for themselves through their now over-simplified trade matching and naive insistence on time-priority right down to time-frames so small that nobody could possibly care outside of the lame system. Think about it - even if you got the news the president was shot, or even if the unemployment numbers are significantly off, you'll only be able to take small bets initially due to the lack of confidence in the news until further confirmation.. And confirmation can only come from reading or hearing words, which even if a computer does that, somebody has to type or say them... which means seconds, not milliseconds, much less microseconds.
- andrew1 17y ago
  The 'news' events they're reacting to are financial events at the level of price changes, order book depths etc. They don't react to things like the president being shot, and I'd imagine that if that did happen then at least some of those traders would be switching their models off - the markets could well go crazy at that point and a model won't have been back-tested in those conditions, hence you have no idea what will happen.
  - ggruschow 17y ago
    I think we agree on what you said, but I take it further and claim it's evidence that the common trade matching rules are badly flawed. As a simple example, consider that common limit order matching rules often let time take priority over anything else (price, quantity, market maker status). That fails their stated goals. They've got concurrency bugs. It initially came from naive implementations of pit trading rules, but it's now often touted as a feature. As a result, customers often get screwed into trades they didn't want, or get worse prices than what's fair. PLUS as a result of all the jockeying, they pay higher fees to support the escalating infrastructure costs.

anonymous 17y ago

I remember talking to a guy working in this field when I was job hunting earlier. For all the talk of CPU speed, there seemed to be little interest in nontraditional architectures (e.g., FPGAs, etc), for the reason that their incoming data rate was the primary bottleneck. Then again, I may have been talking to the wrong one. E.g., maybe a few've gotten some of those fpga ethernet boards? Still, I'd expect a little more VHDL demand than I'm seeing.

known 17y ago

HFT = http://en.wikipedia.org/wiki/Algorithmic_trading http://en.wikipedia.org/wiki/Algorithmic_trading + http://en.wikipedia.org/wiki/Complex_Event_Processing http://en.wikipedia.org/wiki/Complex_Event_Processing

1gor 17y ago

I wonder why the term 'market efficiency' crops up over and over again in the discussions like this? In 1986 Andy Lo and Craig MacKinlay performed a rigorous statistical test of Efficient Market Hypothesis and decisively rejected it, simultaneously supporting previous findings by Benoit Mandelbrot. Unlike Mandelbrot’s their work has been broadly discussed and accepted by the scientific community.

SandB0x 17y ago

A touch off topic, but is anyone else dismayed at seeing so many bright friends disappear into banking as soon as they graduate? Nothing wrong with making money, but I just wonder how many inventions and discoveries are lost this way.

andrew1 17y ago
I suppose maybe the counter-argument to that is that the technological advancements that come about as a result of finance pushing for them, propagate out to the wider world and can help speed up the rate of discoveries as new technologies become available. I also have the feeling that quite often discoveries and advances are not really to do with the discoverer but to do with the fact that an idea's 'time has come'. Take Google for example, they revolutionized search, but supposing the founders had headed off to Goldman Sachs rather than doing their PhDs, what would have filled the gap? I think that the stage was set for new things to happen in the search area and that other researchers would have come up with similar ideas. I wouldn't want to say that the inventor is completely incidental to the discovery; I appreciate that there are brilliant people out there, and moments of inspiration, I just think that the world will steam-roller on with or without a particular person and we'll often end up at roughly the same place.

known 17y ago

I think SEC must confine HFT between 9am - 10am

speek 17y ago

Anybody have any pointers/books about into getting into Quantitative analysis/trading?

SandB0x 17y ago
All these people hang out at: http://www.wilmott.com/ http://www.wilmott.com/