4 ms·

You probably can't do HFT trading because you need to have capital to reduce latency. Maybe you can rent servers very close to the trading centers, but this sti

You probably can't do HFT trading because you need to have capital to reduce latency. Maybe you can rent servers very close to the trading centers, but this still will cost money. You can use http://www.quantopian.com http://www.quantopian.com to try out different algorithms. It will tell you how well your strategy works.

dragthor 9y ago
quantopian.com sounds interesting. I am going to check it out.
hliyan 9y ago
Concur. You won't be able to beat players whose HFT systems are colocated in the same datacenters as the exchange. And you will not be able to beat those systems built using lower level languages like C++ or, worse yet, dedicated hardware. Source: 5+ years in HFT development
- InterestBazinga 9y ago
  Why does a programming language matter in terms of algorithms? Can you elaborate on that?
  - dllthomas 9y ago
    A lot of algorithms are dependent on the ability to execute quickly. If yours is not, then PL doesn't matter (at least, in the respect the parent was discussing).
  - hliyan 9y ago
    Every microsecond counts. We've done some pretty crazy sounding things to get the last bit of juice out of the system: 1. Write a converter to convert a proprietary interpreted business rule language to C++ 2. Make all messages fit the maximum ethernet frame size to avoid fragmentation overhead 3. Make the transported message format the same as the memory object format to avoid the packing/unpacking overhead 4. Predict and pre-allocate object pools in heap memory to avoid heap lock overhead 5. Statically link all libraries 6. Optimize key functions at assembly level 7. Turn all record ID's into zero based indexes and covert most lists and maps into arrays for rapid lookup 8. No indexes or foreign constraints in the rapid-write areas of DB 9. Hand rolled on-disk cache file formats that only operate in append mode to prevent seek overhead 10. And finally (this was after my time) trying to implement the TCP stack in FPGA
  - bluGill 9y ago
    Because big money will trade enough dollar value as to change the price by their action so whoever is second missed the opportunity. Languages like python are immediately out, they make no attempt to be fast (which is fine for their niche). Even languages like Java are out, the JVM is too smart: it turns out that the algorithm needs to analyse a few thousand possible trades where the answer no trade before it gets one where the answer is yes, as a result Java will optimize for the common no path. In C++ the programmer will lie to the optimizer claiming that the yes path is the most likely even though it isn't! As a result code written in C++ will always have the yes path as most optimal: when a trade should be made the java program pays a CPU branch misprediction penalty and the C++ program will not thus C++ wins the trade. Of course for the above to work you need cleaver programmers who spend time at the profiler and know how to make the CPU work for them. C++ has one other useful advantages over some of the competition: it won't waste CPU checking for things like dereferencing a null pointer. Speed is important, you should verify via other means (unit tests, formal methods static analysis, or have the checks that you run on dummy data and compile without them for production) Thus C and C++ are the only useful choices: the compilers have good optimizers (this rules out a lot of other compiled languages), they don't use CPU to check for "can't happen" errors, and you can lie to the optimizer in useful ways. There are other advantages, but the rest are things that a good programmer could easily work around (ie write a new stat structure from your CS textbook)
    - arisAlexis 9y ago
      this blew my mind :)
  - PeterisP 9y ago
    For HFT, it's not that every second counts, is that every millisecond (or even lower) counts. You're competing with other, similar algorithms for picking up opportunities. The assumption is that you're not capital constrained, you (or the competitors) can immediately exploit all the volume of such an opportunity, the deals you submit shift the prices so that it disappears. This means that whoever is not the first to take that opportunity doesn't get it, and if you're reliably a millisecond slower than a competitor then you might as well not even try.
- jameskegel 9y ago
  I’d be interested in picking up what you are putting down. Do you have a blog or other content channel?
  - hliyan 9y ago
    Unfortunately I got tired of that world around 2013 and moved to the food services industry. I usually write here: https://medium.com/@hnlean https://medium.com/@hnlean but it has little to do with HFT anymore... Edit: actually, see my response to the neighboring comment too.
timothylaurent 9y ago
You can also plug Quantopian into RobinHood for $0 trades.
- ARussell 9y ago
  Wasn't support for that removed?
  - westonplatter0 9y ago
    Yes, but check out https://www.oanda.com/forex-trading/tools/algo-lab https://www.oanda.com/forex-trading/tools/algo-lab. (not affiliated with them)
jeffreyrogers 9y ago
There was a great post on HN fairly recently written by someone who used to work in HFT. He talked about how they tapped the incoming network cable to read the incoming prices on an FPGA faster than they could make it through the OS's network stack. I think they were sending out trades in response to the new prices before they would have even made it to userspace on an OS.
- ah- 9y ago
  https://meanderful.blogspot.com https://meanderful.blogspot.com Worth a read.
  - jeffreyrogers 9y ago
    Yep, that's the blog. Specifically this post https://meanderful.blogspot.com/2018/01/the-accidental-hft-firm.html https://meanderful.blogspot.com/2018/01/the-accidental-hft-f...
- kasey_junk 9y ago
  As a bit of context, that technology will not be any where near your most expensive investment for HFT. Years ago I was on a trade where we could rent that technology for 1500 per month. It can only have come down from there. Your time is an order of magnitude more expensive than that. Other expenses besides your time include making sure you have the fee structure to be competitive & getting a clearing partner that is ok with letting you fire off enough orders to make it worth while.

dragthor 9y ago

quantopian.com sounds interesting. I am going to check it out.

hliyan 9y ago

Concur. You won't be able to beat players whose HFT systems are colocated in the same datacenters as the exchange. And you will not be able to beat those systems built using lower level languages like C++ or, worse yet, dedicated hardware. Source: 5+ years in HFT development

InterestBazinga 9y ago
Why does a programming language matter in terms of algorithms? Can you elaborate on that?
- dllthomas 9y ago
  A lot of algorithms are dependent on the ability to execute quickly. If yours is not, then PL doesn't matter (at least, in the respect the parent was discussing).
- hliyan 9y ago
  Every microsecond counts. We've done some pretty crazy sounding things to get the last bit of juice out of the system: 1. Write a converter to convert a proprietary interpreted business rule language to C++ 2. Make all messages fit the maximum ethernet frame size to avoid fragmentation overhead 3. Make the transported message format the same as the memory object format to avoid the packing/unpacking overhead 4. Predict and pre-allocate object pools in heap memory to avoid heap lock overhead 5. Statically link all libraries 6. Optimize key functions at assembly level 7. Turn all record ID's into zero based indexes and covert most lists and maps into arrays for rapid lookup 8. No indexes or foreign constraints in the rapid-write areas of DB 9. Hand rolled on-disk cache file formats that only operate in append mode to prevent seek overhead 10. And finally (this was after my time) trying to implement the TCP stack in FPGA
- bluGill 9y ago
  Because big money will trade enough dollar value as to change the price by their action so whoever is second missed the opportunity. Languages like python are immediately out, they make no attempt to be fast (which is fine for their niche). Even languages like Java are out, the JVM is too smart: it turns out that the algorithm needs to analyse a few thousand possible trades where the answer no trade before it gets one where the answer is yes, as a result Java will optimize for the common no path. In C++ the programmer will lie to the optimizer claiming that the yes path is the most likely even though it isn't! As a result code written in C++ will always have the yes path as most optimal: when a trade should be made the java program pays a CPU branch misprediction penalty and the C++ program will not thus C++ wins the trade. Of course for the above to work you need cleaver programmers who spend time at the profiler and know how to make the CPU work for them. C++ has one other useful advantages over some of the competition: it won't waste CPU checking for things like dereferencing a null pointer. Speed is important, you should verify via other means (unit tests, formal methods static analysis, or have the checks that you run on dummy data and compile without them for production) Thus C and C++ are the only useful choices: the compilers have good optimizers (this rules out a lot of other compiled languages), they don't use CPU to check for "can't happen" errors, and you can lie to the optimizer in useful ways. There are other advantages, but the rest are things that a good programmer could easily work around (ie write a new stat structure from your CS textbook)
  - arisAlexis 9y ago
    this blew my mind :)
- PeterisP 9y ago
  For HFT, it's not that every second counts, is that every millisecond (or even lower) counts. You're competing with other, similar algorithms for picking up opportunities. The assumption is that you're not capital constrained, you (or the competitors) can immediately exploit all the volume of such an opportunity, the deals you submit shift the prices so that it disappears. This means that whoever is not the first to take that opportunity doesn't get it, and if you're reliably a millisecond slower than a competitor then you might as well not even try.
jameskegel 9y ago
I’d be interested in picking up what you are putting down. Do you have a blog or other content channel?
- hliyan 9y ago
  Unfortunately I got tired of that world around 2013 and moved to the food services industry. I usually write here: https://medium.com/@hnlean https://medium.com/@hnlean but it has little to do with HFT anymore... Edit: actually, see my response to the neighboring comment too.

timothylaurent 9y ago

You can also plug Quantopian into RobinHood for $0 trades.

ARussell 9y ago
Wasn't support for that removed?
- westonplatter0 9y ago
  Yes, but check out https://www.oanda.com/forex-trading/tools/algo-lab https://www.oanda.com/forex-trading/tools/algo-lab. (not affiliated with them)

jeffreyrogers 9y ago

There was a great post on HN fairly recently written by someone who used to work in HFT. He talked about how they tapped the incoming network cable to read the incoming prices on an FPGA faster than they could make it through the OS's network stack. I think they were sending out trades in response to the new prices before they would have even made it to userspace on an OS.

ah- 9y ago
https://meanderful.blogspot.com https://meanderful.blogspot.com Worth a read.
- jeffreyrogers 9y ago
  Yep, that's the blog. Specifically this post https://meanderful.blogspot.com/2018/01/the-accidental-hft-firm.html https://meanderful.blogspot.com/2018/01/the-accidental-hft-f...
kasey_junk 9y ago
As a bit of context, that technology will not be any where near your most expensive investment for HFT. Years ago I was on a trade where we could rent that technology for 1500 per month. It can only have come down from there. Your time is an order of magnitude more expensive than that. Other expenses besides your time include making sure you have the fee structure to be competitive & getting a clearing partner that is ok with letting you fire off enough orders to make it worth while.