11 ms·

Donald MacKenzie on high-frequency trading

52 points by joosters 12y ago

joosters 12y ago
IMO this is one of the best articles I have read on HFT. It gives a good, balanced overview of the technology and the fundamental principles behind how automated trading works, without any particular viewpoint or opinion directing the text. The section near the end on FPGAs and programming styles was an eye-opener to me. I wonder how many other firms have taken risk checks out of their code to save some nanoseconds...
- conistonwater 12y ago
  > without any particular viewpoint or opinion directing the text You say that like that's a good thing. I know that a lot of rubbish has been written about HFT and algorithmic trading, but it's sad that no one seems to be able to write about its merits without starting a flame war.
  - Retric 12y ago
    It's an outgrowth of poorly designed matchmaking software at exchanges. The could completly distroy HFT without costing society anything or adding transaction fees.
    - conistonwater 12y ago
      Do you have any specific issues in mind when you say that? A lot of the rules that markets operate by come from standards and regulations, not software design. The software design reflects the kind of markets exchanges want to set up, so it's not a software problem. The people who wrote the standards had specific markets and specific outcomes in mind when they wrote them.
      - harmegido 12y ago
        I work in this industry. I've come to the conclusion that I will be leaving very soon. There are a number of reasons, but the primary one is because it's become clear to me that I spend the vast majority of my time on trying to figure out how to take advantage of flaws in exchange hardware/software. Sorry, I can't go into specifics. That being said, I don't know if it is possible to create an ungame-able matching engine.
mer10z 12y ago
I can't help but think of that article yesterday about how we're avoiding the Big Problems (https://news.ycombinator.com/item?id=8261098 https://news.ycombinator.com/item?id=8261098). It's crazy how much money and brain power people have invested in HFT which is essentially just turning time into money with no benefit to society. What could the human race have achieved if the same amount of effort was focused in other areas?
- afafsd 12y ago
  Don't be mad that some other people's brains are bent on stuff of no particular benefit to you. Their brains, their choice. Just be glad that some people's brains are bent on stuff of benefit to you, and concentrate, if you like, on what of general benefit you can do with your own brain.
- fchollet 12y ago
  It could be that our current economic system implements a pretty terrible resource allocation algorithm, where many workers' jobs consist in undoing the efforts of employees from concurrent companies. In trading, a quant's job is to outsmart or outspeed the other bank's team. They invest considerable brain power, effort and money into producing incremental improvements that are quickly rendered meaningless by the work of the team in the building across the street. Another prime example is advertising, a zero-sum game where competing brands fight to undo other brands' brainwashing efforts and replace it with their own brainwashing. A waste of the time and energy of million of humans. Some more ideas on the subject: http://www.sphere-engineering.com/blog/15-hour-work-week.html http://www.sphere-engineering.com/blog/15-hour-work-week.htm...
  - Retric 12y ago
    Advertizing and HFT are only zero sum for competing products in an established market. For new products it can be a net benifit ex: "talk to your doctor about ED."
- crdoconnor 12y ago
  The economy is geared towards playing such zero sum games. This is pretty much by design - policies such as austerity, ZIRP and financial deregulation shift the allocation of capital to speculation (gambling) and away from productive investment.
tobltobs 12y ago
Goldman Sachs’ Blankfein on Banking: ‘Doing God’s Work’. This kind of self-deception seems to quite prevalent with those banksters.
- GFK_of_xmaspast 12y ago
  They're totally doing god's work, it's just that the god in question is Mammon.
ord-lhr-nyc-nrt 12y ago
I worked on these systems and I can categorically say this is the best summary of HFT I've ever come across. Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space. I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time to get it working well before I decided to pack up and move on. Happy to answer any questions anyone may have.
- harmegido 12y ago
  Also been involved in this line of work. How would you see the specifics of a batch auction working? I'd see it as not doing anything if the matches happened in a FIFO manner (it'd still pay to be first). Would a pro-rata batch auction fix the problem? Or would we have to introduce a random component to match all crossing orders in each batch?
  - joosters 12y ago
    Batch auctions also have the opposite problem, whereby it makes sense to submit your bids as late as possible, to let you utilise the very latest information prior to the auction close. This might actually balance out some of the FIFO advantages?
    - harmegido 12y ago
      In a lot of FIFO markets, you will see price levels that have a large number of orders on them. Being in the front of this line is very valuable to a market maker. Many will have those orders there weeks in advance specifically for the purpose of maintaining a good place in line. I wouldn't see this changing in a batch FIFO auction. But I've never seen such a match before, so who knows.
tfinniga 12y ago
I don't understand why HFT is still around, if it's considered a problem. Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets? Like, right now it appears that trades occur instantly. Instead, queue requests and have the trades occur only once a second. Or 5 seconds. Would this work? I'm assuming that either there's no motivation to solve the problem, or there are difficulties that I'm unaware of. --- I've been wondering this about HFT for a while and this seemed like a good place to ask. Then I went back and finished reading the article, and this solution is proposed at the end. Curious about whether it would work on not.
- joosters 12y ago
  You should read to the end of the article, where they discuss exactly this kind of solution.
  - tfinniga 12y ago
    Yup, just got there. Oops.
- harmegido 12y ago
  Been trying to figure this out myself. What if we have 3 offers and 2 bids crossing at the same price after the 1 second tick is up. Which two offers get filled? If the answer is, the first two, then you've solved nothing. Maybe you randomize it? That could bring up some interesting issues.
  - harryh 12y ago
    Ya, if you randomize it than I have in incentive to submit more bids than I actually want filled because I've estimated what my fill rate is going to be. Of course everyone else has that incentive too which is going to negatively effect my fill rate which seems like it would lead to a pretty unstable system.
- crdoconnor 12y ago
  >I don't understand why HFT is still around, if it's considered a problem. Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power. >Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets? Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?". An almost microscopic tax on transactions would prevent it, though while having a minimal impact on all other trading activity.
guiomie 12y ago
Can someone help me understand the following statement (specifically the don't branch part) ? "Don’t touch the kernel. Don’t touch main memory … Don’t branch.’ (That third commandment means don’t fill your program with ‘if’ statements – ones with the generic form ‘if A then do B else do C’ – because they get in the way of a modern microprocessor’s capacity to do several things in parallel.)"
- kasey_junk 12y ago
  It is probably referring to the fact that code that branches allot will make the prefetcher/branch predictor fail more frequently which causes slower memory loads and less ops. Not sure why it mentions parallelism though.
- bluecalm 12y ago
  Obligatory link: http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array http://stackoverflow.com/questions/11227809/why-is-processin... see most upvoted answer for short lecture about cost of branching in modern CPU's.
NickStefan 12y ago
HFT is a two sided coin. For every person who complains that it's screwed them out, they're forgetting all the times they've saved money on narrower spreads due to extra liquidity. Possible?

52 points by joosters 12y ago

IMO this is one of the best articles I have read on HFT. It gives a good, balanced overview of the technology and the fundamental principles behind how automated trading works, without any particular viewpoint or opinion directing the text. The section near the end on FPGAs and programming styles was an eye-opener to me. I wonder how many other firms have taken risk checks out of their code to save some nanoseconds...

conistonwater 12y ago
> without any particular viewpoint or opinion directing the text You say that like that's a good thing. I know that a lot of rubbish has been written about HFT and algorithmic trading, but it's sad that no one seems to be able to write about its merits without starting a flame war.
- Retric 12y ago
  It's an outgrowth of poorly designed matchmaking software at exchanges. The could completly distroy HFT without costing society anything or adding transaction fees.
  - conistonwater 12y ago
    Do you have any specific issues in mind when you say that? A lot of the rules that markets operate by come from standards and regulations, not software design. The software design reflects the kind of markets exchanges want to set up, so it's not a software problem. The people who wrote the standards had specific markets and specific outcomes in mind when they wrote them.
    - harmegido 12y ago
      I work in this industry. I've come to the conclusion that I will be leaving very soon. There are a number of reasons, but the primary one is because it's become clear to me that I spend the vast majority of my time on trying to figure out how to take advantage of flaws in exchange hardware/software. Sorry, I can't go into specifics. That being said, I don't know if it is possible to create an ungame-able matching engine.

mer10z 12y ago

I can't help but think of that article yesterday about how we're avoiding the Big Problems (https://news.ycombinator.com/item?id=8261098 https://news.ycombinator.com/item?id=8261098). It's crazy how much money and brain power people have invested in HFT which is essentially just turning time into money with no benefit to society. What could the human race have achieved if the same amount of effort was focused in other areas?

afafsd 12y ago
Don't be mad that some other people's brains are bent on stuff of no particular benefit to you. Their brains, their choice. Just be glad that some people's brains are bent on stuff of benefit to you, and concentrate, if you like, on what of general benefit you can do with your own brain.
fchollet 12y ago
It could be that our current economic system implements a pretty terrible resource allocation algorithm, where many workers' jobs consist in undoing the efforts of employees from concurrent companies. In trading, a quant's job is to outsmart or outspeed the other bank's team. They invest considerable brain power, effort and money into producing incremental improvements that are quickly rendered meaningless by the work of the team in the building across the street. Another prime example is advertising, a zero-sum game where competing brands fight to undo other brands' brainwashing efforts and replace it with their own brainwashing. A waste of the time and energy of million of humans. Some more ideas on the subject: http://www.sphere-engineering.com/blog/15-hour-work-week.html http://www.sphere-engineering.com/blog/15-hour-work-week.htm...
- Retric 12y ago
  Advertizing and HFT are only zero sum for competing products in an established market. For new products it can be a net benifit ex: "talk to your doctor about ED."
crdoconnor 12y ago
The economy is geared towards playing such zero sum games. This is pretty much by design - policies such as austerity, ZIRP and financial deregulation shift the allocation of capital to speculation (gambling) and away from productive investment.

tobltobs 12y ago

Goldman Sachs’ Blankfein on Banking: ‘Doing God’s Work’. This kind of self-deception seems to quite prevalent with those banksters.

GFK_of_xmaspast 12y ago
They're totally doing god's work, it's just that the god in question is Mammon.

ord-lhr-nyc-nrt 12y ago

I worked on these systems and I can categorically say this is the best summary of HFT I've ever come across. Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space. I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time to get it working well before I decided to pack up and move on. Happy to answer any questions anyone may have.

harmegido 12y ago
Also been involved in this line of work. How would you see the specifics of a batch auction working? I'd see it as not doing anything if the matches happened in a FIFO manner (it'd still pay to be first). Would a pro-rata batch auction fix the problem? Or would we have to introduce a random component to match all crossing orders in each batch?
- joosters 12y ago
  Batch auctions also have the opposite problem, whereby it makes sense to submit your bids as late as possible, to let you utilise the very latest information prior to the auction close. This might actually balance out some of the FIFO advantages?
  - harmegido 12y ago
    In a lot of FIFO markets, you will see price levels that have a large number of orders on them. Being in the front of this line is very valuable to a market maker. Many will have those orders there weeks in advance specifically for the purpose of maintaining a good place in line. I wouldn't see this changing in a batch FIFO auction. But I've never seen such a match before, so who knows.

tfinniga 12y ago

I don't understand why HFT is still around, if it's considered a problem. Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets? Like, right now it appears that trades occur instantly. Instead, queue requests and have the trades occur only once a second. Or 5 seconds. Would this work? I'm assuming that either there's no motivation to solve the problem, or there are difficulties that I'm unaware of. --- I've been wondering this about HFT for a while and this seemed like a good place to ask. Then I went back and finished reading the article, and this solution is proposed at the end. Curious about whether it would work on not.

joosters 12y ago
You should read to the end of the article, where they discuss exactly this kind of solution.
- tfinniga 12y ago
  Yup, just got there. Oops.
harmegido 12y ago
Been trying to figure this out myself. What if we have 3 offers and 2 bids crossing at the same price after the 1 second tick is up. Which two offers get filled? If the answer is, the first two, then you've solved nothing. Maybe you randomize it? That could bring up some interesting issues.
- harryh 12y ago
  Ya, if you randomize it than I have in incentive to submit more bids than I actually want filled because I've estimated what my fill rate is going to be. Of course everyone else has that incentive too which is going to negatively effect my fill rate which seems like it would lead to a pretty unstable system.
crdoconnor 12y ago
>I don't understand why HFT is still around, if it's considered a problem. Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power. >Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets? Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?". An almost microscopic tax on transactions would prevent it, though while having a minimal impact on all other trading activity.

guiomie 12y ago

Can someone help me understand the following statement (specifically the don't branch part) ? "Don’t touch the kernel. Don’t touch main memory … Don’t branch.’ (That third commandment means don’t fill your program with ‘if’ statements – ones with the generic form ‘if A then do B else do C’ – because they get in the way of a modern microprocessor’s capacity to do several things in parallel.)"

kasey_junk 12y ago
It is probably referring to the fact that code that branches allot will make the prefetcher/branch predictor fail more frequently which causes slower memory loads and less ops. Not sure why it mentions parallelism though.
bluecalm 12y ago
Obligatory link: http://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-an-unsorted-array http://stackoverflow.com/questions/11227809/why-is-processin... see most upvoted answer for short lecture about cost of branching in modern CPU's.

NickStefan 12y ago

HFT is a two sided coin. For every person who complains that it's screwed them out, they're forgetting all the times they've saved money on narrower spreads due to extra liquidity. Possible?