4 ms·

I worked on these systems and I can categorically say this is the best summary of HFT I've ever come across. Many of the systems, fiber networks, data centers,

I worked on these systems and I can categorically say this is the best summary of HFT I've ever come across. Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space. I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time to get it working well before I decided to pack up and move on. Happy to answer any questions anyone may have.

harmegido 12y ago
Also been involved in this line of work. How would you see the specifics of a batch auction working? I'd see it as not doing anything if the matches happened in a FIFO manner (it'd still pay to be first). Would a pro-rata batch auction fix the problem? Or would we have to introduce a random component to match all crossing orders in each batch?
- joosters 12y ago
  Batch auctions also have the opposite problem, whereby it makes sense to submit your bids as late as possible, to let you utilise the very latest information prior to the auction close. This might actually balance out some of the FIFO advantages?
  - harmegido 12y ago
    In a lot of FIFO markets, you will see price levels that have a large number of orders on them. Being in the front of this line is very valuable to a market maker. Many will have those orders there weeks in advance specifically for the purpose of maintaining a good place in line. I wouldn't see this changing in a batch FIFO auction. But I've never seen such a match before, so who knows.
    - joosters 12y ago
      Ah, I was thinking that the concept of these batch auctions was to start with a completely clean order book every round. But as you say, that wouldn't be any good for people wanting long-lasting orders.
  - ord-lhr-nyc-nrt 12y ago
    From what I understand batch auctions would match every participant's first order with everyone else's first order. Then all second orders would be matched together, all third, and so on. I think this is how one of the FX exchanges is doing it now (or will be soon). Caveat: I'm not a trader but was more on the engineering side.
    - kasey_junk 12y ago
      But what do you do when there are more participants on one side of the same price than the other? If it is FIFO you haven't eliminated any of the incentives for speed.
kasey_junk 12y ago
Batch auctions already exist in electronic markets. They still allow for huge advantages for speed. If you want to get rid of the speed element (I'm not sure it's a problem) you should explicitly allow people to pay for priority. The easiest way to do this would be to allow for extremely small fractions of price. So instead of having to pay for faster lasers I could just change my pricing structure to tack on .00001 on to the price.
- yummyfajitas 12y ago
  This proposal, while economically sound and something I have advocated for for years, fails to allow politicians and pundits to express their strong moral disapproval of geeks making money.
Faint 12y ago
Why would the batch auction need to be FIFO? What about something like this: 1) When opening a "new tick", release data about trades in last tick, and start accepting new orders. Information about orders submitted to this tick is kept secret. Allow setting order price to practically arbitrary accuracy. 2) After tick closes, match orders in _random order_. That is, it should not matter at all on at which point in time inside this tick order arrived 3) repeat Allowing arbitrary accuracy would help turn the competition from speed or stuffing the order book with huge number of orders to "spoof" the random matching, towards competition of just providing lower spread. I understand that it would still advantageous to hold submitting trades until very end of the tick, to take into account as much outside information (other markets, etc.) as possible. To counter this, maybe the exact length of the tick could be randomized also. What do you think?
- kasey_junk 12y ago
  So the biggest argument against random matching (whether in batch auction or open market book orders) is that the easiest way to game it is to figure out your expected fill rate. Then you just raise your order volume such that your expected fill rate matches your risk profile. For the advantage of removing the speed advantage, you get the downside of providing advantage to people that have the loosest risk profiles and you further complicate the audit burden such that it is impossible to follow. Think of it from the perspective of a value investor that set a market order 6 months ago. Under the current system once their price becomes top of the book they have a virtual certainty of getting filled. Under a (fair) random match they have the exact same chance of getting filled as someone who literaly just put in their order.

harmegido 12y ago

Also been involved in this line of work. How would you see the specifics of a batch auction working? I'd see it as not doing anything if the matches happened in a FIFO manner (it'd still pay to be first). Would a pro-rata batch auction fix the problem? Or would we have to introduce a random component to match all crossing orders in each batch?

joosters 12y ago
Batch auctions also have the opposite problem, whereby it makes sense to submit your bids as late as possible, to let you utilise the very latest information prior to the auction close. This might actually balance out some of the FIFO advantages?
- harmegido 12y ago
  In a lot of FIFO markets, you will see price levels that have a large number of orders on them. Being in the front of this line is very valuable to a market maker. Many will have those orders there weeks in advance specifically for the purpose of maintaining a good place in line. I wouldn't see this changing in a batch FIFO auction. But I've never seen such a match before, so who knows.
  - joosters 12y ago
    Ah, I was thinking that the concept of these batch auctions was to start with a completely clean order book every round. But as you say, that wouldn't be any good for people wanting long-lasting orders.
- ord-lhr-nyc-nrt 12y ago
  From what I understand batch auctions would match every participant's first order with everyone else's first order. Then all second orders would be matched together, all third, and so on. I think this is how one of the FX exchanges is doing it now (or will be soon). Caveat: I'm not a trader but was more on the engineering side.
  - kasey_junk 12y ago
    But what do you do when there are more participants on one side of the same price than the other? If it is FIFO you haven't eliminated any of the incentives for speed.

kasey_junk 12y ago

Batch auctions already exist in electronic markets. They still allow for huge advantages for speed. If you want to get rid of the speed element (I'm not sure it's a problem) you should explicitly allow people to pay for priority. The easiest way to do this would be to allow for extremely small fractions of price. So instead of having to pay for faster lasers I could just change my pricing structure to tack on .00001 on to the price.

yummyfajitas 12y ago
This proposal, while economically sound and something I have advocated for for years, fails to allow politicians and pundits to express their strong moral disapproval of geeks making money.

Faint 12y ago

Why would the batch auction need to be FIFO? What about something like this: 1) When opening a "new tick", release data about trades in last tick, and start accepting new orders. Information about orders submitted to this tick is kept secret. Allow setting order price to practically arbitrary accuracy. 2) After tick closes, match orders in _random order_. That is, it should not matter at all on at which point in time inside this tick order arrived 3) repeat Allowing arbitrary accuracy would help turn the competition from speed or stuffing the order book with huge number of orders to "spoof" the random matching, towards competition of just providing lower spread. I understand that it would still advantageous to hold submitting trades until very end of the tick, to take into account as much outside information (other markets, etc.) as possible. To counter this, maybe the exact length of the tick could be randomized also. What do you think?

kasey_junk 12y ago
So the biggest argument against random matching (whether in batch auction or open market book orders) is that the easiest way to game it is to figure out your expected fill rate. Then you just raise your order volume such that your expected fill rate matches your risk profile. For the advantage of removing the speed advantage, you get the downside of providing advantage to people that have the loosest risk profiles and you further complicate the audit burden such that it is impossible to follow. Think of it from the perspective of a value investor that set a market order 6 months ago. Under the current system once their price becomes top of the book they have a virtual certainty of getting filled. Under a (fair) random match they have the exact same chance of getting filled as someone who literaly just put in their order.