7 ms·

A detailed data-centric exposé on how the market is rigged (2014)

114 points by chollida1 11y ago

sambe 11y ago
This is a fairly typical Nanex piece: start with loaded language before you've made your case, be "data-centric" yet leave out important details, intentionally exclude obvious details. By now this is all explained publicly many times over and Nanex are definitely aware of it. In the follow-up they admit that by not being stupid you get the fill you want. Hence not rigged or stolen, just that naive execution is a poor choice: same as in unfragmented markets, same as before HFT. A good reference is http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-Frequency-ebook/dp/B00P0QI2M2 http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F....
- RachelF 11y ago
  Why can you cancel orders in a few milliseconds? It looks very much like a great opportunity to rig the marker.
  - fleitz 11y ago
    Because you may no longer wish to sell at that price. Similar to the way that if you have a house on the market and someone makes an offer you don't have to accept it.
    - greghatch 11y ago
      It's not reasonable to assume that a human changed their mind that quickly. The idea of putting out an offer with the intention of changing your mind and letting a computer handle that process cements the point that its not a change of mind here, more likely an intentional misrepresentation of buying pressure - otherwise what's the point? This is not what changing someone's mind looks like.
      - fleitz 11y ago
        The point is to sell your shares at the highest price possible. Computers help you do that by reacting quickly to market information. If someone sends a buy order to the exchange that quote is listed on for the same amount of shares, it will fill the entire order. What is happening here is that there are multiple exchanges and when the algo sees an order come in on one exchange it cancels orders on other exchanges quickly disseminating price information as liquidity evaporates at the current price. The point of price is to disseminate information about the supply and demand of a good. When demand goes up price increases. The buyer is buying because he thinks the price will be higher in the future. Both are motivated by profit, at a certain point the buyer will no longer be interested in buying at that price. The buyer obviously has a piece of information the algo doesn't and is equally trying to capitalize on the algos lack of knowledge of that information. In the case in point the order was ~60% filled. The price the order was submitted at reflects the 'current' demand, not the demand after the order is submitted. It's pretty good to get a 60% fill when you are buying 83% of the currently available stock.
      - omgtehlion 11y ago
        > It's not reasonable to assume that a human changed their mind that quickly Those orders were resting there for a long time, it was not flashing. Like you selling two adjacent houses and see a buyer purchasing one and wanting to purchase another. You understand that he/she wants the house really badly and increase the price for the second house.
  - signet 11y ago
    Market makers put lots of quotes (orders) out on the various exchanges hoping to buy low and sell high, collecting the spread between the two. These days spreads are very small so market makers have to rely on automated quoting in order to make a profit. A side effect of this is that when markets are moving quickly the algorithms tend to take a conservative approach so the firm doesn't end up extremely long or short in a position. So when someone comes in and tries to buy up all the open orders at once they'll cancel their orders on other exchanges that haven't been executed yet so they can reprice to reflect the increased demand. The phantom liquidity really isn't much of a mystery when you look at it this way.
- panic 11y ago
  Which important details are they leaving out?
bluejekyll 11y ago
If this is legit, what amazes me is the time! Network hops can account for approx 100microseconds, which means that these HFT trades would not only have to be in the same datacenter, but within no more than 2-3 hops of the exchange device. Then on top of that, it's making its trade decision within 500 microseconds. I need to sit down to calculate this, but that is guaranteed to mean that they are reading from RAM and not disk. Could the entire stock market fit in a single hashmap in memory? That's just some awesome hardware and super efficient software. Edit:typos
- gruez 11y ago
  >Could the entire stock market fit in a single hashmap in memory? I don't see why not. You also don't have to put the entire market onto one machine - you can shard the market across multiple servers, with one server handling each symbol.
  - bluejekyll 11y ago
    > shard the market across multiple servers Sure, but now you've added a router, consistent hash or the like, to the picture, which is either another hop or a custom physical rack instead of a traditional ToR. It's not impossible at all, it's just impressive that they do all of this in 1ms!
    - deleted 11y ago
      [deleted]
    - deleted 11y ago
      [deleted]
    - gruez 11y ago
      >Sure, but now you've added a router, consistent hash or the like, to the picture not necessarily. If the HFT companies are using some sort of secret internal API (very possible as they like to shave milliseconds off their execution time), it's possible that they are talking directly to relevant server, bypassing any loadbalancers or routers.
bd_at_rivenhill 11y ago
(not so) Detailed Analysis of a Trade: lots of detail is lost here, you can see the problems start right at the beginning with this quote: "The 2nd column is the timestamp of the event - this is from CQS/CTA, the consolidated quote and trade SIP (Securities Information Processor)". Looks like they are determined to show only timestamps from the consolidated tape, which has been pretty much useless as a source of information for about 10 years or more. They omit the firm's receive timestamp for each message and also details of the trade execution reports received from Goldman Sigma X, which they are claiming as the source of information leakage that causes this order to be faded. Regarding Phantom Liquidity, they ask the pertinent question directly in the article: "what kind of liquidity"? but they get the wrong answer. The liquidity where you can dump a 20,000 share order directly on a market with tight spreads and not get faded does not exist now and never existed in the past; spreads were wider back in the day where you could have a hope of doing this. If you want to move large size with low slippage, you need to ship it to someone with good algorithms or spend the money to develop those algorithms yourself (along with the required technical infrastructure such as direct feeds, etc); those are really the only options available.
fleitz 11y ago
Imagine you live in a city and there are currently 1000 houses on the market, you take out an ad in the news paper telling everyone that there are only 1000 and you want to buy 900 of them. Someone wakes up at 6 AM reads the paper and phones every real estate agent in town telling them they will pay the asking price and a little bit more, most agents sell to him and he makes a nice tidy profit off of you who now have to buy from him. Is he front running you or are you just dumb? Especially given that you could take news paper ads out all week long and then let the guy who bought 800 of the houses take the loss on them. Or you could execute a bunch of small 50 trade orders and buy all the houses you want at the asking price.
- webXL 11y ago
  So the solution to this problem is to break up large trades then? Isn't this decreasing liquidity and if so, how big of a problem is that? Seems like trading a large amount of anything has always been challenging.
  - fleitz 11y ago
    The market is already illiquid at this demand / timescale which is why the order is being partially filled, if you spread it out over 5 minutes there'd be no problem finding that much liquidity.
stickfigure 11y ago
I don't understand why stock markets function this way. Why don't they work like a Diplomacy game, in turns? Everyone puts their instructions in, then the turn 'executes' and an algorithm (known to all parties) resolves the actions. Run one turn every second. If you want to buy all 20k shares on offer and nobody else is making the same bid, you get them. Seems reasonably straightforward - what am I missing?
- coderzach 11y ago
  The problem is that the current system is just the digital version of a bunch of guys yelling at each other on a trading floor. No one asked what the market should look like, they just took the existing system and computerized it.
- fleitz 11y ago
  Because people who trade stock prefer it that way. You're free to setup a market that works in this way, and if consumers prefer it you'll make millions. I'd hazard a guess because the point of markets is to disseminate information and the point of diplomacy is to keep information secret. Seriously, look at it like this, this buyer is a big smart human with a big brain that talks to other humans and likely knows 'secret' information about the company, lets say it's Wal-Mart and he's ordered satelite photos, etc to gauge traffic and knows they are doing well. He has all that information and all the algo knows is 'BUY 20000 @ $4.53' the algo disseminates that information to all the other traders by cancelling its quotes. The buyer pays a premium to gain 83% of the available stock on the market. It's like Mark Cuban and his Yahoo stock, if he dumped it all the day he got it the market would tank because there's more supply than anyone wants, so he wisely instructed his broker to sell over a longer period of time. The buyer could have chose to do this, or he could attempt to buy 83% of the available stock all at once in hopes no one noticed.
- camkego 11y ago
  These type systems has been offered for a number of years, it is a good question why they are not more prevalent in the US. http://goo.gl/xXd8So http://goo.gl/xXd8So
gd1 11y ago
I think where Eric Hunsader (and Michael Lewis) go wrong is that they have a big flaw in their mental model of the market. They make the assumption that if say 500 shares are offered at exchange A, and 500 shares at exchange B, then they are entitled to sum those two numbers together and assume that there are 1000 shares on offer. This is wrong. This is the basic mistake that they just can't wrap their heads around. I will try to explain as simply as I can from the point of view of a market maker. Imagine you are a market maker trying to make the spread and provide liquidity back in the old days. There is only a single stock exchange. Your firm obviously has a limit on how much risk it can take. How much capital/money it has available. So let's say you are allowed to post 1000 shares on the offer and 1000 shares on the bid. You are not allowed to hold a position of more than 1000 shares long or short. Life is simple. Then things get complicated. A competing exchange is now set up and begins to thrive. Yay for competition! Yay for the free market! But you still have the same limits on your capital and risk. What do you do? Do you now post 500 shares on the first exchange and 500 shares on the second? You could do that. Then you are guaranteed that you will never be long or short greater than 1000 shares. You'll never break your limit. But what happens when there are 10 different stock exchanges? Do you post only 100 shares at each one? Should the available liquidity in the market be fragmented into 10 pieces? Of course not. So what you do is post your bids and your offers for your full amount on each exchange. And then if you get filled on one exchange, you cancel your orders on the other ones. This isn't crazy, this isn't radical, this isn't some nefarious plot. It happens every day in all sorts of markets. I might put my house for sale on three different real estate websites. When it sells on one, I cancel the listing on the other two. No big deal. Or if I am a fisherman in India, I might send my sons to three different markets to sell my catch. When one finds a buyer, he phones the other two and tells them. (http://qje.oxfordjournals.org/content/122/3/879.abstract http://qje.oxfordjournals.org/content/122/3/879.abstract). This is simple stuff. In more technical terms, the offers and bids are all bona fide ab initio. They are all bona fide when initiated. When posted, you do actually want to sell or buy at that price. When you get filled at one exchange, circumstances (your inventory) have now changed, and you are entitled to cancel. Note in my explanation, the market maker isn't front-running, or even back-running, they are just responding to their own personal changes in inventory level (position) that means they can no longer offer those shares at the other exchanges. Electronic market makers may well have sophisticated models that predicts the short term price move causing them to pull orders at other exchanges, but we need not even consider those models to explain this behaviour. Changes in inventory is ample explanation. Of course, by posting their full available inventory on each exchange, market makers expose themselves to a new risk. Someone might come along at the exact same instant and fill them on every exchange! Then they will be over their limit and massively exposed. Luckily, they have very fast systems and low latency lines to help protect them from this. If they didn't, they would be forced to offer 1/n times their current liquidity, where n is the the number of exchanges. That would suck for all liquidity consumers. Which brings me to the most amusing part of this whole story, is that when the 'hero' of Flash Boys Brad Katsuyama figured out he could precisely fire off orders so that they hit all exchanges at the same instant in time with his 'thor' program, the SEC came knocking. Because behaviour like that, ambushing market makers on multiple exchanges, could in fact force market makers to offer much less liquidity. I don't have a problem with it, but it amuses me that the supposed hero was the one probably endangering market structure.

114 points by chollida1 11y ago

sambe 11y ago

This is a fairly typical Nanex piece: start with loaded language before you've made your case, be "data-centric" yet leave out important details, intentionally exclude obvious details. By now this is all explained publicly many times over and Nanex are definitely aware of it. In the follow-up they admit that by not being stupid you get the fill you want. Hence not rigged or stolen, just that naive execution is a poor choice: same as in unfragmented markets, same as before HFT. A good reference is http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-Frequency-ebook/dp/B00P0QI2M2 http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F....

RachelF 11y ago
Why can you cancel orders in a few milliseconds? It looks very much like a great opportunity to rig the marker.
- fleitz 11y ago
  Because you may no longer wish to sell at that price. Similar to the way that if you have a house on the market and someone makes an offer you don't have to accept it.
  - greghatch 11y ago
    It's not reasonable to assume that a human changed their mind that quickly. The idea of putting out an offer with the intention of changing your mind and letting a computer handle that process cements the point that its not a change of mind here, more likely an intentional misrepresentation of buying pressure - otherwise what's the point? This is not what changing someone's mind looks like.
    - fleitz 11y ago
      The point is to sell your shares at the highest price possible. Computers help you do that by reacting quickly to market information. If someone sends a buy order to the exchange that quote is listed on for the same amount of shares, it will fill the entire order. What is happening here is that there are multiple exchanges and when the algo sees an order come in on one exchange it cancels orders on other exchanges quickly disseminating price information as liquidity evaporates at the current price. The point of price is to disseminate information about the supply and demand of a good. When demand goes up price increases. The buyer is buying because he thinks the price will be higher in the future. Both are motivated by profit, at a certain point the buyer will no longer be interested in buying at that price. The buyer obviously has a piece of information the algo doesn't and is equally trying to capitalize on the algos lack of knowledge of that information. In the case in point the order was ~60% filled. The price the order was submitted at reflects the 'current' demand, not the demand after the order is submitted. It's pretty good to get a 60% fill when you are buying 83% of the currently available stock.
    - omgtehlion 11y ago
      > It's not reasonable to assume that a human changed their mind that quickly Those orders were resting there for a long time, it was not flashing. Like you selling two adjacent houses and see a buyer purchasing one and wanting to purchase another. You understand that he/she wants the house really badly and increase the price for the second house.
- signet 11y ago
  Market makers put lots of quotes (orders) out on the various exchanges hoping to buy low and sell high, collecting the spread between the two. These days spreads are very small so market makers have to rely on automated quoting in order to make a profit. A side effect of this is that when markets are moving quickly the algorithms tend to take a conservative approach so the firm doesn't end up extremely long or short in a position. So when someone comes in and tries to buy up all the open orders at once they'll cancel their orders on other exchanges that haven't been executed yet so they can reprice to reflect the increased demand. The phantom liquidity really isn't much of a mystery when you look at it this way.
panic 11y ago
Which important details are they leaving out?

bluejekyll 11y ago

If this is legit, what amazes me is the time! Network hops can account for approx 100microseconds, which means that these HFT trades would not only have to be in the same datacenter, but within no more than 2-3 hops of the exchange device. Then on top of that, it's making its trade decision within 500 microseconds. I need to sit down to calculate this, but that is guaranteed to mean that they are reading from RAM and not disk. Could the entire stock market fit in a single hashmap in memory? That's just some awesome hardware and super efficient software. Edit:typos

gruez 11y ago
>Could the entire stock market fit in a single hashmap in memory? I don't see why not. You also don't have to put the entire market onto one machine - you can shard the market across multiple servers, with one server handling each symbol.
- bluejekyll 11y ago
  > shard the market across multiple servers Sure, but now you've added a router, consistent hash or the like, to the picture, which is either another hop or a custom physical rack instead of a traditional ToR. It's not impossible at all, it's just impressive that they do all of this in 1ms!
  - deleted 11y ago
    [deleted]
  - deleted 11y ago
    [deleted]
  - gruez 11y ago
    >Sure, but now you've added a router, consistent hash or the like, to the picture not necessarily. If the HFT companies are using some sort of secret internal API (very possible as they like to shave milliseconds off their execution time), it's possible that they are talking directly to relevant server, bypassing any loadbalancers or routers.

bd_at_rivenhill 11y ago

(not so) Detailed Analysis of a Trade: lots of detail is lost here, you can see the problems start right at the beginning with this quote: "The 2nd column is the timestamp of the event - this is from CQS/CTA, the consolidated quote and trade SIP (Securities Information Processor)". Looks like they are determined to show only timestamps from the consolidated tape, which has been pretty much useless as a source of information for about 10 years or more. They omit the firm's receive timestamp for each message and also details of the trade execution reports received from Goldman Sigma X, which they are claiming as the source of information leakage that causes this order to be faded. Regarding Phantom Liquidity, they ask the pertinent question directly in the article: "what kind of liquidity"? but they get the wrong answer. The liquidity where you can dump a 20,000 share order directly on a market with tight spreads and not get faded does not exist now and never existed in the past; spreads were wider back in the day where you could have a hope of doing this. If you want to move large size with low slippage, you need to ship it to someone with good algorithms or spend the money to develop those algorithms yourself (along with the required technical infrastructure such as direct feeds, etc); those are really the only options available.

fleitz 11y ago

Imagine you live in a city and there are currently 1000 houses on the market, you take out an ad in the news paper telling everyone that there are only 1000 and you want to buy 900 of them. Someone wakes up at 6 AM reads the paper and phones every real estate agent in town telling them they will pay the asking price and a little bit more, most agents sell to him and he makes a nice tidy profit off of you who now have to buy from him. Is he front running you or are you just dumb? Especially given that you could take news paper ads out all week long and then let the guy who bought 800 of the houses take the loss on them. Or you could execute a bunch of small 50 trade orders and buy all the houses you want at the asking price.

webXL 11y ago
So the solution to this problem is to break up large trades then? Isn't this decreasing liquidity and if so, how big of a problem is that? Seems like trading a large amount of anything has always been challenging.
- fleitz 11y ago
  The market is already illiquid at this demand / timescale which is why the order is being partially filled, if you spread it out over 5 minutes there'd be no problem finding that much liquidity.

stickfigure 11y ago

I don't understand why stock markets function this way. Why don't they work like a Diplomacy game, in turns? Everyone puts their instructions in, then the turn 'executes' and an algorithm (known to all parties) resolves the actions. Run one turn every second. If you want to buy all 20k shares on offer and nobody else is making the same bid, you get them. Seems reasonably straightforward - what am I missing?

coderzach 11y ago
The problem is that the current system is just the digital version of a bunch of guys yelling at each other on a trading floor. No one asked what the market should look like, they just took the existing system and computerized it.
fleitz 11y ago
Because people who trade stock prefer it that way. You're free to setup a market that works in this way, and if consumers prefer it you'll make millions. I'd hazard a guess because the point of markets is to disseminate information and the point of diplomacy is to keep information secret. Seriously, look at it like this, this buyer is a big smart human with a big brain that talks to other humans and likely knows 'secret' information about the company, lets say it's Wal-Mart and he's ordered satelite photos, etc to gauge traffic and knows they are doing well. He has all that information and all the algo knows is 'BUY 20000 @ $4.53' the algo disseminates that information to all the other traders by cancelling its quotes. The buyer pays a premium to gain 83% of the available stock on the market. It's like Mark Cuban and his Yahoo stock, if he dumped it all the day he got it the market would tank because there's more supply than anyone wants, so he wisely instructed his broker to sell over a longer period of time. The buyer could have chose to do this, or he could attempt to buy 83% of the available stock all at once in hopes no one noticed.
camkego 11y ago
These type systems has been offered for a number of years, it is a good question why they are not more prevalent in the US. http://goo.gl/xXd8So http://goo.gl/xXd8So

gd1 11y ago

I think where Eric Hunsader (and Michael Lewis) go wrong is that they have a big flaw in their mental model of the market. They make the assumption that if say 500 shares are offered at exchange A, and 500 shares at exchange B, then they are entitled to sum those two numbers together and assume that there are 1000 shares on offer. This is wrong. This is the basic mistake that they just can't wrap their heads around. I will try to explain as simply as I can from the point of view of a market maker. Imagine you are a market maker trying to make the spread and provide liquidity back in the old days. There is only a single stock exchange. Your firm obviously has a limit on how much risk it can take. How much capital/money it has available. So let's say you are allowed to post 1000 shares on the offer and 1000 shares on the bid. You are not allowed to hold a position of more than 1000 shares long or short. Life is simple. Then things get complicated. A competing exchange is now set up and begins to thrive. Yay for competition! Yay for the free market! But you still have the same limits on your capital and risk. What do you do? Do you now post 500 shares on the first exchange and 500 shares on the second? You could do that. Then you are guaranteed that you will never be long or short greater than 1000 shares. You'll never break your limit. But what happens when there are 10 different stock exchanges? Do you post only 100 shares at each one? Should the available liquidity in the market be fragmented into 10 pieces? Of course not. So what you do is post your bids and your offers for your full amount on each exchange. And then if you get filled on one exchange, you cancel your orders on the other ones. This isn't crazy, this isn't radical, this isn't some nefarious plot. It happens every day in all sorts of markets. I might put my house for sale on three different real estate websites. When it sells on one, I cancel the listing on the other two. No big deal. Or if I am a fisherman in India, I might send my sons to three different markets to sell my catch. When one finds a buyer, he phones the other two and tells them. (http://qje.oxfordjournals.org/content/122/3/879.abstract http://qje.oxfordjournals.org/content/122/3/879.abstract). This is simple stuff. In more technical terms, the offers and bids are all bona fide ab initio. They are all bona fide when initiated. When posted, you do actually want to sell or buy at that price. When you get filled at one exchange, circumstances (your inventory) have now changed, and you are entitled to cancel. Note in my explanation, the market maker isn't front-running, or even back-running, they are just responding to their own personal changes in inventory level (position) that means they can no longer offer those shares at the other exchanges. Electronic market makers may well have sophisticated models that predicts the short term price move causing them to pull orders at other exchanges, but we need not even consider those models to explain this behaviour. Changes in inventory is ample explanation. Of course, by posting their full available inventory on each exchange, market makers expose themselves to a new risk. Someone might come along at the exact same instant and fill them on every exchange! Then they will be over their limit and massively exposed. Luckily, they have very fast systems and low latency lines to help protect them from this. If they didn't, they would be forced to offer 1/n times their current liquidity, where n is the the number of exchanges. That would suck for all liquidity consumers. Which brings me to the most amusing part of this whole story, is that when the 'hero' of Flash Boys Brad Katsuyama figured out he could precisely fire off orders so that they hit all exchanges at the same instant in time with his 'thor' program, the SEC came knocking. Because behaviour like that, ambushing market makers on multiple exchanges, could in fact force market makers to offer much less liquidity. I don't have a problem with it, but it amuses me that the supposed hero was the one probably endangering market structure.