7 ms·

HFT Quote Stuffing is a software bug

62 points by jpdus 12y ago

mikeash 12y ago
That sure is a lot of hand-wringing for what is a fairly simple and obvious problem. You're reacting to the extremely short-term derivative of a noisy signal. Of course things are going to go crazy from time to time. It's odd that he touches on the obvious solution of rate limiting, but doesn't actually discuss it as a way to stop this problem. If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop? He even goes so far as to say that you can get shut down for doing this, so why isn't there a tripwire to make sure you can never, ever do it?
- lucisferre 12y ago
  The low level of opacity when it comes to electronic trading makes it difficult to make any sense out of exactly what is (or perhaps isn't) happening, which I think is fundamentally the problem.
  - sambe 12y ago
    Do you mean the high level of opacity? There is considerably more opacity than in non-electronic trading, surely (everything on record, all data public, no effect from personal relationships etc)?
    - tptacek 12y ago
      I can't tell whether you're suggesting that "HFT" (heavily automated electronic trading) is more opaque than human trading (ie, the specialist system), but for what it's worth there's a right answer to this question: human market makers are WAY less transparent than algorithms are. Here's a benchmark: human market makers rigged the public markets so that tradable instruments were quoted in fractions of a dollar, rather than in pennies, so they could surreptitiously increase spreads. It's hard to get more brazen than that.
      - hft_throwaway 12y ago
        Human specialists and locals usually either had a monopoly on their product or were buddies with everyone in the pit so of course they would collude. Even on the NASDAQ screen they would collude and intimidate anyone showing tighter prices for showing "Chinese quotes": http://www.acsu.buffalo.edu/~keechung/TEM/Journal%20Articles/Collusion%20A%201994.pdf http://www.acsu.buffalo.edu/~keechung/TEM/Journal%20Articles... Anonymous limit order books are far better. Nobody's price is better than anyone else's and machines have no problem competing spreads down to near-zero.
      - minimax 12y ago
        human market makers rigged the public markets so that tradable instruments were quoted in fractions of a dollar, rather than in pennies, so they could surreptitiously increase spreads. I don't think this is right. Erh, obviously U.S. equities traded in quarters and eights (and later sixteenths before decimalization), but to say that they were using a weird fractional system because they wanted to rip people off isn't documented anywhere in the literature. They could have moved to decimals and still kept the minimum price variation above a penny†, or they could have just added more fractions (32nds, 64ths, 128ths) that would have moved the minimum price variation into roughly the range you get by quoting in decimals. If you think quoting in 128ths sounds silly I won't argue with you, but it's something they do actually do for the US treasury futures contracts on the CME††. I suspect that they just used what worked. In a market of human traders, it's much easier to deal with large round numbers. I think that's why odd lots were treated specially too. E.g. It's a lot easier for humans to keep track of 5000 shares at 50 1/2 than in 100 small orders from 200 shares to 5 shares all priced between 50.40 and 50.60. As markets adopted technology, they naturally moved to a smaller minimum price variation (and eliminated the special treatment of odd lots) because computers don't have any problems dealing with an aggregate of small orders vs one big order. † They are actually talking about doing this for some small cap stocks later this year. http://online.wsj.com/news/articles/SB10001424052702304275304579396861123381346 http://online.wsj.com/news/articles/SB1000142405270230427530... †† ctrl-f "Quotation Practices" -- http://www.cmegroup.com/education/files/understanding-treasury-futures.pdf http://www.cmegroup.com/education/files/understanding-treasu...
        12y ago
  - dllthomas 12y ago
    Though it's worth noting that transparency wasn't the hallmark of earlier systems.
- yummyfajitas 12y ago
  If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop? They do. But a rate limiter is just another signal - you still need to set the rate limit high enough that you can correctly respond to market conditions, but low enough that you don't go wild. "Never, ever do it" is like "zero tolerance for bugs". It's a goal to strive for, but it's not achievable.
  - mikeash 12y ago
    It's completely achievable. Rate limit to one trade per second. Done. You don't want to do that, because it doesn't let you make as much money as you otherwise could? I totally understand. But in that case, quote stuffing is not a bug, it's a deliberate tradeoff. But I suppose that "we let this happen on purpose because it allows higher profits, even though this particular behavior doesn't help us" doesn't sound as nice.
    - yummyfajitas 12y ago
      The point I'm making is that it's undesirable behavior that doesn't directly help the HFT. If they could stop it without going to ridiculous lengths like your proposed "give up trading entirely", they would. And given that quote stuffing doesn't actually hurt anyone, it's not like stopping it is a high priority.
      - mikeash 12y ago
        Well, the point the article is making is that quote stuffing is a bug, but it's not. It's a side effect, but one that's allowed on purpose, because it's necessary to do the things they want to do. I have no strong opinion about HFT. I don't have a lot of interest in the stock market in general. I do, however, have a low tolerance for weaselly language and misdirection.
        yummyfajitas 12y ago
        It's a side effect, but one that's allowed on purpose... Similarly, if you build a system on MySQL and later see dates like `0000-00-00` (yes, really), that's not a bug. That's a "side effect, but one that's allowed on purpose". Pretty much all bugs fall into that category - after all, you could have spent more time on formal verification/testing or just wrote the whole thing in Coq. I'm sorry you found the language "weaselly". My sole intent here was explaining the mechanics behind quote stuffing. How would you prefer I phrased it?
gd1 12y ago
Obviously, well written HFTs avoid this by introducing some hysteresis to the signal. i.e. Insert an order when the signal crosses 0.5, but then only delete it if the signal falls below 0.4. Chris: You've also missed the most common cause of these issues in my experience - feedback from identifying your own orders in the public stream. HFTs need to filter their own orders out of the orderbook they are trading on, or a feedback loop can begin. This can be difficult, since some exchanges don't provide a clear way to identify your own order, or it may be the case that you receive the public UDP packet before the private TCP packet that contains the ID required to match the two. For example (a basic one), say I have an algo that calculates the midpoint based on the best bid and ask volumes. And can only insert within x distance of the midpoint. I add a bid to the top of the book for a 1 lot, get the UDP public depth update, don't know whether this is my order or someone elses. So I re-calculate the midpoint, which pushes the midpoint upwards (greater bid volume), which means I then pull my bid, which means I recalculate the mid again, then re-insert, etc. etc....
- yummyfajitas 12y ago
  I actually mentioned this briefly - see the discussion of the "reconciliation step" in the "Quote Stuffing is not a DOS attack" section. I guess I assumed everyone's reconciliation system was as good as the one I worked on. But you make a very good point, as well as a meta point - this stuff isn't easy.
  - gd1 12y ago
    Sorry, missed that bit
- hft_throwaway 12y ago
  Sure but what if someone else is doing something in the market that perfectly lines up with the edge conditions for your place and cancel thresholds? Nothing is foolproof. If you are operating at a large enough scale this will end up happening a few times a day, especially when you have multiple agents interacting in ways that can't be predicted. A feedback loop can still happen between two unrelated traders running similar rules like "if I'm not the best offer, improve it by a tick".
hft_throwaway 12y ago
http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-features-and-aggressive-incompetence/#A-Bug http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-features... This blog post details similar issues that can lead to rapidly canceled orders. Generally, firms that are smart avoid moving their prices around "too much" since: a.) You lose your spot in the queue. If I am near the front of the price/time queue buying at 100.01 and move to 100.00 and back up again, now I'm behind everyone else and will either lose opportunity or face more negative selection (I'll only trade when someone wants to sell all the contracts bid at 100.01, whereas before I would get some trades where someone only wants to sell a few). b.) Sending orders is more computationally expensive than doing nothing. If you send enough orders you will slow down your own system. c.) Depending on market rules and behavior the messages you send may cause your gateway to queue up or block you from sending more messages which hurts you. I don't think quote flicker can provide any "evil" benefit to the person doing it. It doesn't make sense to do it to slow down competitors, since most feeds are multicasted to everyone and you still have to process all the orders you send, both from the gateway and over the data feed. If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down? I do think it can be disruptive in the sense that it places extra load on the exchange and can annoy participants who have trouble taking prices on their screen. Some markets have quote/trade rules to encourage more efficient messaging which is their right and not a bad idea if the limits are set reasonably. Rules like this make more sense than a minimum resting time, which just penalizes makers at the expense of fast takers. Imagine I am putting quotes out in SPY, an S&P 500 ETF, but they need to rest for half a second. Someone sees the S&P 500 futures or the individual stocks move and then my quote becomes a free option to trade at an unfair price. In a market like this, market makers would be forced to put up wider prices to compensate for these losses. A high quote/trade ratio in itself is not disruptive. A lot of the Nanex charts use thinly traded derivative products like ETFs as their whipping boy. The underlying value of these products will change whenever another stock or commodity changes, so market makers need to update their quotes quite rapidly to avoid trading at unfair prices, even if no trades have occurred in the product itself.
- lmm 12y ago
  > If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down? Suppose your fund has invested a lot in some hefty compute hardware but your trading strategies are on the weak side, whereas you know your competitors have better strategies but their computers are slower. Isn't it then in your interest to stuff the order feed and make everyone more dependent on computer power?
lmg643 12y ago
this is an interesting way to look at the problem but in my reasonably informed opinion, flawed. it seems like this is working backwards from the assumption that quote stuffing must be benign activity, and then trying to rationalize it with signal/noise theories about finding the fair price level. it is totally inaccurate to say that there is more load on the sender than the general public, and also misses the mark on what the DOS theory alleges. the exchange bears considerably more load than the sender of orders. the exchange must (1) ack the order, or reject if any flags are populated out-of-spec, (2) add the order to the order book, (3) check for potential matches based on price of order, if so, cross and update feeds, (4) if non-marketable, publish out to (a) depth-of-book feed, (b) SIP feed, etc. the DOS theory is that excessive activity in a single partition can slow down the publishing of data in that symbol range, which would mess up the quoted price for the exchange in a symbol, which would affect orders priced on the exchange, or anyone using the price in a derived manner (at a dark pool etc).
- yummyfajitas 12y ago
  I'm not making the assumption that it's benign. It's based on my own experience - I've quote stuffed hundreds of times in simulation, and I'm sure it happened in production as well. It's a hard bug to completely avoid. Interesting - the DOS attack is on the exchange rather than other market participants. But could you be more specific on how this helps you make money?
  - kasey_junk 12y ago
    There used to be games where you would attempt to get one order connection on one exchange gateway and then another connection on a different exchange gateway. You'd spam with one connection to slow down all the other participants on that gateway, and use your other connection for your real trades. Other games involved spamming open auctions, settlement trades etc. This gaming was a very brief phenomenon (because it is stupid, doesn't make as much money as traditional trading, and trivial to foil). The exchanges got rid of it almost immediately but it still gets passed around as folklore for one of the "predatory" activities that HFT does.
- 12y ago
personZ 12y ago
The 2. in the series, if I understand it right, is that they overwhelming the derelict SIP feed between market locations, increasing the benefits that their own high speed connections brings. If true, that does make money because they can act "in advance" at other markets before the artificially slowed SIP data gets there.
rsync 12y ago
I always assumed that step 2 was "DoS your competitors". This is crudely done with volume of traffic, but more elegantly done with frequency and amplitude (and so on) given some set of (possibly wrong) assumptions about how competing algorithms will respond to this rapid fire order placement. I know almost nothing (beyond ZH and chapters 1-5, so far, of _Dark Pools_) about HFT, but I can trivially think of ways you could be evil with quote stuffing as it is described in the OP.
- wglb 12y ago
  So since all orders have to go to the exchange first, how does DoS your competitors work?
  - rsync 12y ago
    You're thinking of a crude, direct DoS attack wherein the "traffic" generated by one HFT directly overwhelms another HFT. But there are other ways to DoS - what if I observe your trading activity and I discern something about your underlying algorithms and how expensive those algorithms are in terms of memory or CPU or latency, and I make a bet that a certain trade pattern will dramatically overwhelm your resources via these second and third order effects... It's just speculation, but the point of the OP was that there was no way this could be useful to an HFT algo and I am pointing out that in a few minutes we can come up with all kinds of ways that it could be.
    - wglb 12y ago
      Uh, what if I observe your trading activity Since the traffic is anonymous, how do you propose to do this? [Edit] one HFT directly overwhelms another HFT No, the dos needs to go through the exchange, thus is is not "directly" but indirectly through the exchange.
      - dllthomas 12y ago
        "Since the traffic is anonymous, how do you propose to do this?" Traffic other than your own is anonymous, but that doesn't mean you can't spot patterns that are almost certainly the behaviour of one other agent. On some exchanges you can track the history of resting orders, too, which gives you some more info (though still doesn't tie all of any player's orders together, anywhere).
kasey_junk 12y ago
So again with all HFT conversations we have to be clear on what we mean with the language we are using. I've seen quote stuffing used to mean what Chris mentions here. The implication being that HFTs were using large volumes of low quantity orders for nefarious purposes such as DoS the exchange & SIP or confusing your competitors by making the order flow to heavy to process. These kinds of games did occur, but it was very briefly a long time ago because the exchanges themselves had a vested interest in stopping it and an easy mechanism which to do so (fill ratios). I prefer the term quote spamming for this, and I agree with Chris that most of the time when this happens it's just a bad order management bug. Quote stuffing can also refer to the practice of putting a few large orders into the order book with no intention of trading them in order to make the volume on the book look larger than it is, and then cancelling them quickly to take advantage of the "real" volume. This is a predatory activity designed explicitly to take advantage of market making strategies and is already illegal. Your mileage may vary when it comes to enforcement...
- throwaway13qf85 12y ago
  The second kind of quote-stuffing you mention, where you place larger orders outside the best bid/offer with the intention of distorting the order book (aka spoofing, see comment below) is certainly nefarious. But it's a special kind of nefarious behaviour that really only affects other market participants who can react quickly enough to the information - i.e. it's mostly HFTs taking money from other HFTs.
  - lrm242 12y ago
    Using large non-bonafide orders to lean the book in the direction you want to trade is called 'Spoofing', not quote stuffing. It's been illegal for a long time.
    - kasey_junk 12y ago
      I agree that some people call it spoofing and that is a better name for it, but other people do call it quote stuffing. It's one of my pet peeves that when it comes to this industry there is very little standardization of the language.
jpdus 12y ago
Nanex linked an interesting CS research piece[1] which also discusses other possible reasons for (malicious) quote stuffing including: - walking others into the book, - creating false midpoints and - trying to cause stale pricing and slowing others down. [1]http://de.scribd.com/doc/116761218/CS-HFT-DETECTION http://de.scribd.com/doc/116761218/CS-HFT-DETECTION
- hft_throwaway 12y ago
  But even they admit that a market flashing between 100.01 offer and 100.00 offer is sometimes beneficial for them. It says their execution algo will attempt to "pick off" the 100.00 offer with IOCs if they are representing a buyer. Flashing orders with the intent to manipulate is wrong and is illegal already. It doesn't matter if you do this by hand or with a computer. Putting up a tighter best price, even if it doesn't stay put very long, can only be beneficial. At least some of the time, other traders will get filled at a more advantageous price than they would otherwise.
anigbrowl 12y ago
Except, once in a blue moon this happens when the mean signal approaches the threshold and stays there for a while: I'm having some trouble with the contention that this is just such a rare thing. I do a lot of audio signal processing so I'm used to dealing with noisy signals at Khz rates. Threshold fluctiations are a fact of life if you're trying to do things like pitch detection, envelope following, or sidechain compression (all of which have obvious analogs in the trading context). Obviously you don't want to impose too brutal of a low-pass filter on your input because you would miss important signals, but what prevents you from running the input signal to an LPF in parallel and keying a noise gate when the amplitude of the low-frequency (trend) signal drops? This is such a common problem in dealing with audio that we use specific devices called de-essers for recording vocals, because sibilant sounds otherwise tend to throw off other components of the signal chain: http://en.wikipedia.org/wiki/De-esser http://en.wikipedia.org/wiki/De-esser Ironically, you mention using XKCD-type graphs because 'nobody reads' the explanations and caveats, but just looks at pictures. I wonder if you aren't missing a trick here by over-engineering your HFT systems for speedy response at the inevitable price of positional confusion. Is it possible that you're using Infinite Impulse Response filtering and running into denormalization problems as a result? If so, can you not rectify the problem by adding a small amount of noise? I get your point about the negative aspects (for the HFT trader) of quote stuffing, how it can easily lead to being on the wrong side of a trade or punishment by the broker for flooding the output channel with unintended orders. But, you want to make money from automated signal analysis, that's your problem in much the same way that a sound engineer is responsible for preventing transient pops and feedback howls in an audio network. Sure, these problematic signal conditions are statistically rare, but when you're dealing with a high volume of samples you'r going to run into statistically rare conditions on a regular basis - and indeed you mention that this is the sort thing you might expect to happen once a day.
- hft_throwaway 12y ago
  I think you are right on the money and most traders do use techniques analogous to the ones you describe when designing models: http://www.decal.org/file/2945 http://www.decal.org/file/2945 "Alpha is often nothing more than taking commonly available data and mathematically encoding it in a signal correctly. Correct often means something as simple as using an rate of change instead of a difference, normalizing a value, smoothing a chaotic signal with an EMA, using a heteroscedastic weighted linear regression instead of a simple regression, or handling all numerical errors or edge cases." It's not always easy to get this completely correct in every situation and traders face a reward function that doesn't necessarily reward correctness so much as avoiding Type II errors.
austengary 12y ago
This is fun https://twitter.com/nanexllc/status/458663948800520192 https://twitter.com/nanexllc/status/458663948800520192

62 points by jpdus 12y ago

That sure is a lot of hand-wringing for what is a fairly simple and obvious problem. You're reacting to the extremely short-term derivative of a noisy signal. Of course things are going to go crazy from time to time. It's odd that he touches on the obvious solution of rate limiting, but doesn't actually discuss it as a way to stop this problem. If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop? He even goes so far as to say that you can get shut down for doing this, so why isn't there a tripwire to make sure you can never, ever do it?

lucisferre 12y ago
The low level of opacity when it comes to electronic trading makes it difficult to make any sense out of exactly what is (or perhaps isn't) happening, which I think is fundamentally the problem.
- sambe 12y ago
  Do you mean the high level of opacity? There is considerably more opacity than in non-electronic trading, surely (everything on record, all data public, no effect from personal relationships etc)?
  - tptacek 12y ago
    I can't tell whether you're suggesting that "HFT" (heavily automated electronic trading) is more opaque than human trading (ie, the specialist system), but for what it's worth there's a right answer to this question: human market makers are WAY less transparent than algorithms are. Here's a benchmark: human market makers rigged the public markets so that tradable instruments were quoted in fractions of a dollar, rather than in pennies, so they could surreptitiously increase spreads. It's hard to get more brazen than that.
    - hft_throwaway 12y ago
      Human specialists and locals usually either had a monopoly on their product or were buddies with everyone in the pit so of course they would collude. Even on the NASDAQ screen they would collude and intimidate anyone showing tighter prices for showing "Chinese quotes": http://www.acsu.buffalo.edu/~keechung/TEM/Journal%20Articles/Collusion%20A%201994.pdf http://www.acsu.buffalo.edu/~keechung/TEM/Journal%20Articles... Anonymous limit order books are far better. Nobody's price is better than anyone else's and machines have no problem competing spreads down to near-zero.
    - minimax 12y ago
      human market makers rigged the public markets so that tradable instruments were quoted in fractions of a dollar, rather than in pennies, so they could surreptitiously increase spreads. I don't think this is right. Erh, obviously U.S. equities traded in quarters and eights (and later sixteenths before decimalization), but to say that they were using a weird fractional system because they wanted to rip people off isn't documented anywhere in the literature. They could have moved to decimals and still kept the minimum price variation above a penny†, or they could have just added more fractions (32nds, 64ths, 128ths) that would have moved the minimum price variation into roughly the range you get by quoting in decimals. If you think quoting in 128ths sounds silly I won't argue with you, but it's something they do actually do for the US treasury futures contracts on the CME††. I suspect that they just used what worked. In a market of human traders, it's much easier to deal with large round numbers. I think that's why odd lots were treated specially too. E.g. It's a lot easier for humans to keep track of 5000 shares at 50 1/2 than in 100 small orders from 200 shares to 5 shares all priced between 50.40 and 50.60. As markets adopted technology, they naturally moved to a smaller minimum price variation (and eliminated the special treatment of odd lots) because computers don't have any problems dealing with an aggregate of small orders vs one big order. † They are actually talking about doing this for some small cap stocks later this year. http://online.wsj.com/news/articles/SB10001424052702304275304579396861123381346 http://online.wsj.com/news/articles/SB1000142405270230427530... †† ctrl-f "Quotation Practices" -- http://www.cmegroup.com/education/files/understanding-treasury-futures.pdf http://www.cmegroup.com/education/files/understanding-treasu...
      - 12y ago
- dllthomas 12y ago
  Though it's worth noting that transparency wasn't the hallmark of earlier systems.
yummyfajitas 12y ago
If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop? They do. But a rate limiter is just another signal - you still need to set the rate limit high enough that you can correctly respond to market conditions, but low enough that you don't go wild. "Never, ever do it" is like "zero tolerance for bugs". It's a goal to strive for, but it's not achievable.
- mikeash 12y ago
  It's completely achievable. Rate limit to one trade per second. Done. You don't want to do that, because it doesn't let you make as much money as you otherwise could? I totally understand. But in that case, quote stuffing is not a bug, it's a deliberate tradeoff. But I suppose that "we let this happen on purpose because it allows higher profits, even though this particular behavior doesn't help us" doesn't sound as nice.
  - yummyfajitas 12y ago
    The point I'm making is that it's undesirable behavior that doesn't directly help the HFT. If they could stop it without going to ridiculous lengths like your proposed "give up trading entirely", they would. And given that quote stuffing doesn't actually hurt anyone, it's not like stopping it is a high priority.
    - mikeash 12y ago
      Well, the point the article is making is that quote stuffing is a bug, but it's not. It's a side effect, but one that's allowed on purpose, because it's necessary to do the things they want to do. I have no strong opinion about HFT. I don't have a lot of interest in the stock market in general. I do, however, have a low tolerance for weaselly language and misdirection.
      - yummyfajitas 12y ago
        It's a side effect, but one that's allowed on purpose... Similarly, if you build a system on MySQL and later see dates like `0000-00-00` (yes, really), that's not a bug. That's a "side effect, but one that's allowed on purpose". Pretty much all bugs fall into that category - after all, you could have spent more time on formal verification/testing or just wrote the whole thing in Coq. I'm sorry you found the language "weaselly". My sole intent here was explaining the mechanics behind quote stuffing. How would you prefer I phrased it?

gd1 12y ago

Obviously, well written HFTs avoid this by introducing some hysteresis to the signal. i.e. Insert an order when the signal crosses 0.5, but then only delete it if the signal falls below 0.4. Chris: You've also missed the most common cause of these issues in my experience - feedback from identifying your own orders in the public stream. HFTs need to filter their own orders out of the orderbook they are trading on, or a feedback loop can begin. This can be difficult, since some exchanges don't provide a clear way to identify your own order, or it may be the case that you receive the public UDP packet before the private TCP packet that contains the ID required to match the two. For example (a basic one), say I have an algo that calculates the midpoint based on the best bid and ask volumes. And can only insert within x distance of the midpoint. I add a bid to the top of the book for a 1 lot, get the UDP public depth update, don't know whether this is my order or someone elses. So I re-calculate the midpoint, which pushes the midpoint upwards (greater bid volume), which means I then pull my bid, which means I recalculate the mid again, then re-insert, etc. etc....

yummyfajitas 12y ago
I actually mentioned this briefly - see the discussion of the "reconciliation step" in the "Quote Stuffing is not a DOS attack" section. I guess I assumed everyone's reconciliation system was as good as the one I worked on. But you make a very good point, as well as a meta point - this stuff isn't easy.
- gd1 12y ago
  Sorry, missed that bit
hft_throwaway 12y ago
Sure but what if someone else is doing something in the market that perfectly lines up with the edge conditions for your place and cancel thresholds? Nothing is foolproof. If you are operating at a large enough scale this will end up happening a few times a day, especially when you have multiple agents interacting in ways that can't be predicted. A feedback loop can still happen between two unrelated traders running similar rules like "if I'm not the best offer, improve it by a tick".

hft_throwaway 12y ago

http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-features-and-aggressive-incompetence/#A-Bug http://zacharydavid.com/2014/04/on-hft-part-ii-bugs-features... This blog post details similar issues that can lead to rapidly canceled orders. Generally, firms that are smart avoid moving their prices around "too much" since: a.) You lose your spot in the queue. If I am near the front of the price/time queue buying at 100.01 and move to 100.00 and back up again, now I'm behind everyone else and will either lose opportunity or face more negative selection (I'll only trade when someone wants to sell all the contracts bid at 100.01, whereas before I would get some trades where someone only wants to sell a few). b.) Sending orders is more computationally expensive than doing nothing. If you send enough orders you will slow down your own system. c.) Depending on market rules and behavior the messages you send may cause your gateway to queue up or block you from sending more messages which hurts you. I don't think quote flicker can provide any "evil" benefit to the person doing it. It doesn't make sense to do it to slow down competitors, since most feeds are multicasted to everyone and you still have to process all the orders you send, both from the gateway and over the data feed. If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down? I do think it can be disruptive in the sense that it places extra load on the exchange and can annoy participants who have trouble taking prices on their screen. Some markets have quote/trade rules to encourage more efficient messaging which is their right and not a bad idea if the limits are set reasonably. Rules like this make more sense than a minimum resting time, which just penalizes makers at the expense of fast takers. Imagine I am putting quotes out in SPY, an S&P 500 ETF, but they need to rest for half a second. Someone sees the S&P 500 futures or the individual stocks move and then my quote becomes a free option to trade at an unfair price. In a market like this, market makers would be forced to put up wider prices to compensate for these losses. A high quote/trade ratio in itself is not disruptive. A lot of the Nanex charts use thinly traded derivative products like ETFs as their whipping boy. The underlying value of these products will change whenever another stock or commodity changes, so market makers need to update their quotes quite rapidly to avoid trading at unfair prices, even if no trades have occurred in the product itself.

lmm 12y ago
> If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down? Suppose your fund has invested a lot in some hefty compute hardware but your trading strategies are on the weak side, whereas you know your competitors have better strategies but their computers are slower. Isn't it then in your interest to stuff the order feed and make everyone more dependent on computer power?

lmg643 12y ago

this is an interesting way to look at the problem but in my reasonably informed opinion, flawed. it seems like this is working backwards from the assumption that quote stuffing must be benign activity, and then trying to rationalize it with signal/noise theories about finding the fair price level. it is totally inaccurate to say that there is more load on the sender than the general public, and also misses the mark on what the DOS theory alleges. the exchange bears considerably more load than the sender of orders. the exchange must (1) ack the order, or reject if any flags are populated out-of-spec, (2) add the order to the order book, (3) check for potential matches based on price of order, if so, cross and update feeds, (4) if non-marketable, publish out to (a) depth-of-book feed, (b) SIP feed, etc. the DOS theory is that excessive activity in a single partition can slow down the publishing of data in that symbol range, which would mess up the quoted price for the exchange in a symbol, which would affect orders priced on the exchange, or anyone using the price in a derived manner (at a dark pool etc).

yummyfajitas 12y ago
I'm not making the assumption that it's benign. It's based on my own experience - I've quote stuffed hundreds of times in simulation, and I'm sure it happened in production as well. It's a hard bug to completely avoid. Interesting - the DOS attack is on the exchange rather than other market participants. But could you be more specific on how this helps you make money?
- kasey_junk 12y ago
  There used to be games where you would attempt to get one order connection on one exchange gateway and then another connection on a different exchange gateway. You'd spam with one connection to slow down all the other participants on that gateway, and use your other connection for your real trades. Other games involved spamming open auctions, settlement trades etc. This gaming was a very brief phenomenon (because it is stupid, doesn't make as much money as traditional trading, and trivial to foil). The exchanges got rid of it almost immediately but it still gets passed around as folklore for one of the "predatory" activities that HFT does.
12y ago

personZ 12y ago

The 2. in the series, if I understand it right, is that they overwhelming the derelict SIP feed between market locations, increasing the benefits that their own high speed connections brings. If true, that does make money because they can act "in advance" at other markets before the artificially slowed SIP data gets there.

rsync 12y ago

I always assumed that step 2 was "DoS your competitors". This is crudely done with volume of traffic, but more elegantly done with frequency and amplitude (and so on) given some set of (possibly wrong) assumptions about how competing algorithms will respond to this rapid fire order placement. I know almost nothing (beyond ZH and chapters 1-5, so far, of _Dark Pools_) about HFT, but I can trivially think of ways you could be evil with quote stuffing as it is described in the OP.

wglb 12y ago
So since all orders have to go to the exchange first, how does DoS your competitors work?
- rsync 12y ago
  You're thinking of a crude, direct DoS attack wherein the "traffic" generated by one HFT directly overwhelms another HFT. But there are other ways to DoS - what if I observe your trading activity and I discern something about your underlying algorithms and how expensive those algorithms are in terms of memory or CPU or latency, and I make a bet that a certain trade pattern will dramatically overwhelm your resources via these second and third order effects... It's just speculation, but the point of the OP was that there was no way this could be useful to an HFT algo and I am pointing out that in a few minutes we can come up with all kinds of ways that it could be.
  - wglb 12y ago
    Uh, what if I observe your trading activity Since the traffic is anonymous, how do you propose to do this? [Edit] one HFT directly overwhelms another HFT No, the dos needs to go through the exchange, thus is is not "directly" but indirectly through the exchange.
    - dllthomas 12y ago
      "Since the traffic is anonymous, how do you propose to do this?" Traffic other than your own is anonymous, but that doesn't mean you can't spot patterns that are almost certainly the behaviour of one other agent. On some exchanges you can track the history of resting orders, too, which gives you some more info (though still doesn't tie all of any player's orders together, anywhere).

kasey_junk 12y ago

So again with all HFT conversations we have to be clear on what we mean with the language we are using. I've seen quote stuffing used to mean what Chris mentions here. The implication being that HFTs were using large volumes of low quantity orders for nefarious purposes such as DoS the exchange & SIP or confusing your competitors by making the order flow to heavy to process. These kinds of games did occur, but it was very briefly a long time ago because the exchanges themselves had a vested interest in stopping it and an easy mechanism which to do so (fill ratios). I prefer the term quote spamming for this, and I agree with Chris that most of the time when this happens it's just a bad order management bug. Quote stuffing can also refer to the practice of putting a few large orders into the order book with no intention of trading them in order to make the volume on the book look larger than it is, and then cancelling them quickly to take advantage of the "real" volume. This is a predatory activity designed explicitly to take advantage of market making strategies and is already illegal. Your mileage may vary when it comes to enforcement...

throwaway13qf85 12y ago
The second kind of quote-stuffing you mention, where you place larger orders outside the best bid/offer with the intention of distorting the order book (aka spoofing, see comment below) is certainly nefarious. But it's a special kind of nefarious behaviour that really only affects other market participants who can react quickly enough to the information - i.e. it's mostly HFTs taking money from other HFTs.
- lrm242 12y ago
  Using large non-bonafide orders to lean the book in the direction you want to trade is called 'Spoofing', not quote stuffing. It's been illegal for a long time.
  - kasey_junk 12y ago
    I agree that some people call it spoofing and that is a better name for it, but other people do call it quote stuffing. It's one of my pet peeves that when it comes to this industry there is very little standardization of the language.

jpdus 12y ago

Nanex linked an interesting CS research piece[1] which also discusses other possible reasons for (malicious) quote stuffing including: - walking others into the book, - creating false midpoints and - trying to cause stale pricing and slowing others down. [1]http://de.scribd.com/doc/116761218/CS-HFT-DETECTION http://de.scribd.com/doc/116761218/CS-HFT-DETECTION

hft_throwaway 12y ago
But even they admit that a market flashing between 100.01 offer and 100.00 offer is sometimes beneficial for them. It says their execution algo will attempt to "pick off" the 100.00 offer with IOCs if they are representing a buyer. Flashing orders with the intent to manipulate is wrong and is illegal already. It doesn't matter if you do this by hand or with a computer. Putting up a tighter best price, even if it doesn't stay put very long, can only be beneficial. At least some of the time, other traders will get filled at a more advantageous price than they would otherwise.

anigbrowl 12y ago

Except, once in a blue moon this happens when the mean signal approaches the threshold and stays there for a while: I'm having some trouble with the contention that this is just such a rare thing. I do a lot of audio signal processing so I'm used to dealing with noisy signals at Khz rates. Threshold fluctiations are a fact of life if you're trying to do things like pitch detection, envelope following, or sidechain compression (all of which have obvious analogs in the trading context). Obviously you don't want to impose too brutal of a low-pass filter on your input because you would miss important signals, but what prevents you from running the input signal to an LPF in parallel and keying a noise gate when the amplitude of the low-frequency (trend) signal drops? This is such a common problem in dealing with audio that we use specific devices called de-essers for recording vocals, because sibilant sounds otherwise tend to throw off other components of the signal chain: http://en.wikipedia.org/wiki/De-esser http://en.wikipedia.org/wiki/De-esser Ironically, you mention using XKCD-type graphs because 'nobody reads' the explanations and caveats, but just looks at pictures. I wonder if you aren't missing a trick here by over-engineering your HFT systems for speedy response at the inevitable price of positional confusion. Is it possible that you're using Infinite Impulse Response filtering and running into denormalization problems as a result? If so, can you not rectify the problem by adding a small amount of noise? I get your point about the negative aspects (for the HFT trader) of quote stuffing, how it can easily lead to being on the wrong side of a trade or punishment by the broker for flooding the output channel with unintended orders. But, you want to make money from automated signal analysis, that's your problem in much the same way that a sound engineer is responsible for preventing transient pops and feedback howls in an audio network. Sure, these problematic signal conditions are statistically rare, but when you're dealing with a high volume of samples you'r going to run into statistically rare conditions on a regular basis - and indeed you mention that this is the sort thing you might expect to happen once a day.

hft_throwaway 12y ago
I think you are right on the money and most traders do use techniques analogous to the ones you describe when designing models: http://www.decal.org/file/2945 http://www.decal.org/file/2945 "Alpha is often nothing more than taking commonly available data and mathematically encoding it in a signal correctly. Correct often means something as simple as using an rate of change instead of a difference, normalizing a value, smoothing a chaotic signal with an EMA, using a heteroscedastic weighted linear regression instead of a simple regression, or handling all numerical errors or edge cases." It's not always easy to get this completely correct in every situation and traders face a reward function that doesn't necessarily reward correctness so much as avoiding Type II errors.

austengary 12y ago

This is fun https://twitter.com/nanexllc/status/458663948800520192 https://twitter.com/nanexllc/status/458663948800520192