19 ms·

Nanex Gets $700k Whistleblower Award from SEC

328 points by theandycamps 11y ago

atemerev 11y ago
While I am fully pro-HFT, and in fact work as a consultant for a few HFT shops, this is certainly illegal and anti-competitive. (Somewhat akin to using game engine bugs for artificial lag creation in MMOs on the opponent side).
Someone1234 11y ago
I have two questions about HFT: - Is HFT a healthy or unhealthy part of the economy? Meaning does it help flatten out the highs/lows, or does it emphasise them? - Aside from a micro-tax on trading (e.g. 1c/trade) is there any other mitigations/solutions to HFT?
- SeanLuke 11y ago
  I don't see why we couldn't put a time delay in. Something like this: when you commit to a trade, the actual trade delayed for some N minutes, where N is a randomly-chosen value from some distribution. Wouldn't this eliminate the incentive to do HFT?
  - jgh 11y ago
    Another disincentive might be the trade tax that Bernie Sanders has proposed since it would basically affect HFT way more than anyone else.
    - cheald 11y ago
      No, it would affect everyone - that's why it's so bad. HFT traders would just adjust their minimum spreads and continue on their merry way. Sanders' FTT doesn't attack the mechanisms of HFT, it attacks the markets as a whole - HFT would suffer, but so would everyone else. If you want to hit HFT specifically you need to hit the HFT-specific behaviors - either add a per-message charge (like Canada did) or add a tax to cancelled orders. It's worth noting that after the Canadian regulatory authorities added a per-message fee, bid-ask spreads rose, liquidity dropped, and it was retail traders who suffered the most. http://marginalrevolution.com/marginalrevolution/2014/04/a-study-of-limiting-hft.html http://marginalrevolution.com/marginalrevolution/2014/04/a-s...
      - jgh 11y ago
        I understand that a trade tax would affect everyone, but for someone like me who makes maybe a handful of trades per month it wouldn't really do that much to dissuade me from making those trades. I also don't particularly care one way or the other what happens with HFT, I was just mentioning this as a thing people are thinking about.
        kasey_junk 11y ago
        My big concern with the transaction tax is how it applies to indexed funds/etfs. Those trade all the time so the price impact can be significant for those. Further, depending on implementation, you could easily get taxed multiple times for that as there is a transaction to get in/out of the index, and the internal index transactions.
  - harryh 11y ago
    No, it would just raise the cost of trading for everyone else. If HFT's risk goes up they have to raise their prices.
- kbenson 11y ago
  IIRC, HN's yummyfajitas goes into a lot of details in his blog posts on HFT[1]. My own understanding is that it provides liquidity and reduces bid/ask spreads, resulting in a more accurate stock valuation. I'm not sure the negatives, but I'm not a trader nor do I follow such things all that closely. 1: https://news.ycombinator.com/item?id=3852341 https://news.ycombinator.com/item?id=3852341
  - jobu 11y ago
    From that blog post: "Most HFTs run a market making strategy. What this means is they play both sides of the table - they take no position on whether a stock will go up or down. Instead, they try to offer securities both to buy and sell. If you want to buy, they will sell to you at $20.10. If you want to sell, they'll buy from you at $20. As long as their buys and sells match don't get too out of whack, the HFT will collect $0.10 = $20.10 - 20.00." That does provide more immediate trades for both the buyer and seller, which the pro-HFT camp calls liquidity. Unfortunately it's usually not needed - both the buyer and seller were already in the market and willing to trade, and now they both just had an additional $0.10/share sucked out of their trade by some HFT.
    - arohner 11y ago
      In the past, either the buyer and seller had to wait longer to find a mutually acceptable price, or pay a lot more than $0.10/share. HFT is actually an improvement for both sides, because they can either 1) buy/sell faster or 2) pay lower transaction costs (and probably both!).
    - cdetrio 11y ago
      That's why market makers are also called scalpers, because they skim the $0.10 spread. But in return for paying the spread, what traders get is immediacy. The problem is, how much immediacy do traders actually want? HFT provides immediacy in microseconds; its needed for continuous-time auctions because buyers sellers usually aren't in the market at the exact same microsecond interval (while the HFT is there at every interval). Frequent Batch Auctions[1], where orders are matched in batches with single-price clearing (call auction style) - just like the opening and closing sessions of NYSE, but much more frequently with shorter periods of say 30 seconds, 1 second, or even 100ms per batch. Their models show that frequent batch auctions will reduce spreads even further (because it eliminates mechanical/latency arbitrage rents). In theory, retail traders should get even better prices (because the spreads will be smaller) if they are willing to give up microsecond immediacy for 1 second order matching. http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions-Slides.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
- tptacek 11y ago
  Your first question asks whether HFT increases or decreases volatility. That's an open question, and the answer also depends on the time scale that you're looking at. But volatility isn't the only axis on which to measure the health of the markets. The advent of electronic automated market making is also tied to drastic reduction of spreads, which are a tax that anyone trading has to pay to the market for the privilege of buying at the market. A lot of people talk about spreads as if they were a meaningless abstraction, in sort of the same way as they talk about "liquidity". But in the 1990s, crooked human market makers (read: all market makers) rigged spreads to skim extra money off trades; look up the "odd eighths scandal". That doesn't happen anymore, and, more importantly, spreads (and their attendant cost to trading) are razor thin. That's not the only thing automated electronic trading does to improve the markets. For instance, in the 80s and 90s, simply placing an order with a broker was extremely expensive. Today, there are no-fee brokers. Why? Because automated trading systems purchase the rights to internalize orders from retail brokers; if you're an automated market maker, retail order flow is practically risk free revenue, and all you have to do to earn it is hook your trading system up to brokers.
- CyberDildonics 11y ago
  People who stand to benefit directly from High Frequency Trading will tell you it has benefits. Everyone else will tell you it doesn't. Given the choice, someone buying stocks directly would not choose to buy them from an exchange that allows high frequency trading. People will also dispute that, but you have to ask yourself, if HFT is so good for everyone, who is all their money coming from?
  - tptacek 11y ago
    Accusations of shillage aren't allowed on HN; in fact, they're one of the very few examples of pathological message board behaviors that HN actually explicitly bans. You can't dodge that ban by saying "everyone who believes this is a shill" without naming people; I believe HFT has benefits, and I am not a shill. If your comment invites people to litigate whether I'm a shill, you're doing the one of the few things HN specifically asks you not to do. Please don't write comments like this.
    - CyberDildonics 11y ago
      First of all, I didn't say shill, you did. I also didn't say anything about you in particular yet you seem to take it personally. Saying that people with different incentives will have different biases is observable in virtually every system that involves people. Second, I'll write whatever I want. You'll have to use your words here instead of playing the victim card.
      - tptacek 11y ago
        There is nothing magical about the word "shill". The prohibition isn't simply on using the word. https://hn.algolia.com/?query=author:dang%20shill&sort=byDate&prefix&page=0&dateRange=all&type=comment&storyText=false https://hn.algolia.com/?query=author:dang%20shill&sort=byDat...
        CyberDildonics 11y ago
        All I said was "People who stand to benefit directly from High Frequency Trading will tell you it has benefits." Then is a general statement, I didn't say anything about this site or you. Now you seem to be trying to pull a self righteous reframing of what I said to be against some rule set since you seem to be threatened and unable to confront legitimate criticisms. It had nothing to do with you yet somehow you've attached yourself to it along with the word 'shill' which you seem to be focusing on pretty heavily.
neximo4 11y ago
I never realised the NBBO could cross because of HFT. (Technically a market type order is required to prevent this, but the quote stuffing lags price) Definitely seems wrong.
Dwolb 11y ago
Note, the SEC has a protocol for all whistleblowers. Here's the largest ever award [1] and the wiki article on the topic [2] (sorry for mobile link). I asked my accountant friend about whether he'd ever consider whistleblowing if he saw wrongdoing. His decision criteria was based on whether he'd A) Ever be able to get hired again or B) If the size of the prize was so big he'd never have to worry about it. [1] https://www.sec.gov/News/PressRelease/Detail/PressRelease/1370543011290 https://www.sec.gov/News/PressRelease/Detail/PressRelease/13... [2] https://en.m.wikipedia.org/wiki/SEC_Office_of_the_Whistleblower https://en.m.wikipedia.org/wiki/SEC_Office_of_the_Whistleblo...
chollida1 11y ago
So essentially the exchanges offer multiple data feeds. These feeds are differentiated by a couple of points. 1) How much market data they give out. Typically this is broken into 3 levels: - Level 1, Just the top of book quote, this is the best bid and offer and their corresponding number of shares at this price level. - Level 2 Depth of market at each price level, same as above but you can see all liquidity offered. - Level 3 same as Level 2 but you can see each individual order that contributes to each price level This differentiation is generally considered to be a good thing as most people including most hedge funds can get by with even Level 1 quotes. The other way they differentiate is the speed at which you get updates. Have a look here to see what just the Nasdaq offers. http://www.nasdaqtrader.com/Trader.aspx?id=FeedMIPS http://www.nasdaqtrader.com/Trader.aspx?id=FeedMIPS You can spend anywhere from $500/month to $75,000/month to receive market data from just the Nasdaq, one of 50 trading venues in the US. As you can imagine the difference is speed. The slower methods will use the text based FIX protocol and the fastest methods give you a feed and an FPGA to parse the feed with. It's this later feed that is getting the exchanges into trouble as these feeds notify the user of trades before the SIP can be updated which lets firms have a 300 millisecond second peek at the market before everyone else does. Couple these faster feeds with many dark pools not reporting trades in a timely manner and new order types that allow firms to sweep liquidity at one exchange only( ie not rout-able orders) and you get a situation where latency arb can thrive.
- easytiger 11y ago
  > As you can imagine the difference is speed. The slower methods will use the text based FIX protocol and the fastest methods give you a feed and an FPGA to parse the feed with. Err, No. You can parse FIX with an FPGA if you feel like it. It is up to you how you consume the data. The differential will be CoLocation vs dedicated line vs internet delivery on a base level and then differing subsets of feeds to varying liquidity pools. So really you are paying whomever owns the DC for the privilege of a seat in the same DC as the exchange and for the cross connect right into their cage.
  - chollida1 11y ago
    >Err, No. You can parse FIX with an FPGA if you feel like it. That's true but no one said you couldn't. What I said was that FIX based feeds would be slower than binary ones. I stand behind this. I also said that some binary feeds come with an FPGA card for parsing them, you are of course free to use your own FPGA to parse a fix feed or binary feed, but the binary feed will be faster due to it being smaller. >The differential will be CoLocation vs dedicated line vs internet delivery on a base level and then differing subsets of feeds to varying liquidity pools. This is wrong, but its not your fault, its clear you aren't in the industry or really understand it at all and that's ok. Colocation is part of what you pay for, but the latency and bandwidth of the feed itself also is part of what you pay for. You'll also pay for the type of data you receive and the SLA contract that dictates how fast you'll receive your data at the 99.99% level. Feel free to mail me if you'd like to learn more!
tsunamifury 11y ago
So essentially the NYSE is a provably fixed game, with large HFT houses paying for the right to trade on non-public information, and it hides behind the idea that it was just a few hundred milliseconds. All the while they are selling non-public information as a product that is deliberately used to micro-manipulate the market? I mean, deep down I always knew this, but to have it all spelled out is shocking because it exposes the NYSE as systemically fraudulent. The implication of that is so overwhelming it causes me to want to ignore it out of sheer inability to think of a way to solve the problem.
- tptacek 11y ago
  So, no? From 2008-2012, if you paid NYSE (one of like 15 different exchanges that trade NYSE-listed stocks) for a proprietary feed --- something that costs more than most idle home day traders can afford, but would be affordable by any YC startup --- you were getting an edge in competing with other electronic trading systems. If you were a retail investor or a mutual fund, what NYSE was doing probably didn't impact you at all.
  - CyberDildonics 11y ago
    > what NYSE was doing probably didn't impact you at all. Then where does all the money HFT firms make come from?
    - tptacek 11y ago
      First, HFT firms aren't especially lucrative compared to other finance specialties, so one answer to that question is "nowhere". Second, HFT firms compete with other finance firms, so what money they do make comes from bidding down the costs those firms were imposing on the rest of the market. If you're a retail trader, automated electronic traders make money off you by outbidding the markets to quote good prices to you for your dumb market orders, and then pocketing the spread --- in other words, they make money the same way market makers have always made money; they just make less of it at an instant, but at a far larger scale. The same is true of mutual funds, which is why Vanguard --- the firm most people would say is the most trustworthy in all of finance --- has repeatedly claimed that HFT has lowered their cost of trades.
jobu 11y ago
Pretty sure this is the initial analysis from 2010 - http://www.nanex.net/FlashCrash/FlashCrashAnalysis_LOD.html http://www.nanex.net/FlashCrash/FlashCrashAnalysis_LOD.html "If the average or base quote rate is around 10,000/second, then it only takes an additional 10,000 quotes/second to reach the magic 20,000 quotes/seconds where a corresponding delay is seen in NYSE quote from CQS." This seems like a bug related to system load, so a $5M seems pretty harsh unless they knowingly ignored the issue for the benefit of their high-paying CQS customers.
- Vraxx 11y ago
  I don't think it's really that harsh if you look at it as a bug in a critical system. Certain fields with safety critical systems (such as medical or aviation software) face similarly strict regulations and enforcement for the good of the public. It seems to me with that with the volume of money being affected by this system that the potential danger for faults in the software would be as high as potential damages from medical software or something similar.
lr 11y ago
This might be of interest to people: A NYSE Speed Bump You Weren't Aware Of https://www.iextrading.com/about/press/op-ed/ https://www.iextrading.com/about/press/op-ed/
- tptacek 11y ago
  The "speed bump" here being that NYSE's ancient creaky FIX gateway is slower than the NYSE ARCA gateway that they tell everyone to use instead. Quelle surprise! 80s text network protocol slower than 90s binary protocol: film at 11.
  - minimax 11y ago
    This is almost right. NYSE and Arca are two separate exchanges (both owned by NYSE). Both exchanges offer FIX and binary gateway protocols. Annoyingly, the Arca binary protocol and the NYSE binary protocol are not the same protocol. NYSE does not suggest using the Arca gateway to access NYSE. They suggest you use a NYSE binary gateway to access NYSE.
    - tptacek 11y ago
      Dammit. I even looked this up before I wrote the comment, and got the opposite impression. (I'm a little familiar with the Arca protocol for other reasons). Thanks for correcting me.
  - TheNanex 11y ago
    Pretty sure NYSE's ancient creaky FIX gateway is infinitely faster than the NYSE ARCA gateway - as the latter, while more network efficient, isn't going to ever take your packets to NYSE. NYSE and NYSE ARCA are different exchanges.
epx 11y ago
Nuclear reactors are planned in a way that their response time is in human scale e.g. they take something like 20 seconds to increase or decrease activity. This leaves room for human intervention. IMHO markets should do something similar: create something like 30-second barriers for trades. Trades that cross the bid/ask are carried out. Trades submitted cannot be canceled until the next 30s cycle. Trades outside of the range (bid too low/ask too high) stay open. The open trade book is sent to everybody every 30s. Allowing µs-range trades (and particularly adding and removing bids and asks so fast) is asking for trouble, it is obvious that at some point of time two or three HFT robots will react on each other's trades and do something veeeeery stupid.
- tptacek 11y ago
  Even people who support quantized markets don't propose markets running on human timescales. Meanwhile, if you're the kind of trader that is impacted by trade latency (read: a competitive market maker or decently well capitalized prop trading firm), there are already venues you can trade in that are quantized. The problem is that nobody wants to trade in them.
  - fiatmoney 11y ago
    Actually there have been several proposals for exchanges to work via, eg, every-N-seconds Dutch auctions.
  - cdetrio 11y ago
    Not sure what you mean by "quantized market" (discrete-time auction?). But the opening and closing sessions at the NYSE run at human timescales (twice per day), and most of the daily volume is traded at open and close iirc. edit: "Across Wall Street, less trading is taking place at 3 p.m. or, in fact, most any time but the opening minutes and the final half hour." http://www.wsj.com/articles/traders-pile-in-at-the-close-1432768080 http://www.wsj.com/articles/traders-pile-in-at-the-close-143...
- clarkmoody 11y ago
  The thing about many of our markets today is that they are international, with different rules in each jurisdiction. Supposing you implemented this system in the entire US, the HFT firms would simply shift their efforts to trading venues where order books are not quantized. Then they would take the real-time information and take advantage of the US exchanges. Think about the billions (trillions?) of trades that occur on minescule timescales now and how often we have flash crashes. Sure, HFT algos interact with one another constantly, but flash crash investigations have uncovered both human (manual entry) and algo error as their root causes. Part of what makes flash crashes more severe is when HFT safety protocols kick in and they remove their orders from the market, causing liquidity to dry up and exacerbate the crash.
reverend_gonzo 11y ago
What he's saying there was that due to inefficiencies in NYSE's infrastructure, the feed that went to the consolidation (which provides the NBBO) was delayed. Firms were able to then exploit those inefficiencies by dumping a ton of quotes into the market to cause it slow down those feeds. This is called quote-stuffing and is illegal and against exchange rules. (I don't know if this was illegal then.) From a technical perspective, as orders are sent to the exchange, NYSE sends them out on a direct line to subscribers who listen to that feed. This is not private information, but it is relatively expensive, given you'd need to have servers at a colo, and pay for fast networking, boxes, software, etc. They also then aggregate that data and send them out on a feed to the NBBO which is much more accessible to the majority of traders. Obviously, it is somewhat delayed, as they need to aggregate, and a firm that can aggregate the data faster than NYSE thereby has an advantage. This aggregated data gets set out as soon as possible, and NBBO defines a time range that they need to be accurate. For what its worth, NBBO means "National Best Bid and Offer" and they will aggregate bids and offers from all exchanges and provide the best one in a single consolidated feed. Nanex determined that at certain times during high volume, Nasdaq's aggregate feed was delayed by up to 30 seconds, and this was cause by certain entities quote stuffing (which like I said earlier, is against the rules). That is all that was determined. There is nothing about trading on private information (which there is none here) or front-running (which HFT firms can't do) or any of the other nonsense that gets spewed out. If anything, other firms with good infrastructure would have been able to arb out the inefficiency and make the markets right again, quickly.
- biot 11y ago
  Your summary is incorrect; the 30 second delay was six years ago and was caused by selling access to real-time data vs delaying the data for the public. Quote stuffing resulted in delays of hundreds of milliseconds: > Once again, we detected sizable delays between time-stamps > in the public quote and Open Book. These delays ranged in > the hundreds of milliseconds. Though the delay magnitude was > far lower than the tens of seconds discovered during the flash > crash, it was still hundreds of times higher than expected.
  - reverend_gonzo 11y ago
nostromo 11y ago
Google should open a stock exchange. Sure, they'd be weary of the immense regulation involved. But I imagine they're much more qualified to build a fair, fast and accurate trading system than any of the existing exchanges are.
- tptacek 11y ago
  What would be more "fair" or "fast" or "accurate" about Google's exchange? I've gotten to do software pentests of several huge exchanges. They work pretty much the way you'd expect them to. A lot of things people are just starting to do in web software (message-mediated microservices) were idiomatic in finance in the 1990s.
  - nostromo 11y ago
    Look no further than the linked article. > a high quote rate from any NYSE stock will cause a corresponding delay in all NYSE public quotes! Anyone with this knowledge can easily cause latency on demand across many stocks, by simply blasting quotes in any one stock Or another link from this thread. > U.S. stock exchanges have invested huge sums of money creating two-tier markets - building and offering faster data and technology infrastructures at a price that only a small niche of traders can benefit from or afford, while at the same time continuing to offer slower products to everybody else. https://www.iextrading.com/about/press/op-ed/ https://www.iextrading.com/about/press/op-ed/
    - tptacek 11y ago
      That was true on one exchange in 2012. It's no longer true today on NYSE, and probably wasn't true at any other venue (Hunsader is able to explain how he found this in a relatively small blog post; dollars to donuts he looked everywhere else for it too). So: not delaying public quotes is table stakes. What could Google do better than that? I'm not so much challenging you as I think it's an interesting question, and would like to see it hashed out in technical detail.
mrbill 11y ago
For someone who knows next to nothing about trading, but has read "Flash Boys", any other recommended books on the topic?
- tptacek 11y ago
  Yes: read "Flash Boys: Not So Fast", a very readable and very technical takedown of "Flash Boys", which is a terrible, incoherent book --- without a doubt Lewis' worst. For a better version of "Flash Boys", read "Dark Pools"; Dark Pools also takes a gimlet look at high-speed trading, but also does a great job of documenting the birth of the NASDAQ ECNs and the transition from the early-90s centralized markets to modern, decentralized, electronic trading. If you want to get into the technical detail, the best book is still Larry Harris' _Trading and Exchanges_; it is almost exactly the "TCP/IP Illustrated" of trading.
  - TheNanex 11y ago
    "Flash Boys: Not So Fast" is an absolute joke, and this is known by everyone on Wall Street with a functioning brain (I'm guessing you know this too). Your name is all over this thread like a dirty hand on a white linen dress. Trolling. You've poisoned any chance of a reasonable discussion. If anyone wants their questions on this asked, I'm @nanexllc on twitter. Nanex out..
    - tptacek 11y ago
      I don't believe you're going to persuade anyone here with appeals to your own authority; you are not the only person on the thread with experience with this subject. I know less about market structure than any of the many commenters here that navigate it for a living, but I've had enough exposure to know that you haven't made a coherent argument yet. I'm sure you have one in you!
    - deleted 11y ago
      [deleted]
TheNanex 11y ago
Yikes. Lots of misinformation here. First of all, this has nothing to do with High-Frequency Trading. It's about the NYSE not delivering a product (SIP real-time data) while collecting $100M a year for that service. This was happening for at least 3 years. I am a champion of free markets. The term "High-Frequency Trading critic" is a label others use when they either can't understand and/or refute solid evidence. I'm happy to answer questions.
- kbenson 11y ago
  Can you comment on how hard it was to find another period exhibiting this behavior? The article says We chose this period because there was a noticeable lag in the public quote from the NYSE versus quotes from other exchanges, allowing us to rule out the consolidation process as a source of the delay. Did you have to look for a few periods to see the behavior, or did you look for a few to get a sample that showed the problem more clearly? Do you have any info on how often lag was present, or how often over a specific threshold? Do you know how representative you sample was with respect to when lag was present? I understand you may not have gone into this much detail in your research, but if you have any info like this, I think it helps illustrate the scope of the problem.
  - kdkooo 11y ago
    I second that answers to these types of questions would be very helpful! The more context the better.
  - fnordfnordfnord 11y ago
    http://www.nanex.net/NxResearch/ResearchPage/5/ http://www.nanex.net/NxResearch/ResearchPage/5/
  - TheNanex 11y ago
    It happens all the time (ALL. THE. TIME.) and have documented dozens of examples. There are links to many of those examples on that page (fantaseconds). I've also posted many examples on twitter @nanexllc using recent data (yes, this is still a problem!)
Chefkoochooloo 11y ago
Now that shows you how far the whistleblower laws have gone ever since Enron. A big step up from what there was before.
rcavezza 11y ago
Anyone know what this quote means? When the SEC called Hunsader in June 2015 to tell him about the award, he said to them "I would have accepted $1 if you simply acknowledged me at the time." Was he initially ignored?
- TheNanex 11y ago
  I was ignored by the Trading and Markets people at the SEC, but not ignored (obviously) by the Enforcement people.
  - eewffwe 11y ago
    I don't want to be silly but it appears the SEC ignores the siren often (Madoff is another). Is this due to sheer workload or friends in high places?
    - TheNanex 11y ago
      Wealthy friends with promises of a nice job (and other things)
kpkp 11y ago
Except the NBBO was often stale (on purpose), so you were getting what was wrongly quoted as the NBBO by selling to an HFT who knew that the actual BBO was better, allowing them to arb on that knowledge. That said, a substantial portion of your posting history consists of you being dishonest about the value extracted by HFT (value that is actually created by others), so I'm not surprised that you are lying about the results.
- tptacek 11y ago
  You can't write comments on HN that: * Attempt to synthesize negative things out of people's comment histories to club them with on current threads * Directly accuse commenters of lying * Imply that comments are being written in bad faith or as shills. Your comment manages to do all three. Please don't write comments like this here.
- reverend_gonzo 11y ago
  I speak about what I know. I don't speak about things I don't know, or when there are more knowledgeable people around. Given that I've been in the industry for five years, and I hear a lot of regurgitated nonsense, that's when I speak up. But there's a bunch of people who already have opinions that everything about HFT is dishonest, so be it. Sure, I'm lying about everything.
- dang 11y ago
  We've banned this account for breaking the HN guidelines and detached this subthread from https://news.ycombinator.com/item?id=11211521 https://news.ycombinator.com/item?id=11211521 and marked it off-topic.
itchyouch 11y ago
Nasdaq does go through great pains to ensure that public & private feeds go out "equally"-ish on the wire whether it's a quote going out of the prop or public data feeds. At the packet level, a public quote to the SIP or a private quote via Itch are within the sub ~10u level. But once the packet leaves, the fate of that packet is surrendered to the speed of the SIP system. There's a couple of inherent challenges. The SIP that aggregates the data is by design a separate system from the trading systems. This is for obvious reasons, as every participant provides their quote to the SIP. So no matter how fast the SIP the is, even with a 99.999% latency of 20 microseconds, the SIP will always be 20 micros + travel time to the SIP (1 to 500 to 7000 micros) as opposed to a direct feed. Nasdaq is in Carteret. NYSE is in Mahwah. BATS is at NY4 in Secaucus, etc. So direct feeds will always be the fastest, even with a SIP that can defy the speed of light a little bit. Nasdaq does actually read marketdata directly from other exchange participants and falls back to using the "slower SIP" data as a fall back in the event of any issues. This allows for the latest quotes to be respected. The challenge with the SIPs being hundreds of millis behind is partly due to the criticality & member-driven consensus system in place. Each of the marketcenters NYSE, Nasdaq, Bats, etc have a stake and voting power on how the SIP would be run & operated. Since it's a system that "works" there's really no motivation to continually improve the technology/latency of the SIP. There's also a fear to change the SIP and lead to it going down. If a SIP goes down, by rule, a regulatory halt must be issued and all trading in the securities that particular SIP handles must stop. This is what occurred during the 3 hour outage several years back at Nasdaq. The trading system was fine, but since the SIP was down, everything had to be halted. Halts are detrimental to business since the respective marketcenters make money on trades. Halts = No Trades = Lost Revenue, so there is a lot of motivation to not halt by all the member exchanges. That ambivalence to a tech refresh has since changed since the scathing critique of Flash Boys and the Nasdaq SIP is in process of being refreshed to the same architecture as the trading systems, which should bring median latencies of the SIP down into the tens of microseconds and 99% latencies into the hundreds of microseconds. Current medians and 90th percentiles are at 450/810 microseconds, which is considered an eternity. http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-won-over-sip-committee-with-latency-reductions-and-tech-upgrades/d/d-id/1317316 http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-w... http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Processor%20Stats.pdf http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Pr... On the technical side, one of the challenges with synchronizing packet delivery between a private & public feed is that the absolute fastest NICs available today with kernel bypass can get a packet from wire to memory or memory to wire in about 2-3u. So every hop of a computer costs a minimum of 4-6u. Usually this is Order Port Ingress -> Matching Engine -> Order Port Egress. An FPGA NIC that does the IP Header processing and delivers data to memory can do wire to memory in about 1-2u. One-way PCI-E latency is around 800 nanos. So there's about a 500-1000 nano budget to process a packet in the FPGA, then another 800 nanos to deliver it over PCI-E to memory to be processed. The fastest cut-through switches have a port-to-port latency of around 250 nanos, so switch hops are pretty negligible in the overall latency of a system, but they do play a part. Anyway, making sure that everything is super duper optimized and running as it should be in a semi-continuously deployed environment makes for interesting and quite crazy/bull-shit level challenges.
uptown 11y ago
Ultimately, the market has been setup to almost mandate participation. If you work someplace that provides a matching 401k program, and you choose not to contribute, then you're passing-up on matched dollars if you don't contribute to a 401k. If you're a part of any endowment, pension plan, or other managed retirement plan there's a good chance that plan (and your money) is being invested in the market. So what are your options? Do you not contribute to a 401k or IRA, pay the full tax rate on all of your income today, and invest in real estate or some other investment?
deleted 11y ago
[deleted]
fitzwatermellow 11y ago
Congrats on being rewarded and recognized for the very important work you do in bringing transparency to the financial markets. The whole story of how this went down is just epic! My question is about the business side of Nanex. Are sales of your products such as NxCore enough to sustain the company? Do you also actively trade? And what opportunities do you see currently around data science and machine learning for entrepreneurs in the quantitative finance world?
irascible 11y ago
Hax! Lag switch. :/ bitches.
__d 11y ago
Is there any good reason that the SEC shouldn't ban direct feeds? If everyone were forced to use the SIP, what issues would remain here?

328 points by theandycamps 11y ago

While I am fully pro-HFT, and in fact work as a consultant for a few HFT shops, this is certainly illegal and anti-competitive. (Somewhat akin to using game engine bugs for artificial lag creation in MMOs on the opponent side).

Someone1234 11y ago

I have two questions about HFT: - Is HFT a healthy or unhealthy part of the economy? Meaning does it help flatten out the highs/lows, or does it emphasise them? - Aside from a micro-tax on trading (e.g. 1c/trade) is there any other mitigations/solutions to HFT?

SeanLuke 11y ago
I don't see why we couldn't put a time delay in. Something like this: when you commit to a trade, the actual trade delayed for some N minutes, where N is a randomly-chosen value from some distribution. Wouldn't this eliminate the incentive to do HFT?
- jgh 11y ago
  Another disincentive might be the trade tax that Bernie Sanders has proposed since it would basically affect HFT way more than anyone else.
  - cheald 11y ago
    No, it would affect everyone - that's why it's so bad. HFT traders would just adjust their minimum spreads and continue on their merry way. Sanders' FTT doesn't attack the mechanisms of HFT, it attacks the markets as a whole - HFT would suffer, but so would everyone else. If you want to hit HFT specifically you need to hit the HFT-specific behaviors - either add a per-message charge (like Canada did) or add a tax to cancelled orders. It's worth noting that after the Canadian regulatory authorities added a per-message fee, bid-ask spreads rose, liquidity dropped, and it was retail traders who suffered the most. http://marginalrevolution.com/marginalrevolution/2014/04/a-study-of-limiting-hft.html http://marginalrevolution.com/marginalrevolution/2014/04/a-s...
    - jgh 11y ago
      I understand that a trade tax would affect everyone, but for someone like me who makes maybe a handful of trades per month it wouldn't really do that much to dissuade me from making those trades. I also don't particularly care one way or the other what happens with HFT, I was just mentioning this as a thing people are thinking about.
      - kasey_junk 11y ago
        My big concern with the transaction tax is how it applies to indexed funds/etfs. Those trade all the time so the price impact can be significant for those. Further, depending on implementation, you could easily get taxed multiple times for that as there is a transaction to get in/out of the index, and the internal index transactions.
- harryh 11y ago
  No, it would just raise the cost of trading for everyone else. If HFT's risk goes up they have to raise their prices.
kbenson 11y ago
IIRC, HN's yummyfajitas goes into a lot of details in his blog posts on HFT[1]. My own understanding is that it provides liquidity and reduces bid/ask spreads, resulting in a more accurate stock valuation. I'm not sure the negatives, but I'm not a trader nor do I follow such things all that closely. 1: https://news.ycombinator.com/item?id=3852341 https://news.ycombinator.com/item?id=3852341
- jobu 11y ago
  From that blog post: "Most HFTs run a market making strategy. What this means is they play both sides of the table - they take no position on whether a stock will go up or down. Instead, they try to offer securities both to buy and sell. If you want to buy, they will sell to you at $20.10. If you want to sell, they'll buy from you at $20. As long as their buys and sells match don't get too out of whack, the HFT will collect $0.10 = $20.10 - 20.00." That does provide more immediate trades for both the buyer and seller, which the pro-HFT camp calls liquidity. Unfortunately it's usually not needed - both the buyer and seller were already in the market and willing to trade, and now they both just had an additional $0.10/share sucked out of their trade by some HFT.
  - arohner 11y ago
    In the past, either the buyer and seller had to wait longer to find a mutually acceptable price, or pay a lot more than $0.10/share. HFT is actually an improvement for both sides, because they can either 1) buy/sell faster or 2) pay lower transaction costs (and probably both!).
  - cdetrio 11y ago
    That's why market makers are also called scalpers, because they skim the $0.10 spread. But in return for paying the spread, what traders get is immediacy. The problem is, how much immediacy do traders actually want? HFT provides immediacy in microseconds; its needed for continuous-time auctions because buyers sellers usually aren't in the market at the exact same microsecond interval (while the HFT is there at every interval). Frequent Batch Auctions[1], where orders are matched in batches with single-price clearing (call auction style) - just like the opening and closing sessions of NYSE, but much more frequently with shorter periods of say 30 seconds, 1 second, or even 100ms per batch. Their models show that frequent batch auctions will reduce spreads even further (because it eliminates mechanical/latency arbitrage rents). In theory, retail traders should get even better prices (because the spreads will be smaller) if they are willing to give up microsecond immediacy for 1 second order matching. http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions-Slides.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
tptacek 11y ago
Your first question asks whether HFT increases or decreases volatility. That's an open question, and the answer also depends on the time scale that you're looking at. But volatility isn't the only axis on which to measure the health of the markets. The advent of electronic automated market making is also tied to drastic reduction of spreads, which are a tax that anyone trading has to pay to the market for the privilege of buying at the market. A lot of people talk about spreads as if they were a meaningless abstraction, in sort of the same way as they talk about "liquidity". But in the 1990s, crooked human market makers (read: all market makers) rigged spreads to skim extra money off trades; look up the "odd eighths scandal". That doesn't happen anymore, and, more importantly, spreads (and their attendant cost to trading) are razor thin. That's not the only thing automated electronic trading does to improve the markets. For instance, in the 80s and 90s, simply placing an order with a broker was extremely expensive. Today, there are no-fee brokers. Why? Because automated trading systems purchase the rights to internalize orders from retail brokers; if you're an automated market maker, retail order flow is practically risk free revenue, and all you have to do to earn it is hook your trading system up to brokers.
CyberDildonics 11y ago
People who stand to benefit directly from High Frequency Trading will tell you it has benefits. Everyone else will tell you it doesn't. Given the choice, someone buying stocks directly would not choose to buy them from an exchange that allows high frequency trading. People will also dispute that, but you have to ask yourself, if HFT is so good for everyone, who is all their money coming from?
- tptacek 11y ago
  Accusations of shillage aren't allowed on HN; in fact, they're one of the very few examples of pathological message board behaviors that HN actually explicitly bans. You can't dodge that ban by saying "everyone who believes this is a shill" without naming people; I believe HFT has benefits, and I am not a shill. If your comment invites people to litigate whether I'm a shill, you're doing the one of the few things HN specifically asks you not to do. Please don't write comments like this.
  - CyberDildonics 11y ago
    First of all, I didn't say shill, you did. I also didn't say anything about you in particular yet you seem to take it personally. Saying that people with different incentives will have different biases is observable in virtually every system that involves people. Second, I'll write whatever I want. You'll have to use your words here instead of playing the victim card.
    - tptacek 11y ago
      There is nothing magical about the word "shill". The prohibition isn't simply on using the word. https://hn.algolia.com/?query=author:dang%20shill&sort=byDate&prefix&page=0&dateRange=all&type=comment&storyText=false https://hn.algolia.com/?query=author:dang%20shill&sort=byDat...
      - CyberDildonics 11y ago
        All I said was "People who stand to benefit directly from High Frequency Trading will tell you it has benefits." Then is a general statement, I didn't say anything about this site or you. Now you seem to be trying to pull a self righteous reframing of what I said to be against some rule set since you seem to be threatened and unable to confront legitimate criticisms. It had nothing to do with you yet somehow you've attached yourself to it along with the word 'shill' which you seem to be focusing on pretty heavily.

neximo4 11y ago

I never realised the NBBO could cross because of HFT. (Technically a market type order is required to prevent this, but the quote stuffing lags price) Definitely seems wrong.

Dwolb 11y ago

Note, the SEC has a protocol for all whistleblowers. Here's the largest ever award [1] and the wiki article on the topic [2] (sorry for mobile link). I asked my accountant friend about whether he'd ever consider whistleblowing if he saw wrongdoing. His decision criteria was based on whether he'd A) Ever be able to get hired again or B) If the size of the prize was so big he'd never have to worry about it. [1] https://www.sec.gov/News/PressRelease/Detail/PressRelease/1370543011290 https://www.sec.gov/News/PressRelease/Detail/PressRelease/13... [2] https://en.m.wikipedia.org/wiki/SEC_Office_of_the_Whistleblower https://en.m.wikipedia.org/wiki/SEC_Office_of_the_Whistleblo...

chollida1 11y ago

So essentially the exchanges offer multiple data feeds. These feeds are differentiated by a couple of points. 1) How much market data they give out. Typically this is broken into 3 levels: - Level 1, Just the top of book quote, this is the best bid and offer and their corresponding number of shares at this price level. - Level 2 Depth of market at each price level, same as above but you can see all liquidity offered. - Level 3 same as Level 2 but you can see each individual order that contributes to each price level This differentiation is generally considered to be a good thing as most people including most hedge funds can get by with even Level 1 quotes. The other way they differentiate is the speed at which you get updates. Have a look here to see what just the Nasdaq offers. http://www.nasdaqtrader.com/Trader.aspx?id=FeedMIPS http://www.nasdaqtrader.com/Trader.aspx?id=FeedMIPS You can spend anywhere from $500/month to $75,000/month to receive market data from just the Nasdaq, one of 50 trading venues in the US. As you can imagine the difference is speed. The slower methods will use the text based FIX protocol and the fastest methods give you a feed and an FPGA to parse the feed with. It's this later feed that is getting the exchanges into trouble as these feeds notify the user of trades before the SIP can be updated which lets firms have a 300 millisecond second peek at the market before everyone else does. Couple these faster feeds with many dark pools not reporting trades in a timely manner and new order types that allow firms to sweep liquidity at one exchange only( ie not rout-able orders) and you get a situation where latency arb can thrive.

easytiger 11y ago
> As you can imagine the difference is speed. The slower methods will use the text based FIX protocol and the fastest methods give you a feed and an FPGA to parse the feed with. Err, No. You can parse FIX with an FPGA if you feel like it. It is up to you how you consume the data. The differential will be CoLocation vs dedicated line vs internet delivery on a base level and then differing subsets of feeds to varying liquidity pools. So really you are paying whomever owns the DC for the privilege of a seat in the same DC as the exchange and for the cross connect right into their cage.
- chollida1 11y ago
  >Err, No. You can parse FIX with an FPGA if you feel like it. That's true but no one said you couldn't. What I said was that FIX based feeds would be slower than binary ones. I stand behind this. I also said that some binary feeds come with an FPGA card for parsing them, you are of course free to use your own FPGA to parse a fix feed or binary feed, but the binary feed will be faster due to it being smaller. >The differential will be CoLocation vs dedicated line vs internet delivery on a base level and then differing subsets of feeds to varying liquidity pools. This is wrong, but its not your fault, its clear you aren't in the industry or really understand it at all and that's ok. Colocation is part of what you pay for, but the latency and bandwidth of the feed itself also is part of what you pay for. You'll also pay for the type of data you receive and the SLA contract that dictates how fast you'll receive your data at the 99.99% level. Feel free to mail me if you'd like to learn more!

tsunamifury 11y ago

So essentially the NYSE is a provably fixed game, with large HFT houses paying for the right to trade on non-public information, and it hides behind the idea that it was just a few hundred milliseconds. All the while they are selling non-public information as a product that is deliberately used to micro-manipulate the market? I mean, deep down I always knew this, but to have it all spelled out is shocking because it exposes the NYSE as systemically fraudulent. The implication of that is so overwhelming it causes me to want to ignore it out of sheer inability to think of a way to solve the problem.

tptacek 11y ago
So, no? From 2008-2012, if you paid NYSE (one of like 15 different exchanges that trade NYSE-listed stocks) for a proprietary feed --- something that costs more than most idle home day traders can afford, but would be affordable by any YC startup --- you were getting an edge in competing with other electronic trading systems. If you were a retail investor or a mutual fund, what NYSE was doing probably didn't impact you at all.
- CyberDildonics 11y ago
  > what NYSE was doing probably didn't impact you at all. Then where does all the money HFT firms make come from?
  - tptacek 11y ago
    First, HFT firms aren't especially lucrative compared to other finance specialties, so one answer to that question is "nowhere". Second, HFT firms compete with other finance firms, so what money they do make comes from bidding down the costs those firms were imposing on the rest of the market. If you're a retail trader, automated electronic traders make money off you by outbidding the markets to quote good prices to you for your dumb market orders, and then pocketing the spread --- in other words, they make money the same way market makers have always made money; they just make less of it at an instant, but at a far larger scale. The same is true of mutual funds, which is why Vanguard --- the firm most people would say is the most trustworthy in all of finance --- has repeatedly claimed that HFT has lowered their cost of trades.

jobu 11y ago

Pretty sure this is the initial analysis from 2010 - http://www.nanex.net/FlashCrash/FlashCrashAnalysis_LOD.html http://www.nanex.net/FlashCrash/FlashCrashAnalysis_LOD.html "If the average or base quote rate is around 10,000/second, then it only takes an additional 10,000 quotes/second to reach the magic 20,000 quotes/seconds where a corresponding delay is seen in NYSE quote from CQS." This seems like a bug related to system load, so a $5M seems pretty harsh unless they knowingly ignored the issue for the benefit of their high-paying CQS customers.

Vraxx 11y ago
I don't think it's really that harsh if you look at it as a bug in a critical system. Certain fields with safety critical systems (such as medical or aviation software) face similarly strict regulations and enforcement for the good of the public. It seems to me with that with the volume of money being affected by this system that the potential danger for faults in the software would be as high as potential damages from medical software or something similar.

lr 11y ago

This might be of interest to people: A NYSE Speed Bump You Weren't Aware Of https://www.iextrading.com/about/press/op-ed/ https://www.iextrading.com/about/press/op-ed/

tptacek 11y ago
The "speed bump" here being that NYSE's ancient creaky FIX gateway is slower than the NYSE ARCA gateway that they tell everyone to use instead. Quelle surprise! 80s text network protocol slower than 90s binary protocol: film at 11.
- minimax 11y ago
  This is almost right. NYSE and Arca are two separate exchanges (both owned by NYSE). Both exchanges offer FIX and binary gateway protocols. Annoyingly, the Arca binary protocol and the NYSE binary protocol are not the same protocol. NYSE does not suggest using the Arca gateway to access NYSE. They suggest you use a NYSE binary gateway to access NYSE.
  - tptacek 11y ago
    Dammit. I even looked this up before I wrote the comment, and got the opposite impression. (I'm a little familiar with the Arca protocol for other reasons). Thanks for correcting me.
- TheNanex 11y ago
  Pretty sure NYSE's ancient creaky FIX gateway is infinitely faster than the NYSE ARCA gateway - as the latter, while more network efficient, isn't going to ever take your packets to NYSE. NYSE and NYSE ARCA are different exchanges.

epx 11y ago

Nuclear reactors are planned in a way that their response time is in human scale e.g. they take something like 20 seconds to increase or decrease activity. This leaves room for human intervention. IMHO markets should do something similar: create something like 30-second barriers for trades. Trades that cross the bid/ask are carried out. Trades submitted cannot be canceled until the next 30s cycle. Trades outside of the range (bid too low/ask too high) stay open. The open trade book is sent to everybody every 30s. Allowing µs-range trades (and particularly adding and removing bids and asks so fast) is asking for trouble, it is obvious that at some point of time two or three HFT robots will react on each other's trades and do something veeeeery stupid.

tptacek 11y ago
Even people who support quantized markets don't propose markets running on human timescales. Meanwhile, if you're the kind of trader that is impacted by trade latency (read: a competitive market maker or decently well capitalized prop trading firm), there are already venues you can trade in that are quantized. The problem is that nobody wants to trade in them.
- fiatmoney 11y ago
  Actually there have been several proposals for exchanges to work via, eg, every-N-seconds Dutch auctions.
- cdetrio 11y ago
  Not sure what you mean by "quantized market" (discrete-time auction?). But the opening and closing sessions at the NYSE run at human timescales (twice per day), and most of the daily volume is traded at open and close iirc. edit: "Across Wall Street, less trading is taking place at 3 p.m. or, in fact, most any time but the opening minutes and the final half hour." http://www.wsj.com/articles/traders-pile-in-at-the-close-1432768080 http://www.wsj.com/articles/traders-pile-in-at-the-close-143...
clarkmoody 11y ago
The thing about many of our markets today is that they are international, with different rules in each jurisdiction. Supposing you implemented this system in the entire US, the HFT firms would simply shift their efforts to trading venues where order books are not quantized. Then they would take the real-time information and take advantage of the US exchanges. Think about the billions (trillions?) of trades that occur on minescule timescales now and how often we have flash crashes. Sure, HFT algos interact with one another constantly, but flash crash investigations have uncovered both human (manual entry) and algo error as their root causes. Part of what makes flash crashes more severe is when HFT safety protocols kick in and they remove their orders from the market, causing liquidity to dry up and exacerbate the crash.

reverend_gonzo 11y ago

What he's saying there was that due to inefficiencies in NYSE's infrastructure, the feed that went to the consolidation (which provides the NBBO) was delayed. Firms were able to then exploit those inefficiencies by dumping a ton of quotes into the market to cause it slow down those feeds. This is called quote-stuffing and is illegal and against exchange rules. (I don't know if this was illegal then.) From a technical perspective, as orders are sent to the exchange, NYSE sends them out on a direct line to subscribers who listen to that feed. This is not private information, but it is relatively expensive, given you'd need to have servers at a colo, and pay for fast networking, boxes, software, etc. They also then aggregate that data and send them out on a feed to the NBBO which is much more accessible to the majority of traders. Obviously, it is somewhat delayed, as they need to aggregate, and a firm that can aggregate the data faster than NYSE thereby has an advantage. This aggregated data gets set out as soon as possible, and NBBO defines a time range that they need to be accurate. For what its worth, NBBO means "National Best Bid and Offer" and they will aggregate bids and offers from all exchanges and provide the best one in a single consolidated feed. Nanex determined that at certain times during high volume, Nasdaq's aggregate feed was delayed by up to 30 seconds, and this was cause by certain entities quote stuffing (which like I said earlier, is against the rules). That is all that was determined. There is nothing about trading on private information (which there is none here) or front-running (which HFT firms can't do) or any of the other nonsense that gets spewed out. If anything, other firms with good infrastructure would have been able to arb out the inefficiency and make the markets right again, quickly.

biot 11y ago
Your summary is incorrect; the 30 second delay was six years ago and was caused by selling access to real-time data vs delaying the data for the public. Quote stuffing resulted in delays of hundreds of milliseconds: > Once again, we detected sizable delays between time-stamps > in the public quote and Open Book. These delays ranged in > the hundreds of milliseconds. Though the delay magnitude was > far lower than the tens of seconds discovered during the flash > crash, it was still hundreds of times higher than expected.
- reverend_gonzo 11y ago

nostromo 11y ago

Google should open a stock exchange. Sure, they'd be weary of the immense regulation involved. But I imagine they're much more qualified to build a fair, fast and accurate trading system than any of the existing exchanges are.

tptacek 11y ago
What would be more "fair" or "fast" or "accurate" about Google's exchange? I've gotten to do software pentests of several huge exchanges. They work pretty much the way you'd expect them to. A lot of things people are just starting to do in web software (message-mediated microservices) were idiomatic in finance in the 1990s.
- nostromo 11y ago
  Look no further than the linked article. > a high quote rate from any NYSE stock will cause a corresponding delay in all NYSE public quotes! Anyone with this knowledge can easily cause latency on demand across many stocks, by simply blasting quotes in any one stock Or another link from this thread. > U.S. stock exchanges have invested huge sums of money creating two-tier markets - building and offering faster data and technology infrastructures at a price that only a small niche of traders can benefit from or afford, while at the same time continuing to offer slower products to everybody else. https://www.iextrading.com/about/press/op-ed/ https://www.iextrading.com/about/press/op-ed/
  - tptacek 11y ago
    That was true on one exchange in 2012. It's no longer true today on NYSE, and probably wasn't true at any other venue (Hunsader is able to explain how he found this in a relatively small blog post; dollars to donuts he looked everywhere else for it too). So: not delaying public quotes is table stakes. What could Google do better than that? I'm not so much challenging you as I think it's an interesting question, and would like to see it hashed out in technical detail.

mrbill 11y ago

For someone who knows next to nothing about trading, but has read "Flash Boys", any other recommended books on the topic?

tptacek 11y ago
Yes: read "Flash Boys: Not So Fast", a very readable and very technical takedown of "Flash Boys", which is a terrible, incoherent book --- without a doubt Lewis' worst. For a better version of "Flash Boys", read "Dark Pools"; Dark Pools also takes a gimlet look at high-speed trading, but also does a great job of documenting the birth of the NASDAQ ECNs and the transition from the early-90s centralized markets to modern, decentralized, electronic trading. If you want to get into the technical detail, the best book is still Larry Harris' _Trading and Exchanges_; it is almost exactly the "TCP/IP Illustrated" of trading.
- TheNanex 11y ago
  "Flash Boys: Not So Fast" is an absolute joke, and this is known by everyone on Wall Street with a functioning brain (I'm guessing you know this too). Your name is all over this thread like a dirty hand on a white linen dress. Trolling. You've poisoned any chance of a reasonable discussion. If anyone wants their questions on this asked, I'm @nanexllc on twitter. Nanex out..
  - tptacek 11y ago
    I don't believe you're going to persuade anyone here with appeals to your own authority; you are not the only person on the thread with experience with this subject. I know less about market structure than any of the many commenters here that navigate it for a living, but I've had enough exposure to know that you haven't made a coherent argument yet. I'm sure you have one in you!
  - deleted 11y ago
    [deleted]

TheNanex 11y ago

Yikes. Lots of misinformation here. First of all, this has nothing to do with High-Frequency Trading. It's about the NYSE not delivering a product (SIP real-time data) while collecting $100M a year for that service. This was happening for at least 3 years. I am a champion of free markets. The term "High-Frequency Trading critic" is a label others use when they either can't understand and/or refute solid evidence. I'm happy to answer questions.

kbenson 11y ago
Can you comment on how hard it was to find another period exhibiting this behavior? The article says We chose this period because there was a noticeable lag in the public quote from the NYSE versus quotes from other exchanges, allowing us to rule out the consolidation process as a source of the delay. Did you have to look for a few periods to see the behavior, or did you look for a few to get a sample that showed the problem more clearly? Do you have any info on how often lag was present, or how often over a specific threshold? Do you know how representative you sample was with respect to when lag was present? I understand you may not have gone into this much detail in your research, but if you have any info like this, I think it helps illustrate the scope of the problem.
- kdkooo 11y ago
  I second that answers to these types of questions would be very helpful! The more context the better.
- fnordfnordfnord 11y ago
  http://www.nanex.net/NxResearch/ResearchPage/5/ http://www.nanex.net/NxResearch/ResearchPage/5/
- TheNanex 11y ago
  It happens all the time (ALL. THE. TIME.) and have documented dozens of examples. There are links to many of those examples on that page (fantaseconds). I've also posted many examples on twitter @nanexllc using recent data (yes, this is still a problem!)

Chefkoochooloo 11y ago

Now that shows you how far the whistleblower laws have gone ever since Enron. A big step up from what there was before.

rcavezza 11y ago

Anyone know what this quote means? When the SEC called Hunsader in June 2015 to tell him about the award, he said to them "I would have accepted $1 if you simply acknowledged me at the time." Was he initially ignored?

TheNanex 11y ago
I was ignored by the Trading and Markets people at the SEC, but not ignored (obviously) by the Enforcement people.
- eewffwe 11y ago
  I don't want to be silly but it appears the SEC ignores the siren often (Madoff is another). Is this due to sheer workload or friends in high places?
  - TheNanex 11y ago
    Wealthy friends with promises of a nice job (and other things)

kpkp 11y ago

Except the NBBO was often stale (on purpose), so you were getting what was wrongly quoted as the NBBO by selling to an HFT who knew that the actual BBO was better, allowing them to arb on that knowledge. That said, a substantial portion of your posting history consists of you being dishonest about the value extracted by HFT (value that is actually created by others), so I'm not surprised that you are lying about the results.

tptacek 11y ago
You can't write comments on HN that: * Attempt to synthesize negative things out of people's comment histories to club them with on current threads * Directly accuse commenters of lying * Imply that comments are being written in bad faith or as shills. Your comment manages to do all three. Please don't write comments like this here.
reverend_gonzo 11y ago
I speak about what I know. I don't speak about things I don't know, or when there are more knowledgeable people around. Given that I've been in the industry for five years, and I hear a lot of regurgitated nonsense, that's when I speak up. But there's a bunch of people who already have opinions that everything about HFT is dishonest, so be it. Sure, I'm lying about everything.
dang 11y ago
We've banned this account for breaking the HN guidelines and detached this subthread from https://news.ycombinator.com/item?id=11211521 https://news.ycombinator.com/item?id=11211521 and marked it off-topic.

itchyouch 11y ago

Nasdaq does go through great pains to ensure that public & private feeds go out "equally"-ish on the wire whether it's a quote going out of the prop or public data feeds. At the packet level, a public quote to the SIP or a private quote via Itch are within the sub ~10u level. But once the packet leaves, the fate of that packet is surrendered to the speed of the SIP system. There's a couple of inherent challenges. The SIP that aggregates the data is by design a separate system from the trading systems. This is for obvious reasons, as every participant provides their quote to the SIP. So no matter how fast the SIP the is, even with a 99.999% latency of 20 microseconds, the SIP will always be 20 micros + travel time to the SIP (1 to 500 to 7000 micros) as opposed to a direct feed. Nasdaq is in Carteret. NYSE is in Mahwah. BATS is at NY4 in Secaucus, etc. So direct feeds will always be the fastest, even with a SIP that can defy the speed of light a little bit. Nasdaq does actually read marketdata directly from other exchange participants and falls back to using the "slower SIP" data as a fall back in the event of any issues. This allows for the latest quotes to be respected. The challenge with the SIPs being hundreds of millis behind is partly due to the criticality & member-driven consensus system in place. Each of the marketcenters NYSE, Nasdaq, Bats, etc have a stake and voting power on how the SIP would be run & operated. Since it's a system that "works" there's really no motivation to continually improve the technology/latency of the SIP. There's also a fear to change the SIP and lead to it going down. If a SIP goes down, by rule, a regulatory halt must be issued and all trading in the securities that particular SIP handles must stop. This is what occurred during the 3 hour outage several years back at Nasdaq. The trading system was fine, but since the SIP was down, everything had to be halted. Halts are detrimental to business since the respective marketcenters make money on trades. Halts = No Trades = Lost Revenue, so there is a lot of motivation to not halt by all the member exchanges. That ambivalence to a tech refresh has since changed since the scathing critique of Flash Boys and the Nasdaq SIP is in process of being refreshed to the same architecture as the trading systems, which should bring median latencies of the SIP down into the tens of microseconds and 99% latencies into the hundreds of microseconds. Current medians and 90th percentiles are at 450/810 microseconds, which is considered an eternity. http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-won-over-sip-committee-with-latency-reductions-and-tech-upgrades/d/d-id/1317316 http://www.wallstreetandtech.com/infrastructure/nasdaq-omx-w... http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Processor%20Stats.pdf http://www.utpplan.com/DOC/UTP%202015-Q4%20Stats%20with%20Pr... On the technical side, one of the challenges with synchronizing packet delivery between a private & public feed is that the absolute fastest NICs available today with kernel bypass can get a packet from wire to memory or memory to wire in about 2-3u. So every hop of a computer costs a minimum of 4-6u. Usually this is Order Port Ingress -> Matching Engine -> Order Port Egress. An FPGA NIC that does the IP Header processing and delivers data to memory can do wire to memory in about 1-2u. One-way PCI-E latency is around 800 nanos. So there's about a 500-1000 nano budget to process a packet in the FPGA, then another 800 nanos to deliver it over PCI-E to memory to be processed. The fastest cut-through switches have a port-to-port latency of around 250 nanos, so switch hops are pretty negligible in the overall latency of a system, but they do play a part. Anyway, making sure that everything is super duper optimized and running as it should be in a semi-continuously deployed environment makes for interesting and quite crazy/bull-shit level challenges.

uptown 11y ago

Ultimately, the market has been setup to almost mandate participation. If you work someplace that provides a matching 401k program, and you choose not to contribute, then you're passing-up on matched dollars if you don't contribute to a 401k. If you're a part of any endowment, pension plan, or other managed retirement plan there's a good chance that plan (and your money) is being invested in the market. So what are your options? Do you not contribute to a 401k or IRA, pay the full tax rate on all of your income today, and invest in real estate or some other investment?

deleted 11y ago

[deleted]

fitzwatermellow 11y ago

Congrats on being rewarded and recognized for the very important work you do in bringing transparency to the financial markets. The whole story of how this went down is just epic! My question is about the business side of Nanex. Are sales of your products such as NxCore enough to sustain the company? Do you also actively trade? And what opportunities do you see currently around data science and machine learning for entrepreneurs in the quantitative finance world?

irascible 11y ago

Hax! Lag switch. :/ bitches.

__d 11y ago

Is there any good reason that the SEC shouldn't ban direct feeds? If everyone were forced to use the SIP, what issues would remain here?