20 ms·

Online Algorithms in High-Frequency Trading

116 points by stevewepay 12y ago

expertonkappa 12y ago
Is there a mirror? Seems like too many connections at the moment.
- galaktor 12y ago
  google cache should do the trick if it's down http://webcache.googleusercontent.com/search?q=cache:WqysSpS-SBoJ:queue.acm.org/detail.cfm%3Fid%3D2534976+&cd=1&hl=en&ct=clnk&gl=ie http://webcache.googleusercontent.com/search?q=cache:WqysSpS...
amelius 12y ago
Could somebody explain what the use is of HFT to society? Because I don't get it. Links are also welcome.
- res0nat0r 12y ago
  Shortening the bid/ask spread and increasing liquidity among other things. http://en.wikipedia.org/wiki/High-frequency_trading http://en.wikipedia.org/wiki/High-frequency_trading
- reverend_gonzo 12y ago
  Before HFT, there was only a few market makers, which meant the bid-ask was high (if you wanted to buy or sell right now, you'd pay more), liquidity was low (it'd be harder for you to get in or out of a large position and in hard times it would take longer for the market to recover - See 80s crash vs flash crash). With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market. The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).
  - djrobstep 12y ago
    Which brings us to the next question: What is the social benefit of a slightly smaller bid-ask spread and a few milliseconds of faster liquidity? And are these benefits worth the massive amount of wealth extraction from the rest of us that HFTers are engaged in?
    - gd1 12y ago
      That's pretty simple to answer. The smaller the spread, the less going into the pocket of middlemen. Which is where we are now. Less going into the pockets of middlemen than ever before. >And are these benefits worth the massive amount of wealth extraction What "massive amount of wealth extraction"? Are you just making shit up?
      - jeffreyrogers 12y ago
        > What "massive amount of wealth extraction"? Are you just making shit up? I think he was probably just confused about how HFT works, since unless you're familiar with it it is very easy to assume it is screwing someone.
    - deleted 12y ago
      [deleted]
      - danbruc 12y ago
        The total peak profit of HFT companies is usually estimated a bit north of 5 billion Dollar in 2009. After that it fell to about 1 billion Dollar per year as far as I can tell. For me 5 billion Dollar or even only 1 billion Dollar per year seems quite a lot for not producing a single salable good and service. The 2009 profit was the equivalent of 175,000 average yearly US incomes, i.e. 175,000 people worked somewhere on factory floors to produce the profits for HFT companies.
        gd1 12y ago
        Can you quote me some stats on how much the equivalent middle men made in earlier decades? Both absolute and relative to the size of the market?
- cheepin 12y ago
  Unfortunately, there are a couple types of HFT. At it's core, it just means trading using a computer to execute more/faster trades. That seems justifiable in making the markets more efficient. Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins simply because their latency was low enough. An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.
  - kasey_junk 12y ago
    > Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins Luckily that is not allowed (nor technically possible) on any credible exchange. > An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal. You probably meant: https://news.ycombinator.com/item?id=8977198 https://news.ycombinator.com/item?id=8977198 Spoofing, which is what you are describing, is not useful for countering HFT, it is used to change the market price of an instrument by exaggerating the demand for it. This has obvious detrimental impact on any one, HFT or otherwise, who is trying to price the instrument.
- Paradigma11 12y ago
  Is it necessary for things to "have a use to society" to be allowed? Where is the commitee that decides on the allowed actions?
  - calibraxis 12y ago
    The committees are called "governments" and rule life in nation-states. The dominant ones have a capitalist morality and therefore make laws to create and enforce it.
    - Paradigma11 12y ago
      Depends on the nation: http://en.wikipedia.org/wiki/Everything_which_is_not_forbidden_is_allowed http://en.wikipedia.org/wiki/Everything_which_is_not_forbidd...
- gadders 12y ago
  This guy's blog is pretty good: https://scottlocklin.wordpress.com/2011/03/01/the-arguments-against-hft/ https://scottlocklin.wordpress.com/2011/03/01/the-arguments-... https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-shilling-for-the-buyside/ https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-...
- lmm 12y ago
  We see crazy competition on latency because these firms can no longer compete on price. HFT has taken the spreads down from $.25 to $.01 (putting a lot of market-makers out of business, saving a lot of money for anyone who is actually trying to buy or sell shares). But since the price isn't allowed to go below $.01 (sub-penny rule) the firms are now all in this wasteful zero-sum race to get that $.01.
- wodzu 12y ago
  While you will hear many negatives here I believe there are few positive outcomes. The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.
- Adlai 12y ago
  https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2012/hft_whats_broken.html https://www.chrisstucchio.com/blog/2012/hft_whats_broken.htm... The second link specifically talks about social utility.
- innguest 12y ago
  There is no benefit. HFT does not lower spreads in reality, only nominally. They put in orders but back away when a matching order comes in. The myth is that HFT improves liquidity. It may look like that. But those orders aren't real and can't be fulfilled. This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.
  - throwaway283719 12y ago
    The trouble with this argument is that, on basically every exchange in the world, you do not get to find out if someone has matched with your passive order until after the trade has happened. There is no possibility to "back away when a matching order comes in" because the first time you find out about the matching order is when you are informed about the trade that you just did! What can happen is that a market maker will post liquidity on multiple exchanges A, B and C, and will remove their orders from exchanges B and C if they are filled on exchange A (assuming they are fast enough to do it!) This is not shady or nefarious. In fact, it is the only way that a distributed market could possibly work. The market maker has some finite capacity to provide liquidity (e.g. they have position limits, they need to post margin, they have certain risk tolerances). They want to maximise the chances of their orders being traded on (i.e. providing liquidity), so they post liquidity on all available markets. When they match on one of those markets, their capacity to provide liquidity has diminished, until they manage to offset their position somewhere (either on the same market, or on a different one). Moreover, they have just received a new piece of information about the supply/demand imbalance for this particular stock. For both of these reasons, it is natural to assume that they would want to adjust the prices they are offering on other exchanges. The market cannot function any other way.
    - tptacek 12y ago
      I like asking programmers to think about the CAP theorem when they consider this problem.
- aet 12y ago
  You have the wrong question. The question is: What is the optimal market structure? Trying to strike a balance between the benefits of competition and the troubles of market fragmentation. Fragmented markets = HFT + competition. Unified markets = monopoly power + liquidity. You choose...
  - tptacek 12y ago
    Wait, why would monopolies on trading venues and market-making provide more liquidity? Shouldn't liquidity be on the other side of the comparison?
    - aet 12y ago
      A single exchange brings all traders together reducing search and monitoring costs and thereby reducing trading costs. The single exchange as a monopoly can extract rents from members and has little incentive to improve stuff, e.g. infrastructure. So the tension is between the monopoly power and the liquidity externality arrising from bringing traders together to the same place. The market makers do not necessarily have a monopoly, they can compete against each other, but face monopoly level fees, potentially. (This is my amateur economic analysis, but I think basically this is what economists think.) A fragmented market increases competition on fees and infrastructure, but traders face search, monitoring costs and the cost of dealing with HFTs.
- vasilipupkin 12y ago
  I find this question soooo frustrating. I mean, we live in a free ( relatively speaking ) capitalist society. Why does there have to be use to society for anything, unless the issue in question has negative externalities on others ?
  - pgeorgi 12y ago
    HFT may have negative externalities on others. The money they make has to come from somewhere. For example: Buyers pay more or seller get less than they'd get without HFTs acting on the market. The (quite common) liquidity argument isn't a very good one IMHO, given that HFT have no interest in holding positions for any long time, so they won't buy what they're not reasonably sure they can sell quickly (ie. where there already is liquidity). The other argument here is that before HFT, traders wanted even larger fees. HFT can't be the cheapest possible option out there, if they can afford to lay their own fiber between continents to operate. So what is it that makes HFT desirable enough that we (as a society) shouldn't find means to make trading happen without them?
    - tptacek 12y ago
      The money they make comes from a premium the market's "officers" used to extract from the market's "customers". Market officers used to scalp dimes and quarters off the price of every share transacted by deliberately quoting wide spreads. Automated traders shrunk those spreads, so much so that they're pushing the boundary of a single penny. The great thing about this is, automated traders don't capture the value they're creating by bidding the spread down: the buy-side does! Automated traders are getting paid to liberate money that used to go to fees for sell-side middle-men. The sell-side is, as you can read subtextually from _Flash Boys_, pretty furious about this.
      - pgeorgi 12y ago
        So what specific part of HFT operation makes them able to do that? Is it their own (really public) instance of Core War (https://en.wikipedia.org/wiki/Core_War https://en.wikipedia.org/wiki/Core_War)? Is it the milliseconds advantage gained through custom transatlantic fiber? If either of these aren't core to what enables them to dump prices, we could get them down even more by getting rid of these toys.
abeppu 12y ago
The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that, and I'd expect that their relevance to HFT would be immediately obvious. I've never worked in HFT, but I've probably written the online variance algorithm at 3 of 4 companies I've worked. If you care about metrics such as "how quickly are each of my customers sending me requests?", then this stuff is relevant to you. But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?
- kasey_junk 12y ago
  a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else. b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about. c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.
  - vasilipupkin 12y ago
    regarding a), not all technical professions require deep knowledge of massive data processing and time series modeling, so this statement is misleading regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.
    - dllthomas 12y ago
      I think people are reading your comment as "they need to know these things, therefore they are elite" - but I think you rather meant "it's not about being elite, just that they have to know XYZ so they are more likely to know W".
CyberDildonics 12y ago
Anyone who thinks HFT is something needed should go read Flash Boys. HFT is basically front running with some variations. There is a small amount that is necessary arbitration that will have to be done at some speed, but by and large it is not performing any sort of a service, it is taking advantage of the people trying to buy and trade stock by being a manipulative middle man.
- tptacek 12y ago
  There are three sentences in this comment. The first two are wrong, and the third is hard to understand, but if I understand it correctly, it too is false. 1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad. 2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening. 3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.
  - jaybaxter 12y ago
    Could you please link to the technical reviews you mentioned? I would be very interested to read them because as an outsider, Flash Boys seemed overly simplistic, but I haven't been able to figure out what the real story is yet.
    - tptacek 12y ago
      Here's a good free starting point: http://www.amazon.com/review/R3PJO6KJGRMWUE/ref%3Dcm_cr_pr_viewpnt#R3PJO6KJGRMWUE http://www.amazon.com/review/R3PJO6KJGRMWUE/ref%3Dcm_cr_pr_v... Again: a single positive technical review would be interesting. I'm not being hyperbolic: I haven't read a single one. The sense I get is that everyone who's ever formatted a FIX message thinks _Flash Boys_ is terrible.
tptacek 12y ago
The sad thing about any thread including the term "High-Frequency Trading" is that we won't be able to talk about the technology, and instead will need to relitigate automated trading. I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_. I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).
- CyberDildonics 12y ago
  So what is your opinion on the book Flash Boys?
  - throwaway283719 12y ago
    tptacek already responded to your previous comment about Flash Boys below [0] [0] https://news.ycombinator.com/item?id=9045150 https://news.ycombinator.com/item?id=9045150
- kylebrown 12y ago
  The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response[1][2] is a must-read. The tldr; is that a continuous mechanism (sequential trade processing) only looks efficient in time-space, but is actually inefficient in volume-space (most of the volume trades at stale prices). The solution is discrete batch auctions (batch trade processing). 1. slides: http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions-Slides.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre... 2. paper: http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
  - tptacek 12y ago
    This is really interesting, but doesn't it presume that there's only one trading venue for each instrument, and that instruments aren't correlated? You can make time discrete for a single auction, but the time between auctions still has to be continuous unless everything's traded in a single exchange, which can't in reality happen even if we want it to (and we probably don't). As I understand it, most of the most dramatic HFT stuff is already targeting multi-venue trades or correlations.
crazypyro 12y ago
Needs a date stamp (2013).

116 points by stevewepay 12y ago

expertonkappa 12y ago

Is there a mirror? Seems like too many connections at the moment.

galaktor 12y ago
google cache should do the trick if it's down http://webcache.googleusercontent.com/search?q=cache:WqysSpS-SBoJ:queue.acm.org/detail.cfm%3Fid%3D2534976+&cd=1&hl=en&ct=clnk&gl=ie http://webcache.googleusercontent.com/search?q=cache:WqysSpS...

amelius 12y ago

Could somebody explain what the use is of HFT to society? Because I don't get it. Links are also welcome.

res0nat0r 12y ago
Shortening the bid/ask spread and increasing liquidity among other things. http://en.wikipedia.org/wiki/High-frequency_trading http://en.wikipedia.org/wiki/High-frequency_trading
reverend_gonzo 12y ago
Before HFT, there was only a few market makers, which meant the bid-ask was high (if you wanted to buy or sell right now, you'd pay more), liquidity was low (it'd be harder for you to get in or out of a large position and in hard times it would take longer for the market to recover - See 80s crash vs flash crash). With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market. The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).
- djrobstep 12y ago
  Which brings us to the next question: What is the social benefit of a slightly smaller bid-ask spread and a few milliseconds of faster liquidity? And are these benefits worth the massive amount of wealth extraction from the rest of us that HFTers are engaged in?
  - gd1 12y ago
    That's pretty simple to answer. The smaller the spread, the less going into the pocket of middlemen. Which is where we are now. Less going into the pockets of middlemen than ever before. >And are these benefits worth the massive amount of wealth extraction What "massive amount of wealth extraction"? Are you just making shit up?
    - jeffreyrogers 12y ago
      > What "massive amount of wealth extraction"? Are you just making shit up? I think he was probably just confused about how HFT works, since unless you're familiar with it it is very easy to assume it is screwing someone.
  - deleted 12y ago
    [deleted]
    - danbruc 12y ago
      The total peak profit of HFT companies is usually estimated a bit north of 5 billion Dollar in 2009. After that it fell to about 1 billion Dollar per year as far as I can tell. For me 5 billion Dollar or even only 1 billion Dollar per year seems quite a lot for not producing a single salable good and service. The 2009 profit was the equivalent of 175,000 average yearly US incomes, i.e. 175,000 people worked somewhere on factory floors to produce the profits for HFT companies.
      - gd1 12y ago
        Can you quote me some stats on how much the equivalent middle men made in earlier decades? Both absolute and relative to the size of the market?
cheepin 12y ago
Unfortunately, there are a couple types of HFT. At it's core, it just means trading using a computer to execute more/faster trades. That seems justifiable in making the markets more efficient. Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins simply because their latency was low enough. An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.
- kasey_junk 12y ago
  > Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins Luckily that is not allowed (nor technically possible) on any credible exchange. > An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal. You probably meant: https://news.ycombinator.com/item?id=8977198 https://news.ycombinator.com/item?id=8977198 Spoofing, which is what you are describing, is not useful for countering HFT, it is used to change the market price of an instrument by exaggerating the demand for it. This has obvious detrimental impact on any one, HFT or otherwise, who is trying to price the instrument.
Paradigma11 12y ago
Is it necessary for things to "have a use to society" to be allowed? Where is the commitee that decides on the allowed actions?
- calibraxis 12y ago
  The committees are called "governments" and rule life in nation-states. The dominant ones have a capitalist morality and therefore make laws to create and enforce it.
  - Paradigma11 12y ago
    Depends on the nation: http://en.wikipedia.org/wiki/Everything_which_is_not_forbidden_is_allowed http://en.wikipedia.org/wiki/Everything_which_is_not_forbidd...
gadders 12y ago
This guy's blog is pretty good: https://scottlocklin.wordpress.com/2011/03/01/the-arguments-against-hft/ https://scottlocklin.wordpress.com/2011/03/01/the-arguments-... https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-shilling-for-the-buyside/ https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-...
lmm 12y ago
We see crazy competition on latency because these firms can no longer compete on price. HFT has taken the spreads down from $.25 to $.01 (putting a lot of market-makers out of business, saving a lot of money for anyone who is actually trying to buy or sell shares). But since the price isn't allowed to go below $.01 (sub-penny rule) the firms are now all in this wasteful zero-sum race to get that $.01.
wodzu 12y ago
While you will hear many negatives here I believe there are few positive outcomes. The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.
Adlai 12y ago
https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2012/hft_whats_broken.html https://www.chrisstucchio.com/blog/2012/hft_whats_broken.htm... The second link specifically talks about social utility.
innguest 12y ago
There is no benefit. HFT does not lower spreads in reality, only nominally. They put in orders but back away when a matching order comes in. The myth is that HFT improves liquidity. It may look like that. But those orders aren't real and can't be fulfilled. This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.
- throwaway283719 12y ago
  The trouble with this argument is that, on basically every exchange in the world, you do not get to find out if someone has matched with your passive order until after the trade has happened. There is no possibility to "back away when a matching order comes in" because the first time you find out about the matching order is when you are informed about the trade that you just did! What can happen is that a market maker will post liquidity on multiple exchanges A, B and C, and will remove their orders from exchanges B and C if they are filled on exchange A (assuming they are fast enough to do it!) This is not shady or nefarious. In fact, it is the only way that a distributed market could possibly work. The market maker has some finite capacity to provide liquidity (e.g. they have position limits, they need to post margin, they have certain risk tolerances). They want to maximise the chances of their orders being traded on (i.e. providing liquidity), so they post liquidity on all available markets. When they match on one of those markets, their capacity to provide liquidity has diminished, until they manage to offset their position somewhere (either on the same market, or on a different one). Moreover, they have just received a new piece of information about the supply/demand imbalance for this particular stock. For both of these reasons, it is natural to assume that they would want to adjust the prices they are offering on other exchanges. The market cannot function any other way.
  - tptacek 12y ago
    I like asking programmers to think about the CAP theorem when they consider this problem.
aet 12y ago
You have the wrong question. The question is: What is the optimal market structure? Trying to strike a balance between the benefits of competition and the troubles of market fragmentation. Fragmented markets = HFT + competition. Unified markets = monopoly power + liquidity. You choose...
- tptacek 12y ago
  Wait, why would monopolies on trading venues and market-making provide more liquidity? Shouldn't liquidity be on the other side of the comparison?
  - aet 12y ago
    A single exchange brings all traders together reducing search and monitoring costs and thereby reducing trading costs. The single exchange as a monopoly can extract rents from members and has little incentive to improve stuff, e.g. infrastructure. So the tension is between the monopoly power and the liquidity externality arrising from bringing traders together to the same place. The market makers do not necessarily have a monopoly, they can compete against each other, but face monopoly level fees, potentially. (This is my amateur economic analysis, but I think basically this is what economists think.) A fragmented market increases competition on fees and infrastructure, but traders face search, monitoring costs and the cost of dealing with HFTs.
vasilipupkin 12y ago
I find this question soooo frustrating. I mean, we live in a free ( relatively speaking ) capitalist society. Why does there have to be use to society for anything, unless the issue in question has negative externalities on others ?
- pgeorgi 12y ago
  HFT may have negative externalities on others. The money they make has to come from somewhere. For example: Buyers pay more or seller get less than they'd get without HFTs acting on the market. The (quite common) liquidity argument isn't a very good one IMHO, given that HFT have no interest in holding positions for any long time, so they won't buy what they're not reasonably sure they can sell quickly (ie. where there already is liquidity). The other argument here is that before HFT, traders wanted even larger fees. HFT can't be the cheapest possible option out there, if they can afford to lay their own fiber between continents to operate. So what is it that makes HFT desirable enough that we (as a society) shouldn't find means to make trading happen without them?
  - tptacek 12y ago
    The money they make comes from a premium the market's "officers" used to extract from the market's "customers". Market officers used to scalp dimes and quarters off the price of every share transacted by deliberately quoting wide spreads. Automated traders shrunk those spreads, so much so that they're pushing the boundary of a single penny. The great thing about this is, automated traders don't capture the value they're creating by bidding the spread down: the buy-side does! Automated traders are getting paid to liberate money that used to go to fees for sell-side middle-men. The sell-side is, as you can read subtextually from _Flash Boys_, pretty furious about this.
    - pgeorgi 12y ago
      So what specific part of HFT operation makes them able to do that? Is it their own (really public) instance of Core War (https://en.wikipedia.org/wiki/Core_War https://en.wikipedia.org/wiki/Core_War)? Is it the milliseconds advantage gained through custom transatlantic fiber? If either of these aren't core to what enables them to dump prices, we could get them down even more by getting rid of these toys.

abeppu 12y ago

The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that, and I'd expect that their relevance to HFT would be immediately obvious. I've never worked in HFT, but I've probably written the online variance algorithm at 3 of 4 companies I've worked. If you care about metrics such as "how quickly are each of my customers sending me requests?", then this stuff is relevant to you. But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?

kasey_junk 12y ago
a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else. b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about. c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.
- vasilipupkin 12y ago
  regarding a), not all technical professions require deep knowledge of massive data processing and time series modeling, so this statement is misleading regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.
  - dllthomas 12y ago
    I think people are reading your comment as "they need to know these things, therefore they are elite" - but I think you rather meant "it's not about being elite, just that they have to know XYZ so they are more likely to know W".

CyberDildonics 12y ago

Anyone who thinks HFT is something needed should go read Flash Boys. HFT is basically front running with some variations. There is a small amount that is necessary arbitration that will have to be done at some speed, but by and large it is not performing any sort of a service, it is taking advantage of the people trying to buy and trade stock by being a manipulative middle man.

tptacek 12y ago
There are three sentences in this comment. The first two are wrong, and the third is hard to understand, but if I understand it correctly, it too is false. 1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad. 2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening. 3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.
- jaybaxter 12y ago
  Could you please link to the technical reviews you mentioned? I would be very interested to read them because as an outsider, Flash Boys seemed overly simplistic, but I haven't been able to figure out what the real story is yet.
  - tptacek 12y ago
    Here's a good free starting point: http://www.amazon.com/review/R3PJO6KJGRMWUE/ref%3Dcm_cr_pr_viewpnt#R3PJO6KJGRMWUE http://www.amazon.com/review/R3PJO6KJGRMWUE/ref%3Dcm_cr_pr_v... Again: a single positive technical review would be interesting. I'm not being hyperbolic: I haven't read a single one. The sense I get is that everyone who's ever formatted a FIX message thinks _Flash Boys_ is terrible.

tptacek 12y ago

The sad thing about any thread including the term "High-Frequency Trading" is that we won't be able to talk about the technology, and instead will need to relitigate automated trading. I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_. I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).

CyberDildonics 12y ago
So what is your opinion on the book Flash Boys?
- throwaway283719 12y ago
  tptacek already responded to your previous comment about Flash Boys below [0] [0] https://news.ycombinator.com/item?id=9045150 https://news.ycombinator.com/item?id=9045150
kylebrown 12y ago
The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response[1][2] is a must-read. The tldr; is that a continuous mechanism (sequential trade processing) only looks efficient in time-space, but is actually inefficient in volume-space (most of the volume trades at stale prices). The solution is discrete batch auctions (batch trade processing). 1. slides: http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions-Slides.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre... 2. paper: http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
- tptacek 12y ago
  This is really interesting, but doesn't it presume that there's only one trading venue for each instrument, and that instruments aren't correlated? You can make time discrete for a single auction, but the time between auctions still has to be continuous unless everything's traded in a single exchange, which can't in reality happen even if we want it to (and we probably don't). As I understand it, most of the most dramatic HFT stuff is already targeting multi-venue trades or correlations.

crazypyro 12y ago

Needs a date stamp (2013).