5 ms·

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. There is no chaos monkey for the stock marke

by pmb 14y ago

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. There is no chaos monkey for the stock market, and transaction rollback is available only in the most extreme circumstances. I have yet to see a convincing rationale for how high frequency trading adds value to the system - it certainly doesn't seem to add pricing stability. Because this is hacker news, I suspect a few people reading this either work in the industry and/or have strong opinions about it, and I would love to hear why I am wrong. Because I do like it when clever computer scientists make money, and that seems to be pretty much the only social benefit of HFT, at the cost of flash crashes and things like the example above.

fr0sty 14y ago
The 'Does HFT add value?' question gets flogged to death every time one of these articles gets posted. http://news.ycombinator.com/item?id=3894302 http://news.ycombinator.com/item?id=3894302 http://news.ycombinator.com/item?id=3852341 http://news.ycombinator.com/item?id=3852341 http://news.ycombinator.com/item?id=2828538 http://news.ycombinator.com/item?id=2828538
patio11 14y ago
These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. This implies that humans are actually better than machines at making data-driven decisions at high rates. They're not. They're astoundingly not once you compute the cost of humans versus the cost of machines on a per-decision basis. Liquidity in the market used to be provided by large groups of sweaty, overpaid alpha males yelling at each other. We replaced them with dueling robots. The robots can provide liquidity for a fraction of the price. The alpha males really, really hate competing with robots, because the alpha males invariably lose, so they complain that robots are stealing the money that the alpha males used to extract from their customers by right of being the one with a license to be shouting and sweaty at a particular physical location. Occasionally a robot blows up. Not a problem -- robots are easy to replace. Besides, humans blow up all the time. We only ignore their ridiculous strictly-inferior-in-every-way-at-this-task nature because they look more like us than the robots do, and because these particular humans being displaced used to be rich, whereas e.g. telephone operators tasked with manually doing call routing (also clearly inferior to highly reliable distributed algorithms) were poor.
- JonnieCache 14y ago
  That's the first time I've heard an explanation of the benefits of HFT that actually adds up. Maybe that's because it's appealing to my prejudices, but thanks.
  - patio11 14y ago
    I think I stole "dueling robots" from tptacek or yummyfajitas, but sweaty alpha males are totally mine. Edit to add: The dueling robot at HNsearch says a) it was tptacek and b) I should trade with it because it remembers HN comments better than I could ever hope to.
- theorique 14y ago
  Well put. A lot of this "but ... but ... but ... we NEED a human in the loop" handwringing that we're seeing on the news channels is obsolete people who used to be ridiculously overpaid complaining that they have been replaced by fast computers and efficient code. (Disclaimer: Non-sweaty alpha male HFT algorithm writer here ... ha.)
- danso 14y ago
  Another way to look at this: it's a well-known phenomenon that expert radiologists will misdiagnose an X-ray image, even contradicting a past judgment they made on the same X-ray. It's feasible that a properly machine-learned computer could make better judgments on a long-running average basis. But when a computer screws up an edge case, versus the many, many times a doctor will screw up a diagnosis, which instance will get more attention. People are scared of black box machines and will take particular note of the times an algorithm has screwed them over, regardless of the times a human has screwed them up. Also, when the possibility of examination exists, computerized decision making is far, far easier to audit (i.e. determine blame)
  - jlgreco 14y ago
    Exactly. It is the same as the problem with robotic cars. Cars with human drivers kill thousands, but nobody blinks. The day a robotic car kills a single person, all the news networks will have a field day.
Permit 14y ago
People make massive mistakes all the time. http://dealbook.nytimes.com/2012/06/28/jpmorgan-trading-loss-may-reach-9-billion/ http://dealbook.nytimes.com/2012/06/28/jpmorgan-trading-loss...
- harshreality 14y ago
  What would happen if a computer trading operation used the JPMorgan London unit's flawed risk model and a computer executed trades based on it as many times as the algo saw "opportunities"... until the humans who created the model noticed a problem with it? Tens of billions? Hundreds?
veyron 14y ago
"These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop" You write a risk check gateway once. It rarely if ever needs to be touched. If you do this right, you roll out changes on one machine with strict risk controls (1000 share position, for example) to prevent such a blowout. Then you transition everything else. Ironically, this should be exactly what other tech firms should do: roll out changes to a small area, check basic problems, and then move on to a larger rollout.
- fr0sty 14y ago
  > You write a risk check gateway once. It rarely if ever needs to be touched. Such checks are only as good as the imagination of the author. > Then you transition everything else. Which is all well and good until you have an issue where two independent systems creat a feedback loop. Such problems are only evident 'at scale'. I'm not blaming or trying to exonerate anyone. I love armchair quarterbacking as much as the next guy but a small trading shop runs a little differently than america's largest electronic market maker.
  - veyron 14y ago
    I went through this process myself in setting up a self-clearing BD, so I am fully aware of all of the relevant rules and regulations. >Such checks are only as good as the imagination of the author. SEC and FINRA regulations require very specific risk checks. One of those involves looking at potential positions if all of your (buy or sell) orders get filled. You have to sum your max potential gross position across all symbols and decide if you will trip a limit. If so you have to halt. This is not imagination; this is codified. If you send an order for 100 shares, you HAVE TO ASSUME they were filled when you send the next order. That's the rule. This was not left to imagination. "Which is all well and good until you have an issue where two independent systems creat a feedback loop. Such problems are only evident 'at scale'." There should be no feedback where the trading engine affects risk check. In fact, the new rules require separate code bases and separate legal entities.
    - 14y ago
paperwork 14y ago
The usual criticism of high frequency traders doesn't really apply to Knight. Knight is required by law to execute orders that are sent to them, instead of 'investing' in stocks for the long term.
jellicle 14y ago
Nobody except for HFT people has ever argued that they add value to the system. Everyone else believes that HFT exists because it's profitable, and that it subtracts value from the system. You can choose who you want to believe.
- harryh 14y ago
  Market makers used to be humans. Now, due to advances in technology, computers can do that job. Computers are (generally) cheaper than humans so they can do the job for less money. You can see this in the fact that spreads have decreased as market making has been turned over to computers. That's the value they are adding. Market making has always been work so it's always cost money. But now it's cheaper. There's always been money subtracted from the system, but now it's less than it used to be.
tbenst 14y ago
To return the discussion back to your main question, there is strong evidence that high frequency trading, or algorithmic trading (AT), increases liquidity: http://www.afajof.org/afa/forthcoming/6130p.pdf http://www.afajof.org/afa/forthcoming/6130p.pdf (See Figure 2). Increasing liquidity and reducing the bid/ask spread adds value to the system by increasing efficiency and information dissemination. For a period after the introduction of autoquote in 2003, providers of liquidity captured most of the surplus and enjoyed larger margins on trades (See Figure 3). However, this advantage quickly dissipated as more parties implemented AT. The first-mover advantage doesn't apply in the world of equity markets; competitors quickly duplicated AT strategies and competition swiftly lowered spreads in the second half of 2003. Today, spreads on equities are much lower than pre-2003 largely thanks to AT. As for pricing stability, let me disregard AT glitches for the moment. Algorithms can tirelessly monitor market information, whether media reporting, filings, event rumors (eg M&A), order trends, etc. Humans are are relatively constricted to a few information sources when executing trades in comparison to AT. In addition, AT reacts faster to new information sources and can adjust bid/ask near-instantly. Therefore, price volatility increases as a result of increased information efficiency. Glitches and fast-crashes are a negative counter-example to the information efficiency argument above. I leave it to the reader to decide if liquidity benefits justify the occasional flash-crash. However, recognize that this phenomena is not exclusive to AT: many human traders have caused similar crashes of their own -- I'm looking at you London Whale.

by pmb 14y ago

fr0sty 14y ago

The 'Does HFT add value?' question gets flogged to death every time one of these articles gets posted. http://news.ycombinator.com/item?id=3894302 http://news.ycombinator.com/item?id=3894302 http://news.ycombinator.com/item?id=3852341 http://news.ycombinator.com/item?id=3852341 http://news.ycombinator.com/item?id=2828538 http://news.ycombinator.com/item?id=2828538

patio11 14y ago

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. This implies that humans are actually better than machines at making data-driven decisions at high rates. They're not. They're astoundingly not once you compute the cost of humans versus the cost of machines on a per-decision basis. Liquidity in the market used to be provided by large groups of sweaty, overpaid alpha males yelling at each other. We replaced them with dueling robots. The robots can provide liquidity for a fraction of the price. The alpha males really, really hate competing with robots, because the alpha males invariably lose, so they complain that robots are stealing the money that the alpha males used to extract from their customers by right of being the one with a license to be shouting and sweaty at a particular physical location. Occasionally a robot blows up. Not a problem -- robots are easy to replace. Besides, humans blow up all the time. We only ignore their ridiculous strictly-inferior-in-every-way-at-this-task nature because they look more like us than the robots do, and because these particular humans being displaced used to be rich, whereas e.g. telephone operators tasked with manually doing call routing (also clearly inferior to highly reliable distributed algorithms) were poor.

JonnieCache 14y ago
That's the first time I've heard an explanation of the benefits of HFT that actually adds up. Maybe that's because it's appealing to my prejudices, but thanks.
- patio11 14y ago
  I think I stole "dueling robots" from tptacek or yummyfajitas, but sweaty alpha males are totally mine. Edit to add: The dueling robot at HNsearch says a) it was tptacek and b) I should trade with it because it remembers HN comments better than I could ever hope to.
theorique 14y ago
Well put. A lot of this "but ... but ... but ... we NEED a human in the loop" handwringing that we're seeing on the news channels is obsolete people who used to be ridiculously overpaid complaining that they have been replaced by fast computers and efficient code. (Disclaimer: Non-sweaty alpha male HFT algorithm writer here ... ha.)
danso 14y ago
Another way to look at this: it's a well-known phenomenon that expert radiologists will misdiagnose an X-ray image, even contradicting a past judgment they made on the same X-ray. It's feasible that a properly machine-learned computer could make better judgments on a long-running average basis. But when a computer screws up an edge case, versus the many, many times a doctor will screw up a diagnosis, which instance will get more attention. People are scared of black box machines and will take particular note of the times an algorithm has screwed them over, regardless of the times a human has screwed them up. Also, when the possibility of examination exists, computerized decision making is far, far easier to audit (i.e. determine blame)
- jlgreco 14y ago
  Exactly. It is the same as the problem with robotic cars. Cars with human drivers kill thousands, but nobody blinks. The day a robotic car kills a single person, all the news networks will have a field day.

Permit 14y ago

People make massive mistakes all the time. http://dealbook.nytimes.com/2012/06/28/jpmorgan-trading-loss-may-reach-9-billion/ http://dealbook.nytimes.com/2012/06/28/jpmorgan-trading-loss...

harshreality 14y ago
What would happen if a computer trading operation used the JPMorgan London unit's flawed risk model and a computer executed trades based on it as many times as the algo saw "opportunities"... until the humans who created the model noticed a problem with it? Tens of billions? Hundreds?

veyron 14y ago

"These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop" You write a risk check gateway once. It rarely if ever needs to be touched. If you do this right, you roll out changes on one machine with strict risk controls (1000 share position, for example) to prevent such a blowout. Then you transition everything else. Ironically, this should be exactly what other tech firms should do: roll out changes to a small area, check basic problems, and then move on to a larger rollout.

fr0sty 14y ago
> You write a risk check gateway once. It rarely if ever needs to be touched. Such checks are only as good as the imagination of the author. > Then you transition everything else. Which is all well and good until you have an issue where two independent systems creat a feedback loop. Such problems are only evident 'at scale'. I'm not blaming or trying to exonerate anyone. I love armchair quarterbacking as much as the next guy but a small trading shop runs a little differently than america's largest electronic market maker.
- veyron 14y ago
  I went through this process myself in setting up a self-clearing BD, so I am fully aware of all of the relevant rules and regulations. >Such checks are only as good as the imagination of the author. SEC and FINRA regulations require very specific risk checks. One of those involves looking at potential positions if all of your (buy or sell) orders get filled. You have to sum your max potential gross position across all symbols and decide if you will trip a limit. If so you have to halt. This is not imagination; this is codified. If you send an order for 100 shares, you HAVE TO ASSUME they were filled when you send the next order. That's the rule. This was not left to imagination. "Which is all well and good until you have an issue where two independent systems creat a feedback loop. Such problems are only evident 'at scale'." There should be no feedback where the trading engine affects risk check. In fact, the new rules require separate code bases and separate legal entities.
  - 14y ago

paperwork 14y ago

The usual criticism of high frequency traders doesn't really apply to Knight. Knight is required by law to execute orders that are sent to them, instead of 'investing' in stocks for the long term.

jellicle 14y ago

Nobody except for HFT people has ever argued that they add value to the system. Everyone else believes that HFT exists because it's profitable, and that it subtracts value from the system. You can choose who you want to believe.

harryh 14y ago
Market makers used to be humans. Now, due to advances in technology, computers can do that job. Computers are (generally) cheaper than humans so they can do the job for less money. You can see this in the fact that spreads have decreased as market making has been turned over to computers. That's the value they are adding. Market making has always been work so it's always cost money. But now it's cheaper. There's always been money subtracted from the system, but now it's less than it used to be.

tbenst 14y ago

To return the discussion back to your main question, there is strong evidence that high frequency trading, or algorithmic trading (AT), increases liquidity: http://www.afajof.org/afa/forthcoming/6130p.pdf http://www.afajof.org/afa/forthcoming/6130p.pdf (See Figure 2). Increasing liquidity and reducing the bid/ask spread adds value to the system by increasing efficiency and information dissemination. For a period after the introduction of autoquote in 2003, providers of liquidity captured most of the surplus and enjoyed larger margins on trades (See Figure 3). However, this advantage quickly dissipated as more parties implemented AT. The first-mover advantage doesn't apply in the world of equity markets; competitors quickly duplicated AT strategies and competition swiftly lowered spreads in the second half of 2003. Today, spreads on equities are much lower than pre-2003 largely thanks to AT. As for pricing stability, let me disregard AT glitches for the moment. Algorithms can tirelessly monitor market information, whether media reporting, filings, event rumors (eg M&A), order trends, etc. Humans are are relatively constricted to a few information sources when executing trades in comparison to AT. In addition, AT reacts faster to new information sources and can adjust bid/ask near-instantly. Therefore, price volatility increases as a result of increased information efficiency. Glitches and fast-crashes are a negative counter-example to the information efficiency argument above. I leave it to the reader to decide if liquidity benefits justify the occasional flash-crash. However, recognize that this phenomena is not exclusive to AT: many human traders have caused similar crashes of their own -- I'm looking at you London Whale.