6 ms·

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. This implies that humans are actually bette

by patio11 14y ago

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. This implies that humans are actually better than machines at making data-driven decisions at high rates. They're not. They're astoundingly not once you compute the cost of humans versus the cost of machines on a per-decision basis. Liquidity in the market used to be provided by large groups of sweaty, overpaid alpha males yelling at each other. We replaced them with dueling robots. The robots can provide liquidity for a fraction of the price. The alpha males really, really hate competing with robots, because the alpha males invariably lose, so they complain that robots are stealing the money that the alpha males used to extract from their customers by right of being the one with a license to be shouting and sweaty at a particular physical location. Occasionally a robot blows up. Not a problem -- robots are easy to replace. Besides, humans blow up all the time. We only ignore their ridiculous strictly-inferior-in-every-way-at-this-task nature because they look more like us than the robots do, and because these particular humans being displaced used to be rich, whereas e.g. telephone operators tasked with manually doing call routing (also clearly inferior to highly reliable distributed algorithms) were poor.

JonnieCache 14y ago
That's the first time I've heard an explanation of the benefits of HFT that actually adds up. Maybe that's because it's appealing to my prejudices, but thanks.
- patio11 14y ago
  I think I stole "dueling robots" from tptacek or yummyfajitas, but sweaty alpha males are totally mine. Edit to add: The dueling robot at HNsearch says a) it was tptacek and b) I should trade with it because it remembers HN comments better than I could ever hope to.
theorique 14y ago
Well put. A lot of this "but ... but ... but ... we NEED a human in the loop" handwringing that we're seeing on the news channels is obsolete people who used to be ridiculously overpaid complaining that they have been replaced by fast computers and efficient code. (Disclaimer: Non-sweaty alpha male HFT algorithm writer here ... ha.)
danso 14y ago
Another way to look at this: it's a well-known phenomenon that expert radiologists will misdiagnose an X-ray image, even contradicting a past judgment they made on the same X-ray. It's feasible that a properly machine-learned computer could make better judgments on a long-running average basis. But when a computer screws up an edge case, versus the many, many times a doctor will screw up a diagnosis, which instance will get more attention. People are scared of black box machines and will take particular note of the times an algorithm has screwed them over, regardless of the times a human has screwed them up. Also, when the possibility of examination exists, computerized decision making is far, far easier to audit (i.e. determine blame)
- jlgreco 14y ago
  Exactly. It is the same as the problem with robotic cars. Cars with human drivers kill thousands, but nobody blinks. The day a robotic car kills a single person, all the news networks will have a field day.
  - mc32 14y ago
    I think you're right about this. Maybe it has to do with appearance of control/self determination. It was a mistake (an accident), but it was my mistake/accident (as opposed to something one didn't have control over).
  - laserDinosaur 14y ago
    I once posited the question to a group of friends, would you rather drive a car with a 60% chance of crashing fatally in your lifetime, or use a robotic car that as a 2% failure rate. Everyone chose the 60%, even though statistically it makes absolutely no sense.
brittohalloran 14y ago
I agree with your premise -- the issue is that when humans blow up it doesn't cause a cascading runaway failure like robots are susceptible to.
- KevinEldon 14y ago
  Humans are susceptible to cascading runaway "ethical" failures. I'm thinking the $2 billion UBS loss. Humans also don't scale very well so risk tends to be distributed across several of us. Perhaps robots should be setup in the same way; that there are several of them with different algorithms tuned for different levels of risk... or perhaps it really doesn't matter all that much because this is one firm and not the entire system so the risk is already distributed correctly.
msellout 14y ago
Not (entirely) true. Robots are not strictly superior to humans in this case because the emergent properties of the system change due to the speed of robot action. As more robots crowd into the very fast strategy space, there tends to be more collision of strategies and thus more extreme price movements as many robots all try to buy/sell the same thing at the same time. If they all pile onto one side of the market, liquidity disappears and price movements become dramatic. In the very fast strategy space, there is less capability for complex strategies requiring long memories. If you want to save some nanoseconds to win at this winner-take-all market, you need to reduce your memory and processing time. http://arxiv.org/PS_cache/arxiv/pdf/1006/1006.5490v1.pdf http://arxiv.org/PS_cache/arxiv/pdf/1006/1006.5490v1.pdf This isn't saying that humans would never display herd behavior, but that they do so less often.
- theorique 14y ago
  Humans absolutely do display herd behavior. This has been documented in financial markets and other venues. However, they do it (by definition) on a time scale that is perceptible to other humans.
alwaysinshade 14y ago
There are cases where having a human in the loop is appropriate, particularly manufacturing. A human, thanks to a combination of slowness and observation skill, will pick up on a fault after the first error while a machine, thanks to a combination of speed and lack of observation skill, will produce fault after fault until things come to a crashing halt or someone notices before that happens. This $440m example shows that a human should be thrown into the mix. Let the human/machine strengths cancel out each other's weaknesses.

by patio11 14y ago

JonnieCache 14y ago

That's the first time I've heard an explanation of the benefits of HFT that actually adds up. Maybe that's because it's appealing to my prejudices, but thanks.

patio11 14y ago
I think I stole "dueling robots" from tptacek or yummyfajitas, but sweaty alpha males are totally mine. Edit to add: The dueling robot at HNsearch says a) it was tptacek and b) I should trade with it because it remembers HN comments better than I could ever hope to.

theorique 14y ago

Well put. A lot of this "but ... but ... but ... we NEED a human in the loop" handwringing that we're seeing on the news channels is obsolete people who used to be ridiculously overpaid complaining that they have been replaced by fast computers and efficient code. (Disclaimer: Non-sweaty alpha male HFT algorithm writer here ... ha.)

danso 14y ago

Another way to look at this: it's a well-known phenomenon that expert radiologists will misdiagnose an X-ray image, even contradicting a past judgment they made on the same X-ray. It's feasible that a properly machine-learned computer could make better judgments on a long-running average basis. But when a computer screws up an edge case, versus the many, many times a doctor will screw up a diagnosis, which instance will get more attention. People are scared of black box machines and will take particular note of the times an algorithm has screwed them over, regardless of the times a human has screwed them up. Also, when the possibility of examination exists, computerized decision making is far, far easier to audit (i.e. determine blame)

jlgreco 14y ago
Exactly. It is the same as the problem with robotic cars. Cars with human drivers kill thousands, but nobody blinks. The day a robotic car kills a single person, all the news networks will have a field day.
- mc32 14y ago
  I think you're right about this. Maybe it has to do with appearance of control/self determination. It was a mistake (an accident), but it was my mistake/accident (as opposed to something one didn't have control over).
- laserDinosaur 14y ago
  I once posited the question to a group of friends, would you rather drive a car with a 60% chance of crashing fatally in your lifetime, or use a robotic car that as a 2% failure rate. Everyone chose the 60%, even though statistically it makes absolutely no sense.

brittohalloran 14y ago

I agree with your premise -- the issue is that when humans blow up it doesn't cause a cascading runaway failure like robots are susceptible to.

KevinEldon 14y ago
Humans are susceptible to cascading runaway "ethical" failures. I'm thinking the $2 billion UBS loss. Humans also don't scale very well so risk tends to be distributed across several of us. Perhaps robots should be setup in the same way; that there are several of them with different algorithms tuned for different levels of risk... or perhaps it really doesn't matter all that much because this is one firm and not the entire system so the risk is already distributed correctly.

msellout 14y ago

Not (entirely) true. Robots are not strictly superior to humans in this case because the emergent properties of the system change due to the speed of robot action. As more robots crowd into the very fast strategy space, there tends to be more collision of strategies and thus more extreme price movements as many robots all try to buy/sell the same thing at the same time. If they all pile onto one side of the market, liquidity disappears and price movements become dramatic. In the very fast strategy space, there is less capability for complex strategies requiring long memories. If you want to save some nanoseconds to win at this winner-take-all market, you need to reduce your memory and processing time. http://arxiv.org/PS_cache/arxiv/pdf/1006/1006.5490v1.pdf http://arxiv.org/PS_cache/arxiv/pdf/1006/1006.5490v1.pdf This isn't saying that humans would never display herd behavior, but that they do so less often.

theorique 14y ago
Humans absolutely do display herd behavior. This has been documented in financial markets and other venues. However, they do it (by definition) on a time scale that is perceptible to other humans.

alwaysinshade 14y ago

There are cases where having a human in the loop is appropriate, particularly manufacturing. A human, thanks to a combination of slowness and observation skill, will pick up on a fault after the first error while a machine, thanks to a combination of speed and lack of observation skill, will produce fault after fault until things come to a crashing halt or someone notices before that happens. This $440m example shows that a human should be thrown into the mix. Let the human/machine strengths cancel out each other's weaknesses.