8 ms·

The stock market as a single, very big piece of multithreaded software

58 points by olefoo 16y ago

thraxil 16y ago
Were you actually intending to link directly to the comments section rather than to the top of the article? Am I the only one that finds that annoying?
davidw 16y ago
I think a bunch of message-passing Erlang processes seems like a more apt description. Any one of them can crash, and others will take over, if things are set up properly. Indeed, after a half an hour, things were mostly sorted out.
gte910h 16y ago
With this idea as "stock market as multithreaded program", I find it ironic then that many derivatives now have to specify their function in python...
yummyfajitas 16y ago
It is hard to agree with his conclusion that the market is unstable. The flash crash was a huge exogenous shock of unknown (at the time) origin and the market self corrected in a few minutes. The market did such a good job that the author did not even notice the crash until a specialist pointed it out. If only all multithreaded programs behaved that well. [edit: added parenthetical "(at the time)", thanks chr15.]
- chr15 16y ago
  If people are actually interested in what caused the flash crash, it was a sale of $4.1 billion of E-Mini contracts by a single fund. Report of SEC finding [PDF]: http://www.sec.gov/news/studies/2010/marketevents-report.pdf http://www.sec.gov/news/studies/2010/marketevents-report.pdf
  - yummyfajitas 16y ago
    To clarify, I meant "unknown at the time."
    - dpapathanasiou 16y ago
      Well, you would have noticed it if your order book had open limit orders on PG: a sudden drop and recovery like that would have cleared them all.
  - gaius 16y ago
    One thing that I think isn't getting enough attention is it wasn't even a Wall Street "master of the universe" like GS, just a relatively staid firm out on Main Street in the Midwest.
  - salemh 16y ago
    From the report..and where much of the ludicrousness surround HFT exists: "Between 2:45:13 and 2:45:27, HFTs traded over 27,000 contracts, which accounted for about 49 percent of the total trading volume, while buying only about 200 additional contracts net... " While buying only 200 additional contracts...HFT algo's pinging the market with orders never placed :D
    - yummyfajitas 16y ago
      You don't know what you are talking about. A) You can't ping the market with "orders never placed." You can place orders which may or may not be filled. In this case, they were filled. B) They are talking about trade volume, not order volume. Further, "200 additional contracts net". Thus means the HFTs immediately purchased 27,000 contracts and later resold 26,800 of them to someone else. Lastly, your laptop can probably process 27,000 trades in less than one second.
  - nomurrcy 16y ago
    This really isn't what caused the flash crash. The SEC either doesn't have a complete understanding of what happened, or is omitting swaths of data from their report. Nanex did a much better job writing this all up, and weeks before the SEC published their report. http://www.nanex.net/FlashCrashFinal/FlashCrashSummary.html http://www.nanex.net/FlashCrashFinal/FlashCrashSummary.html
- SHOwnsYou 16y ago
  I agree that his conclusion of instability is difficult and will offer the idea that of all the critiques of the stock market, instability is probably the least credible. A stock market with round the clock, instantaneous transactions is one of the most efficient markets that exist. Instability would suggest that it frequently allows for inefficient outcomes. A stock market's prices are determined exactly by buyer demand and seller supply. It corrects itself in the event of freak transactions (like the flash crash) and if it doesn't correct itself, that means the supply and demand have changed and the price of stocks nearly instantly reflect those changes.
- jbarham 16y ago
  It's all very well to say that the market "self corrected" unless you lost a bundle w/ stop-loss orders that were triggered on the down-spike.
  - yummyfajitas 16y ago
    Very true. If you made the problem worse by placing "sell at any price" orders, you lost money. On the other hand, if you (as many HFT's did) tried to buy at the bottom and push the price back up to normal, you made money. This is why the market tends to correct errors quickly. Fixing things == making money, breaking things == losing money.
    - borism 16y ago
      Nobody said people placed simple STOP orders. I think it's pretty obvious to even n00biest of the traders and investors that in such volatile markets as we had in last couple of years you should use STOP LIMIT. So all those people that wanted to protect their hard earned investments (incl. 401k's) at a reasonable level "broke" things and HFT "corrected" them!? In my world that's actually called stealing.
      - yummyfajitas 16y ago
        From the comment I responded to: "...unless you lost a bundle w/ stop-loss orders..." If you placed stop limit orders, you didn't lose a bundle and you also didn't harm the market very much. If you placed a stop limit on Accenture at 40, you lost $1/share. That's a hit, but not a ridiculous one. That's the penalty for telling the market that Accenture should be priced at $40 when it should really be priced at $41. A lot of retail investors actually did place stop loss orders. That's why Accenture traded at $0.01 (note: those trades were broken). Bargain seeking HFT's and daytraders bought at $0.01, sold at $0.05, and quickly pushed the price of Accenture back up to $40.0 or so. The market will hurt you if you give it bad information. The worse the information, the more you lose. If you don't understand this you should not trade.
        16y ago
sz 16y ago
Humanity as a single, very big piece of multithreaded software: Minds are processes, reality is shared memory. Crashes happen under suitably bad conditions (e.g. WWII). I don't see how this per se offers any real insight. You can apply the analogy to almost anything.
- tomjen3 16y ago
  Actually because you can apply it to anything, it means that the things we learned in CS class are that much more useful.
- dpapathanasiou 16y ago
  Well, Douglas Adams thought of it, too. In HHGG, the Earth is just a massive computer for calculating the question to the answer of life the universe and everything. All the biological components are merely instances and sub-processes spawned by the main program.
- keefe 16y ago
  argh and it's a crappy analogy honestly. There's no real deadlocks as far as cycles of dependencies on critical resources. It's basically saying the stock market is a giant distributed system with a bunch of independent agents, which erm it is... this analogy to threading just doesn't add anything, it's just the author's model for agents acting concurrently, threading is a low level OS thing not the fundamentals of concurrent action.
contextfree 16y ago
I do tend to think of financial markets as a giant computer whose function is to allocate capital in the economy, and which just happens to use a really weird architecture/computational model that incorporates human brains as computational nodes, sort of like The Matrix if they hadn't gone with the goofy energy-extraction motivation instead. It's probably an analogy with limited applicability, but I do think it might be interesting to examine the computational properties of financial markets, try to see to what extent arbitrary programs can be encoded in them, etc.
- yummyfajitas 16y ago
  It's actually an analogy with great applicability. Quite a bit of work has been done on this topic - see the work of Pennock at Yahoo. One result: completely efficient prediction market is a general purpose computer. Such an object also requires a ridiculous number of market participants, since it can solve NP complete problems. http://dpennock.com/publications.html http://dpennock.com/publications.html One very popular result (major media attention) along these lines is this paper, which proves that pricing a certain type of CDO is an NP-complete problem: http://www.cs.princeton.edu/~rongge/derivative.pdf http://www.cs.princeton.edu/~rongge/derivative.pdf (Side note: this paper really pisses me off because they did not cite ANY of the prediction market literature. It's hard to believe this was not on purpose.)
rgarcia 16y ago
So when the HFTs got hold of the futures contracts that the Waddell & Reed algorithm was selling, they would've first looked for legit buyers to flip them too. And, not finding any legit buyers, they just started rapidly buying and selling amongst themselves to at least pick up some rebate money. A couple of things: 1. E-mini futures don't give rebates 2. He seems to suggest that different algos from different companies were knowingly trying to collude with each other to get these nonexistent rebates. This seems pretty far-fetched.
- gxti 16y ago
  And even if there were rebates (as there are on stocks) you can't make a profit just passing them back and forth because taking liquidity costs more than is rebated to the offerer. If liquidity hot potato were profitable then that would be the sole purpose of the stock market until the exchanges ran out of money, because it'd be coming straight out of their pockets.
  - ScottBurson 16y ago
    This must be true. But then, what did happen? Why were there 27,000 trades in 14 seconds (IIRC)?
    - gxti 16y ago
      I'm not saying it didn't happen, just that it's not something that a "rational" algorithm would do, because the one that's taking liquidity is losing money. Unless they're profiting for reasons other than rebates, that is, which is another small but nonzero possibility. My favorite theory is a purely technical one. I don't have much supporting evidence beyond the linked site though: """In summary, quotes from NYSE began to queue, but because they were time stamped after exiting the queue, the delay was undetectable to systems processing those quotes. The delay was small enough to cause the NYSE bid to be just slightly higher than the lowest offer price from competing exchanges. This caused sell order flow to route to NYSE -- thus removing any buying power that existed on other exchanges. When these sell orders arrived at NYSE, the actual bid price was lower because new lower quotes were still waiting to exit a queue for dissemination.""" Original report: http://www.nanex.net/20100506/FlashCrashAnalysis_Intro.html http://www.nanex.net/20100506/FlashCrashAnalysis_Intro.html More news from the same company: http://www.nanex.net/FlashCrash/FlashCrashAnalysis.html http://www.nanex.net/FlashCrash/FlashCrashAnalysis.html In addition to the crash itself, they also have interesting charts from various algorithms behaving strangely.
fleitz 16y ago
Not to be pedantic but it's more like a piece of functional message passing software with many agents each with their own immutable state (the final record is the ticker tape and it's generally immutable).
grandalf 16y ago
Multithreaded? Maybe, but then exchanges are mutexes.

58 points by olefoo 16y ago

Were you actually intending to link directly to the comments section rather than to the top of the article? Am I the only one that finds that annoying?

davidw 16y ago

I think a bunch of message-passing Erlang processes seems like a more apt description. Any one of them can crash, and others will take over, if things are set up properly. Indeed, after a half an hour, things were mostly sorted out.

gte910h 16y ago

With this idea as "stock market as multithreaded program", I find it ironic then that many derivatives now have to specify their function in python...

yummyfajitas 16y ago

It is hard to agree with his conclusion that the market is unstable. The flash crash was a huge exogenous shock of unknown (at the time) origin and the market self corrected in a few minutes. The market did such a good job that the author did not even notice the crash until a specialist pointed it out. If only all multithreaded programs behaved that well. [edit: added parenthetical "(at the time)", thanks chr15.]

chr15 16y ago
If people are actually interested in what caused the flash crash, it was a sale of $4.1 billion of E-Mini contracts by a single fund. Report of SEC finding [PDF]: http://www.sec.gov/news/studies/2010/marketevents-report.pdf http://www.sec.gov/news/studies/2010/marketevents-report.pdf
- yummyfajitas 16y ago
  To clarify, I meant "unknown at the time."
  - dpapathanasiou 16y ago
    Well, you would have noticed it if your order book had open limit orders on PG: a sudden drop and recovery like that would have cleared them all.
- gaius 16y ago
  One thing that I think isn't getting enough attention is it wasn't even a Wall Street "master of the universe" like GS, just a relatively staid firm out on Main Street in the Midwest.
- salemh 16y ago
  From the report..and where much of the ludicrousness surround HFT exists: "Between 2:45:13 and 2:45:27, HFTs traded over 27,000 contracts, which accounted for about 49 percent of the total trading volume, while buying only about 200 additional contracts net... " While buying only 200 additional contracts...HFT algo's pinging the market with orders never placed :D
  - yummyfajitas 16y ago
    You don't know what you are talking about. A) You can't ping the market with "orders never placed." You can place orders which may or may not be filled. In this case, they were filled. B) They are talking about trade volume, not order volume. Further, "200 additional contracts net". Thus means the HFTs immediately purchased 27,000 contracts and later resold 26,800 of them to someone else. Lastly, your laptop can probably process 27,000 trades in less than one second.
- nomurrcy 16y ago
  This really isn't what caused the flash crash. The SEC either doesn't have a complete understanding of what happened, or is omitting swaths of data from their report. Nanex did a much better job writing this all up, and weeks before the SEC published their report. http://www.nanex.net/FlashCrashFinal/FlashCrashSummary.html http://www.nanex.net/FlashCrashFinal/FlashCrashSummary.html
SHOwnsYou 16y ago
I agree that his conclusion of instability is difficult and will offer the idea that of all the critiques of the stock market, instability is probably the least credible. A stock market with round the clock, instantaneous transactions is one of the most efficient markets that exist. Instability would suggest that it frequently allows for inefficient outcomes. A stock market's prices are determined exactly by buyer demand and seller supply. It corrects itself in the event of freak transactions (like the flash crash) and if it doesn't correct itself, that means the supply and demand have changed and the price of stocks nearly instantly reflect those changes.
jbarham 16y ago
It's all very well to say that the market "self corrected" unless you lost a bundle w/ stop-loss orders that were triggered on the down-spike.
- yummyfajitas 16y ago
  Very true. If you made the problem worse by placing "sell at any price" orders, you lost money. On the other hand, if you (as many HFT's did) tried to buy at the bottom and push the price back up to normal, you made money. This is why the market tends to correct errors quickly. Fixing things == making money, breaking things == losing money.
  - borism 16y ago
    Nobody said people placed simple STOP orders. I think it's pretty obvious to even n00biest of the traders and investors that in such volatile markets as we had in last couple of years you should use STOP LIMIT. So all those people that wanted to protect their hard earned investments (incl. 401k's) at a reasonable level "broke" things and HFT "corrected" them!? In my world that's actually called stealing.
    - yummyfajitas 16y ago
      From the comment I responded to: "...unless you lost a bundle w/ stop-loss orders..." If you placed stop limit orders, you didn't lose a bundle and you also didn't harm the market very much. If you placed a stop limit on Accenture at 40, you lost $1/share. That's a hit, but not a ridiculous one. That's the penalty for telling the market that Accenture should be priced at $40 when it should really be priced at $41. A lot of retail investors actually did place stop loss orders. That's why Accenture traded at $0.01 (note: those trades were broken). Bargain seeking HFT's and daytraders bought at $0.01, sold at $0.05, and quickly pushed the price of Accenture back up to $40.0 or so. The market will hurt you if you give it bad information. The worse the information, the more you lose. If you don't understand this you should not trade.
      - 16y ago

sz 16y ago

Humanity as a single, very big piece of multithreaded software: Minds are processes, reality is shared memory. Crashes happen under suitably bad conditions (e.g. WWII). I don't see how this per se offers any real insight. You can apply the analogy to almost anything.

tomjen3 16y ago
Actually because you can apply it to anything, it means that the things we learned in CS class are that much more useful.
dpapathanasiou 16y ago
Well, Douglas Adams thought of it, too. In HHGG, the Earth is just a massive computer for calculating the question to the answer of life the universe and everything. All the biological components are merely instances and sub-processes spawned by the main program.
keefe 16y ago
argh and it's a crappy analogy honestly. There's no real deadlocks as far as cycles of dependencies on critical resources. It's basically saying the stock market is a giant distributed system with a bunch of independent agents, which erm it is... this analogy to threading just doesn't add anything, it's just the author's model for agents acting concurrently, threading is a low level OS thing not the fundamentals of concurrent action.

contextfree 16y ago

I do tend to think of financial markets as a giant computer whose function is to allocate capital in the economy, and which just happens to use a really weird architecture/computational model that incorporates human brains as computational nodes, sort of like The Matrix if they hadn't gone with the goofy energy-extraction motivation instead. It's probably an analogy with limited applicability, but I do think it might be interesting to examine the computational properties of financial markets, try to see to what extent arbitrary programs can be encoded in them, etc.

yummyfajitas 16y ago
It's actually an analogy with great applicability. Quite a bit of work has been done on this topic - see the work of Pennock at Yahoo. One result: completely efficient prediction market is a general purpose computer. Such an object also requires a ridiculous number of market participants, since it can solve NP complete problems. http://dpennock.com/publications.html http://dpennock.com/publications.html One very popular result (major media attention) along these lines is this paper, which proves that pricing a certain type of CDO is an NP-complete problem: http://www.cs.princeton.edu/~rongge/derivative.pdf http://www.cs.princeton.edu/~rongge/derivative.pdf (Side note: this paper really pisses me off because they did not cite ANY of the prediction market literature. It's hard to believe this was not on purpose.)

rgarcia 16y ago

So when the HFTs got hold of the futures contracts that the Waddell & Reed algorithm was selling, they would've first looked for legit buyers to flip them too. And, not finding any legit buyers, they just started rapidly buying and selling amongst themselves to at least pick up some rebate money. A couple of things: 1. E-mini futures don't give rebates 2. He seems to suggest that different algos from different companies were knowingly trying to collude with each other to get these nonexistent rebates. This seems pretty far-fetched.

gxti 16y ago
And even if there were rebates (as there are on stocks) you can't make a profit just passing them back and forth because taking liquidity costs more than is rebated to the offerer. If liquidity hot potato were profitable then that would be the sole purpose of the stock market until the exchanges ran out of money, because it'd be coming straight out of their pockets.
- ScottBurson 16y ago
  This must be true. But then, what did happen? Why were there 27,000 trades in 14 seconds (IIRC)?
  - gxti 16y ago
    I'm not saying it didn't happen, just that it's not something that a "rational" algorithm would do, because the one that's taking liquidity is losing money. Unless they're profiting for reasons other than rebates, that is, which is another small but nonzero possibility. My favorite theory is a purely technical one. I don't have much supporting evidence beyond the linked site though: """In summary, quotes from NYSE began to queue, but because they were time stamped after exiting the queue, the delay was undetectable to systems processing those quotes. The delay was small enough to cause the NYSE bid to be just slightly higher than the lowest offer price from competing exchanges. This caused sell order flow to route to NYSE -- thus removing any buying power that existed on other exchanges. When these sell orders arrived at NYSE, the actual bid price was lower because new lower quotes were still waiting to exit a queue for dissemination.""" Original report: http://www.nanex.net/20100506/FlashCrashAnalysis_Intro.html http://www.nanex.net/20100506/FlashCrashAnalysis_Intro.html More news from the same company: http://www.nanex.net/FlashCrash/FlashCrashAnalysis.html http://www.nanex.net/FlashCrash/FlashCrashAnalysis.html In addition to the crash itself, they also have interesting charts from various algorithms behaving strangely.

fleitz 16y ago

Not to be pedantic but it's more like a piece of functional message passing software with many agents each with their own immutable state (the final record is the ticker tape and it's generally immutable).

grandalf 16y ago

Multithreaded? Maybe, but then exchanges are mutexes.