10 ms·

The stock market is controlled by algorithms that are fighting with each other

94 points by DanielRibeiro 16y ago

acrum 16y ago
If you've done any daytrading, you've seen these algorithms in action. I know a few folks who could make a small fortune daily just by "fooling" the algorithms by setting extraordinarily high or low order prices, and rarely actually filling one of those orders. There is lots of money to be had (and lost) by mathematicians in the stock market, and there is a lot to be said for technical analysis... but it's a jungle out there.
- joshu 16y ago
  there's nothing to be said for technical analysis. it doesn't work.
  - riffer 16y ago
    PBS documentary on Paul Tudor Jones from 1986: http://www.tudou.com/programs/view/XH5W4vffBbY http://www.tudou.com/programs/view/XH5W4vffBbY Sorry that the first 10 seconds are shady japanese yogurt commercials, last time I looked the videos were $300 on ebay reportedly because PTJ had gone around buying up all the tapes (maybe the glasses?)
  - kliao 16y ago
    even though technical analysis may not have any merit, many people do use it in their trading decisions. thus, sometimes it is worth paying attention to.
  - firefoxman1 16y ago
    If you depend solely on technical analysis and you aren't a total pro, then it won't work. For most people it should part of their overall trading strategy.
- joshu 16y ago
  there's nothing to be said for technical analysis. it doesn't work.
- fr0sty 16y ago
  "Could make a small fortune..." This spikes my BS mater just a bit. Are any of these people actually doing this? Doing it consistently? Making a fortune? (How small is small, btw). If there was free money for the taking, someone would be taking it and squashing the opportunity out for the others. It's how markets (and especially the stock market) works.
joshu 16y ago
Sigh. VWAP isn't an algorithm, it's a benchmark. Volume-weighted Average Price. An algorithm that is attempting to trade VWAP is trying to get the day's average price, whatever it is. And a good algorithm is quite hard to detect.
- wrong 16y ago
  Most shops call their VWAP-benched algo "VWAP". http://www.thetradenews.com/algorithmic-trading http://www.thetradenews.com/algorithmic-trading VWAP algos try to beat VWAP over the order's timeframe, which is almost never the entire day.
- drstrangevibes 16y ago
  no you read it wrong, they are detecting the true volume weighted price of an instrument, by sniffing all the little sneaky trades jettisoned into the market.
deleted 16y ago
[deleted]
AlexC04 16y ago
Artificial Intelligences warring with each other on stock markets? Witness the birth of Skynet, it's not going to be a military machine but a predatory corporate AI. Originally designed to devour competing AIs on the capital markets only to turn on its masters when the funds run out. Woah. Spooky. Someone tell WIlliam Gibson and Vernor Vinge. Hell, I'd write that story if I had the time! EDIT oh no! It's a predatory Corporate AI that has built in new monitoring functions. The news monitoring subprocess has discovered that there's a predictable correlation between WAR and stock market fluctuation. What does it do, suddenly places a whole stack of extremem long shot PUTS and SHORTS then hacks into the military computers to instigate a war between two nations. We've got a grizzled reporter or plucky young cop (maybe an SEC investigator?) who's doing routine investigations into this and that and uncovers a series of small time conflicts in bananna republics. (these were A/B split testing 'dry runs' the damned AI was reading the visual website optimizer's blog posts!) Just as he (or she's) connecting the dots on the small time stuff there's a major tunabout and the AI has put a bunch of shorts ON US SOIL.... !!! (oh noes! all of america is at risk!) Boom! Race around trying to uncover the source... but now the AI is on to him (or her) and has started hacking public records to declare him (or her) as public enemy number one. Then ... ROBOT ATTACK!! Just as he or she is on the final sprint to disconnect the AI's network cables - military robots spring out of their boxes and start shooting. PEW PEW PEW. ALso - after the cable gets unplugged of course we learn that it's already made it out into the CLOUD! I love this story :) Just the right balance of total cheese and hard Sci Fi.
- jerf 16y ago
  "Woah. Spooky. Someone tell William Gibson and Vernor Vinge." cstross, actually: http://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html http://www.antipope.org/charlie/blog-static/fiction/accelera...
- nikcub 16y ago
  the news discovery portion fascinates me. I know that they are sucking in all of Twitter, as much of Facebook as they can, tons of RSS, etc. recently there was a case where the actress anne hathaway was trending on twitter and it caused the bots to short berkshire hathaway stock, which dipped ~6% arms race going on which will only open more opportunities to exploit these situations. feeding in all this public news data and automating trades makes a mockery of SEC insider trading regulation (because it is so easy to hide - the computer did it!) I would love to read a full feature of what goes on inside GS et al but I know that the quants aren't allowed to speak about it and the companies do not discuss it. We get drip fed small pieces of info but nobody has really assembled a complete picture of what is going on. I do know that the investment banks are some of the largest customers for most of the large enterprise IT co's (IBM, Sun, Microsoft, EMC, etc.). The IB's are also hiring mathematicians and compsci graduates like crazy - the tech these guys are producing must be cutting edge and at a very very large scale
ignifero 16y ago
Shouldn't we be hoping that more and more bots enter the markets? Sure, there will be a period when hackers will be able to exploit their weaknesses, but in the due time they will evolve and market will equilibrate, making today's speculative traders' job much harder.
baberuth 16y ago
Brutal title (Not OP's fault). The article basically suggests that algos are reducing market volatility. Algos don't try really hard to get VWAP. VWAP is REALLY easy to achieve, _beating_ VWAP is what most banks are trying to do when they trade large blocks. A lot of the stories about warring algos and detecting each other are just that: stories. Yes, it has happened, but by and large thats not the game thats going on.
- fr0sty 16y ago
  Citation needed for your assertion that VWAP is "REALLY easy to achieve".
  - vecter 16y ago
    In the extreme case, consider you were on the other side of every other trade that happened in the market. You would by definition have achieved VWAP over some time period. Trading randomly during any given time period probably comes close to achieving VWAP. I'm sure something a little more intelligent could probably achieve it in expectation.
    - joncooper 16y ago
      VWAP is "volume-weighted average price," not "average price". Therefore, to replicate it, you need to participate in line with volume-at-time, not 1/N per time bucket. This requires you to estimate the volume envelope over the time period of interest. (Often a day, or in the case of hedging certain new issue convertible bonds, up to several days.)
      - vecter 16y ago
        Yeah, that's why I said in the extreme case you have to participate in all the volume. Of course randomly trading doesn't prove that you achieve VWAP, but I'm just saying that it's probably not terribly far off.
        16y ago
btilly 16y ago
Something that I've never understood. We have markets that are based on allowing trades to happen at any agreed on price, as fast as people's minds can meet. Which can therefore result in very high volatility. Why hasn't anyone tried building a market that goes the opposite way? Prices drift at a fixed rate (eg min(1 cent, 0.05% of price)/minute) in a direction determined by whether there is an excess of buy or sell orders, and trades execute as soon as both a buy and a sell are available at the right price. This would give greatly reduced instant volatility in return for uncertainty about how quickly orders will complete. The advantage of the latter strategy being that it is very resistant to attempts to compromise it via high frequency trading techniques. That 1/100th of a second time margin doesn't matter because the price changes too slowly. Therefore a long-term investor knows that they aren't getting charged "rent" via the automated algorithms. It might be objected that lack of trading volume would make such a market initially useless. But arbitrage would keep its price close to the mainstream market. Hmmm..I guess it initially comes down to whether you're paying more in rent on arbitrage or on HFT...
- waqf 16y ago
  This reminds me of my idea for a compiler that guarantees performance but not correctness, instead of the other way round.
  - regehr 16y ago
    I have a compiler like that. It turns any code into "return 0".
  - btilly 16y ago
    You mean C? http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
  - repsilat 16y ago
    3D rendering and media compression are almost entirely based on the idea that "close enough" is vastly cheaper than "do it right". At the code level, though... You'd have to be fairly careful about what you allowed, and you'd probably have to be able to enforce correctness to some degree to get any meaningful use out of it. I like the idea of a compiler making lossy simplifications to mathematical formulas for speed - "You told me to divide by seven. This (faster) machine code divides by 8 most of the time, and divides by 4 every now and then." Blindly trusting the branch-predictor could be fun, too, as long as it took into account the correct answer in its subsequent predictions. Were you thinking along similar lines, or did you have something completely different in mind?
CountHackulus 16y ago
For those curious, here's a link to the full text of the Isaac Asimov short story that was mentioned in the text, The Evitable Conflict: http://members.multimania.co.uk/shortstories/asimovconflict.html http://members.multimania.co.uk/shortstories/asimovconflict....
fdb 16y ago
For people who speak dutch, there's a great interactive documentary for the iPad called "Money & Speed" that examines the flash crash and high-frequency trading: http://itunes.apple.com/nl/app/money-speed-inside-black-box/id411884445 http://itunes.apple.com/nl/app/money-speed-inside-black-box/...
d0ne 16y ago
Source article: http://www.lrb.co.uk/2011/05/19/donald-mackenzie/how-to-make-money-in-microseconds http://www.lrb.co.uk/2011/05/19/donald-mackenzie/how-to-make... Not set to be published until the 19th of this month.
iwwr 16y ago
The Core Wars, on a huge canvas.
known 16y ago
HFT is the legalized way of insider trading and front running
- drstrangevibes 16y ago
  Absolutely! If the computer 'decided' on these illegal strategies theres impunity!

94 points by DanielRibeiro 16y ago

acrum 16y ago

If you've done any daytrading, you've seen these algorithms in action. I know a few folks who could make a small fortune daily just by "fooling" the algorithms by setting extraordinarily high or low order prices, and rarely actually filling one of those orders. There is lots of money to be had (and lost) by mathematicians in the stock market, and there is a lot to be said for technical analysis... but it's a jungle out there.

joshu 16y ago
there's nothing to be said for technical analysis. it doesn't work.
- riffer 16y ago
  PBS documentary on Paul Tudor Jones from 1986: http://www.tudou.com/programs/view/XH5W4vffBbY http://www.tudou.com/programs/view/XH5W4vffBbY Sorry that the first 10 seconds are shady japanese yogurt commercials, last time I looked the videos were $300 on ebay reportedly because PTJ had gone around buying up all the tapes (maybe the glasses?)
- kliao 16y ago
  even though technical analysis may not have any merit, many people do use it in their trading decisions. thus, sometimes it is worth paying attention to.
- firefoxman1 16y ago
  If you depend solely on technical analysis and you aren't a total pro, then it won't work. For most people it should part of their overall trading strategy.
joshu 16y ago
there's nothing to be said for technical analysis. it doesn't work.
fr0sty 16y ago
"Could make a small fortune..." This spikes my BS mater just a bit. Are any of these people actually doing this? Doing it consistently? Making a fortune? (How small is small, btw). If there was free money for the taking, someone would be taking it and squashing the opportunity out for the others. It's how markets (and especially the stock market) works.

joshu 16y ago

Sigh. VWAP isn't an algorithm, it's a benchmark. Volume-weighted Average Price. An algorithm that is attempting to trade VWAP is trying to get the day's average price, whatever it is. And a good algorithm is quite hard to detect.

wrong 16y ago
Most shops call their VWAP-benched algo "VWAP". http://www.thetradenews.com/algorithmic-trading http://www.thetradenews.com/algorithmic-trading VWAP algos try to beat VWAP over the order's timeframe, which is almost never the entire day.
drstrangevibes 16y ago
no you read it wrong, they are detecting the true volume weighted price of an instrument, by sniffing all the little sneaky trades jettisoned into the market.

deleted 16y ago

[deleted]

AlexC04 16y ago

Artificial Intelligences warring with each other on stock markets? Witness the birth of Skynet, it's not going to be a military machine but a predatory corporate AI. Originally designed to devour competing AIs on the capital markets only to turn on its masters when the funds run out. Woah. Spooky. Someone tell WIlliam Gibson and Vernor Vinge. Hell, I'd write that story if I had the time! EDIT oh no! It's a predatory Corporate AI that has built in new monitoring functions. The news monitoring subprocess has discovered that there's a predictable correlation between WAR and stock market fluctuation. What does it do, suddenly places a whole stack of extremem long shot PUTS and SHORTS then hacks into the military computers to instigate a war between two nations. We've got a grizzled reporter or plucky young cop (maybe an SEC investigator?) who's doing routine investigations into this and that and uncovers a series of small time conflicts in bananna republics. (these were A/B split testing 'dry runs' the damned AI was reading the visual website optimizer's blog posts!) Just as he (or she's) connecting the dots on the small time stuff there's a major tunabout and the AI has put a bunch of shorts ON US SOIL.... !!! (oh noes! all of america is at risk!) Boom! Race around trying to uncover the source... but now the AI is on to him (or her) and has started hacking public records to declare him (or her) as public enemy number one. Then ... ROBOT ATTACK!! Just as he or she is on the final sprint to disconnect the AI's network cables - military robots spring out of their boxes and start shooting. PEW PEW PEW. ALso - after the cable gets unplugged of course we learn that it's already made it out into the CLOUD! I love this story :) Just the right balance of total cheese and hard Sci Fi.

jerf 16y ago
"Woah. Spooky. Someone tell William Gibson and Vernor Vinge." cstross, actually: http://www.antipope.org/charlie/blog-static/fiction/accelerando/accelerando-intro.html http://www.antipope.org/charlie/blog-static/fiction/accelera...
nikcub 16y ago
the news discovery portion fascinates me. I know that they are sucking in all of Twitter, as much of Facebook as they can, tons of RSS, etc. recently there was a case where the actress anne hathaway was trending on twitter and it caused the bots to short berkshire hathaway stock, which dipped ~6% arms race going on which will only open more opportunities to exploit these situations. feeding in all this public news data and automating trades makes a mockery of SEC insider trading regulation (because it is so easy to hide - the computer did it!) I would love to read a full feature of what goes on inside GS et al but I know that the quants aren't allowed to speak about it and the companies do not discuss it. We get drip fed small pieces of info but nobody has really assembled a complete picture of what is going on. I do know that the investment banks are some of the largest customers for most of the large enterprise IT co's (IBM, Sun, Microsoft, EMC, etc.). The IB's are also hiring mathematicians and compsci graduates like crazy - the tech these guys are producing must be cutting edge and at a very very large scale

ignifero 16y ago

Shouldn't we be hoping that more and more bots enter the markets? Sure, there will be a period when hackers will be able to exploit their weaknesses, but in the due time they will evolve and market will equilibrate, making today's speculative traders' job much harder.

baberuth 16y ago

Brutal title (Not OP's fault). The article basically suggests that algos are reducing market volatility. Algos don't try really hard to get VWAP. VWAP is REALLY easy to achieve, _beating_ VWAP is what most banks are trying to do when they trade large blocks. A lot of the stories about warring algos and detecting each other are just that: stories. Yes, it has happened, but by and large thats not the game thats going on.

fr0sty 16y ago
Citation needed for your assertion that VWAP is "REALLY easy to achieve".
- vecter 16y ago
  In the extreme case, consider you were on the other side of every other trade that happened in the market. You would by definition have achieved VWAP over some time period. Trading randomly during any given time period probably comes close to achieving VWAP. I'm sure something a little more intelligent could probably achieve it in expectation.
  - joncooper 16y ago
    VWAP is "volume-weighted average price," not "average price". Therefore, to replicate it, you need to participate in line with volume-at-time, not 1/N per time bucket. This requires you to estimate the volume envelope over the time period of interest. (Often a day, or in the case of hedging certain new issue convertible bonds, up to several days.)
    - vecter 16y ago
      Yeah, that's why I said in the extreme case you have to participate in all the volume. Of course randomly trading doesn't prove that you achieve VWAP, but I'm just saying that it's probably not terribly far off.
      - 16y ago

btilly 16y ago

Something that I've never understood. We have markets that are based on allowing trades to happen at any agreed on price, as fast as people's minds can meet. Which can therefore result in very high volatility. Why hasn't anyone tried building a market that goes the opposite way? Prices drift at a fixed rate (eg min(1 cent, 0.05% of price)/minute) in a direction determined by whether there is an excess of buy or sell orders, and trades execute as soon as both a buy and a sell are available at the right price. This would give greatly reduced instant volatility in return for uncertainty about how quickly orders will complete. The advantage of the latter strategy being that it is very resistant to attempts to compromise it via high frequency trading techniques. That 1/100th of a second time margin doesn't matter because the price changes too slowly. Therefore a long-term investor knows that they aren't getting charged "rent" via the automated algorithms. It might be objected that lack of trading volume would make such a market initially useless. But arbitrage would keep its price close to the mainstream market. Hmmm..I guess it initially comes down to whether you're paying more in rent on arbitrage or on HFT...

waqf 16y ago
This reminds me of my idea for a compiler that guarantees performance but not correctness, instead of the other way round.
- regehr 16y ago
  I have a compiler like that. It turns any code into "return 0".
- btilly 16y ago
  You mean C? http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
- repsilat 16y ago
  3D rendering and media compression are almost entirely based on the idea that "close enough" is vastly cheaper than "do it right". At the code level, though... You'd have to be fairly careful about what you allowed, and you'd probably have to be able to enforce correctness to some degree to get any meaningful use out of it. I like the idea of a compiler making lossy simplifications to mathematical formulas for speed - "You told me to divide by seven. This (faster) machine code divides by 8 most of the time, and divides by 4 every now and then." Blindly trusting the branch-predictor could be fun, too, as long as it took into account the correct answer in its subsequent predictions. Were you thinking along similar lines, or did you have something completely different in mind?

CountHackulus 16y ago

For those curious, here's a link to the full text of the Isaac Asimov short story that was mentioned in the text, The Evitable Conflict: http://members.multimania.co.uk/shortstories/asimovconflict.html http://members.multimania.co.uk/shortstories/asimovconflict....

fdb 16y ago

For people who speak dutch, there's a great interactive documentary for the iPad called "Money & Speed" that examines the flash crash and high-frequency trading: http://itunes.apple.com/nl/app/money-speed-inside-black-box/id411884445 http://itunes.apple.com/nl/app/money-speed-inside-black-box/...

d0ne 16y ago

Source article: http://www.lrb.co.uk/2011/05/19/donald-mackenzie/how-to-make-money-in-microseconds http://www.lrb.co.uk/2011/05/19/donald-mackenzie/how-to-make... Not set to be published until the 19th of this month.

iwwr 16y ago

The Core Wars, on a huge canvas.

known 16y ago

HFT is the legalized way of insider trading and front running

drstrangevibes 16y ago
Absolutely! If the computer 'decided' on these illegal strategies theres impunity!