6 ms·

Algorithmic surrealism: A slow-motion guide to high-frequency trading

169 points by Gigamouse 11y ago

washedup 11y ago
This is one of the better overviews of HFT I have come across. It hits on all the points I consider to be important (having worked in HFT for over six years); especially the Techno-Leviathan, trader psyche, and the perception of constant "war". I interviewed at Ronin once, and at the time I was super impressed by the beauty of their offices. Reading this helped me realize what a fucking circus it really is. One facet which always fascinated me was the dispersion of trading ideas, including the code behind algorithms and any sort of research. Successful ideas are constantly being updated, adapted, and often times stolen. Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The rest are just copying what they have been exposed to and hoping that it sticks when they throw it all the wall, which eventually runs each successful idea into the ground as the value being captured disappears quickly. Either way, it was a great school for learning how to program and use statistics effectively. As my interviewer at Ronin told me after a I failed the interview (we both knew it): "This is all a game, you just need to learn the rules"
- chollida1 11y ago
  > This is one of the better overviews of HFT I have come across. Really, I found it to be long on words and almost completely devoid of any content what so ever. I mean this in all seriousness, What specifically can one point to in this article that has any actual content and isnt' just a collection of links to other sites for common definitions. At no point did the article actually say how HFT firms design their systems or even how their algorithms might work. > t hits on all the points I consider to be important (having worked in HFT for over six years) I too am in the industry:)
  - dragontamer 11y ago
    I concur with this sentiment. Especially: > Really, I found it to be long on words and almost completely devoid of any content what so ever. But I don't work in the financial industry. Still, the article offered nothing new to me
  - washedup 11y ago
    It's not technical, that's true, but I really appreciate the philosophical take. It describes the foundation of everything and how the industry works from a high level very well. It was fun for me to read.
- mrchicity 11y ago
  > the perception of constant "war". It's not a perception though. Short-term trading opportunities are zero-sum. To win trades you either must undercut your competitor by trading for less edge, or beat him to realizing the opportunity by either predicting better or getting faster, usually some combination of all those. All of that means effectively working your ass off to take food off another man's table before he takes it from yours. That probably happened less in the pit days because there was a camaraderie between the locals facing off against paper. Nowadays I never see the guy I'm trying to beat into the queue. We aren't going out for beers. It's just business. I'd say being successful in this business feels less like war and more like riding a bear. You never get to rest and there's a constant fear that someone, somewhere is hungrier and smarter than you are, plus the bear might snap and eat you. >Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The better firms tend to hire people out of school and train them up so they can do novel research. It's a smarter long-term play. I agree that few people are up to the task, even the best and brightest. It does amuse me that some guy a couple blocks away probably runs a super similar model with a different cool code-name and we battle each other for years in Sisyphean fashion. I'll never even know his name. A zipless fuck. I don't like the author's characterization of HFTs as "parasites." Someone can be a middle-man and still add value. Market users don't need to trade with HFT dealers. They can easily work their own orders or wait around in a block trading facility hoping to find a natural match instead. For most of them, it's a bad trade-off. Even though the HFTs make money on average, trading with them costs less than becoming an expert in market microstructure, finding good hedges, facing non-execution risk, etc. CarMax makes money when they buy/sell your car in the same way, but it's usually more convenient or cheaper than doing it yourself. In a way, HFT is like insurance in that they only provide a valuable service when they lose, but overall people prefer paying for certainty. Almost every winning trade I make is one where two natural counter parties could have found one another had they waited a short period of time or were in the right place/product, but I lose on a huge % of trades as well. If HFTs weren't in a competitive environment or if one firm had a clairvoyant pricing model, this would be problematic but unfortunately for me, that's not the case.
  - eru 11y ago
- deleted 11y ago
  [deleted]
- lmm 11y ago
  > One facet which always fascinated me was the dispersion of trading ideas, including the code behind algorithms and any sort of research. Successful ideas are constantly being updated, adapted, and often times stolen. Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The rest are just copying what they have been exposed to and hoping that it sticks when they throw it all the wall, which eventually runs each successful idea into the ground as the value being captured disappears quickly. So exactly like Silicon Valley?
  - washedup 11y ago
    Successful ideas in general work this way, but I am impressed with the SPEED in which this dispersion takes place in the trading world. All this brain and computing power cycles through ideas as quickly as possible to find new sources of income. It's a giant complex network trying to find an ever-changing optimal solution to the current market state. Quite incredible.
themartorana 11y ago
Having zero real knowledge of trading in any fashion at all, recently I've been wondering if there was a niche for "medium-speed trading." I know I can't get close to the exchanges, and I don't have the capital to hire experienced trader/engineers to develop the latest algorithms. Basically, is there a slice of the pie in trading much faster than humans, but much slower than HST? It's an academic exercise, but one I've been toying with.
- gnaritas 11y ago
  Since people still make money trading manually, yes, there's room at all levels.
- kvcc01 11y ago
  Yes, there’s a "frequency spectrum" of sorts and HFT is at the short end of it. Here, your models monitor and react to trading tick-by-tick, and execution speed is paramount. It’s also where you have the fiercest arms race for the fastest systems, colocation, FPGAs, etc. Next up comes what’s often called “statistical arbitrage,” where you have models that no longer look at tick-by-tick trading but may look at what happens at 30-second, 1-minute, or 5-minute windows. Here you have more interesting relations emerge between stocks and the market, e.g., what does IBM do relative to the tech index, or relative to MSFT, etc. (Such cross sectional relationships don’t seem to matter as much in HFT.) Actually stat-arb was the domain where the earliest statistical approaches such as "pairs trading" emerged. Next up come models that trade daily (or less frequently), and here you begin to see the long-short market-neutral relative-value type of approaches, where some quantitative [mutual] funds may operate. Next up will be the traditional mutual funds, and beyond that you have your Warren Buffett’s, etc. One thing to keep in mind is that the higher your trading frequency, the smaller the price moves you can hope to capture, which limits how much capital you can deploy in your models. This is why HFT models are usually small in size but have high Sharpe ratios. As you reduce your trading frequency, you can expect to capture larger price movements and deploy more capital but you’ll also be exposed to more of the vicissitudes of the general market, so your Sharpe ratio will decline. Market participants usually carve themselves a happy spot on this frequency spectrum and stay there. I don’t know of any firm who is successful at every spot.
Natsu 11y ago
> if you'd like to support my ongoing Creative Commons writing, please consider buying me a virtual beer. In the spirit of the article's talk about financialization, I wonder if there's yet a way to buy the author virtual beer options?
MichaelCrawford 11y ago
I have some experience with quantitative investment coding and so am frequently asked to apply for HFT jobs. I don't pursue it because I came to regard the practice as unethical.
- bcook 11y ago
  Care to explain why? (unethical) Aside from stating your own skills, employability, and high standard of ethics, your post is lacking useful info about HFT.
  - getsat 11y ago
    I just spent 15 minutes Googling this guy because his site in his profile was interesting. He humblebrags about employability, but he's incapable of finding work. He also makes bomb threats: http://blog.up.co/2012/04/30/not-even-bmob-threats-could-deter-portlands-entrepreneurs-at-startup-weekend/ http://blog.up.co/2012/04/30/not-even-bmob-threats-could-det...
    - MichaelCrawford 11y ago
      No, I don't make bomb threats. I was using Twitter to lecture the Portland Startup Weekend on engineering ethics. I resigned in protest from an HMI/SCADA vendor because of the poor code quality of their product.
      - getsat 11y ago
        I see. Thanks for your side of the story. You are a very interesting person! I'm going to read more of your site later.
        MichaelCrawford 11y ago
        Thank you. From time to time I toy with the idea of suing that joker for libel, but the reason I don't is that I support the notion that the best response to hate speech is more speech.
brentis 11y ago
It's only going to get worse (or better) depending on how you look at it. So many behavioral factors will be included to shake stops and squeeze shorts at just the right time to make huge profits from the bounce. It's part of the reason as a trader I'm now just chasing Momentum, ignoring most other signals as noise. So much so, I'm building an app that supports discovering these momentum breakouts. In the unlikely event anyone is interested - www.mometic.com
deleted 11y ago
[deleted]
Xcelerate 11y ago
> "People are routinely worried about harmless things, and routinely completely unworried about incredibly harmful things." God what a quote. I'm stealing this and using it everywhere I can. It basically sums up my entire attitude toward humanity.
- washedup 11y ago
  Bad quote, I agree, but it does express how risk management in the HFT world is often short sighted.

169 points by Gigamouse 11y ago

This is one of the better overviews of HFT I have come across. It hits on all the points I consider to be important (having worked in HFT for over six years); especially the Techno-Leviathan, trader psyche, and the perception of constant "war". I interviewed at Ronin once, and at the time I was super impressed by the beauty of their offices. Reading this helped me realize what a fucking circus it really is. One facet which always fascinated me was the dispersion of trading ideas, including the code behind algorithms and any sort of research. Successful ideas are constantly being updated, adapted, and often times stolen. Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The rest are just copying what they have been exposed to and hoping that it sticks when they throw it all the wall, which eventually runs each successful idea into the ground as the value being captured disappears quickly. Either way, it was a great school for learning how to program and use statistics effectively. As my interviewer at Ronin told me after a I failed the interview (we both knew it): "This is all a game, you just need to learn the rules"

chollida1 11y ago
> This is one of the better overviews of HFT I have come across. Really, I found it to be long on words and almost completely devoid of any content what so ever. I mean this in all seriousness, What specifically can one point to in this article that has any actual content and isnt' just a collection of links to other sites for common definitions. At no point did the article actually say how HFT firms design their systems or even how their algorithms might work. > t hits on all the points I consider to be important (having worked in HFT for over six years) I too am in the industry:)
- dragontamer 11y ago
  I concur with this sentiment. Especially: > Really, I found it to be long on words and almost completely devoid of any content what so ever. But I don't work in the financial industry. Still, the article offered nothing new to me
- washedup 11y ago
  It's not technical, that's true, but I really appreciate the philosophical take. It describes the foundation of everything and how the industry works from a high level very well. It was fun for me to read.
mrchicity 11y ago
> the perception of constant "war". It's not a perception though. Short-term trading opportunities are zero-sum. To win trades you either must undercut your competitor by trading for less edge, or beat him to realizing the opportunity by either predicting better or getting faster, usually some combination of all those. All of that means effectively working your ass off to take food off another man's table before he takes it from yours. That probably happened less in the pit days because there was a camaraderie between the locals facing off against paper. Nowadays I never see the guy I'm trying to beat into the queue. We aren't going out for beers. It's just business. I'd say being successful in this business feels less like war and more like riding a bear. You never get to rest and there's a constant fear that someone, somewhere is hungrier and smarter than you are, plus the bear might snap and eat you. >Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The better firms tend to hire people out of school and train them up so they can do novel research. It's a smarter long-term play. I agree that few people are up to the task, even the best and brightest. It does amuse me that some guy a couple blocks away probably runs a super similar model with a different cool code-name and we battle each other for years in Sisyphean fashion. I'll never even know his name. A zipless fuck. I don't like the author's characterization of HFTs as "parasites." Someone can be a middle-man and still add value. Market users don't need to trade with HFT dealers. They can easily work their own orders or wait around in a block trading facility hoping to find a natural match instead. For most of them, it's a bad trade-off. Even though the HFTs make money on average, trading with them costs less than becoming an expert in market microstructure, finding good hedges, facing non-execution risk, etc. CarMax makes money when they buy/sell your car in the same way, but it's usually more convenient or cheaper than doing it yourself. In a way, HFT is like insurance in that they only provide a valuable service when they lose, but overall people prefer paying for certainty. Almost every winning trade I make is one where two natural counter parties could have found one another had they waited a short period of time or were in the right place/product, but I lose on a huge % of trades as well. If HFTs weren't in a competitive environment or if one firm had a clairvoyant pricing model, this would be problematic but unfortunately for me, that's not the case.
- eru 11y ago
deleted 11y ago
[deleted]
lmm 11y ago
> One facet which always fascinated me was the dispersion of trading ideas, including the code behind algorithms and any sort of research. Successful ideas are constantly being updated, adapted, and often times stolen. Traders are generally hired for the trading strategies they have been exposed to and the potential value within. There are very few individuals who create new and successful ideas. The rest are just copying what they have been exposed to and hoping that it sticks when they throw it all the wall, which eventually runs each successful idea into the ground as the value being captured disappears quickly. So exactly like Silicon Valley?
- washedup 11y ago
  Successful ideas in general work this way, but I am impressed with the SPEED in which this dispersion takes place in the trading world. All this brain and computing power cycles through ideas as quickly as possible to find new sources of income. It's a giant complex network trying to find an ever-changing optimal solution to the current market state. Quite incredible.

themartorana 11y ago

Having zero real knowledge of trading in any fashion at all, recently I've been wondering if there was a niche for "medium-speed trading." I know I can't get close to the exchanges, and I don't have the capital to hire experienced trader/engineers to develop the latest algorithms. Basically, is there a slice of the pie in trading much faster than humans, but much slower than HST? It's an academic exercise, but one I've been toying with.

gnaritas 11y ago
Since people still make money trading manually, yes, there's room at all levels.
kvcc01 11y ago
Yes, there’s a "frequency spectrum" of sorts and HFT is at the short end of it. Here, your models monitor and react to trading tick-by-tick, and execution speed is paramount. It’s also where you have the fiercest arms race for the fastest systems, colocation, FPGAs, etc. Next up comes what’s often called “statistical arbitrage,” where you have models that no longer look at tick-by-tick trading but may look at what happens at 30-second, 1-minute, or 5-minute windows. Here you have more interesting relations emerge between stocks and the market, e.g., what does IBM do relative to the tech index, or relative to MSFT, etc. (Such cross sectional relationships don’t seem to matter as much in HFT.) Actually stat-arb was the domain where the earliest statistical approaches such as "pairs trading" emerged. Next up come models that trade daily (or less frequently), and here you begin to see the long-short market-neutral relative-value type of approaches, where some quantitative [mutual] funds may operate. Next up will be the traditional mutual funds, and beyond that you have your Warren Buffett’s, etc. One thing to keep in mind is that the higher your trading frequency, the smaller the price moves you can hope to capture, which limits how much capital you can deploy in your models. This is why HFT models are usually small in size but have high Sharpe ratios. As you reduce your trading frequency, you can expect to capture larger price movements and deploy more capital but you’ll also be exposed to more of the vicissitudes of the general market, so your Sharpe ratio will decline. Market participants usually carve themselves a happy spot on this frequency spectrum and stay there. I don’t know of any firm who is successful at every spot.

Natsu 11y ago

> if you'd like to support my ongoing Creative Commons writing, please consider buying me a virtual beer. In the spirit of the article's talk about financialization, I wonder if there's yet a way to buy the author virtual beer options?

MichaelCrawford 11y ago

I have some experience with quantitative investment coding and so am frequently asked to apply for HFT jobs. I don't pursue it because I came to regard the practice as unethical.

bcook 11y ago
Care to explain why? (unethical) Aside from stating your own skills, employability, and high standard of ethics, your post is lacking useful info about HFT.
- getsat 11y ago
  I just spent 15 minutes Googling this guy because his site in his profile was interesting. He humblebrags about employability, but he's incapable of finding work. He also makes bomb threats: http://blog.up.co/2012/04/30/not-even-bmob-threats-could-deter-portlands-entrepreneurs-at-startup-weekend/ http://blog.up.co/2012/04/30/not-even-bmob-threats-could-det...
  - MichaelCrawford 11y ago
    No, I don't make bomb threats. I was using Twitter to lecture the Portland Startup Weekend on engineering ethics. I resigned in protest from an HMI/SCADA vendor because of the poor code quality of their product.
    - getsat 11y ago
      I see. Thanks for your side of the story. You are a very interesting person! I'm going to read more of your site later.
      - MichaelCrawford 11y ago
        Thank you. From time to time I toy with the idea of suing that joker for libel, but the reason I don't is that I support the notion that the best response to hate speech is more speech.

brentis 11y ago

It's only going to get worse (or better) depending on how you look at it. So many behavioral factors will be included to shake stops and squeeze shorts at just the right time to make huge profits from the bounce. It's part of the reason as a trader I'm now just chasing Momentum, ignoring most other signals as noise. So much so, I'm building an app that supports discovering these momentum breakouts. In the unlikely event anyone is interested - www.mometic.com

deleted 11y ago

[deleted]

Xcelerate 11y ago

> "People are routinely worried about harmless things, and routinely completely unworried about incredibly harmful things." God what a quote. I'm stealing this and using it everywhere I can. It basically sums up my entire attitude toward humanity.

washedup 11y ago
Bad quote, I agree, but it does express how risk management in the HFT world is often short sighted.