9 ms·

The clickbait headline is amusing, but "Wall Street" has been doing this > 10 years. I'm genuinely not sure if they're trying to "beat wall street" or simply be

by SEJeff 10y ago

The clickbait headline is amusing, but "Wall Street" has been doing this > 10 years. I'm genuinely not sure if they're trying to "beat wall street" or simply be yet another quantitative hedge fund that happens to use machine learning and neural networks to power their strategies. Most real electronic trading firms are putting those algorithms in hardware. Source: Have worked in HFT the past 10 years.

frankc 10y ago
Longer than that. This article was passed around at my job to laugh at.
- cr0sh 10y ago
  Definitely much longer - I'm pretty sure I read about things like this in Omni Magazine back in the 1980s...
jredwards 10y ago
HFT is a different kind of 'beating the market', though, don't you think?
- SEJeff 10y ago
  It isn't beating the market, it is literally "making the market": https://en.wikipedia.org/wiki/Market_maker https://en.wikipedia.org/wiki/Market_maker
  - raw23 10y ago
    Arguable whether or not HFT increases liquidity in the market.
    - SEJeff 10y ago
      It is arguable if more liquidity is a good or a bad thing from some sides, but I'm not sure how HFT increasing liquidity was ever a question? It is a matter of volume and there is more automated than non-automated volume in most equities and futures exchanges.
    - digler999 10y ago
      However, given that you want to make a trade, it's not arguable whether you want more market participants or fewer.
- dsacco 10y ago
  It is "beating the market" if your strategy results in higher net fee-adjusted returns than the overall market (typically ~7-11%). But yes it's different because you're not deploying a trading strategy, you're being a market maker for other traders.
refurb 10y ago
Machine learning and neural networks? SHUT UP AND TAKE MY MONEY!
6stringmerc 10y ago
I used to have a stock broker co-worker who had one rule for investing if day trading or even trying to employ a strategy: "You can't fight the Street." His point was trying to bet against a momentum trade or some other scenario was foolish. He watched time and again. There were opportunities to get with the crowd, or be on the wrong side.
- got2surf 10y ago
  Great point. And even if you are "with the crowd" more often than you are not, your timing/sizing on each trade you make as a day trader or mom-and-pop investor is very likely to be worse than an institutional trading firm. So you can be right on a "buy" vs "sell" decision 51% of the time, but an algo trading firm has probably executed with better timing/trade size and beaten you to a percentage of the profit. A friend who works in quant finance said something along the lines of "if a trader can identify a mom-and-pop investor, it's like taking candy from a baby" in terms of making an easy profit.
  - Ntrails 10y ago
    I'd rather just try really hard to get people to understand that active trading is negative sum, and that unless (As a mom and pop investor) you are properly bench-marking against a relevant index - you're just lying to yourself. Sure HFT guys are faster than you to get short term signals and make money on the micro scale - but even ignoring that way too many people are thinking they're beating the market and they really really aren't.
    - SEJeff 10y ago
      You just nailed it! I work for a very large HFT and my portfolio for my 401k and extra retirement (in the market) is primarily low cost / fee vanguard index funds. You really can't beat a FPGA with tick to trade times in the milliseconds. Your optical nerve likely hasn't even processed a number changing yet.
      - hendzen 10y ago
jbmorgado 10y ago
> Source: Have worked in HFT the past 10 years. This is not about High Frequency Trading, but about machine learning applied to the stock market.
- SEJeff 10y ago
  Which every single HFT firm that isn't a pure market maker (such as Virtu Financial, my previous employer) does already and has done for years. That is my point. Here's a great example of Citadel talking about how they use Hashicorp nomad to spin up 100,000 cpu research jobs on AWS. What do you think their 100k cpu jobs are doing? Machine learning and quantative analysis, what else? https://www.youtube.com/watch?v=MRtRwhL5lwM https://www.youtube.com/watch?v=MRtRwhL5lwM
  - jbmorgado 10y ago
    I think you are willingly failing to understand what is being said by me and more people on the comments. This is not about exploiting a privileged access to market conditions (as in the sub millisecond access HFT firms have) in order to gain an advantage. This is about using machine learning to predict future (as in hours/days) moves on the market and take advantage of them.
    - SEJeff 10y ago
      Yes and I think you are willingly failing to understand reality from someone who has worked in technology in finance for a long time. HFT or perhaps more accurately, Electronic Trading, is more than simply passive trading and "doing something faster than everyone else". That is a part of it absolutely, but the industry also hires people with PHDs in quantum physics, or builds supercomputers at national labs to build private ones, or teach mathematics at ivy league universities. For much of this math, speed is beneficial, but it isn't universally and always required. Speed is nice, but smarts are better. "Machine Learning" is the new buzzword for a way to mathematically deducing future outcomes based on data science and quantitative analysis. That is a grotesque simplifications and there are specific ways of approaching this (Tensorflow/Keras from Google, Torch from Facebook, etc). This could be nanoseconds in the futures, it could be hours, or even weeks / months. And this is the part you're failing to understand by trying to assume (incorrectly) that all HFT is made up of is bid sniping and latency arbitrage (often referred to as front running). It isn't really that privileged access if you yourself can pay any exchange, such as NYSE, for a feed to your house. TL;DNR: Your understanding of electronic / HFT trading is wrong. If you try to understand it first, you'll realize your statements bear absolutely no basis whatsoever in facts. Facts are that Electronic Trading firms have been using machine learning literally since the invention of machine learning and quantitative analysis to make money. A new firm trying to do this can simply join the club.
digler999 10y ago
> Source: Have worked in HFT the past 10 years. OT question: how would I get into acquiring market data to try my own backtesting strategies ? Is there anyway to "get in" without putting up $5k + for the data ? I'm not green to this, I worked at a prop shop for a year writing market feed handlers. I know the dangers of overfitting data and how hard it is to make money (so I'm not deluded by a fantasy). I want to start out just for fun doing backtesting and see if I can scrape out a (paper) profit, but accessing historical data seems to be the hard part. I dont think online services that offer this would work for me because I want to use intraday data and replay many times with different parameters. I want access to "the whole firehose" if you will.
- kybernetikos 10y ago
  Not a personal recommendation as I haven't used it, but you might find quantopian.com fun if you've got any python.
- hendzen 10y ago
  You are not going to get usable intraday data for free. Best bet is to pony up for NYSE TAQ. http://www.nyxdata.com/Data-Products/Daily-TAQ http://www.nyxdata.com/Data-Products/Daily-TAQ
- pliny 10y ago
  TickData[1] are cheap and good for US data, I have had bad experience with them for EU data. They have a minimum of (IIRC) $1K but you can get a ton for that. Some exchanges offer free historical data, for instance BM&F BOVESPA[2], trading in some markets (such as Brasil and China) is often difficult if you are a foreigner. [1] https://www.tickdata.com/ https://www.tickdata.com/ [2] http://www.bmfbovespa.com.br/en_us/services/market-data/historical-data/ http://www.bmfbovespa.com.br/en_us/services/market-data/hist...
- jmalicki 10y ago
  IQFeed has many many years of historical data at the minute level, and tick data for 6 months (for equities), at ~$80/month... they many not have the highest quality data from what I've heard, or the most tick data, but are so much cheaper than any of the more professional solutions a large fund would use that it's a pretty decent start for a hobbyist.

by SEJeff 10y ago

frankc 10y ago

Longer than that. This article was passed around at my job to laugh at.

cr0sh 10y ago
Definitely much longer - I'm pretty sure I read about things like this in Omni Magazine back in the 1980s...

jredwards 10y ago

HFT is a different kind of 'beating the market', though, don't you think?

SEJeff 10y ago
It isn't beating the market, it is literally "making the market": https://en.wikipedia.org/wiki/Market_maker https://en.wikipedia.org/wiki/Market_maker
- raw23 10y ago
  Arguable whether or not HFT increases liquidity in the market.
  - SEJeff 10y ago
    It is arguable if more liquidity is a good or a bad thing from some sides, but I'm not sure how HFT increasing liquidity was ever a question? It is a matter of volume and there is more automated than non-automated volume in most equities and futures exchanges.
  - digler999 10y ago
    However, given that you want to make a trade, it's not arguable whether you want more market participants or fewer.
dsacco 10y ago
It is "beating the market" if your strategy results in higher net fee-adjusted returns than the overall market (typically ~7-11%). But yes it's different because you're not deploying a trading strategy, you're being a market maker for other traders.

refurb 10y ago

Machine learning and neural networks? SHUT UP AND TAKE MY MONEY!

6stringmerc 10y ago

I used to have a stock broker co-worker who had one rule for investing if day trading or even trying to employ a strategy: "You can't fight the Street." His point was trying to bet against a momentum trade or some other scenario was foolish. He watched time and again. There were opportunities to get with the crowd, or be on the wrong side.

got2surf 10y ago
Great point. And even if you are "with the crowd" more often than you are not, your timing/sizing on each trade you make as a day trader or mom-and-pop investor is very likely to be worse than an institutional trading firm. So you can be right on a "buy" vs "sell" decision 51% of the time, but an algo trading firm has probably executed with better timing/trade size and beaten you to a percentage of the profit. A friend who works in quant finance said something along the lines of "if a trader can identify a mom-and-pop investor, it's like taking candy from a baby" in terms of making an easy profit.
- Ntrails 10y ago
  I'd rather just try really hard to get people to understand that active trading is negative sum, and that unless (As a mom and pop investor) you are properly bench-marking against a relevant index - you're just lying to yourself. Sure HFT guys are faster than you to get short term signals and make money on the micro scale - but even ignoring that way too many people are thinking they're beating the market and they really really aren't.
  - SEJeff 10y ago
    You just nailed it! I work for a very large HFT and my portfolio for my 401k and extra retirement (in the market) is primarily low cost / fee vanguard index funds. You really can't beat a FPGA with tick to trade times in the milliseconds. Your optical nerve likely hasn't even processed a number changing yet.
    - hendzen 10y ago

jbmorgado 10y ago

> Source: Have worked in HFT the past 10 years. This is not about High Frequency Trading, but about machine learning applied to the stock market.

SEJeff 10y ago
Which every single HFT firm that isn't a pure market maker (such as Virtu Financial, my previous employer) does already and has done for years. That is my point. Here's a great example of Citadel talking about how they use Hashicorp nomad to spin up 100,000 cpu research jobs on AWS. What do you think their 100k cpu jobs are doing? Machine learning and quantative analysis, what else? https://www.youtube.com/watch?v=MRtRwhL5lwM https://www.youtube.com/watch?v=MRtRwhL5lwM
- jbmorgado 10y ago
  I think you are willingly failing to understand what is being said by me and more people on the comments. This is not about exploiting a privileged access to market conditions (as in the sub millisecond access HFT firms have) in order to gain an advantage. This is about using machine learning to predict future (as in hours/days) moves on the market and take advantage of them.
  - SEJeff 10y ago
    Yes and I think you are willingly failing to understand reality from someone who has worked in technology in finance for a long time. HFT or perhaps more accurately, Electronic Trading, is more than simply passive trading and "doing something faster than everyone else". That is a part of it absolutely, but the industry also hires people with PHDs in quantum physics, or builds supercomputers at national labs to build private ones, or teach mathematics at ivy league universities. For much of this math, speed is beneficial, but it isn't universally and always required. Speed is nice, but smarts are better. "Machine Learning" is the new buzzword for a way to mathematically deducing future outcomes based on data science and quantitative analysis. That is a grotesque simplifications and there are specific ways of approaching this (Tensorflow/Keras from Google, Torch from Facebook, etc). This could be nanoseconds in the futures, it could be hours, or even weeks / months. And this is the part you're failing to understand by trying to assume (incorrectly) that all HFT is made up of is bid sniping and latency arbitrage (often referred to as front running). It isn't really that privileged access if you yourself can pay any exchange, such as NYSE, for a feed to your house. TL;DNR: Your understanding of electronic / HFT trading is wrong. If you try to understand it first, you'll realize your statements bear absolutely no basis whatsoever in facts. Facts are that Electronic Trading firms have been using machine learning literally since the invention of machine learning and quantitative analysis to make money. A new firm trying to do this can simply join the club.

digler999 10y ago

> Source: Have worked in HFT the past 10 years. OT question: how would I get into acquiring market data to try my own backtesting strategies ? Is there anyway to "get in" without putting up $5k + for the data ? I'm not green to this, I worked at a prop shop for a year writing market feed handlers. I know the dangers of overfitting data and how hard it is to make money (so I'm not deluded by a fantasy). I want to start out just for fun doing backtesting and see if I can scrape out a (paper) profit, but accessing historical data seems to be the hard part. I dont think online services that offer this would work for me because I want to use intraday data and replay many times with different parameters. I want access to "the whole firehose" if you will.

kybernetikos 10y ago
Not a personal recommendation as I haven't used it, but you might find quantopian.com fun if you've got any python.
hendzen 10y ago
You are not going to get usable intraday data for free. Best bet is to pony up for NYSE TAQ. http://www.nyxdata.com/Data-Products/Daily-TAQ http://www.nyxdata.com/Data-Products/Daily-TAQ
pliny 10y ago
TickData[1] are cheap and good for US data, I have had bad experience with them for EU data. They have a minimum of (IIRC) $1K but you can get a ton for that. Some exchanges offer free historical data, for instance BM&F BOVESPA[2], trading in some markets (such as Brasil and China) is often difficult if you are a foreigner. [1] https://www.tickdata.com/ https://www.tickdata.com/ [2] http://www.bmfbovespa.com.br/en_us/services/market-data/historical-data/ http://www.bmfbovespa.com.br/en_us/services/market-data/hist...
jmalicki 10y ago
IQFeed has many many years of historical data at the minute level, and tick data for 6 months (for equities), at ~$80/month... they many not have the highest quality data from what I've heard, or the most tick data, but are so much cheaper than any of the more professional solutions a large fund would use that it's a pretty decent start for a hobbyist.