6 ms·
True high-frequency is very hard. Most of these guys got started in the 1999/2000 time frame and didn't have to deal with many of the start-up issues that new e
by lrm242 16y ago
True high-frequency is very hard. Most of these guys got started in the
1999/2000 time frame and didn't have to deal with many of the start-up
issues that new entrants face. For example, just consider the data you
need to trade that quickly. Not only is the real-time feed expensive,
but if you want to source the data from the execution venue then you
have a lot of code to write. The amount of data is also quite large. For
example, the US equities market executes approximately 45 million trades
per day and there are about 700 million quotes per day, not including
the non-top of book quote activity in the various ECNs. I collect
between 2 and 4 GB of data per day, if you were to go direct to the
execution venues you'd be looking at >50 GB per day, probably.
So, anyway, the "true" high-frequency game is very tough today because
we're already 10+ years into it. The markets have changed and the edge
has gotten smaller, but there is still plently for a lowly individual
automated trader to scratch away at.
Instead of looking for millisecond opportunities, look for second or
minute opportunities. Go where the big guys can't because there isn't
enough capacity. Can you find an edge that, on average, keeps you in a
trade 30 seconds, for example? There's still plenty of alpha left, just
don't step onto their playground and expect to get onto the swing set.
A few practical notes:
* You need to look into unbundled or cost-plus commission structures.
These fee structures charge a per share commission and pass through
all fees and rebates from the executing venue. This is required to do
any sort of size with reasonable cost.
* Most "retail" brokers are not sufficient for any sort of high-volume
algorithmic trading. Interactive Brokers is barely ok if you are in
any way interested in limit order trading because they have fairly
large cancel fees for direct routed orders. If you're model doesn't
rely heavily on strictly offering liquidity or you're ok with letting
IB route your order then IB is ok and offers an unbundled commission
structure. Lightspeed Trading and Lime Brokerage are two that cater to
active individual and institutions.
* Data storage is a big deal. Effective storage of regular (evenly
spaced) and irregular time series will require you to engineer
something. There are commercial solutions, but you can't afford them.
When you're dealing with high-volume intraday trading this is one of
the first issues you'll face. How do you store, query, and manipulate
data that includes 45 million new rows per day? The relational DBs
fall apart pretty quickly and even if they didn't they won't give you
the time series operations you need/want.
* Data feeds are expensive, but required. Look at DTN NxCore. It is a
full market feed that will give you the best you can get w/o going
direct to the exchange. Some brokers will give you access to raw
exchange feeds, but you'll need to engineer feed handlers and a ticker
plant for them. This is a non-trivial task, but not impossible. Once
you've done that, you'll need to figure out how to get all that lovely
data off your co-located server and back to your home base for
analysis. Network engineering will be required because your broker
doesn't want you pushing 10-20GB per day through their network
connection, so you'll need a circuit from an on-premise carrier.
* The banks are players in HFT, but not the original or best. Most of
the guys that started it are still independent. Look at GETCO, RGM
Advisors, etc.
* Flash orders, what most folks in this thread are refering to when they
say the exchange gives the HFT firm a first look, are no more. That
edge existed, and I'm sure HFT took advantage, but no HFT firm was
built on flash orders. When they started flash orders didn't exist.
It is possible to be a successful, independent, automated trader. It is
even possible to do it on a purely intraday, high-volume basis. Don't get caught up
in the hype of needing to be high-frequency or not.
---
EDIT: The other thing I forgot to mention is that naked sponsored access is likely going away. This is a near-requirement for "true" HFT. Any future regulation won't affect any of the existing players because they're all grown up now and most have their own broker/dealers. Some form of sponsored access will likely still exist, but pre-trade risk checks will probably be required and will therefore still leave you're broker between you and the market.
- keltex 16y agoNPR's Podcast Planet Money has an interesting show on this subject called "The Million Dollar Microsecond": http://www.npr.org/blogs/money/2010/06/08/127563433/the-tuesday-podcast-the-million-dollar-microsecond http://www.npr.org/blogs/money/2010/06/08/127563433/the-tues...
- anigbrowl 16y ago(very off topic) How do you store, query, and manipulate data that includes 45 million new rows per day? Perhaps one shouldn't? I've always been a bit fascinated by the analytical tools in brokerage software - with a pretty good understanding of DSP and an appreciation for the fact that asset prices are somewhat periodic, it's hard to overlook the fundamental similarities between stock graphs and audio waveforms. Once you start performing FFTs or wavelet transforms and get a 'feel' for dealing with signals, patterns become very seductive...possibly too seductive: http://en.wikipedia.org/wiki/Pareidolia http://en.wikipedia.org/wiki/Pareidolia and http://en.wikipedia.org/wiki/Tetris_effect http://en.wikipedia.org/wiki/Tetris_effect Might there be another approach? You wouldn't prepare to go to the store by reviewing and analyzing the 1287 individual footsteps of your previous trip, or try to predict the content of a HN thread by textual analysis of all previous threads. Do we do so at a subconscious level, then? Not really - or rather, our subconscious tends to forget about things as soon as they cease to be important, which in the case of things like walking is a period of seconds or less. Processing large volumes of data is computationally expensive, but it turns out that simple rules can yield results that are both complex and useful, as in flocking and swarming behavior: see http://en.wikipedia.org/wiki/Boids http://en.wikipedia.org/wiki/Boids and http://en.wikipedia.org/wiki/Swarm_Intelligence http://en.wikipedia.org/wiki/Swarm_Intelligence, plus everything from the wisdom of crowds to nonlinear dynamic systems (aka chaotic ones) like Newton's basin or the logistic equation. I feel there are two other fundamental problems with the massive dataset + analysis approach. One is that you're not working in a closed system, and there's no sensible way to quantify unexpected events. 'Bigcorp CEO in Sex Scandal!' might cause the price of Bigcorp to tank if it's a major distraction or their largest customer base is among rural conservatives. If Bigcorp makes racing cars, it might just be good publicity! Now you can do some kinds of interesting posthoc analysis (eg for news stories that contain a stock symbol, measure the correlation between # of textually similar stories and stock volume/prices using a distributed windowing function) but we're a long way from having a browser plugin that trades based on the contents of your RSS feed. Another problem is that of feedback. As you've discussed so ably above, people who spot an arbitrage opportunity will mine the hell out of it. And as we all know, traders are extremely subject to herding behavior even though all training suggests they do otherwise. Sure there are systematic contrarians, but I bet that if you just want to do academic analysis you could find a contrarian coefficient and quantify its damping effect on price or volume movements. So rather than crunching vast quantities of stored data, I wonder if it might be better to treat price movements not as absolutes which you hope will reach a particular ceiling or floor, but as differential vector data with a short half-life. So far AI and modeling approaches seem to have focused on prediction (surprise) and don't perform especially well. I think it would be more interesting to map correlation variations for as large a number of nodes (listed securities) as possible - think how we intuitively appreciate the dynamics of a school of fish when watching a nature documentary, without performing any detailed analysis of individual fish trajectories. Of course this still involves processing a lot of data, but storing it is less important because are only seeking to become more familiar with high-level behaviors inside that system. There's more to fishing than running trawlers!