11 ms·
High Frequency Trading Development Kit
- eftpotrm 16y agoIn all seriousness, I would rather see a market designed explicitly around the barring of high frequency trades. All trades taking a fixed and relatively long period (say, 30 minutes) to complete. This sort of investment strategy doesn't seem to be creating long-term market value, it's exploiting short-term edge effects in a zero-sum strategy. The arms race to pursue ever more inhumanly fast automatic trades seems to have no net value at all yet is consuming resources to impoverish the algorithmically and connectively poorer at the expense of the richer. Markets seem to be favoured because they allow prices to find their natural level from supply and demand. How is this facility helped by permitting trades to be completed and then reversed faster than any human can ever perceive? This is algorithmic gambling on edge effects far more than a true market.
- secretasiandan 16y agoWhat information/experience informs your critique of markets/HFT?
- vecter 16y agoI can see from your post that you're very misinformed about how markets operate and the role of high frequency traders (a.k.a. liquidity providers). Read my response to a previous comment: http://news.ycombinator.com/item?id=2164458 http://news.ycombinator.com/item?id=2164458
- ldng 16y agoThey're just piggybacking trend and making profit of it, that how there used. Maybe not why and how they were designed. Still, IMHO, it is not fair and goes against the very definition of stock exchange.
- uwsnyc 16y agoHigh frequency traders are simply are able to provide the most competitive price in the market. They usually do this with a strong hunch that they will be able to get out of that position in a short period of time at a better price. Are they providing a service? Absolutely. When HF traders are involved in a transaction, risk is transferred to them. No matter how short the interval is that they're trading on, they're still taking on risk. The party trading against them doesn't have to worry about what way the market is going to move after the trade is complete and they are willing to pay for that.
- tricky 16y agoI have nothing against HFT but I don't buy your liquidity argument. Liquidity is needed most when markets are falling outside the norm. Any algorithm with a fail-safe or kill switch will immediately shut off when times get bad thereby ending their contribution to liquidity at a time when it is needed most. Please help me understand this better if I'm incorrect. (edited for clarity)
- secretasiandan 16y ago"Liquidity is needed most when markets are falling outside the norm" True but humans have fail-safes and kill switches too, thus this is not a valid argument against HFT in favor of human market makers. At any time, humans could step in and provide liquidity during the flash crash and during other crashes.
- tricky 16y ago"At any time, humans could step in and provide liquidity during the flash crash and during other crashes." Humans can, and do. But algorithms can't and won't.
- vecter 16y agoShort answer: there is a large demand for liquidity throughout most of the trading day. I'll give you the long answer when I get back from work.
- vecter 16y agoThe main point that you're missing is that there is a huge demand for liquidity during a majority of market hours. On average, 7 billion shares of US equities are traded every day. That is a monumentally large number if you think about it. According to the TABB consulting group, roughly 50-70% of American stock trades are done by HFT [1]. Let's assume that 10-20% of trading is non-profit motivated (utilitarian, you could say). That means that roughly 1 billion shares are exchanged every day by utilitarian traders. That in itself should be clear evidence that there is a large demand for liquidity. HFT market makers play an important role in those transactions. Specifically, they make it cheaper to buy and sell stocks by (1) tightening the bid-ask spread and (2) providing more quantity at each price, so that the average cost of executing an order is less. Not only that, when markets become tighter, they actually enable transactions to occur that would not have happened before. In other words, previously where buyers and sellers would NOT have traded because the transaction costs of crossing the bid-ask spread were too high, those two parties can now trade. Specifically, without HFT, there would be far fewer than 1 billion shares traded by utilitarian traders on a daily basis. To address the other point of providing liquidity "when the markets need it most", let's take a step back. When you say that "market makers should step in to provide liquidity [for society's benefit]", you're implying that there's some externality to lack of liquidity in financial markets (if so, this is yet another reason that we need HFT on a daily basis). Suppose that this is the case: there is some negative externality to society when markets are illiquid, as is oft to happen when things go crazy in the world. During those times, volatility is insanely high because the risk of being in any position is also insanely high. Remember, market makers get compensated (on average) for holding risk that you don't want. If risk is higher, naturally, the compensation should be also. This is manifested in higher costs of execution: spreads widen and the available quantity at each level decreases. If you want to force HFT market makers, which are private corporations, to step in to provide more liquidity, then you are forcing these companies to pay for that externality. In effect, they would take on huge risk for far diminished expected returns. That doesn't make any particular sense to me. However, if society as a whole has this view that some private corporations need to pay for public externalities, then that should be a matter of regulation. But if that's the case, why pick on HFT in particular? Why not force McDonalds and Whole Foods to give food to hungry people during famines? Surely, there is an externality to the food industry NOT stepping in during periods of extended hunger, "just when people need it the most." [1] http://georgewashington2.blogspot.com/2010/10/yes-70-of-us-equity-trades-are-high.html http://georgewashington2.blogspot.com/2010/10/yes-70-of-us-e...
- socillion 16y agoConsider a hypothetical stock currently trading at 34.05 bid, 34.06 ask. Trader A puts in a bid for 100 @ 34.06, matching Trader B's ask of 34.06. Normally, B would then sell to A, but thanks to HFT, institute C has the unique and intrinsically unfair position of being able to fulfill A's order. Assuming a moderate holding of stock, you are then able to manipulate the market up and down second-to-second by selectively fulfilling buy or sell orders and so causing supply or demand to pile up temporarily. In a zero sum game where knowledge is power, you know exactly who is buying and who is selling - before John Q. Public knows. To compound this, you can fulfill orders using immediate or cancel (IOC) orders before they hit the markets - making it impossible for others to know how much or even whether you are buying/selling.[1] In itself, there is really not much wrong with this - market makers have been doing this for decades, possibly even centuries. And as long as there is a level playing, where everyone sees orders at the same time, there really is no problem - an invaluable service in the form of liquidity is provided. Having the advantage of being able to front run the markets is not limited to just market making, however. Limits are one of the ubiquitous features of stock markets - the ability to say, I will buy or sell X shares of this stick up until it hits price Y. This can be exploited easily by HFT/front running - you have 30 ms [2] to probe out where the limit is with small IOC transactions. In this way, you are able to meet orders at prices that are different than the currently trading price - effectively bypassing all competition. Just to respond to one of your statements from the linked post: > A common misconception is that high frequency trading is like operating a money printing machine... This is false. Peruse this Forbes article, if you will - http://blogs.forbes.com/afontevecchia/2011/02/15/bernanke-put-allows-jp-morgan-to-post-profits-on-96-9-of-2010-trading-days/ http://blogs.forbes.com/afontevecchia/2011/02/15/bernanke-pu... I am interested in any civil responses to this. I am not professing to be any sort of expert on HFT. [3] Normally I would not bother responding to posts like this, preferring to avoid talking to anyone, but your slightly condescending tone was infuriating. 1. This may not be correct, in any case, the IOC would only be of the required size and so not provide any indication of holding size. 2. http://rusingwithronestar.blogspot.com/2009/07/goldman-effectively-front-running.html http://rusingwithronestar.blogspot.com/2009/07/goldman-effec... Cannot find any mainstream media confirmations that it is indeed 30ms, but this number seems well corroborated. 3. I have used HFT and front running as synonyms because they seem to be inescapably tied together. Why the need to have picosecond advantages if you aren't trying to beat another bottom feeder to the prey? edit: bah, I really dislike how HN treats newlines - I can't find any way to simply start a new line without a space in between the two sentences. Comes out like a block of text at 1366x768, or ends up looking too spread out.
- dlauer 16y agoActually this is a far better explanation of High Frequency Trading: http://www.thedailyshow.com/watch/wed-september-30-2009/cash-cow---high-frequency-trading http://www.thedailyshow.com/watch/wed-september-30-2009/cash...
- dnsworks 16y agoAm I the only one who read the URL as Stoner I'd Getechnology.com?
- swah 16y agoI wonder what exactly the FPGA does, and how they can make it practical and flexible. Of course they could always just test with C and then compile that to HDL.
- tghw 16y agoA month or two ago I spoke to someone working in these shops. Apparently, they're using the FPGAs to run many of the high-bandwidth algorithms, because going from NIC to motherboard to CPU and back again added too much latency. So they program the algorithms into the FPGA (which has its own Ethernet port) so it can receive, process, and send the data without ever hitting circuitry not dedicated to the task.
- cdavid 16y agoThis seems a bit surprising: what is the latency coming from the NIC to the motherboard ? I somehow doubt the latency due to distance within one piece of hardware matters (current CPU cannot do much more than a few 1000s cycles for the electricity to go around a few times in a motherboard, which is nothing to do any kind of advanced processing).
- bnoordhuis 16y agoThe problem is that a high-bandwidth link overruns the CPU with thousands of interrupts per second. It can paralyse the system, where the CPU is so busy handling I/O that it doesn't get around to actually scheduling processes. Disclaimer: I sometimes consult for HFTs but we don't use FPGAs (yet).
- cdavid 16y agoSo it is more an architecture issue with conventional hardware ? It makes much more sense to me with this explanation - the issue is the same as in most real-time systems then (like in embedded hardware for audio gears, which is more familiar to me than HFT). Don't things like real time linux or soft real time patches for linux help, though ?
- swah 16y agoAnyone knows how much something like this costs? My guess is around 20K.
- brownegg 16y agoMost solutions of this type are an order of magnitude higher than that. It's very easy to spend $20k/mo. To give an idea of scale / endpoint, a top-tier HFT infrastructure costs on the order of $100k/day. There are probably only a half dozen total of those in the world, though.
- swah 16y agoOuch! That's a great "incentive" for the system to work well!
- philwise 16y agoWhy is this interesting?
- JonnieCache 16y agoSee this article for an explanation of HFT: http://arstechnica.com/tech-policy/news/2010/08/the-sorcerers-apprentice-but-with-money-another-look-at-hft.ars http://arstechnica.com/tech-policy/news/2010/08/the-sorcerer...
- profquail 16y agoIt looks like you got a few downvotes because people assumed you're being snarky (maybe you are?) -- but I think that most people (even hackers) who work outside of the financial sector are going to wonder why this is on the front page. In short though, it's a 1U blade server with a full-size FPGA in it; implementing your algorithms in hardware makes them extremely fast (thus the mention of the C-to-HDL compiler), which is obviously of critical importance in high-frequency trading.
- ortatherox 16y agoThanks, I came to ask that question also. So is there any reason that this can't be done using OpenCL instead?
- chollida1 16y agoWe've tried using OpenCL for pricing, the biggest bottle neck is getting data into and out of video card memory. It's just too slow to move it from main to video memory and back. Also OpenCL works best when working on large pieces of data at once, with High frequency trading often you want to work on very small pieces, 50 bytes, and process them as fast as possible. Batching up the data and working on it in a SIMD type fashion just takes too long:(
- dman 16y agoAPU's should fix some of that.
- keithwarren 16y agoI am waiting for one of these shops to open up the opportunity to hobbyists/devs. I would love to try and create my own decision/trade algorithms but dont have the kind of money it would take to mitigate the cost effect of the trade volume (ie, at $8 per trade and making 1000 trades per day with an average return of 2% per day you would need to be working with more than 400K to even cover your tx cost). Strikes me that it would be a great research incubator for them to understand/learn new ideas and techniques and hunt for talent.
- bwhite 16y agoThere are plenty of prop shops that would let you trade with a very modest (low 5-figure or even 4-figure) dollar amount. Some will leverage your money, some will flat-out stake you in addition to your input. These places have the benefit of offering you relatively low costs -- minuscule compared to the cost of a retail trade at your average discount broker -- but they do not necessarily have the sort of investment in supercharged hardware. It is a good thing that your actual trading costs are not anywhere near those at the retail brokers you have in mind. 2%/day is an outrageously huge return. $1,000 at 2% compounded 250 times (approximate number of trading days in a earth year) is over $140k; you would return 14,000% annually. The bottom line is that it is harder than it looks.
- keithwarren 16y agoIf there are plenty, could you please point one out?
- bwhite 16y agoStart at: http://www.traderslog.com/proprietarytradingfirms/ http://www.traderslog.com/proprietarytradingfirms/ Some of these are purely manual trading but many have software APIs.
- roel_v 16y agoSet it up, trade 'virtually' (i.e., just simulate), prove your algos, get bankrolled by someone who is convinced of the algos worth. Quite straightforward. I'm highly skeptical of a hobbyist beating the competition though (maybe in micro niches, where one would add domain specific knowledge to get an edge - that's my pet idea on this topic :) )
- charlief 16y agoEven if this product allowed you to have picosecond processing of market data and generation of trading signals, if you don't have exchange co-location, inherent network latency in your market data from providers and order routing to exchanges will make this thing pretty useless. For a hobbyist, throw leased lines out the door, now a Linode box, even a desktop running Windows XP with full-blown Nagle's algorithm, would have good-enough performance unless you can pour serious capital into a project. They're also very vague on the software architecture or framework "kit". They don't say which feeds they support or if they provide any domain-specific data structures/standard trading algorithms like VWAP (slicing an order up over the day to give you "better" fills). I do see mention of an optional Feed Parser reference design, which I can't ascertain much value from. I am guessing it is mostly a generic kit for writing on their specialized hardware, and not specific to trading. There will be a lot of pain, maybe impossible to take vendor or open source libraries for market feed handling and compile it down using their embedded libraries. You'll be stuck doing tons of TCP/IP work, fun for hacking, but relative to the latency issues in your market data/order routing, it won't help you profit. Most people buy these appliances with included feed support. With a big budget and co-location, this may be useful in bootstrapping some really low-level processing of market data, although there are a variety of hardware and software messaging solutions that would be a good alternative in driving latency lower if you have some money, Solace (HW), Tibco, 29West, etc to name a few. I wish there was a hobbyist solution out there, but sorry, my initial impression says this isn't it. I would like more information, but maybe there are better hobbyist efforts out there. Interactive Brokers API isn't bad, and here's a full-blown real open-source trading kit that has a similar business model to MySQL. http://www.marketcetera.com/ http://www.marketcetera.com/ Note: I feel that there are a few issues with this as well in some of the technologies they chose (JMS, Eclipse RCP).
- mneedham 16y agoDespite all of this, I find the most worrying thing about the recent HFT developments is the simple fact that hobbyists want to get involved. Surely turning amateurs loose in such a necessarily complex world will result in large sums of money being lost. Oh wait... I get it.
- tricky 16y agoWhat's that quote that says something like in a gold rush you're better off selling the picks and shovels than trying to strike gold?
- deleted 16y ago[deleted]
- tlrobinson 16y agoSaw this quote most recently in a Chris Dixon post: http://cdixon.org/2011/02/05/selling-pickaxes-during-a-gold-rush/ http://cdixon.org/2011/02/05/selling-pickaxes-during-a-gold-...
- Gaussian 16y agoEven with this solution and well-designed code, a trader needs external infrastructure that plays on the same low-latency level. Most HFT strategies aren't as secret as you think; there are usually dozens of shops running similar trades. For that reason, something like this needs to be jacked directly into an exchange co-lo or connected through dedicated fiber. In short, the cost of this blade is just the start if you want to play with the gunslingers. Somebody else mentioned that there are prop shops that will give you the infrastructure you need for a cut of your profits. Some will even front the cash you need to trade. One such place is HTG Capital Partners of Chicago, which is where much of the HFT world is centered.
- shn21 16y agoAnd that's exactly what the world needed. As if Wall Street was not enough to destroy the world. I also question the validity of posting this link to where entrepreneurs, a.k.a value creating people, frequent.
- veyron 16y agoMy guess, as is usually the case with these HFT technology vendors, is that this company is trying to sell the residual technology IP from a hedge fund that bellied up. To those "hobbyists", what would you look for in a solution?
- rauljara 16y agoVirtually no hobbyist would be able to compete against big financial firms' hardware and algorithms. It might be a smart move for a large high frequency trading firm to put out these sorts of tools for free. An incredibly easy to use free api with good (though not superior) speed would be pretty popular. More subpar competition == more money for them. Though, admittedly, probably nowhere near what they are making now.
- spitfire 16y agoAlready exists, It's called ETrade.
- yummyfajitas 16y agoVirtually no hobbyist would be able to compete against big financial firms' hardware and algorithms. This simply isn't true. There are many HFT firms started by hobbyists/amateurs. The company I work for started this way. HFT is a field which has many niches and relatively low barriers to entry (thought certainly not as low as renting a linode).
- SkyMarshal 16y agoYeah, they're running out of marks and suckers, so it's time to get the average joe into the HFT game, just like they did with stocks and currencies.
- tlrobinson 16y agoIs there something like AWS for automated trading? Located extremely close to the exchanges and offering specialized hardware like this?
- rgarcia 16y agocertain exchanges are starting to standardize access to colocation(for example: http://www.cmegroup.com/globex/trading-cme-group-products/co-location-services.html http://www.cmegroup.com/globex/trading-cme-group-products/co...), but there is nothing that is exchange-agnostic that I know of.
- ShabbyDoo 16y agoSo, if we limit our consideration of HFT to strategies which rely primarily upon speed-based exploits (trade decisions based on the examination of current market data w/o correlation to external data sources like, say, Twitter feeds for market "sentiment"), can we conclude from the trading stack proposed for this box that most arbitrage opportunities have been exhausted by fierce competition? If the current state of the art is measured in microseconds, how much more room is there for improvement? I understand that there's a whole world of data streams from which one might attempt to predict markets, and I am explicitly limiting my question to the genre of trading which seems to be profitable via latency reduction alone.
- astrodust 16y agoThat there's a market for products like this is a very bad sign. How many parasites can the market handle before it becomes completely ineffective?