9 ms·

Mark Cuban: High-Frequency Traders Are the Ultimate Hackers

104 points by adventureful 14y ago

ezl 14y ago
sigh i got really excited about this because I was considering the interpretation of the word "hackers" like we use it in "Hacker News" (relevant: http://paulgraham.com/gba.html http://paulgraham.com/gba.html) I know that the startup community at large doesn't largely respect short term trading, particularly HFT. I understand it, though I honestly think its misguided and the outrage is disproportionately large, but thats another story. The reason why this article TITLE excited me was because I think its totally true, or least close to true. I think awesome HTFs and algo traders have a very similar skillset to top notch startup hackers. Small prop shops ARE startups. You: 1. wear a lot of hats 2. release early, release often 3. write really fast code, really fast 4. validate theories against the market (or consumers) 5. if you're right, keep going, if not, pivot 6. success is dependent on understanding lots of different sub-specialties 7. lots of overlapping skills: big data, distributed computing, machine learning At some point, I'd love to get a bunch of hackers together and start a small prop shop. The jobs are different, but in so many ways it takes the same kind of person to succeed at either.
- kjhughes 14y ago
  Definition of prop shop, for those unfamiliar with the term: http://www.investopedia.com/terms/p/prop_shop.asp http://www.investopedia.com/terms/p/prop_shop.asp
- theorique 14y ago
  Exactly my thought - I guessed that MC was praising us. Sigh, you'd think we HFT practitioners were, I dunno, Goldman Sachs or something like that. ;) (jk, GS, you know we love you ...) Good list, also. I work with some really smart programmers and tech people and HFT is a great laboratory to work on tough technical problems in the real-world, with real results.
- chewxy 14y ago
  I was very interested in HFT when I was in uni. Then I discovered the minimum sunk costs of running a HFT startup and my balls shrank. So now I create my own matching markets which is still based on high speed trading theories, but without the pressure. But if ever you're interested, my email is my username here at gmail
  - javert 14y ago
    Then I discovered the minimum sunk costs of running a HFT startup and my balls shrank. Care to elaborate?
    - chewxy 14y ago
      Everything matters if you're doing HFT (or back in the day, high speed algorithmic trading). In Australia, the support for algorithmic trading means you're required to house your server in ASX itself (ASX has recently introduced their HFT platform in 2011). And that meant ball-shrinking dollar values from place rental alone, not to mention the hardware and software times. So I concluded that it's not where I could be.
      - javert 14y ago
        And that meant ball-shrinking dollar values from place rental alone, not to mention the hardware and software times. I got the first part of that, but what do you mean by "hardware and software times"? Yes, HFT relies on fast hardware/networks/software. Was that your point, or is there something I'm missing? FWIW, despite a strong entrepreneurial bent, I would (possibly even will) look to join an existing HFT operation, rather than start my own. (Unless you want to do it in the Bitcoin world - there is a Bitcoin stock exchange now and it's exciting times, my friend!)
      - chewxy 14y ago
        as in time spent to tune your hardware and software. I do currently trade BTC. Not at a large volume though. My startups require more money than my hobby
      - 14y ago
- Retric 14y ago
  The problem with HFT algorithms in the eyes of most hackers is they are almost entirely a zero sum game. Assuming minimal volatility and free trades the absolutely dumbest algorithm* will make money. *Buy at X sell at X + 1cent with a simple ratchet of buy orders from 1 cent up to market price + a small number of shares to be sold if the market price increases. PS: Random walk that averages +/- one cent 10 times a second = 5 cent's profit a second assuming ~2,000 trading hours a year = 360k profit. Cost to make that for a stock worth ~20$ = ~30,000$ of capital + whatever it takes to get on the HFT floor.
- AznHisoka 14y ago
  I find HFT is a lot similar to SEO, and the perception is similar as well. People just don't like either because they feel it's not bringing any extra value to society, and it's all about gaming something for your own personal wealth. But most people aren't looking out for your personal interest anyway, so you should ignore them.
  - yaix 14y ago
    I disagree. SEO serves a purpose. It makes web sites more accessible to users, by correctly using HTML (title, h1, alt="", etc.) and it makes the web more machine readable and thus easier to find useful things. But I guess you mainly refer to the spammer SEOs that try to trick Google. HTF otoh does not serve any purpose whatsoever. On the contrary, it brings instability into the market and makes the market less useful. But my guess it that it will need a major market crash before HFT gets banned.
    - sseveran 14y ago
      SEO gets crappy websites into Google. HFT serves many purposes, mainly replacing human market makers with computers.
      - yaix 14y ago
        What positive purposes does HFT server, appart from making yourself money at the expense of the stability of the overall economy? """replacing human market makers with computers""" That is not HFT, that is just automated trading. The damaging part if the "HF", not the "T".
        skorgu 14y ago
        The structure of many markets causes trading to be latency sensitive, full stop. There's no bright line between automated trading and hft afaict.
        sseveran 14y ago
        No. The dominate strategies of HFT firms are market making and related arbitrage such as latency arbitrage between venues. There is nothing nefarious about having a computer do this in general as it is much more efficient than a human doing this. These strategies are not very complicated and instead require a lot of infrastructure to run. It used to be you needed to buy or rent a floor seat at the exchange to be able to run these types of strategies. Today that money goes to servers, hosting and engineers. Automated trading, such as statistical arbitrage would be lower frequency, although often still high frequency by most peoples standards. This algorithms often are liquidity takers from HFT.
- joe_the_user 14y ago
  Oddly enough, I was originally expecting that same thing. Still, I think it's reasonable to ask the question: where does the boundary between good, useful hacking and evil, insidious hacking lie? And I think it's reasonable to not assume that the boundary lies at what is legal or that the boundary lies at what is "interesting". I mean, how you produce and profit from spam is a hard, interesting problem. Flame and Stuxnet might, in some courts, be legal. In fact, a lot of "Black hats" satisfy many if not all of your list items. The argument that HFT is working with a nearly zero-sum situation needs to be addressed. A "normal" startup is usually trying to create "new usefulness". Where do "props" fit in to this question?
  - cynicalkane 14y ago
    Trading is a zero-sum game, but investing is not, and trading exists to make markets and snipe inefficiencies. The amount of money that can be made here can be surprising, until you consider that they're oiling the gears for manifold trillions of dollars in commerce and wealth. Some of that is gambling, but most of it isn't. Stuff like institutional hedgers and investment funds are huge sources of market activity.
    - jasonwatkinspdx 14y ago
      It's is very hard to see how predicting short term market position unrolling by individual institutions is making a market or sniping inefficiency. That is an arms race that creates a minimum cost to enter to compete. When success at sniping depends upon privileged network access, both human and technical, it's more a matter of profiting on an inefficiency you enforce upon everyone else.
    - Variance 14y ago
      I want to make an important note here: trading is not a zero-sum game. It generates value through the exact mechanisms that you noted, market making (liquidity) and inefficiency elimination (accuracy). Via the efficient market hypothesis, the value created by adding this asset liquidity and price accuracy finds its way back into the hands of the traders, often through real value increases in the assets going to traders with call positions. This functions to keep demand for trading moving by making the average market return stay close to or above the market discount rate. So stock trading doesn't end up as zero-sum, but rather benefits parties at the market discount rate. If it didn't, no one would trade but those seeking to gamble, and there would be no notion of fundamental analysis in the market. In reality, outside of an ideally efficient market, it's easy to intuitively suspect that the average return associated with trading is above the discount rate, and that more savvy investors obviously earn higher returns. This is more or less correct: the S&P averaged around 8.5% annual returns over the last few decades [1], and that only increases the farther you go back and lower the more recent your range's early bound is, corroborating the association between unideal market efficiency and high returns. Both "investing" and "trading" are less distinct in the stock market than in, say, VC, and so that 8.5% applies equally in aggregate. But trading is most certainly not a zero-sum game. Like all market activity, it converges to producing value at the market discount rate in an efficient market. What that rate is is incredibly hard to say, but it can vaguely be associated with S&P returns, which have hovered around 5-6% in recent years on time-weighted rolling averages. [1]http://moneyover55.about.com/od/howtoinvest/a/marketreturns.htm http://moneyover55.about.com/od/howtoinvest/a/marketreturns....
- jasonwatkinspdx 14y ago
  That you can, does not mean you should. And who pays for your failures?
  - chii 14y ago
    For a corporation, the psycopathic behaviour is that if it makes a profit and is not illegal (yet), then do it. The real problem is that large financial institutions have somehow ingrained themselves in society in such a way that they bring us all down when they fail, and so society would _have_ to bail them out. It is an ingenious stragegy - by aligning failure with the downfall of many others, these large financial institutions gets the protection they don't deserve. I wish it was me who was reaping the benefits, but unfortunately, being a "commoner", that aint so.
    - jasonwatkinspdx 14y ago
      It's interesting how many other areas of the economy have pulled off the same hat trick, but largely go unnoticed. Regulatory capture is a good marker because it's as conspicuous as it is essencial.
kristianp 14y ago
Here is Cuban's blog entry in which he states that HFT-ers are (the bad type of) hackers: http://blogmaverick.com/2010/05/09/what-business-is-wall-street-in/ http://blogmaverick.com/2010/05/09/what-business-is-wall-str...
kristianp 14y ago
I for one don't agree with HFT. Why should our best hackers and mathematical minds be wasted on something so shallow as gaming the market? Would a small randomised delay introduced by the exchange into each stock trade (or price datum) reduce the incentive for HFT?
- twoodfin 14y ago
  Why should our best hackers and mathematical minds be wasted on something so shallow as gaming the market? If you think they're wasting their time, don't give them your money.
  - drblast 14y ago
    I don't think it's as simple as all that. There are legitimate parties at either end of a stock transaction, and the HFT's are pretty much just siphoning off small amounts of the proceeds. The reason this can occur is that our government ensures that public companies have to do business this way. And all of these rules are in place for good reasons and it works pretty well, but HFT is an aberration. A good analogy is credit card companies; it's perfectly rational for a customer to use a rewards credit card and pay it off at the end of every month, and it's perfectly rational for a merchant to allow credit card transactions for convenience. Arguably, however, the credit card company is little else but a 3-5% drain on the economy, especially for brick-and-mortar stores. That said, credit card companies provide some value in the form of convenience, so maybe the transaction fee is worth it. HFT's, however, provide no benefit to anyone but themselves. The transactions would still occur whether or not the HFT existed. The distinction Mark Cuban makes between "investors" and "traders" is an astute one.
    - deleted 14y ago
      [deleted]
    - jamii 14y ago
      Brick-and-mortar stores themselves don't create anything. They buy in bulk and sell at a markup, thus adding liquidity to the grocery market. Imagine if you had to buy carrots by the tonne.
programminggeek 14y ago
I have no problem with HFT, but only because it is turning machines into things that are doing what humans did before, only at a speed that we can't keep up with as humans with our inherently physical interfaces. How is HFT different than traditional arbitrage except for the fact that it is faster and potentially at a scale that normal traders would never be able to keep up with? How is buying something one second and selling the next inherently different than buying one day and selling the next? Speed? Does computers being faster make it any more unfair than when one trader has access to millions or billions of dollars when I only have access to hundreds or thousands? I don't think so. Now, one could argue that HFT disrupts the way that Wall Street has worked for decades, but really, it is shifting the profit from traders, bankers, and fund managers to HFT firms. Sure, Wall Street probably doesn't like it, but does anyone like it when someone disrupts their market and takes their profit? You could also argue that HFT doesn't add value because HFT firms inherently don't do anything other than trade millions of stocks only for the sake of trading them, but how is this worse than day trading or running any other fund other than the turnover rate? Perhaps I'm blind to the real issues, but I just don't see a huge difference between HFT and traditional Wall St. trading, except maybe that it magnifies the problems inherent in the system due to speed and scale.
- wmf 14y ago
  The difference is that a crash can happen faster than any human can stop it. It's fine if you trust your high-frequency circuit breakers.
  - unabridged 14y ago
    What exactly is a flash crash and why should individuals care? Isn't it just bank's computers temporarily offering stock at a severe discount and then the price going back to normal. It doesn't hurt anyone but the people with poorly programmed algorithms. No trades should ever be broken just because someone can't control their trading bots.
    - mistermann 14y ago
      Are you familiar with stop loss orders?
evolve2k 14y ago
'Is the market supposed to be a platform for companies to raise money for growth and to create liquidity and opportunity for shareholders as it has been in the past? Or is the stock market a laissez-faire platform that evolves however it evolves? The missing link in all the discussions is: What is the purpose of the stock market?'
javert 14y ago
It's clear why we need investors. Why do we need day traders? Well, they can only exist because the investors exist, and they provide the function of pricing the market accurately. And provide liquidity to investors. Nobody is arguing that we should ban day trading. Why do we need market makers? Well, they definitely benefit the day traders, because they do lower the bid/ask spread, once again making pricing more accurate. And provide liquidity to day traders. So what I'm proposing is that we only "really need" investors - but day traders are good for investors, and market makers are good for day traders. Thus, market makers are good for investors. I'd be legitemitely interested to hear criticism of the above argument. I've been learning about market making recently, and I'm certainly sure that there's still more for me to learn.
- _delirium 14y ago
  I don't disagree in broad terms, but I think it's also quite possible that some configurations of trading will result in the "support" tail wagging the investment dog. At least, that would be the expected result if you think of markets like a dynamical system; in non-trivial dynamical systems (which I'd assume the economy is), the usual case is that there are significant internal dynamics (feedback loops, attractors, etc.) that in a number of cases will modulate, and may even dominate, the movements caused by external inputs. If that transfers to trading, there may well be pathological market dynamics that have little to do with the fundamentals of providing services to (ultimately) investors. Whether the dis-value of those pathologies produced outweighs the value of the services provided would then be the relevant question. Viewed as a machine, does the machine that includes the HFT lubricants run better in general, or is the reduced friction offset by a undesired effects such as greater risk of malfunction?
wikkiwa 14y ago
Here's an interview with a pro high frequency trader from a top shop if you're interested. http://techzinglive.com/page/1049/185-tz-interview-james-thomas-headlands-technologies http://techzinglive.com/page/1049/185-tz-interview-james-tho...
romansanchez 14y ago
Does anyone know if the software used for HFT is built in house or is there a particular vendor that specializes in this particular market?
- rurounijones 14y ago
  In-house. very very much in-house.
- josephagoss 14y ago
  From what I have read, its all in house, the software (including the mathematical algorithms) is the most important center point of a HFT shop. No vender would sell a competitive software system because they would make more money using it themselves.
yaix 14y ago
HFT is just another area where everybody knows that it is broken, it does not serve any useful purpose for society, and everybody knows that eventually it will lead to a huge crash. And nobody does anything, because there is just too much quick money to be made. Humans...
bokonist 14y ago
"As far as narrowing spreads, that’s absolutely true, but in absolute terms what does it translate into? For the individual investor it might save them a quarter a month." In a properly designed, information age stock market there should not be a spread. All stocks should trade via a programmed, black box auction that runs on an interval. The HFT practice of creating phony orders that are immediately canceled, just to gain visibility into the current bids and asks, would be eliminated. No seeing other people's bids, and then front running them, no canceling orders. You put in a limit order for the value of the stock, and the stock goes to the highest bidder. All the extra pennies and quarters go to the shareholder, nothing to the HFT algorithms (unless they provide actually value such as market making or smoothing irrational volatility). It's the most efficient design for stock market trading possible.
- freshhawk 14y ago
  > It's the most efficient design for stock market trading possible Precisely why it doesn't work that way.
  - bokonist 14y ago
    In theory, if this system were to be adopted, the Stock Exchange Operator could charge a small tariff so that some of the money that now goes to HFT's goes to the stock exchange, with the rest going to the shareholders. So in theory, adopting this more optimal system would be financially beneficially to both the corporations and the Stock Exchange (and in theory, those two are the only two players who's agreement is needed to change the system). So I wonder why the optimal, auction based system does not get adopted - is there some reason for the status quo other than the normal inertia and stupidity of large institutions?
    - freshhawk 14y ago
      > In theory Not a good theory. All the players in this game believe they are above average among players in this game. They all want the system to be exploitable because they all believe they will exploit it better than their peers. At the very least better than the investors (the real suckers here). You are falling into the game theory economics trap. People are highly irrational actors who don't even act in their own self interest most of the time, nevermind some form of optimal rational behavior.
ajtaylor 14y ago
"To say they’re adding liquidity is like saying spitting in a thunderstorm is adding liquidity." Best quote of the whole article!
ajays 14y ago
I keep hearing about HFT, and have read about it a little. But there's a surprising dearth of data. Can someone come up with a data set? For example: here are N inputs to the program. If the program can make a decision D in time T, then it can make money. Or something to that effect. I have no idea right now what the inputs to these HFT programs are; what the expected actions under the time constraints are; and how the effectiveness is measured (in terms of the given data). Someone please enlighten me.
- ajtulloch 14y ago
  Sure: 1. Data set - order book of your chosen stock exchange. Events being streamed in at rates up to 1Gb/s [1]. This is the entire set of actions affecting the order book - bids, offers, cancellations, adjustments, etc. - it's huge. If you want to get fancy - most do - you would typically pull in several of these feeds (or subscribe to a consolidated feed) containing several exchanges, and look to arb any price/book inefficiencies. 2. Actions - given the changes to the order book, figure out how you need to change your positioning to make money. Time is everything here - steps here are counted in tenths of milliseconds, and overall response times in low ms. HFT firms even try to minimise cable lengths within the colocated datacentres to shave the time to response down even further. 3. Effectiveness - if you didn't blow up, and if so, if you made money - typically you'd examine intraday PnL volatility, returns, etc. and assess yourself on those and other metrics.
  - ajays 14y ago
    How about releasing some data? Do some feature generation and come up with features, along with the desired action (the best possible action, given that you know how things are going to go in the future). Then us clueless types can see what all the hubbub is all about :-)
mkramlich 14y ago
HFT has some qualities which make it unattractive to me: - strategically, it's an unbounded arms race, due to competing against lots of other HFT forces all trying to squeeze profits quickly out of micro opportunities in bid/ask numbers, news leak/announce gaps, etc. - due to above, there are high technical and talent costs just to have the minimum setup needed to play in that game - also following from the first thing, HFT becomes an additional profit motive for adding destructive, dishonest political vectors to society, such as corruption and propaganda, in order to time trades, do front running, insider trading, war profiteering, bankster conspiracies, etc. - short-term thinking - lastly, you're fundamentally not building anything real or positive for the rest of society outside HFT and your direct clients/owners; or making the world a better place; to others, you're just a parasite on the money flows, adding more noise and chaos, and arguably risk, to the economy I like to compare it to the alternative, strategically-speaking: LFT (low f.t.) - lower costs (your fixed & marginal costs "scale down" better, which is also better in the startup phase. Think lemonade stand, a small simple web SaaS, contract development -> software product company, etc.) - longer-term thinking & bets (buy & hold; future trends, etc.) - actually building, investing, helping others, bringing new products/services to the market, solving problems for others for profit LFT could be used to describe traditional stock investing, it could include personal relationship investing, personal health/skills investing, building lifestyle businesses, building startups to flip, doing R&D, invention & marketing, etc. Both HFT and LFT can make you wealthy on the top end scenarios. But LFT scales down to the low end scenarios and early phases better, and you can tangibly and visibly see improvements to the world that you've made happen. Both can benefit from hacker thinking. But to me, it's clear which is the healthier choice for the individual and society, in the long run.
cletus 14y ago
This post is being used as somewhat of a stalking horse for any number of other agendas and soapboxes, which I guess isn't surprising, so I'll join the fray. 1. There is a misconception about market makers and the nature of a spread. In particularly illiquid markets, market makers allow you to buy and sell immediately. This is a service. The spread pays for this service. The more liquid a market is the lower the spread (eg it is essentially if not actually zero on US Treasuries); 2. For those questioning our brightest and best being "wasted" on HFT, I see comparatively better value on all the bright minds being wasted on social networks; 3. Finance is a constant arms race. The flavour of the month is HFT. In 5-10 years it'll be something else where low-latency transactions are just taken for granted. This kind of market is a gold mine for innovators as the existing players need a certain amount of innovation just to stand still (relative to the other players). There is classic "mine the miners" opportunity here; 4. There is a lot of highly-leveraged trading that goes on, automated and otherwise, beyond the world of HFT. (I believe) Warren Buffett describes this as "picking up nickels in front of a bulldozer" [1]. In extreme market situations, market models go out the window and the highly leveraged are the first to go to the flames. Just ask Bear Stearns [2] with its 35:1 leveraged ratio. 5. Financial institutions have access to way more data and cheaper (typically free) transactions compared to the average investor. It's common practice to do things like whipsaw the market to hit stop loss orders and the like. Honestly, unless you're a bank in the short term it is an insider's game; 6. A lot of the problems with the lack of transparency, assymmetry of information between parties and bad behaviour (like flooding exchanges) comes down to a failure of regulation. The SEC is either toothless or has been subverted by political interests. Much of what happens in the US just doesn't happen in Australia (to the same degree at least). Just take all the fake documentation in the US housing bubble. How bank executives, loan originators and the like didn't go to jail for this is really astounding; 7. Banks and market funds have limited to no downside risk. If things go horribly pear-shaped the US Federal government will--and has--bail them out. This breeds an adverse appetite for risk. I subscribe to the view that the Federal Reserve, the World Bank and the IMF are in large part welfare for investment bankers; and 8. Banks and funds are too large. If something is "too big to fail" then it's too big and needs to be broken up. That is all. [1]: http://en.wikipedia.org/wiki/Long-Term_Capital_Management http://en.wikipedia.org/wiki/Long-Term_Capital_Management [2]: http://en.wikipedia.org/wiki/Bear_Stearns http://en.wikipedia.org/wiki/Bear_Stearns
avstraliitski 14y ago
Real hackers use their powers for good.
infinii 14y ago
There's the sentiment that exchanges allow this because they are "businesses". What's to to stop exchanges from front running the HFT front runners without their knowledge and stealing all the marbles?
asto 14y ago
If a person is successful in one business should we subject ourselves to his idiotic views on another business? By definition they can’t go into an equity unless there already is liquidity. HFTs who are usually market makers [1] maintain liquidity in the market by constantly having buy/sell orders open for many instruments/scrips. In their absence, a counter trade may not exist for matching against when you or I put out a sell/buy order. Being able to sell/buy a scrip/instrument within seconds is a valuable service that investors appreciate. In India, the securities exchange regulator mandates that all market makers make a minimum value of trades per day. I am not sure of the exact current values in each segment but it's in the order of millions of rupees. This is how regulators create conditions of liquidity so that markets are nicer to trade in especially for retail investors. The missing link in all the discussions is: What is the purpose of the stock market? The stock market can be divided into a primary market and a secondary market. The primary market exists to finance companies. When they IPO, cash is generated which they can use to fund business activities. The secondary market exists to encourage purchases in the primary market and to efficiently discover the price of the securities. Without a liquid secondary market, very few investors would be interested in a primary market. HFT is only doing what was already done since many years ago manually. Humans make the same mistakes computers are being programmed to make today. In their defence, they execute exactly as asked - they feel no fear, doubt or greed - unlike a human trader . Should we treat them as being harmful because computers make mistakes at a significantly faster rate? HFT is a still nascent concept. With sufficient regulation, we will likely be able to get rid of the problems and retain the advantages. [1] http://en.wikipedia.org/wiki/Market_maker http://en.wikipedia.org/wiki/Market_maker
- JonnieCache 14y ago
  >hould we treat them as being harmful because computers make mistakes at a significantly faster rate? Yes. Significantly doesn't really cover it. When you increase the rate of an activity by many orders of magnitude and put an AI behind it, it becomes a totally different activity, even if the mechanics are notionally the same.
bogger 14y ago
MC is confused. High frequency trading covers a broad range of trading strategies, some of which are beneficial to market participants. In particular, were all automated market making to be banned, we would see a huge jump in the cost of trading. The fuss about HFT is distracting people from the much more serious problem of good old-fashioned insider trading, which remains rampant.
dm_mongodb 14y ago
it seems to me that the exchange adding a small random delay (say 1ms) on message reception would dampen the benefits of speed and then there would be less work invested in saving a few microseconds here and there. you still have program trading but a little bit of silly dynamics is dampened.

104 points by adventureful 14y ago

ezl 14y ago

sigh i got really excited about this because I was considering the interpretation of the word "hackers" like we use it in "Hacker News" (relevant: http://paulgraham.com/gba.html http://paulgraham.com/gba.html) I know that the startup community at large doesn't largely respect short term trading, particularly HFT. I understand it, though I honestly think its misguided and the outrage is disproportionately large, but thats another story. The reason why this article TITLE excited me was because I think its totally true, or least close to true. I think awesome HTFs and algo traders have a very similar skillset to top notch startup hackers. Small prop shops ARE startups. You: 1. wear a lot of hats 2. release early, release often 3. write really fast code, really fast 4. validate theories against the market (or consumers) 5. if you're right, keep going, if not, pivot 6. success is dependent on understanding lots of different sub-specialties 7. lots of overlapping skills: big data, distributed computing, machine learning At some point, I'd love to get a bunch of hackers together and start a small prop shop. The jobs are different, but in so many ways it takes the same kind of person to succeed at either.

kjhughes 14y ago
Definition of prop shop, for those unfamiliar with the term: http://www.investopedia.com/terms/p/prop_shop.asp http://www.investopedia.com/terms/p/prop_shop.asp
theorique 14y ago
Exactly my thought - I guessed that MC was praising us. Sigh, you'd think we HFT practitioners were, I dunno, Goldman Sachs or something like that. ;) (jk, GS, you know we love you ...) Good list, also. I work with some really smart programmers and tech people and HFT is a great laboratory to work on tough technical problems in the real-world, with real results.
chewxy 14y ago
I was very interested in HFT when I was in uni. Then I discovered the minimum sunk costs of running a HFT startup and my balls shrank. So now I create my own matching markets which is still based on high speed trading theories, but without the pressure. But if ever you're interested, my email is my username here at gmail
- javert 14y ago
  Then I discovered the minimum sunk costs of running a HFT startup and my balls shrank. Care to elaborate?
  - chewxy 14y ago
    Everything matters if you're doing HFT (or back in the day, high speed algorithmic trading). In Australia, the support for algorithmic trading means you're required to house your server in ASX itself (ASX has recently introduced their HFT platform in 2011). And that meant ball-shrinking dollar values from place rental alone, not to mention the hardware and software times. So I concluded that it's not where I could be.
    - javert 14y ago
      And that meant ball-shrinking dollar values from place rental alone, not to mention the hardware and software times. I got the first part of that, but what do you mean by "hardware and software times"? Yes, HFT relies on fast hardware/networks/software. Was that your point, or is there something I'm missing? FWIW, despite a strong entrepreneurial bent, I would (possibly even will) look to join an existing HFT operation, rather than start my own. (Unless you want to do it in the Bitcoin world - there is a Bitcoin stock exchange now and it's exciting times, my friend!)
    - chewxy 14y ago
      as in time spent to tune your hardware and software. I do currently trade BTC. Not at a large volume though. My startups require more money than my hobby
    - 14y ago
Retric 14y ago
The problem with HFT algorithms in the eyes of most hackers is they are almost entirely a zero sum game. Assuming minimal volatility and free trades the absolutely dumbest algorithm* will make money. *Buy at X sell at X + 1cent with a simple ratchet of buy orders from 1 cent up to market price + a small number of shares to be sold if the market price increases. PS: Random walk that averages +/- one cent 10 times a second = 5 cent's profit a second assuming ~2,000 trading hours a year = 360k profit. Cost to make that for a stock worth ~20$ = ~30,000$ of capital + whatever it takes to get on the HFT floor.
AznHisoka 14y ago
I find HFT is a lot similar to SEO, and the perception is similar as well. People just don't like either because they feel it's not bringing any extra value to society, and it's all about gaming something for your own personal wealth. But most people aren't looking out for your personal interest anyway, so you should ignore them.
- yaix 14y ago
  I disagree. SEO serves a purpose. It makes web sites more accessible to users, by correctly using HTML (title, h1, alt="", etc.) and it makes the web more machine readable and thus easier to find useful things. But I guess you mainly refer to the spammer SEOs that try to trick Google. HTF otoh does not serve any purpose whatsoever. On the contrary, it brings instability into the market and makes the market less useful. But my guess it that it will need a major market crash before HFT gets banned.
  - sseveran 14y ago
    SEO gets crappy websites into Google. HFT serves many purposes, mainly replacing human market makers with computers.
    - yaix 14y ago
      What positive purposes does HFT server, appart from making yourself money at the expense of the stability of the overall economy? """replacing human market makers with computers""" That is not HFT, that is just automated trading. The damaging part if the "HF", not the "T".
      - skorgu 14y ago
        The structure of many markets causes trading to be latency sensitive, full stop. There's no bright line between automated trading and hft afaict.
      - sseveran 14y ago
        No. The dominate strategies of HFT firms are market making and related arbitrage such as latency arbitrage between venues. There is nothing nefarious about having a computer do this in general as it is much more efficient than a human doing this. These strategies are not very complicated and instead require a lot of infrastructure to run. It used to be you needed to buy or rent a floor seat at the exchange to be able to run these types of strategies. Today that money goes to servers, hosting and engineers. Automated trading, such as statistical arbitrage would be lower frequency, although often still high frequency by most peoples standards. This algorithms often are liquidity takers from HFT.
joe_the_user 14y ago
Oddly enough, I was originally expecting that same thing. Still, I think it's reasonable to ask the question: where does the boundary between good, useful hacking and evil, insidious hacking lie? And I think it's reasonable to not assume that the boundary lies at what is legal or that the boundary lies at what is "interesting". I mean, how you produce and profit from spam is a hard, interesting problem. Flame and Stuxnet might, in some courts, be legal. In fact, a lot of "Black hats" satisfy many if not all of your list items. The argument that HFT is working with a nearly zero-sum situation needs to be addressed. A "normal" startup is usually trying to create "new usefulness". Where do "props" fit in to this question?
- cynicalkane 14y ago
  Trading is a zero-sum game, but investing is not, and trading exists to make markets and snipe inefficiencies. The amount of money that can be made here can be surprising, until you consider that they're oiling the gears for manifold trillions of dollars in commerce and wealth. Some of that is gambling, but most of it isn't. Stuff like institutional hedgers and investment funds are huge sources of market activity.
  - jasonwatkinspdx 14y ago
    It's is very hard to see how predicting short term market position unrolling by individual institutions is making a market or sniping inefficiency. That is an arms race that creates a minimum cost to enter to compete. When success at sniping depends upon privileged network access, both human and technical, it's more a matter of profiting on an inefficiency you enforce upon everyone else.
  - Variance 14y ago
    I want to make an important note here: trading is not a zero-sum game. It generates value through the exact mechanisms that you noted, market making (liquidity) and inefficiency elimination (accuracy). Via the efficient market hypothesis, the value created by adding this asset liquidity and price accuracy finds its way back into the hands of the traders, often through real value increases in the assets going to traders with call positions. This functions to keep demand for trading moving by making the average market return stay close to or above the market discount rate. So stock trading doesn't end up as zero-sum, but rather benefits parties at the market discount rate. If it didn't, no one would trade but those seeking to gamble, and there would be no notion of fundamental analysis in the market. In reality, outside of an ideally efficient market, it's easy to intuitively suspect that the average return associated with trading is above the discount rate, and that more savvy investors obviously earn higher returns. This is more or less correct: the S&P averaged around 8.5% annual returns over the last few decades [1], and that only increases the farther you go back and lower the more recent your range's early bound is, corroborating the association between unideal market efficiency and high returns. Both "investing" and "trading" are less distinct in the stock market than in, say, VC, and so that 8.5% applies equally in aggregate. But trading is most certainly not a zero-sum game. Like all market activity, it converges to producing value at the market discount rate in an efficient market. What that rate is is incredibly hard to say, but it can vaguely be associated with S&P returns, which have hovered around 5-6% in recent years on time-weighted rolling averages. [1]http://moneyover55.about.com/od/howtoinvest/a/marketreturns.htm http://moneyover55.about.com/od/howtoinvest/a/marketreturns....
jasonwatkinspdx 14y ago
That you can, does not mean you should. And who pays for your failures?
- chii 14y ago
  For a corporation, the psycopathic behaviour is that if it makes a profit and is not illegal (yet), then do it. The real problem is that large financial institutions have somehow ingrained themselves in society in such a way that they bring us all down when they fail, and so society would _have_ to bail them out. It is an ingenious stragegy - by aligning failure with the downfall of many others, these large financial institutions gets the protection they don't deserve. I wish it was me who was reaping the benefits, but unfortunately, being a "commoner", that aint so.
  - jasonwatkinspdx 14y ago
    It's interesting how many other areas of the economy have pulled off the same hat trick, but largely go unnoticed. Regulatory capture is a good marker because it's as conspicuous as it is essencial.

kristianp 14y ago

Here is Cuban's blog entry in which he states that HFT-ers are (the bad type of) hackers: http://blogmaverick.com/2010/05/09/what-business-is-wall-street-in/ http://blogmaverick.com/2010/05/09/what-business-is-wall-str...

kristianp 14y ago

I for one don't agree with HFT. Why should our best hackers and mathematical minds be wasted on something so shallow as gaming the market? Would a small randomised delay introduced by the exchange into each stock trade (or price datum) reduce the incentive for HFT?

twoodfin 14y ago
Why should our best hackers and mathematical minds be wasted on something so shallow as gaming the market? If you think they're wasting their time, don't give them your money.
- drblast 14y ago
  I don't think it's as simple as all that. There are legitimate parties at either end of a stock transaction, and the HFT's are pretty much just siphoning off small amounts of the proceeds. The reason this can occur is that our government ensures that public companies have to do business this way. And all of these rules are in place for good reasons and it works pretty well, but HFT is an aberration. A good analogy is credit card companies; it's perfectly rational for a customer to use a rewards credit card and pay it off at the end of every month, and it's perfectly rational for a merchant to allow credit card transactions for convenience. Arguably, however, the credit card company is little else but a 3-5% drain on the economy, especially for brick-and-mortar stores. That said, credit card companies provide some value in the form of convenience, so maybe the transaction fee is worth it. HFT's, however, provide no benefit to anyone but themselves. The transactions would still occur whether or not the HFT existed. The distinction Mark Cuban makes between "investors" and "traders" is an astute one.
  - deleted 14y ago
    [deleted]
  - jamii 14y ago
    Brick-and-mortar stores themselves don't create anything. They buy in bulk and sell at a markup, thus adding liquidity to the grocery market. Imagine if you had to buy carrots by the tonne.

programminggeek 14y ago

I have no problem with HFT, but only because it is turning machines into things that are doing what humans did before, only at a speed that we can't keep up with as humans with our inherently physical interfaces. How is HFT different than traditional arbitrage except for the fact that it is faster and potentially at a scale that normal traders would never be able to keep up with? How is buying something one second and selling the next inherently different than buying one day and selling the next? Speed? Does computers being faster make it any more unfair than when one trader has access to millions or billions of dollars when I only have access to hundreds or thousands? I don't think so. Now, one could argue that HFT disrupts the way that Wall Street has worked for decades, but really, it is shifting the profit from traders, bankers, and fund managers to HFT firms. Sure, Wall Street probably doesn't like it, but does anyone like it when someone disrupts their market and takes their profit? You could also argue that HFT doesn't add value because HFT firms inherently don't do anything other than trade millions of stocks only for the sake of trading them, but how is this worse than day trading or running any other fund other than the turnover rate? Perhaps I'm blind to the real issues, but I just don't see a huge difference between HFT and traditional Wall St. trading, except maybe that it magnifies the problems inherent in the system due to speed and scale.

wmf 14y ago
The difference is that a crash can happen faster than any human can stop it. It's fine if you trust your high-frequency circuit breakers.
- unabridged 14y ago
  What exactly is a flash crash and why should individuals care? Isn't it just bank's computers temporarily offering stock at a severe discount and then the price going back to normal. It doesn't hurt anyone but the people with poorly programmed algorithms. No trades should ever be broken just because someone can't control their trading bots.
  - mistermann 14y ago
    Are you familiar with stop loss orders?

evolve2k 14y ago

'Is the market supposed to be a platform for companies to raise money for growth and to create liquidity and opportunity for shareholders as it has been in the past? Or is the stock market a laissez-faire platform that evolves however it evolves? The missing link in all the discussions is: What is the purpose of the stock market?'

javert 14y ago

It's clear why we need investors. Why do we need day traders? Well, they can only exist because the investors exist, and they provide the function of pricing the market accurately. And provide liquidity to investors. Nobody is arguing that we should ban day trading. Why do we need market makers? Well, they definitely benefit the day traders, because they do lower the bid/ask spread, once again making pricing more accurate. And provide liquidity to day traders. So what I'm proposing is that we only "really need" investors - but day traders are good for investors, and market makers are good for day traders. Thus, market makers are good for investors. I'd be legitemitely interested to hear criticism of the above argument. I've been learning about market making recently, and I'm certainly sure that there's still more for me to learn.

_delirium 14y ago
I don't disagree in broad terms, but I think it's also quite possible that some configurations of trading will result in the "support" tail wagging the investment dog. At least, that would be the expected result if you think of markets like a dynamical system; in non-trivial dynamical systems (which I'd assume the economy is), the usual case is that there are significant internal dynamics (feedback loops, attractors, etc.) that in a number of cases will modulate, and may even dominate, the movements caused by external inputs. If that transfers to trading, there may well be pathological market dynamics that have little to do with the fundamentals of providing services to (ultimately) investors. Whether the dis-value of those pathologies produced outweighs the value of the services provided would then be the relevant question. Viewed as a machine, does the machine that includes the HFT lubricants run better in general, or is the reduced friction offset by a undesired effects such as greater risk of malfunction?

wikkiwa 14y ago

Here's an interview with a pro high frequency trader from a top shop if you're interested. http://techzinglive.com/page/1049/185-tz-interview-james-thomas-headlands-technologies http://techzinglive.com/page/1049/185-tz-interview-james-tho...

romansanchez 14y ago

Does anyone know if the software used for HFT is built in house or is there a particular vendor that specializes in this particular market?

rurounijones 14y ago
In-house. very very much in-house.
josephagoss 14y ago
From what I have read, its all in house, the software (including the mathematical algorithms) is the most important center point of a HFT shop. No vender would sell a competitive software system because they would make more money using it themselves.

yaix 14y ago

HFT is just another area where everybody knows that it is broken, it does not serve any useful purpose for society, and everybody knows that eventually it will lead to a huge crash. And nobody does anything, because there is just too much quick money to be made. Humans...

bokonist 14y ago

"As far as narrowing spreads, that’s absolutely true, but in absolute terms what does it translate into? For the individual investor it might save them a quarter a month." In a properly designed, information age stock market there should not be a spread. All stocks should trade via a programmed, black box auction that runs on an interval. The HFT practice of creating phony orders that are immediately canceled, just to gain visibility into the current bids and asks, would be eliminated. No seeing other people's bids, and then front running them, no canceling orders. You put in a limit order for the value of the stock, and the stock goes to the highest bidder. All the extra pennies and quarters go to the shareholder, nothing to the HFT algorithms (unless they provide actually value such as market making or smoothing irrational volatility). It's the most efficient design for stock market trading possible.

freshhawk 14y ago
> It's the most efficient design for stock market trading possible Precisely why it doesn't work that way.
- bokonist 14y ago
  In theory, if this system were to be adopted, the Stock Exchange Operator could charge a small tariff so that some of the money that now goes to HFT's goes to the stock exchange, with the rest going to the shareholders. So in theory, adopting this more optimal system would be financially beneficially to both the corporations and the Stock Exchange (and in theory, those two are the only two players who's agreement is needed to change the system). So I wonder why the optimal, auction based system does not get adopted - is there some reason for the status quo other than the normal inertia and stupidity of large institutions?
  - freshhawk 14y ago
    > In theory Not a good theory. All the players in this game believe they are above average among players in this game. They all want the system to be exploitable because they all believe they will exploit it better than their peers. At the very least better than the investors (the real suckers here). You are falling into the game theory economics trap. People are highly irrational actors who don't even act in their own self interest most of the time, nevermind some form of optimal rational behavior.

ajtaylor 14y ago

"To say they’re adding liquidity is like saying spitting in a thunderstorm is adding liquidity." Best quote of the whole article!

ajays 14y ago

I keep hearing about HFT, and have read about it a little. But there's a surprising dearth of data. Can someone come up with a data set? For example: here are N inputs to the program. If the program can make a decision D in time T, then it can make money. Or something to that effect. I have no idea right now what the inputs to these HFT programs are; what the expected actions under the time constraints are; and how the effectiveness is measured (in terms of the given data). Someone please enlighten me.

ajtulloch 14y ago
Sure: 1. Data set - order book of your chosen stock exchange. Events being streamed in at rates up to 1Gb/s [1]. This is the entire set of actions affecting the order book - bids, offers, cancellations, adjustments, etc. - it's huge. If you want to get fancy - most do - you would typically pull in several of these feeds (or subscribe to a consolidated feed) containing several exchanges, and look to arb any price/book inefficiencies. 2. Actions - given the changes to the order book, figure out how you need to change your positioning to make money. Time is everything here - steps here are counted in tenths of milliseconds, and overall response times in low ms. HFT firms even try to minimise cable lengths within the colocated datacentres to shave the time to response down even further. 3. Effectiveness - if you didn't blow up, and if so, if you made money - typically you'd examine intraday PnL volatility, returns, etc. and assess yourself on those and other metrics.
- ajays 14y ago
  How about releasing some data? Do some feature generation and come up with features, along with the desired action (the best possible action, given that you know how things are going to go in the future). Then us clueless types can see what all the hubbub is all about :-)

mkramlich 14y ago

HFT has some qualities which make it unattractive to me: - strategically, it's an unbounded arms race, due to competing against lots of other HFT forces all trying to squeeze profits quickly out of micro opportunities in bid/ask numbers, news leak/announce gaps, etc. - due to above, there are high technical and talent costs just to have the minimum setup needed to play in that game - also following from the first thing, HFT becomes an additional profit motive for adding destructive, dishonest political vectors to society, such as corruption and propaganda, in order to time trades, do front running, insider trading, war profiteering, bankster conspiracies, etc. - short-term thinking - lastly, you're fundamentally not building anything real or positive for the rest of society outside HFT and your direct clients/owners; or making the world a better place; to others, you're just a parasite on the money flows, adding more noise and chaos, and arguably risk, to the economy I like to compare it to the alternative, strategically-speaking: LFT (low f.t.) - lower costs (your fixed & marginal costs "scale down" better, which is also better in the startup phase. Think lemonade stand, a small simple web SaaS, contract development -> software product company, etc.) - longer-term thinking & bets (buy & hold; future trends, etc.) - actually building, investing, helping others, bringing new products/services to the market, solving problems for others for profit LFT could be used to describe traditional stock investing, it could include personal relationship investing, personal health/skills investing, building lifestyle businesses, building startups to flip, doing R&D, invention & marketing, etc. Both HFT and LFT can make you wealthy on the top end scenarios. But LFT scales down to the low end scenarios and early phases better, and you can tangibly and visibly see improvements to the world that you've made happen. Both can benefit from hacker thinking. But to me, it's clear which is the healthier choice for the individual and society, in the long run.

cletus 14y ago

This post is being used as somewhat of a stalking horse for any number of other agendas and soapboxes, which I guess isn't surprising, so I'll join the fray. 1. There is a misconception about market makers and the nature of a spread. In particularly illiquid markets, market makers allow you to buy and sell immediately. This is a service. The spread pays for this service. The more liquid a market is the lower the spread (eg it is essentially if not actually zero on US Treasuries); 2. For those questioning our brightest and best being "wasted" on HFT, I see comparatively better value on all the bright minds being wasted on social networks; 3. Finance is a constant arms race. The flavour of the month is HFT. In 5-10 years it'll be something else where low-latency transactions are just taken for granted. This kind of market is a gold mine for innovators as the existing players need a certain amount of innovation just to stand still (relative to the other players). There is classic "mine the miners" opportunity here; 4. There is a lot of highly-leveraged trading that goes on, automated and otherwise, beyond the world of HFT. (I believe) Warren Buffett describes this as "picking up nickels in front of a bulldozer" [1]. In extreme market situations, market models go out the window and the highly leveraged are the first to go to the flames. Just ask Bear Stearns [2] with its 35:1 leveraged ratio. 5. Financial institutions have access to way more data and cheaper (typically free) transactions compared to the average investor. It's common practice to do things like whipsaw the market to hit stop loss orders and the like. Honestly, unless you're a bank in the short term it is an insider's game; 6. A lot of the problems with the lack of transparency, assymmetry of information between parties and bad behaviour (like flooding exchanges) comes down to a failure of regulation. The SEC is either toothless or has been subverted by political interests. Much of what happens in the US just doesn't happen in Australia (to the same degree at least). Just take all the fake documentation in the US housing bubble. How bank executives, loan originators and the like didn't go to jail for this is really astounding; 7. Banks and market funds have limited to no downside risk. If things go horribly pear-shaped the US Federal government will--and has--bail them out. This breeds an adverse appetite for risk. I subscribe to the view that the Federal Reserve, the World Bank and the IMF are in large part welfare for investment bankers; and 8. Banks and funds are too large. If something is "too big to fail" then it's too big and needs to be broken up. That is all. [1]: http://en.wikipedia.org/wiki/Long-Term_Capital_Management http://en.wikipedia.org/wiki/Long-Term_Capital_Management [2]: http://en.wikipedia.org/wiki/Bear_Stearns http://en.wikipedia.org/wiki/Bear_Stearns

avstraliitski 14y ago

Real hackers use their powers for good.

infinii 14y ago

There's the sentiment that exchanges allow this because they are "businesses". What's to to stop exchanges from front running the HFT front runners without their knowledge and stealing all the marbles?

asto 14y ago

If a person is successful in one business should we subject ourselves to his idiotic views on another business? By definition they can’t go into an equity unless there already is liquidity. HFTs who are usually market makers [1] maintain liquidity in the market by constantly having buy/sell orders open for many instruments/scrips. In their absence, a counter trade may not exist for matching against when you or I put out a sell/buy order. Being able to sell/buy a scrip/instrument within seconds is a valuable service that investors appreciate. In India, the securities exchange regulator mandates that all market makers make a minimum value of trades per day. I am not sure of the exact current values in each segment but it's in the order of millions of rupees. This is how regulators create conditions of liquidity so that markets are nicer to trade in especially for retail investors. The missing link in all the discussions is: What is the purpose of the stock market? The stock market can be divided into a primary market and a secondary market. The primary market exists to finance companies. When they IPO, cash is generated which they can use to fund business activities. The secondary market exists to encourage purchases in the primary market and to efficiently discover the price of the securities. Without a liquid secondary market, very few investors would be interested in a primary market. HFT is only doing what was already done since many years ago manually. Humans make the same mistakes computers are being programmed to make today. In their defence, they execute exactly as asked - they feel no fear, doubt or greed - unlike a human trader . Should we treat them as being harmful because computers make mistakes at a significantly faster rate? HFT is a still nascent concept. With sufficient regulation, we will likely be able to get rid of the problems and retain the advantages. [1] http://en.wikipedia.org/wiki/Market_maker http://en.wikipedia.org/wiki/Market_maker

JonnieCache 14y ago
>hould we treat them as being harmful because computers make mistakes at a significantly faster rate? Yes. Significantly doesn't really cover it. When you increase the rate of an activity by many orders of magnitude and put an AI behind it, it becomes a totally different activity, even if the mechanics are notionally the same.

bogger 14y ago

MC is confused. High frequency trading covers a broad range of trading strategies, some of which are beneficial to market participants. In particular, were all automated market making to be banned, we would see a huge jump in the cost of trading. The fuss about HFT is distracting people from the much more serious problem of good old-fashioned insider trading, which remains rampant.

dm_mongodb 14y ago

it seems to me that the exchange adding a small random delay (say 1ms) on message reception would dampen the benefits of speed and then there would be less work invested in saving a few microseconds here and there. you still have program trading but a little bit of silly dynamics is dampened.