14 ms·

How I made $500k with machine learning and high frequency trading

551 points by jspaulding 14y ago

jspaulding 14y ago
If anyone has questions for me happy to answer as best I can.
- fr0sty 14y ago
  Wouldn't the term "Statistical Arbitrage" be a more apt description of what you were doing? If you are relying on a broker-supplied pricefeed over the internet you are far outside the real of what is traditionally understood as "High Frequency Trading".
  - jspaulding 14y ago
    It wasn't anything over the internet. I had a server rented at my broker who were situated close to the exchanges in Chicago and had direct lines. For sure I was not the fastest but only behind by a couple milliseconds perhaps.
    - mej10 14y ago
      How much does that cost?
    - fr0sty 14y ago
      Ok, that wasn't clear to me. You talked about programming hotkeys (and then automating the hotkeys) so I assumed this was running on your desktop.
      - jspaulding 14y ago
        oh right.. well in the very beginning it was.
- kamaal 14y ago
  Do you know of any resources/books to learn trading/HFT in general?
- jcromartie 14y ago
  I have one question: Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this? EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?
  - amalag 14y ago
    I tried at one point to do it with Ninjatrader. The programming skills for the trading software is not complicated. What is complicated is tweaking it so it will make money, there are tons of indicators out there and many people have tried this with neural networks and the like. A lot of effort is put into it. I personally thought there were smarter people than me who barely had an edge. I honestly didn't think it was within the grasp of a single programmer nowadays, but this author has proved me wrong. He of course has much more sophisticated algorithms than what I was attempting.
  - jetti 14y ago
    "Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?" I have flirted with HFT in a hobby-like manner and it isn't the programming that will get you the money, it is the domain specific knowledge coupled with the programming. On top of that, there are quite a few risks and potential to lose a lot of money.
  - beagle3 14y ago
    > You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. A profitable predictor is a much, much harder problem. At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough. Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.
    - otakucode 14y ago
- dude_abides 14y ago
  Thanks for the post. Is this the software you used? http://www.ctsfutures.com/t4.aspx http://www.ctsfutures.com/t4.aspx Would be even more awesome if you could share some code on github.
  - jspaulding 14y ago
    Yes, however, I forgot to mention that pretty early on I converted my program to use an API from https://www.tradingtechnologies.com https://www.tradingtechnologies.com. It's odd, but I can't remember exactly why. I think it was simply because I found a broker who could offer me a lower commission rate and they only supported TT. With regard to posting code yes I may do that. We'll see.
    - briancurtin 14y ago
      XTAPI, or did you write something to talk to a FIX gateway? (former TT employee)
      - jspaulding 14y ago
        I'm pretty sure it was XTAPI. Is that the simpler one? I used the simpler one. Whatever the case.. the code that dealt with the API was trivial compared to the rest of my program. So good work!
    - kami8845 14y ago
      >With regard to posting code yes I may do that. We'll see. What does it depend on?
      - briancurtin 14y ago
        If I was correct in guessing that he used XTAPI (via X_TRADER Pro), IIRC that costs around $1k/mo depending on who you go through for the front end license. I was only a developer so I'm not sure what licensing costs are for gateways, but those are the servers you connect to in order to place orders on the exchanges. He was trading the Russel on ICE, and DAX on Xetra (maybe Eurex?), so he would have needed two gateways. If he posts the code, you're a long way from running it.
- mej10 14y ago
  Do you think you got lucky or that your skill as a trader made you this money? Or rather, which of these do you think mattered most?
  - jspaulding 14y ago
    1) Being connected to friends who taught me a scalping style of day trading 2) Being really good at designing algorithms. 3) I only wish I could have started automated trading back in 2001
- retube 14y ago
  why do you think your technique stopped working over time?
- psionic_ 14y ago
  I wonder if running your program nowadays can have same successful results as two years ago.. anyone is going to try? :)
- mgl 14y ago
  How do (did) you cope with increased stress level? Trading futures, especially in a an automated way, can easily drain your margin unless the algorithm is really well tested for edge cases.
  - jspaulding 14y ago
    My automated program was much less stressful than trading manually. The best was going to Hawaii, waking up, and having the entire day done. I was making like 6k every day on that vacation. The best!! :)
stickydink 14y ago
Looking at your first chart there, is there a reason (other than market conditions) you were making significantly more at the end of '09 than mid '10?
- jspaulding 14y ago
  To be honest I don't know exactly what happened. My theory is that over time more and more market participants started integrating the types of analysis I was doing which rendered my program ineffectual. It's a pretty normal pattern that there is some inefficiency in the market and over time it disappears.
  - mgl 14y ago
    It's possible that your algorithm is sensitive to market volatility.
  - stickydink 14y ago
    Being a machine learning program, how much of it did you tell it to forget? Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between? (I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)
    - achy 14y ago
      This. If the 2009 model worked well, did you try letting your algorithm to 'forget' the 2010 data and see if the model worked better?
wtvanhest 14y ago
You said "Growth stopped" but your P&L shows negative growth approaching zero. Would it be more fair to say that your profitability turned to zero?
- jspaulding 14y ago
  To be honest I'd given up and moved on to other stuff well before I actually shut down the program.
nrmn 14y ago
Could you comment on how your "curve fitting" algorithm worked? Did you end up with an equation for each curve? Im working on something that requires curve fitting and any kind of tip would be helpful.
- jspaulding 14y ago
  I basically just brute forced it. I came up with a cost function which would measure the difference between a possible curve and each data point. I think you're supposed to do the squared difference but I can't remember if I did that. With a cost function in place it's just a matter of zooming in on variables that minimize the cost function.
davidw 14y ago
Looks like it's written in C# - is that correct?
- binxbolling 14y ago
  Doesn't the post explicitly say it was?
- jetti 14y ago
  It stated in the article it was C#. "In 2008 I was “manually” day trading futures using software called T4. I’d been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."
  - davidw 14y ago
    Yes, that's what I was going off of. I was curious about that aspect of it, if it had stayed C# or what.
    - jspaulding 14y ago
      Actually I believe when I switched from T4 to TT it was C++ I was using.
amalag 14y ago
What contracts did you trade? Looks like you did futures contracts? Did you look into forex at all, or was this strictly equites.
- jspaulding 14y ago
  I was trading stock market index futures: Russell 2000, DAX, and a bit of Nasdaq. I experimented with other stuff as well.
pdog 14y ago
Would you be able to open source any of the code behind your trading system? Maybe not the "secret sauce", but it would be interesting to see how you processed the data feeds, modeled the data, entered orders, etc.
- jspaulding 14y ago
  I certainly could open source it. I may just wait a bit on the off chance that somebody wants to purchase it.
  - Juuumanji 14y ago
    that would require some proof that it works today.
    - beagle3 14y ago
      Not really. A lot of people in the business would pay e.g. $5,000 for exclusive rights for something that worked this well in 2009 (with proof that it worked in 2009, e.g. verifiable broker statements), and a smaller amount (say, $5,00) for non exclusive rights. If he claimed it still works but he wants to sell it, it is a completely different game -- because when these things work, they are cash cows.
      - jspaulding 14y ago
        Yes i'm pretty sure it wouldn't work today. By buying the code I realistically mean hiring me to work for them based on what I achieved.
        beagle3 14y ago
        The guys you want to work for (2sigma, RenTec, Jane Street, Susquehanna, ...) are unlikely to call you up as a result of this blog post / hacker news exposure/discussion. If you want to go back to trading, you'll probably have to actively try to get a job -- at the very least, let someone who's still in the business know that you are looking. In my experience in this field, word of mouth and friends-of-friends are infinitely more successful hiring strategies, for both sides. (Re:releasing the source - I would like to have a look at the strategy, but I would recommend against releasing anything that is even close to being useful, unless you want to spend the next year screening "where can I get a good XTAPI broker" and "I've got XTrader_PRO set up, but I'm getting error 10013, what gives?" emails). the Nuclear Phynance message board is probably a better place to look for business offers.
tarr11 14y ago
This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who were market makers. The quants called guys like him "retail" investors and they gleefully picked off all those trades. It's how they made all their money. So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.
- marshallp 14y ago
  I don't get that. It would be true if he just made a few trades, but the author claimed to be making 2000 trades a day. Over a period of months winning that wouldn't qualify as blind luck.
  - DanBC 14y ago
    >Over a period of months winning that wouldn't qualify as blind luck. Why not?
    - marshallp 14y ago
      By that definition you could start claiming everything as blind luck. Why bother doing anything at all, let luck do the work.
    - crntaylor 14y ago
      It's simple statistics. The author was up 4k/day over 120 days. He doesn't say what his daily volatility was, but let's assume 2k (which squares pretty well with his claim that his worst day was a 2k loss). With a quick bit of R code, we can simulate his PnL over 120 days multiple times, assuming he has no skill, and see what the probability of him being up 4k/day is. I'll use a t-distribution with 3 degrees of freedom, which allows big up and down swings (again, accentuating the effect of luck). > pnl <- c() > for (i in 1:1000) pnl[i] <- mean(2000 * rt(120, df=3)) > mean(pnl > 4000) 0.0 That is, there's a zero percent chance that he would have made those returns if he had no skill. And remember that this simulation is overestimating the effect of luck.
vinayan3 14y ago
Have you ever thought of making a trading system that would buy tons of stock when a flash crash happens? It is going to happen again. If your system is ready and you buy before they shut the market down or roll back orders you could make a hefty profit.
- crntaylor 14y ago
  The difficulty is in identifying what is a 'flash crash' (i.e. a temporary downward blip in prices caused by computer or human error) and what is a genuine downward price movement. If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses. Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.
  - beagle3 14y ago
    > The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential. This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash). That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.
    - fr0sty 14y ago
      Nitpick: When exchanges have busted trades there is a "at or below $XX.XX" condition as well as the "between XX:XX and YY:YY" condition. In the Flash Crash as well as the Knight Capital incident "up/down 30% from the Previous Close" was the price collar (anything outside that was busted and anything inside stood). Of course there is no guarantee that the same criteria will be used the next time around so caveat emptor.
lrm242 14y ago
Take a look at the VIX from 2009 until today and you'll understand why you stopped making money.
- S_A_P 14y ago
  http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol=%5Evix;range=5y;compare=;indicator=volume;charttype=area;crosshair=on;ohlcvalues=0;logscale=off;source=undefined http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...; very interesting, a spike in sept-dec 2009 that roughly follows his PNL...
  - jcampbell1 14y ago
    Without a ton of volatility, any homebrew HFT is going to lose to commissions and spread. Basically, he was trading in one of the few periods where is was possible to make money.
Judson 14y ago
This reminds me of an AMA from a few years ago. Really interesting if you are into stuff like this: http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automated_independent_retail_trader_i/ http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...
- jspaulding 14y ago
  thanks for posting hadn't seen that
tomp 14y ago
This article is missing a crucial piece of data: what was the initial investment.. earning 500k with 10k initial investment is genious, with 10M initial investment it's just another year on the stock market.
- beagle3 14y ago
  This is a different style of trading than what investors do. He said that he was never more than a few contracts in. A single contract needs $10K day trading margin usually (depends on time frame and specific contract, but it's a reasonable estimate). Therefore, if he was never more than 10 contracts long or short at the same time, the risk was of $100K. Making $500K on $100K over 6 months is good business.
  - jspaulding 14y ago
    You are correct. I think I had to have $10-30k in my account. I was extremely low risk so they weren't concerned.
- brown9-2 14y ago
  Especially considering that from it's lowest point in 2009 through the end of 2010, the S&P500 rose by 71%. Need to know what the starting capital was to be able to figure out if his return beat the market.
curiousdannii 14y ago
I'm sceptical that HFT is good for the public. What did you do to ensure your system wouldn't make a flash crash worse?
- beagle3 14y ago
  Why is it his business to defend other market participants?
  - curiousdannii 14y ago
    I didn't say it was. But everyone should be in the business of not being evil and not exasperating the problems of others. In trading that means not contributing to flash crashes. I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.
rhplus 14y ago
The charts show he was trading between Jun 2009 and Oct 2010. How much of his gains could be attributed to the market recovery in general? The Dow went from about 7000 to 11000, the Russell from about 600 to 800.
- washedup 14y ago
  Well, in the article he said tat he did not care about direction, he would simple buy when his expected price was up, and sell when down. However, there could have easily been a bias in his model that "preferred" and performed better during upward movements. If so, he got lucky.
  - jspaulding 14y ago
    Longs and shorts were 50/50 and my program showed no preference for up or down days. High volatility and high volume was what it liked.
    - debacle 14y ago
      I guess the real question is: what was your alpha in that timeframe?
      - jspaulding 14y ago
        I'm probably showing my ignorance here but what do you mean by alpha? And how is it quantified?
        solutionyogi 14y ago
        Alpha is how much excess return you had over the market (or risk free return( E.g. if you made 10% when overall market was up 15% for the year, you have negative alpha. [As someone could have bought index and held it through year to generate better return] If you made 20% when market was up 10%, you have positive alpha. That is why everyone in the investment community is 'seeking alpha'.
junto 14y ago
Out of interest, how much capital did you start off with?
- jspaulding 14y ago
  $10k loan from dad. Built it up to 30k trading manually before my automated program went live.
  - intel4004 14y ago
    What were your average transaction costs per trade? 1000 to 4000 trades per day, lets say 2500 on average, translates to about 625K transactions per year. I assume you did not have to pay something like ETrades 3$ commission per future contract, which would result in almost 2M of fees per year.
danso 14y ago
I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines. I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-chess-master-and-the-computer/?pagination=false http://www.nybooks.com/articles/archives/2010/feb/11/the-che... He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers: > The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process. So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?
- bloaf 14y ago
  Good point, I also wonder about the potential to exploit the algorithms used by the "professionals." In other words, if you can come up with a reasonable approximation of what the pros will do, can you use that information to beat them?
  - aggronn 14y ago
    'Theoretically', no. Its hard to be optimistic about these two ideas because while the chess example is a good story, its not analogous for many reasons, ranging from disparity in available information to players to a difference of several magnitudes in saturation. Not to mention HFT just isn't chess. HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right. -- If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.
Sharma 14y ago
Trust me, you earned that much because of your luck. Otherwise Andrew Ng would have partnered with another finance professor and they would have been the richest people on earth!! Imagine trading with their expert systems on global markets. I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!
- zdwalter 14y ago
  I like to trade Forex using mql4, any suggestion? Thanks.
arbuge 14y ago
I am curious as to exactly why the profitability decreased steadily and rather rapidly all the way to ~zero. The article doesn't seem to expound on that unless I missed something. Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.
- baq 14y ago
  another comment explained: http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol=%5Evix;range=5y;compare=;indicator=volume;charttype=area;crosshair=on;ohlcvalues=0;logscale=off;source=undefined http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...;
- jspaulding 14y ago
  I don't think anyone was adapting to what I was doing in particular but rather simply adapting to the opportunities in the market.
  - arbuge 14y ago
    Thanks. Makes sense... I guess those opportunities tend to get ironed out rather fast then.
dschiptsov 14y ago
Basically this is a story about a guy who was smart enough to script up his trading tool (he discovered that there is an API and wrote some code to use it). He trade other people's money, using other people's (probably employer's) account and resources, I suppose. His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.) The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.) But this is just my guess.
- mattyppants 14y ago
  This is not even close to an accurate summary. He never stated that he had any employer backing, and he wasn't collecting market making fees. In fact he was paying brokerage fees which is the exact opposite.
  - dschiptsov 14y ago
    He used his own money to test his algos? Come on. Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system. This is why my guess is that he was an employee.
    - oijaf888 14y ago
      An employee of what? He states above that he was paying roughly $200/month for a server and $1800/month for the software/data connections to his broker.
      - dschiptsov 14y ago
        Prior to setting up my automated trading program I’d had 2 years experience as a “manual” day trader. - mom's and pap's 401k?
    - mattyppants 14y ago
      No, he said he backtested them against data he collected from the feeds. He paid for the feeds just like everyone else does with his own money.
iandanforth 14y ago
I'm glad to see a healthy respect for investment among the hacker community. It's traders like this who commit to nearly a full 10 seconds of ownership that are the backbone of economic growth for this country.
- unreal37 14y ago
  I think there is a healthy respect for computer code that can let loose upon the world and make money.
  - mattyppants 14y ago
    Your missing the point that owning part of a company for 10 seconds brings zero social value to the economy. Which I also tend to agree with. Slightly off-topic from the point of the article, which was that he achieved it without backing of a fairly large institution.
    - OldSchool 14y ago
      True it adds no social value, but it is arguably the ultimate hacker's game :) You can actually add a little social value by placing only orders that increase liquidity; those where someone else "takes out" your standing offer or bid. That and the fact that your profits are taxed as ordinary income in the US for equities at least.
    - unreal37 14y ago
      I didn't miss that point - I just chose not to address it. Lots of things add zero (or negative) social value to the economy including Farmville or the gazillionth Instagram clone. That doesn't mean they're not worth doing. And HFT or day-trading does bring some value to the world. It enables companies to go to the public markets and raise capital. And investors to sell their shares without having to wait too long for a buyer. But those positives come along with negatives as well.
OldSchool 14y ago
Great work, very interesting to me. Counter to what we're constantly told through the media this stuff can be done. Doing it year after year seems to be the elusive part. Intuitively, once you've proven your technique on 1000+ trades it's not luck. I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower. The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities. I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.
mempko 14y ago
Wow, fuck that guy. no really. Should be title "How I stole 500k with machine learning and high frequency trading".
- oijaf888 14y ago
  Who did he steal it from?
deleted 14y ago
[deleted]
hafabnew 14y ago
I've been considering trying HFT myself for a while. I'm competent with Machine Learning and am a Software Developer by day, so I can program and can sysadmin well enough to get something up and running without any trouble at all. But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering. It seems like the logical course of single-programmer HFT trading being: - Find sample data - Build your trading program using sample data - When you're happy: connect to live API and set your trading program loose - Iterate. However, the first step and the third step seem like the ones which require the most research. Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program? Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.
- sseveran 14y ago
  Nothing in HFT is free. People doing this for a living use precision time protocol in a colocated data center to build their own timestamps. However you can get started by buying ITCH, OpenBook, etc... data which has the full market depth feeds for the various exchanges. There are a couple of brokers out there specializing in the space. If the easy part was building a working model either you got incredibly lucky or the model is wrong.
- aiskunkworks 14y ago
  Find sample data: https://quantquote.com/ https://quantquote.com/ API (Colo): http://www.lightspeedinstitutional.com/automated-trading/ http://www.lightspeedinstitutional.com/automated-trading/
- monksy 14y ago
  Quality sample data and historical data to backtest is hard. Symbols change, stocks split, dividends are announced etc.
hnruss 14y ago
Just because you CAN do it doesn't mean you SHOULD. Even if you don't think of it as "gambling", you're still taking in tons of money without providing any tangible benefit to society. If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.
- jspaulding 14y ago
  I didn't lose any sleep but I do indeed hope to contribute more to society with future projects.
- wyan 14y ago
  You mean, something like Instagram?
  - sjm 14y ago
    Not really defending the parent, but believe it or not Instagram provides something that makes a lot of people happy. Winning a bunch of money on the stock market does nothing for anyone but the winner. Not really a fair comparison.
quant123 14y ago
There is an air of either incredibility or sheer jealousy in these comments. Nevertheless, I just wanted to tell the OP that he did a great job. Thanks for sharing. I work in the finance industry as a quantitative software developer, and it certainly is not an easy job for one person to do. In fact, I tried (independent of my professional work) doing this myself, and I ended up losing a lot of money. If people are trying to do this, please please be careful. Big companies, like ones I have worked at, have technical and human resources that are vastly more powerful.
- jspaulding 14y ago
  thanks quant123 ;)
- stmartin 14y ago
  Did you mean incredulity instead? I'm not jealous of the money he made - I'm a sw developer, I have a good salary, my wife's an accountant, and we do well - what I am 'jealous' of is not having the time to do something like this, regardless of the outcome. Clearly, I'm prepared to take the risk... but if you have 2 children, you may think differently about spending your time doing this instead of on your family. Money cannot buy happiness. It can pave the way to it, but happiness only comes through surrender to another person or a cause greater than what one can accomplish by him or herself, and in no other way. And when it comes, it's an unintended side-effect, rather than something that one can pursue (which is one of the reasons I hate the 1st amendment and the way it's worded).
loup-vaillant 14y ago
While this was quite fascinating, I couldn't see this form of trading as anything but a zero-sum game. Some players win, the other lose, like in any other game. Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?
- loup-vaillant 14y ago
  Okay, enough with the downvotes. What's wrong with my question? I didn't mean to bash a field about which I know next to nothing. My question is genuine. Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had. Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth. But. The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum. Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"
- gd1 14y ago
  A strange position to take, I'm guessing you've absorbed it from the media somehow. His machine was in the market enough to trade 4000 times a day (I would suggest passively, or he would have been eaten by the cost of crossing the spread), so he was basically continuously offering a service to the market - an offer to sell and a bid to buy at the price he thought fair. Do you demand to know what value your local 7-11 provides by selling you milk? The value is that you can buy milk 24 hours a day, so you don't begrudge them the 15c a carton they are making. And they probably charge more than your local supermarket for the convenience too. The dirty thieves.
dwk9080 14y ago
If you're interested in working on this kind of stuff in the San Francisco area, send a resume to Headlands Technologies. careers@headlandstech.com
ianstallings 14y ago
I could do the same with poker. Or I could lose it all. That variance thing is a bitch.
mrchess 14y ago
Cool article but I hope people don't start trying to follow this path. Ask yourself -- why did he stop?
- jspaulding 14y ago
  yes. and yes.
stcredzero 14y ago
Is there a way to do this with Python or Ruby? I could just as well program this in C#, but I have a friend who can code a little, but doesn't really need everything in C# to do what he wants. The value add of offering the simulator, including the taking into account the bid/ask prices and a stochastic model for latency. Combine this with a web based code editor and easy hosting, and I think this would be a viable product.
- HockeyPlayer 14y ago
  Sure, his program could easily have been done in almost any language. He wasn't competing on speed, which might have excluded languages like Python.
- owyn 14y ago
  Nope, there is absolutely no way to do this with python or ruby. :) Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks: http://www.rapidquant.com/features http://www.rapidquant.com/features A C++ library with python bindings: http://quantlib.org/index.shtml http://quantlib.org/index.shtml And a low level library for data analysis: http://pandas.pydata.org http://pandas.pydata.org
  - grumps 14y ago
    If you use their library/SDK/Framework.... how do you actually trade, do you still need a brokerage? Sorry a bit new to this field.
    - owyn 14y ago
      Yep, almost any of them have an API these days. Your software would make the list of trades which is uploaded and executed... Oh, I forgot about another backtesting framework for python. It has a built in IDE as well. http://www.quantopian.com http://www.quantopian.com
      - 14y ago
unreal37 14y ago
There is a coursera course called "Computational Investing, Part I" that I am taking that aims to build a market trading simulator to test a trading model. It just started so it's not too late to join. https://www.coursera.org/course/compinvesting1 https://www.coursera.org/course/compinvesting1
impotter 14y ago
Might I suggest an alternate title / next startup? "Learn how this unemployed soccer dad made over $45k a month working from home! His winning system can be yours for only $200!!!!"
runjake 14y ago
Relax with the vitriol. The guy is sharing an interesting personal story, not providing a step-by-step HOWTO or recommending people follow his suit. In fact, the article is really an ad for his startup Courseware.
- jspaulding 14y ago
  CourseTalk. CourseTalk! http://coursetalk.org http://coursetalk.org .. You are right kind of :) But I've made a decision to start reaching out generally so I can attract cool people to work with on whatever projects I may be interested in in the future.
  - runjake 14y ago
    By the way, no offense meant by the advertising thing. I enjoyed the article. Although, as a technical person, would've enjoyed more details on the code and algorithms.
ww520 14y ago
Very cool article. Gave a detail explanation of the ENTIRE process. Thanks!
ChuckMcM 14y ago
The high point of my trading was October 2009 when I made almost 100k. After this I continued to spend the next four months trying to improve my program despite decreased profit each month. Unfortunately by this point I guess I’d implemented all my best ideas because nothing I tried seemed to help much. It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.
jcfrei 14y ago
It is pretty clear from his own graph that this stopped working in october '10. which was an eternity ago in terms of financial markets. algorithmic trading has increased manifold since then, so finding another arbitrage opportunity like he did is only going to be more difficult.
rbc 14y ago
I started the infrastructure for this kind of thing a while ago. It's BSD licensed. It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems. Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at: http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ I won't try to advertise it as a complete solution, but it might be the start of one.
padobson 14y ago
Don't do this with your own money. Found a startup building HFT tools, and then raise money for it, and use other people's money to test your tools. If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do. This is risk management.
nesu 14y ago
Wait. This is not HFT. There's a huge difference between automated and high frequency trading. What he does is only automated scalping at best (or at the fastest). Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders. HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.
- crucialfelix 14y ago
  quite true. but HFT has become a buzzword like "the cloud" and even many financial industry specialists claim to do HFT but in fact they are just market making. auto-scalping or even hedge funds that day trade with robots are not HFT. this: http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first-order autocorrelation of returns. it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.
HockeyPlayer 14y ago
I run a 12 person HFT group in Denver. This is an excellent description of how an individual can bootstrap themselves into success. Great story, and nicely presented. The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities." Which doesn't really tell you much.
- rorrr 14y ago
  How well are you guys doing? How much money did you make?
- jspaulding 14y ago
  Thank you. I'm looking back through my code and there are really a lot of indicators. It would get pretty technical to explain them. They are all explainable it's just that each one corresponds to slightly different market conditions and I just didn't want to get into it.
  - HockeyPlayer 14y ago
    Fair enough. Most people considering trying this probably have a few ideas for indicators. But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.
    - jhales 14y ago
      Yes, I would find this very interesting. Doesn't matter if the indicator is now defunct.
  - deathflute 14y ago
    Thanks for an excellent post. I would like to see any one indicator explained in detail as well. I am sure it would be of interest to a lot of people.
mcarvin 14y ago
Very important to understand that making $500k speculatively is not evidence of an 'edge', nor is trading frequency evidence of the absence of luck. From March 2009 through much of 2010, the market was strongly bullish - if his algorithm showed a positive market bias then his returns would primarily be a function of timing (read luck: and there are a million variants on the nature of the bias that could be unwittingly responsible for his returns, despite the frequency of trades). We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with? Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.
- fr0sty 14y ago
  1. Author graphed his daily returns which should give you a handle on his volatility. He also stated his max drawdown was $2k. 2. Author has stated elsewhere than he began with $10k in seed capital. 3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number. 4. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped. Edited to fix numbers.
  - mcarvin 14y ago
    1. I cannot get even a remote sense for the nature of his risk exposure from looking at his daily returns. 2. ok. 3. The point here is that a systematic bias in his algorithm will expose his trading strategy to the good graces of market fortune (luck) regardless of whether he trades a million, billion or once a day. The source of the bias is irrelevant. 4. did not see where he said that but that very much confirms 'timing' / which in this case I interpret as luck as being at least a contributing factor.
namank 14y ago
Might've been what it was a couple of years ago but this post, dated today, is the perfect advertisement for the author's current business. Well done! PS: no sarcasm intended, it truly is an excellent advert.
hellsten 14y ago
A rising tide lifts all boats. "Price move predictions" sounds as effective as tea leaf reading.
doki_pen 14y ago
Isn't profit meaningless without knowing initial investment? I read the first few paragraphs and got bored. Why not say upfront what the bankroll was to start? edit: I found what I was looking for in the comments.
- JoblessWonder 14y ago
  Just to save someone else from having to look for it like we both did, his initial investment was $10k loan from dad. Built it up to 30k trading manually before my automated program went live.
thisismyname 14y ago
How much did you start with?
dmmalam 14y ago
Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset. I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US. Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest. Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money. I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.
- jspaulding 14y ago
  I don't know the exact definition of HFT but I did run my algorithm from a server collocated with my broker close to the exchange. I modelled lag time in simulation and not having it collocated certainly would have hurt.
- beagle3 14y ago
  In the US, HFT is mostly synonymous with "all out tech war, flooding the order queue so your less-equipped peers get lags". (Nanex publishes analysis on these events, which are not occuring several times a month and keep accelerating). In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions). > Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt. Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.
shizzy0 14y ago
And nothing of value was created. Just kidding. It was a fascinating article. Thanks for sharing.
suyash 14y ago
@Author ? Can you create an online course and teach us all? Thanks
photorized 14y ago
I am skeptical for two reasons: 1. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2. You don't just stop using it. There should be more to this story.
- frankster 14y ago
  Can I recommend that you read the article and you will find therein the answers you seek!
gubatron 14y ago
of course no firm would respond to his noob low yield model. The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this. Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm. Kudos though for taking on the task of learning how to code and making money with ML.
- jspaulding 14y ago
  This is such an odd comment. What motivates you to write such a comment? And what makes my system amateur and noob in your eyes?
derryl 14y ago
This might be a dumb question but... what system does your algorithm interface with? Last time I checked, NASDAQ and NYSE don't exactly publish API's
deleted 14y ago
[deleted]
nashequilibrium 14y ago
Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster. The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.
- jspaulding 14y ago
  There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck. Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone. Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading. I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.
  - quant123 14y ago
    agreed. jspaulding got it right although I can see nashequilibrium's skepticism as the US futures market is incredibly crowded (read: competitive, no free cookies).
confluence 14y ago
A lot of people are stating that this is like gambling - it is - but not in the sense that you think. Firstly he doesn't use his entire bankroll on each trade, secondly he goes long-short consistently over very short periods of time, thirdly he's too tiny to actually move markets, and fourthly he is in and out within a day - where his max var. is 100% on thousands of tiny trades. Think of it like this - he has $100 - he bets $10 of that each day. He can blow that $10 - no problems. Worst case he runs out of capital over a period of weeks. He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks. Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+). In the short term however (sub-decade) - they can't price jack. Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that. There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw). Both work. Both work well. And will continue to do so as long as markets exist.
JuDue 14y ago
"Since I'm no longer running my [half million dollar] program I’m happy to tell all" This does not ring true.
sejje 14y ago
I don't understand trading enough to even understand many of the terms in the article, however I'm curious to one thing: is it possible that a program could be written specifically to exploit yours? And/or is that a potential reason it became unprofitable?
FireBeyond 14y ago
Hopefully not buried too deep, but any books recommended for getting into day-trading, either manual, or algorithmic? Kindle preferred, but definitely not the deciding factor.
leminhhai 14y ago
Amazing, love your project
dataisfun 14y ago
This is an activity that adds no value to our world.
MonteChristo 14y ago
The prototypical example of why a tax on financial transactions is urgently needed.
curationary 14y ago
how is it "machine learning"? A few if-else doesn't make it "machine learning", or does it? - @curationary
- jspaulding 14y ago
  It takes data (last 4 weeks of market update data) and learns how to predict price movements and how to make profitable trades.
vincegata 14y ago
Thanks for the post, it's very inspirational. Could you explain this part, specifically what do you mean by "bucket". "To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
vincegata 14y ago
Thanks for the post, it's very inspirational. Could you explain this part, specifically what do you mean by "bucket"? "To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
mrs1238 14y ago
One comment...everything is stated in this book "Why You Could Be Destined To Fail in Trading and How You Can Avoid it!" http://outskirtspress.com/estetrader/ http://outskirtspress.com/estetrader/
witoldc 14y ago
Let's see the real meat of the article: your trading program and algorithms.
datashaman 14y ago
Excellent article on this posted here: http://www.lrb.co.uk/v33/n10/donald-mackenzie/how-to-make-money-in-microseconds http://www.lrb.co.uk/v33/n10/donald-mackenzie/how-to-make-mo...

551 points by jspaulding 14y ago

jspaulding 14y ago

If anyone has questions for me happy to answer as best I can.

fr0sty 14y ago
Wouldn't the term "Statistical Arbitrage" be a more apt description of what you were doing? If you are relying on a broker-supplied pricefeed over the internet you are far outside the real of what is traditionally understood as "High Frequency Trading".
- jspaulding 14y ago
  It wasn't anything over the internet. I had a server rented at my broker who were situated close to the exchanges in Chicago and had direct lines. For sure I was not the fastest but only behind by a couple milliseconds perhaps.
  - mej10 14y ago
    How much does that cost?
  - fr0sty 14y ago
    Ok, that wasn't clear to me. You talked about programming hotkeys (and then automating the hotkeys) so I assumed this was running on your desktop.
    - jspaulding 14y ago
      oh right.. well in the very beginning it was.
kamaal 14y ago
Do you know of any resources/books to learn trading/HFT in general?
jcromartie 14y ago
I have one question: Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this? EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?
- amalag 14y ago
  I tried at one point to do it with Ninjatrader. The programming skills for the trading software is not complicated. What is complicated is tweaking it so it will make money, there are tons of indicators out there and many people have tried this with neural networks and the like. A lot of effort is put into it. I personally thought there were smarter people than me who barely had an edge. I honestly didn't think it was within the grasp of a single programmer nowadays, but this author has proved me wrong. He of course has much more sophisticated algorithms than what I was attempting.
- jetti 14y ago
  "Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?" I have flirted with HFT in a hobby-like manner and it isn't the programming that will get you the money, it is the domain specific knowledge coupled with the programming. On top of that, there are quite a few risks and potential to lose a lot of money.
- beagle3 14y ago
  > You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. A profitable predictor is a much, much harder problem. At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough. Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.
  - otakucode 14y ago
dude_abides 14y ago
Thanks for the post. Is this the software you used? http://www.ctsfutures.com/t4.aspx http://www.ctsfutures.com/t4.aspx Would be even more awesome if you could share some code on github.
- jspaulding 14y ago
  Yes, however, I forgot to mention that pretty early on I converted my program to use an API from https://www.tradingtechnologies.com https://www.tradingtechnologies.com. It's odd, but I can't remember exactly why. I think it was simply because I found a broker who could offer me a lower commission rate and they only supported TT. With regard to posting code yes I may do that. We'll see.
  - briancurtin 14y ago
    XTAPI, or did you write something to talk to a FIX gateway? (former TT employee)
    - jspaulding 14y ago
      I'm pretty sure it was XTAPI. Is that the simpler one? I used the simpler one. Whatever the case.. the code that dealt with the API was trivial compared to the rest of my program. So good work!
  - kami8845 14y ago
    >With regard to posting code yes I may do that. We'll see. What does it depend on?
    - briancurtin 14y ago
      If I was correct in guessing that he used XTAPI (via X_TRADER Pro), IIRC that costs around $1k/mo depending on who you go through for the front end license. I was only a developer so I'm not sure what licensing costs are for gateways, but those are the servers you connect to in order to place orders on the exchanges. He was trading the Russel on ICE, and DAX on Xetra (maybe Eurex?), so he would have needed two gateways. If he posts the code, you're a long way from running it.
mej10 14y ago
Do you think you got lucky or that your skill as a trader made you this money? Or rather, which of these do you think mattered most?
- jspaulding 14y ago
  1) Being connected to friends who taught me a scalping style of day trading 2) Being really good at designing algorithms. 3) I only wish I could have started automated trading back in 2001
retube 14y ago
why do you think your technique stopped working over time?
psionic_ 14y ago
I wonder if running your program nowadays can have same successful results as two years ago.. anyone is going to try? :)
mgl 14y ago
How do (did) you cope with increased stress level? Trading futures, especially in a an automated way, can easily drain your margin unless the algorithm is really well tested for edge cases.
- jspaulding 14y ago
  My automated program was much less stressful than trading manually. The best was going to Hawaii, waking up, and having the entire day done. I was making like 6k every day on that vacation. The best!! :)

stickydink 14y ago

Looking at your first chart there, is there a reason (other than market conditions) you were making significantly more at the end of '09 than mid '10?

jspaulding 14y ago
To be honest I don't know exactly what happened. My theory is that over time more and more market participants started integrating the types of analysis I was doing which rendered my program ineffectual. It's a pretty normal pattern that there is some inefficiency in the market and over time it disappears.
- mgl 14y ago
  It's possible that your algorithm is sensitive to market volatility.
- stickydink 14y ago
  Being a machine learning program, how much of it did you tell it to forget? Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between? (I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)
  - achy 14y ago
    This. If the 2009 model worked well, did you try letting your algorithm to 'forget' the 2010 data and see if the model worked better?

wtvanhest 14y ago

You said "Growth stopped" but your P&L shows negative growth approaching zero. Would it be more fair to say that your profitability turned to zero?

jspaulding 14y ago
To be honest I'd given up and moved on to other stuff well before I actually shut down the program.

nrmn 14y ago

Could you comment on how your "curve fitting" algorithm worked? Did you end up with an equation for each curve? Im working on something that requires curve fitting and any kind of tip would be helpful.

jspaulding 14y ago
I basically just brute forced it. I came up with a cost function which would measure the difference between a possible curve and each data point. I think you're supposed to do the squared difference but I can't remember if I did that. With a cost function in place it's just a matter of zooming in on variables that minimize the cost function.

davidw 14y ago

Looks like it's written in C# - is that correct?

binxbolling 14y ago
Doesn't the post explicitly say it was?
jetti 14y ago
It stated in the article it was C#. "In 2008 I was “manually” day trading futures using software called T4. I’d been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."
- davidw 14y ago
  Yes, that's what I was going off of. I was curious about that aspect of it, if it had stayed C# or what.
  - jspaulding 14y ago
    Actually I believe when I switched from T4 to TT it was C++ I was using.

amalag 14y ago

What contracts did you trade? Looks like you did futures contracts? Did you look into forex at all, or was this strictly equites.

jspaulding 14y ago
I was trading stock market index futures: Russell 2000, DAX, and a bit of Nasdaq. I experimented with other stuff as well.

pdog 14y ago

Would you be able to open source any of the code behind your trading system? Maybe not the "secret sauce", but it would be interesting to see how you processed the data feeds, modeled the data, entered orders, etc.

jspaulding 14y ago
I certainly could open source it. I may just wait a bit on the off chance that somebody wants to purchase it.
- Juuumanji 14y ago
  that would require some proof that it works today.
  - beagle3 14y ago
    Not really. A lot of people in the business would pay e.g. $5,000 for exclusive rights for something that worked this well in 2009 (with proof that it worked in 2009, e.g. verifiable broker statements), and a smaller amount (say, $5,00) for non exclusive rights. If he claimed it still works but he wants to sell it, it is a completely different game -- because when these things work, they are cash cows.
    - jspaulding 14y ago
      Yes i'm pretty sure it wouldn't work today. By buying the code I realistically mean hiring me to work for them based on what I achieved.
      - beagle3 14y ago
        The guys you want to work for (2sigma, RenTec, Jane Street, Susquehanna, ...) are unlikely to call you up as a result of this blog post / hacker news exposure/discussion. If you want to go back to trading, you'll probably have to actively try to get a job -- at the very least, let someone who's still in the business know that you are looking. In my experience in this field, word of mouth and friends-of-friends are infinitely more successful hiring strategies, for both sides. (Re:releasing the source - I would like to have a look at the strategy, but I would recommend against releasing anything that is even close to being useful, unless you want to spend the next year screening "where can I get a good XTAPI broker" and "I've got XTrader_PRO set up, but I'm getting error 10013, what gives?" emails). the Nuclear Phynance message board is probably a better place to look for business offers.

tarr11 14y ago

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who were market makers. The quants called guys like him "retail" investors and they gleefully picked off all those trades. It's how they made all their money. So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.

marshallp 14y ago
I don't get that. It would be true if he just made a few trades, but the author claimed to be making 2000 trades a day. Over a period of months winning that wouldn't qualify as blind luck.
- DanBC 14y ago
  >Over a period of months winning that wouldn't qualify as blind luck. Why not?
  - marshallp 14y ago
    By that definition you could start claiming everything as blind luck. Why bother doing anything at all, let luck do the work.
  - crntaylor 14y ago
    It's simple statistics. The author was up 4k/day over 120 days. He doesn't say what his daily volatility was, but let's assume 2k (which squares pretty well with his claim that his worst day was a 2k loss). With a quick bit of R code, we can simulate his PnL over 120 days multiple times, assuming he has no skill, and see what the probability of him being up 4k/day is. I'll use a t-distribution with 3 degrees of freedom, which allows big up and down swings (again, accentuating the effect of luck). > pnl <- c() > for (i in 1:1000) pnl[i] <- mean(2000 * rt(120, df=3)) > mean(pnl > 4000) 0.0 That is, there's a zero percent chance that he would have made those returns if he had no skill. And remember that this simulation is overestimating the effect of luck.

vinayan3 14y ago

Have you ever thought of making a trading system that would buy tons of stock when a flash crash happens? It is going to happen again. If your system is ready and you buy before they shut the market down or roll back orders you could make a hefty profit.

crntaylor 14y ago
The difficulty is in identifying what is a 'flash crash' (i.e. a temporary downward blip in prices caused by computer or human error) and what is a genuine downward price movement. If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses. Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.
- beagle3 14y ago
  > The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential. This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash). That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.
  - fr0sty 14y ago
    Nitpick: When exchanges have busted trades there is a "at or below $XX.XX" condition as well as the "between XX:XX and YY:YY" condition. In the Flash Crash as well as the Knight Capital incident "up/down 30% from the Previous Close" was the price collar (anything outside that was busted and anything inside stood). Of course there is no guarantee that the same criteria will be used the next time around so caveat emptor.

lrm242 14y ago

Take a look at the VIX from 2009 until today and you'll understand why you stopped making money.

S_A_P 14y ago
http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol=%5Evix;range=5y;compare=;indicator=volume;charttype=area;crosshair=on;ohlcvalues=0;logscale=off;source=undefined http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...; very interesting, a spike in sept-dec 2009 that roughly follows his PNL...
- jcampbell1 14y ago
  Without a ton of volatility, any homebrew HFT is going to lose to commissions and spread. Basically, he was trading in one of the few periods where is was possible to make money.

Judson 14y ago

This reminds me of an AMA from a few years ago. Really interesting if you are into stuff like this: http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automated_independent_retail_trader_i/ http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...

jspaulding 14y ago
thanks for posting hadn't seen that

tomp 14y ago

This article is missing a crucial piece of data: what was the initial investment.. earning 500k with 10k initial investment is genious, with 10M initial investment it's just another year on the stock market.

beagle3 14y ago
This is a different style of trading than what investors do. He said that he was never more than a few contracts in. A single contract needs $10K day trading margin usually (depends on time frame and specific contract, but it's a reasonable estimate). Therefore, if he was never more than 10 contracts long or short at the same time, the risk was of $100K. Making $500K on $100K over 6 months is good business.
- jspaulding 14y ago
  You are correct. I think I had to have $10-30k in my account. I was extremely low risk so they weren't concerned.
brown9-2 14y ago
Especially considering that from it's lowest point in 2009 through the end of 2010, the S&P500 rose by 71%. Need to know what the starting capital was to be able to figure out if his return beat the market.

curiousdannii 14y ago

I'm sceptical that HFT is good for the public. What did you do to ensure your system wouldn't make a flash crash worse?

beagle3 14y ago
Why is it his business to defend other market participants?
- curiousdannii 14y ago
  I didn't say it was. But everyone should be in the business of not being evil and not exasperating the problems of others. In trading that means not contributing to flash crashes. I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.

rhplus 14y ago

The charts show he was trading between Jun 2009 and Oct 2010. How much of his gains could be attributed to the market recovery in general? The Dow went from about 7000 to 11000, the Russell from about 600 to 800.

washedup 14y ago
Well, in the article he said tat he did not care about direction, he would simple buy when his expected price was up, and sell when down. However, there could have easily been a bias in his model that "preferred" and performed better during upward movements. If so, he got lucky.
- jspaulding 14y ago
  Longs and shorts were 50/50 and my program showed no preference for up or down days. High volatility and high volume was what it liked.
  - debacle 14y ago
    I guess the real question is: what was your alpha in that timeframe?
    - jspaulding 14y ago
      I'm probably showing my ignorance here but what do you mean by alpha? And how is it quantified?
      - solutionyogi 14y ago
        Alpha is how much excess return you had over the market (or risk free return( E.g. if you made 10% when overall market was up 15% for the year, you have negative alpha. [As someone could have bought index and held it through year to generate better return] If you made 20% when market was up 10%, you have positive alpha. That is why everyone in the investment community is 'seeking alpha'.

junto 14y ago

Out of interest, how much capital did you start off with?

jspaulding 14y ago
$10k loan from dad. Built it up to 30k trading manually before my automated program went live.
- intel4004 14y ago
  What were your average transaction costs per trade? 1000 to 4000 trades per day, lets say 2500 on average, translates to about 625K transactions per year. I assume you did not have to pay something like ETrades 3$ commission per future contract, which would result in almost 2M of fees per year.

danso 14y ago

I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines. I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-chess-master-and-the-computer/?pagination=false http://www.nybooks.com/articles/archives/2010/feb/11/the-che... He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers: > The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process. So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?

bloaf 14y ago
Good point, I also wonder about the potential to exploit the algorithms used by the "professionals." In other words, if you can come up with a reasonable approximation of what the pros will do, can you use that information to beat them?
- aggronn 14y ago
  'Theoretically', no. Its hard to be optimistic about these two ideas because while the chess example is a good story, its not analogous for many reasons, ranging from disparity in available information to players to a difference of several magnitudes in saturation. Not to mention HFT just isn't chess. HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right. -- If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.

Sharma 14y ago

Trust me, you earned that much because of your luck. Otherwise Andrew Ng would have partnered with another finance professor and they would have been the richest people on earth!! Imagine trading with their expert systems on global markets. I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!

zdwalter 14y ago
I like to trade Forex using mql4, any suggestion? Thanks.

arbuge 14y ago

I am curious as to exactly why the profitability decreased steadily and rather rapidly all the way to ~zero. The article doesn't seem to expound on that unless I missed something. Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.

baq 14y ago
another comment explained: http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol=%5Evix;range=5y;compare=;indicator=volume;charttype=area;crosshair=on;ohlcvalues=0;logscale=off;source=undefined http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...;
jspaulding 14y ago
I don't think anyone was adapting to what I was doing in particular but rather simply adapting to the opportunities in the market.
- arbuge 14y ago
  Thanks. Makes sense... I guess those opportunities tend to get ironed out rather fast then.

dschiptsov 14y ago

Basically this is a story about a guy who was smart enough to script up his trading tool (he discovered that there is an API and wrote some code to use it). He trade other people's money, using other people's (probably employer's) account and resources, I suppose. His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.) The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.) But this is just my guess.

mattyppants 14y ago
This is not even close to an accurate summary. He never stated that he had any employer backing, and he wasn't collecting market making fees. In fact he was paying brokerage fees which is the exact opposite.
- dschiptsov 14y ago
  He used his own money to test his algos? Come on. Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system. This is why my guess is that he was an employee.
  - oijaf888 14y ago
    An employee of what? He states above that he was paying roughly $200/month for a server and $1800/month for the software/data connections to his broker.
    - dschiptsov 14y ago
      Prior to setting up my automated trading program I’d had 2 years experience as a “manual” day trader. - mom's and pap's 401k?
  - mattyppants 14y ago
    No, he said he backtested them against data he collected from the feeds. He paid for the feeds just like everyone else does with his own money.

iandanforth 14y ago

I'm glad to see a healthy respect for investment among the hacker community. It's traders like this who commit to nearly a full 10 seconds of ownership that are the backbone of economic growth for this country.

unreal37 14y ago
I think there is a healthy respect for computer code that can let loose upon the world and make money.
- mattyppants 14y ago
  Your missing the point that owning part of a company for 10 seconds brings zero social value to the economy. Which I also tend to agree with. Slightly off-topic from the point of the article, which was that he achieved it without backing of a fairly large institution.
  - OldSchool 14y ago
    True it adds no social value, but it is arguably the ultimate hacker's game :) You can actually add a little social value by placing only orders that increase liquidity; those where someone else "takes out" your standing offer or bid. That and the fact that your profits are taxed as ordinary income in the US for equities at least.
  - unreal37 14y ago
    I didn't miss that point - I just chose not to address it. Lots of things add zero (or negative) social value to the economy including Farmville or the gazillionth Instagram clone. That doesn't mean they're not worth doing. And HFT or day-trading does bring some value to the world. It enables companies to go to the public markets and raise capital. And investors to sell their shares without having to wait too long for a buyer. But those positives come along with negatives as well.

OldSchool 14y ago

Great work, very interesting to me. Counter to what we're constantly told through the media this stuff can be done. Doing it year after year seems to be the elusive part. Intuitively, once you've proven your technique on 1000+ trades it's not luck. I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower. The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities. I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.

mempko 14y ago

Wow, fuck that guy. no really. Should be title "How I stole 500k with machine learning and high frequency trading".

oijaf888 14y ago
Who did he steal it from?

deleted 14y ago

[deleted]

hafabnew 14y ago

I've been considering trying HFT myself for a while. I'm competent with Machine Learning and am a Software Developer by day, so I can program and can sysadmin well enough to get something up and running without any trouble at all. But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering. It seems like the logical course of single-programmer HFT trading being: - Find sample data - Build your trading program using sample data - When you're happy: connect to live API and set your trading program loose - Iterate. However, the first step and the third step seem like the ones which require the most research. Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program? Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.

sseveran 14y ago
Nothing in HFT is free. People doing this for a living use precision time protocol in a colocated data center to build their own timestamps. However you can get started by buying ITCH, OpenBook, etc... data which has the full market depth feeds for the various exchanges. There are a couple of brokers out there specializing in the space. If the easy part was building a working model either you got incredibly lucky or the model is wrong.
aiskunkworks 14y ago
Find sample data: https://quantquote.com/ https://quantquote.com/ API (Colo): http://www.lightspeedinstitutional.com/automated-trading/ http://www.lightspeedinstitutional.com/automated-trading/
monksy 14y ago
Quality sample data and historical data to backtest is hard. Symbols change, stocks split, dividends are announced etc.

hnruss 14y ago

Just because you CAN do it doesn't mean you SHOULD. Even if you don't think of it as "gambling", you're still taking in tons of money without providing any tangible benefit to society. If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.

jspaulding 14y ago
I didn't lose any sleep but I do indeed hope to contribute more to society with future projects.
wyan 14y ago
You mean, something like Instagram?
- sjm 14y ago
  Not really defending the parent, but believe it or not Instagram provides something that makes a lot of people happy. Winning a bunch of money on the stock market does nothing for anyone but the winner. Not really a fair comparison.

quant123 14y ago

There is an air of either incredibility or sheer jealousy in these comments. Nevertheless, I just wanted to tell the OP that he did a great job. Thanks for sharing. I work in the finance industry as a quantitative software developer, and it certainly is not an easy job for one person to do. In fact, I tried (independent of my professional work) doing this myself, and I ended up losing a lot of money. If people are trying to do this, please please be careful. Big companies, like ones I have worked at, have technical and human resources that are vastly more powerful.

jspaulding 14y ago
thanks quant123 ;)
stmartin 14y ago
Did you mean incredulity instead? I'm not jealous of the money he made - I'm a sw developer, I have a good salary, my wife's an accountant, and we do well - what I am 'jealous' of is not having the time to do something like this, regardless of the outcome. Clearly, I'm prepared to take the risk... but if you have 2 children, you may think differently about spending your time doing this instead of on your family. Money cannot buy happiness. It can pave the way to it, but happiness only comes through surrender to another person or a cause greater than what one can accomplish by him or herself, and in no other way. And when it comes, it's an unintended side-effect, rather than something that one can pursue (which is one of the reasons I hate the 1st amendment and the way it's worded).

loup-vaillant 14y ago

While this was quite fascinating, I couldn't see this form of trading as anything but a zero-sum game. Some players win, the other lose, like in any other game. Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?

loup-vaillant 14y ago
Okay, enough with the downvotes. What's wrong with my question? I didn't mean to bash a field about which I know next to nothing. My question is genuine. Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had. Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth. But. The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum. Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"
gd1 14y ago
A strange position to take, I'm guessing you've absorbed it from the media somehow. His machine was in the market enough to trade 4000 times a day (I would suggest passively, or he would have been eaten by the cost of crossing the spread), so he was basically continuously offering a service to the market - an offer to sell and a bid to buy at the price he thought fair. Do you demand to know what value your local 7-11 provides by selling you milk? The value is that you can buy milk 24 hours a day, so you don't begrudge them the 15c a carton they are making. And they probably charge more than your local supermarket for the convenience too. The dirty thieves.

dwk9080 14y ago

If you're interested in working on this kind of stuff in the San Francisco area, send a resume to Headlands Technologies. careers@headlandstech.com

ianstallings 14y ago

I could do the same with poker. Or I could lose it all. That variance thing is a bitch.

mrchess 14y ago

Cool article but I hope people don't start trying to follow this path. Ask yourself -- why did he stop?

jspaulding 14y ago
yes. and yes.

stcredzero 14y ago

Is there a way to do this with Python or Ruby? I could just as well program this in C#, but I have a friend who can code a little, but doesn't really need everything in C# to do what he wants. The value add of offering the simulator, including the taking into account the bid/ask prices and a stochastic model for latency. Combine this with a web based code editor and easy hosting, and I think this would be a viable product.

HockeyPlayer 14y ago
Sure, his program could easily have been done in almost any language. He wasn't competing on speed, which might have excluded languages like Python.
owyn 14y ago
Nope, there is absolutely no way to do this with python or ruby. :) Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks: http://www.rapidquant.com/features http://www.rapidquant.com/features A C++ library with python bindings: http://quantlib.org/index.shtml http://quantlib.org/index.shtml And a low level library for data analysis: http://pandas.pydata.org http://pandas.pydata.org
- grumps 14y ago
  If you use their library/SDK/Framework.... how do you actually trade, do you still need a brokerage? Sorry a bit new to this field.
  - owyn 14y ago
    Yep, almost any of them have an API these days. Your software would make the list of trades which is uploaded and executed... Oh, I forgot about another backtesting framework for python. It has a built in IDE as well. http://www.quantopian.com http://www.quantopian.com
    - 14y ago

unreal37 14y ago

There is a coursera course called "Computational Investing, Part I" that I am taking that aims to build a market trading simulator to test a trading model. It just started so it's not too late to join. https://www.coursera.org/course/compinvesting1 https://www.coursera.org/course/compinvesting1

impotter 14y ago

Might I suggest an alternate title / next startup? "Learn how this unemployed soccer dad made over $45k a month working from home! His winning system can be yours for only $200!!!!"

runjake 14y ago

Relax with the vitriol. The guy is sharing an interesting personal story, not providing a step-by-step HOWTO or recommending people follow his suit. In fact, the article is really an ad for his startup Courseware.

jspaulding 14y ago
CourseTalk. CourseTalk! http://coursetalk.org http://coursetalk.org .. You are right kind of :) But I've made a decision to start reaching out generally so I can attract cool people to work with on whatever projects I may be interested in in the future.
- runjake 14y ago
  By the way, no offense meant by the advertising thing. I enjoyed the article. Although, as a technical person, would've enjoyed more details on the code and algorithms.

ww520 14y ago

Very cool article. Gave a detail explanation of the ENTIRE process. Thanks!

ChuckMcM 14y ago

The high point of my trading was October 2009 when I made almost 100k. After this I continued to spend the next four months trying to improve my program despite decreased profit each month. Unfortunately by this point I guess I’d implemented all my best ideas because nothing I tried seemed to help much. It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.

jcfrei 14y ago

It is pretty clear from his own graph that this stopped working in october '10. which was an eternity ago in terms of financial markets. algorithmic trading has increased manifold since then, so finding another arbitrage opportunity like he did is only going to be more difficult.

rbc 14y ago

I started the infrastructure for this kind of thing a while ago. It's BSD licensed. It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems. Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at: http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ http://home.rbcarleton.com/rbc/software/smalltalk/VSA/ I won't try to advertise it as a complete solution, but it might be the start of one.

padobson 14y ago

Don't do this with your own money. Found a startup building HFT tools, and then raise money for it, and use other people's money to test your tools. If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do. This is risk management.

nesu 14y ago

Wait. This is not HFT. There's a huge difference between automated and high frequency trading. What he does is only automated scalping at best (or at the fastest). Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders. HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.

crucialfelix 14y ago
quite true. but HFT has become a buzzword like "the cloud" and even many financial industry specialists claim to do HFT but in fact they are just market making. auto-scalping or even hedge funds that day trade with robots are not HFT. this: http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first-order autocorrelation of returns. it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.

HockeyPlayer 14y ago

I run a 12 person HFT group in Denver. This is an excellent description of how an individual can bootstrap themselves into success. Great story, and nicely presented. The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities." Which doesn't really tell you much.

rorrr 14y ago
How well are you guys doing? How much money did you make?
jspaulding 14y ago
Thank you. I'm looking back through my code and there are really a lot of indicators. It would get pretty technical to explain them. They are all explainable it's just that each one corresponds to slightly different market conditions and I just didn't want to get into it.
- HockeyPlayer 14y ago
  Fair enough. Most people considering trying this probably have a few ideas for indicators. But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.
  - jhales 14y ago
    Yes, I would find this very interesting. Doesn't matter if the indicator is now defunct.
- deathflute 14y ago
  Thanks for an excellent post. I would like to see any one indicator explained in detail as well. I am sure it would be of interest to a lot of people.

mcarvin 14y ago

Very important to understand that making $500k speculatively is not evidence of an 'edge', nor is trading frequency evidence of the absence of luck. From March 2009 through much of 2010, the market was strongly bullish - if his algorithm showed a positive market bias then his returns would primarily be a function of timing (read luck: and there are a million variants on the nature of the bias that could be unwittingly responsible for his returns, despite the frequency of trades). We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with? Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.

fr0sty 14y ago
1. Author graphed his daily returns which should give you a handle on his volatility. He also stated his max drawdown was $2k. 2. Author has stated elsewhere than he began with $10k in seed capital. 3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number. 4. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped. Edited to fix numbers.
- mcarvin 14y ago
  1. I cannot get even a remote sense for the nature of his risk exposure from looking at his daily returns. 2. ok. 3. The point here is that a systematic bias in his algorithm will expose his trading strategy to the good graces of market fortune (luck) regardless of whether he trades a million, billion or once a day. The source of the bias is irrelevant. 4. did not see where he said that but that very much confirms 'timing' / which in this case I interpret as luck as being at least a contributing factor.

namank 14y ago

Might've been what it was a couple of years ago but this post, dated today, is the perfect advertisement for the author's current business. Well done! PS: no sarcasm intended, it truly is an excellent advert.

hellsten 14y ago

A rising tide lifts all boats. "Price move predictions" sounds as effective as tea leaf reading.

doki_pen 14y ago

Isn't profit meaningless without knowing initial investment? I read the first few paragraphs and got bored. Why not say upfront what the bankroll was to start? edit: I found what I was looking for in the comments.

JoblessWonder 14y ago
Just to save someone else from having to look for it like we both did, his initial investment was $10k loan from dad. Built it up to 30k trading manually before my automated program went live.

thisismyname 14y ago

How much did you start with?

dmmalam 14y ago

Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset. I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US. Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest. Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money. I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.

jspaulding 14y ago
I don't know the exact definition of HFT but I did run my algorithm from a server collocated with my broker close to the exchange. I modelled lag time in simulation and not having it collocated certainly would have hurt.
beagle3 14y ago
In the US, HFT is mostly synonymous with "all out tech war, flooding the order queue so your less-equipped peers get lags". (Nanex publishes analysis on these events, which are not occuring several times a month and keep accelerating). In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions). > Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt. Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.

shizzy0 14y ago

And nothing of value was created. Just kidding. It was a fascinating article. Thanks for sharing.

suyash 14y ago

@Author ? Can you create an online course and teach us all? Thanks

photorized 14y ago

I am skeptical for two reasons: 1. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2. You don't just stop using it. There should be more to this story.

frankster 14y ago
Can I recommend that you read the article and you will find therein the answers you seek!

gubatron 14y ago

of course no firm would respond to his noob low yield model. The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this. Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm. Kudos though for taking on the task of learning how to code and making money with ML.

jspaulding 14y ago
This is such an odd comment. What motivates you to write such a comment? And what makes my system amateur and noob in your eyes?

derryl 14y ago

This might be a dumb question but... what system does your algorithm interface with? Last time I checked, NASDAQ and NYSE don't exactly publish API's

deleted 14y ago

[deleted]

nashequilibrium 14y ago

Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster. The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.

jspaulding 14y ago
There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck. Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone. Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading. I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.
- quant123 14y ago
  agreed. jspaulding got it right although I can see nashequilibrium's skepticism as the US futures market is incredibly crowded (read: competitive, no free cookies).

confluence 14y ago

A lot of people are stating that this is like gambling - it is - but not in the sense that you think. Firstly he doesn't use his entire bankroll on each trade, secondly he goes long-short consistently over very short periods of time, thirdly he's too tiny to actually move markets, and fourthly he is in and out within a day - where his max var. is 100% on thousands of tiny trades. Think of it like this - he has $100 - he bets $10 of that each day. He can blow that $10 - no problems. Worst case he runs out of capital over a period of weeks. He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks. Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+). In the short term however (sub-decade) - they can't price jack. Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that. There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw). Both work. Both work well. And will continue to do so as long as markets exist.

JuDue 14y ago

"Since I'm no longer running my [half million dollar] program I’m happy to tell all" This does not ring true.

sejje 14y ago

I don't understand trading enough to even understand many of the terms in the article, however I'm curious to one thing: is it possible that a program could be written specifically to exploit yours? And/or is that a potential reason it became unprofitable?

FireBeyond 14y ago

Hopefully not buried too deep, but any books recommended for getting into day-trading, either manual, or algorithmic? Kindle preferred, but definitely not the deciding factor.

leminhhai 14y ago

Amazing, love your project

dataisfun 14y ago

This is an activity that adds no value to our world.

MonteChristo 14y ago

The prototypical example of why a tax on financial transactions is urgently needed.

curationary 14y ago

how is it "machine learning"? A few if-else doesn't make it "machine learning", or does it? - @curationary

jspaulding 14y ago
It takes data (last 4 weeks of market update data) and learns how to predict price movements and how to make profitable trades.

vincegata 14y ago

Thanks for the post, it's very inspirational. Could you explain this part, specifically what do you mean by "bucket". "To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."

vincegata 14y ago

Thanks for the post, it's very inspirational. Could you explain this part, specifically what do you mean by "bucket"? "To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."

mrs1238 14y ago

One comment...everything is stated in this book "Why You Could Be Destined To Fail in Trading and How You Can Avoid it!" http://outskirtspress.com/estetrader/ http://outskirtspress.com/estetrader/

witoldc 14y ago

Let's see the real meat of the article: your trading program and algorithms.

datashaman 14y ago

Excellent article on this posted here: http://www.lrb.co.uk/v33/n10/donald-mackenzie/how-to-make-money-in-microseconds http://www.lrb.co.uk/v33/n10/donald-mackenzie/how-to-make-mo...