14 ms·
Money Machines – An Interview with an Anonymous Algorithmic Trader
- organicdude 8y agoThis was a really good read. Thank you.
- TheOtherHobbes 8y agoAgreed. Very thoughtful and interesting.
- nodesocket 8y agoI've been playing with the trading platform Alpaca[1] and wrote some code that essentially pulls trade information (each tick) for a handful of tech companies I am interested in. It's a lot of data, but once I have the data stored, I'm now thinking... Ok now what? I have volume, price, time, and a lot of other metrics. Not sure where to go from here honestly. Should the code evaluate the data in real-time for each trade, or wait to the end/start of the trading day and do some evaluations then make a decision? Obviously, I am not going to be able to compete with the big boys using really sophisticated code (written in a low level language like C), executing in colocation right next to the exchanges, running Apache Hadoop and Spark. Still, I am interested in just tinkering with simple algorithms with the data. [1] https://alpaca.markets/ https://alpaca.markets/
- web007 8y agoI don't know about Alpaca, but Quantopian lets you do the equivalent and then also backtesting on historic data.
- KloudTrader 8y agoShameless plug: We built a hosted Heroku-style platform for algo trading - https://kloudtrader.com/narwhal https://kloudtrader.com/narwhal Created it after not being able to do livetrading on Quantopian. Here's the associated python library and documentation: https://docs.kloudtrader.com/ https://docs.kloudtrader.com/ https://github.com/KloudTrader/libkloudtrader https://github.com/KloudTrader/libkloudtrader
- casual_slacker 8y agoThe python library Backtrader is popular and has good intro docs. https://www.backtrader.com/docu/quickstart/quickstart.html https://www.backtrader.com/docu/quickstart/quickstart.html
- khold_stare 8y agoI'd say play around and see if you can predict the price - treat it like any machine learning/optimization problem - if you want to learn and have fun. If your goal is to make money, unfortunately it'll take a lot more. I've worked at a high frequency trading company, and now work at an algorithmic hedge fund, and you really need to scale horizontally to make money overall. There's a lot I can talk about but here are some points: - Most of the time you don't trade instruments in isolation. They are all correlated (even if inversely) one way or another , so movements in one will affect others. Having a myopic view of just one instrument will have too much unexplainable randomness. - The other way to scale is having more data inform your pricing - i.e. don't just look at the prices of one or more instruments. Look at twitter, look at weather reports, news etc. It kind of suggested that in the article. The meaningful movements in the market are most of the time due to truths outside of the price itself. - You have to decide whether you want to trade directionally and long term (like a hedge fund) and take on positions over time, or trade in and out of positions quickly (like high frequency trading firms) to minimize risk. You can be anywhere along that spectrum. The faster you are, the more myopic you can be and just react to current trading activity of other participants. The more long-term you are, the more you have to look at the big picture, more data, more instruments, making sure you have the right balance in your portfolio (whatever that means to you). I'd be happy to expand more :)
- richardw 8y agoVery interesting. No obligation but if you're bored feel free to expand!
- khold_stare 8y agoSure! I'll just ramble a bit about HFTs. I'd say starting a "pure" HFT firm is incredibly hard nowadays. There's a big upfront investment and huge operating costs, and there's a lot of consolidation happening in the industry. In general, most hardware and software is tailored to high-throughput, and a huge part of the effort of HFT is fighting all that to prioritize latency. If you want to compete with the very best firms, that means renting rackspace in a colocation with the exchanges, overclocked CPUs, expensive network cards, FPGAs, paying out the nose for the most accurate market-data feeds, and many order-entry connections. Then you have to tweak the BIOS, kernel settings, isolate your processes on cores, write your own userspace networking stack etc. etc. etc. On top of that if you want to trade between different colocations, you might need a network of microwave towers because fiber is too slow (light travels about 2/3 slower in fiber because it effectively "bounces" along the fiber). At the end of the day though, if your algorithm or strategy doesn't work, you won't make any money - speed is only a part of it. At the previous firm, we had a market making desk that was very profitable, but was quite "slow" by HFT standards (triple digit microseconds vs sub-microsecond response times). You still need to be smart about it. Having said all that, I think HFT gets a bad rap, about being a waste of resources/talent, and somehow "stealing" from the common man. The truth is, they are only "stealing" from each other - they compete to remove the tiniest inefficiencies in pricing. They ensure the cost of trading is actually as low as it can possibly be. By themselves they don't have enough capital to move the market in any meaningful way. In fact, most flash crashes occur when HFT firms step out of the way! They move out of the way of huge market movements because they can't cushion the blow without taking an unnecessary risk as a business. It's like asking your plumber to fix your sink while your house is on fire. Market orders, stop orders and panic are most of the problem when it comes to flash crashes. PSA: Please use limit orders :) I might ruffle some feathers with the comments, but I think overall HFTs aren't the bogeyman most people think them to be. Sure there are some bad apples, but that's true of any industry, and it comes down to the people - not the system itself.
- lowdest 8y agoI've spent a huge amount of time playing with similar stuff. The goal is to predict anything that you can make money on -- this can be obviously be price, but you can also try volatility, or predicting a probability distribution. Try relaxing constraints to find your signal: price ranges instead of exact prices, date ranges instead of exact dates. If you can output a probability, distribution, or confidence interval as well, that information can be used for position sizing (see Kelly criterion.) If you start expanding beyond a small set of specific companies, don't forget to find data for delisted companies to avoid survivorship bias.
- xiphias2 8y agoI looked at Kelly criterion, and what's interesting is that I was just thinking yesterday about whether I should optimize E(log(wealth)) - c * std(log(wealth)) or E(wealth) - c * std(wealth) The first one is better for building up wealth over a long time, but when I looked at Modern Portfolio Theory, it is suggesting to use the second formula. I'm also planning to spend about 5% of my wealth every year, and the second number gives me lower risk in the short time frame, so I think something in the middle would be a better criterion.
- lowdest 8y agoKelly sizing is optimal long term, but is highly volatile. A common practice is to size by a fraction of Kelly, to reduce variance and since risk of ruin increases fast after 1.0 * Kelly. It also allows room for error in the probability predictions, to avoid going over 1.0 * Kelly by accident. I try not to worry about stdev and focus on the soundness of my process, but in practice I have money split into higher-risk where I try to be clever, and low-risk failsafe investments to raise the floor of the worst case scenario.
- marketgod 8y agoThis is kind of a backwards philosophy because there are so many things you can test in the market. What you want to try to do is find an edge and then capitalize on it with the tools at your disposal, just my opinion. Try this strategy and see if you can automate it. It's called the wheel strategy. What you want to do is setup a $200K account, and sell puts with certain deltas, and then if assigned, sell calls with certain deltas, until you get called. This will get you a relative return of up to 20% per year if done well. You can use your skills to figure out the best delta position to trade, you can look for high volatility to take advantage of. Once you get this completed, you can look for other trends. I always recommend starting with this because it's the easiest way to make some money in the market. It's a lot of work to do it manually, but automated, it would let you earn income while you are not doing much. You will also want to read and understand what calls and puts are, and what selling/buy calls/puts entails. Probably want to do this with stocks like Disney, GE, and so forth, something with large market caps, small movements, and something you are willing to hold long term and which will not go bankrupt/drop significantly in the near future, disclaimer: I do not like GE at all, however, you also want to try this with 1 contract to start, because 1 contract is 100 shares. You can probably work with a smaller starting capital. If you get this implemented and working, it can generate passive income for you in a sense, or you can blow an account. Edit: You want to settle/close out around earnings because the volatility is insane at that time.
- tomp 8y agoOption selling is “picking pennies in front of a train”. Small wins, big losses.
- marketgod 8y agoI only buy calls or puts, however, you have to look at the wheel in a different view. You are making pennies in front of a steam roller, but you are getting paid to buy the stock you want. Most people say, if AAPL goes to $140, I will buy. Well this lets you sell puts until AAPL hits $140, then you get assigned, so you get paid every week or 3 weeks it goes up. Then when you have the stock, you sell a call against it which helps reduce your overall cost basis. This is a fairly easy strategy to automate and you can use a small account to do so.
- floatrock 8y agoFrom the article: > You’ve already got robo-advisors, which use algorithms to manage assets for retail investors. We’re also probably only a few years away from you being able to log into a brokerage account and run a sophisticated institutional-grade algorithm yourself. > People tend to assume that the diffusion of these technologies is a good thing. I’m more ambivalent. I think it could be a big mistake to have the population at large play around with algorithms. Some people who are very good at it might benefit from having access to this broadened toolset. But most people will just end up paying too much or make bad decisions because they’re being given access to a technology that they aren’t equipped to do anything useful with. They can lose money with it, however. Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges.
- infinite8s 8y ago> Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges. Those folks with fiber lines into the exchanges are definitely not running hadoop.
- kaolti 8y agoFrom my limited experience it seems the game is less about "predicting" price - which seems to be impossible, and more about placing strategic bets that will give you a positive return over a long period of time.
- planktons 8y agoThe key bit here is where he says that a lot of people will lose their jobs soon. There are tons of jobs that exist only because of momentum. The financial incentive to pay all these people will dry up and it’s going to hurt. And we won’t opt for UBI until many people have suffered a great deal. People do not seem to appreciate the gravity of what is coming. Basically every time I bring up jobs, ai and society someone comes out of the woodwork to insult my character or sarcastically dismiss me. Never any substantive counter-arguments. I’m really looking forward to what you guys will come up with this time. Always such a pleasure.
- scottlocklin 8y agoOK, here you go: what large group of jobs has been eliminated by "AI" so far? No time limit answering, unless you die first, in which case "time's up!" My assertion, made in public and under my real name is that barring some giant breakthrough, nothing like this is going to happen any time soon. As far as I can tell, as an active worker in the field (erstwhile finance FWIIW), "AI" is a force multiplier for statisticians, and basically that's it. Jobs have been eliminated in America because of poor industrial and trade policy. Nothing to do with "AI." I get extremely irate when people make the assertion that "oh, well, the jobs are going away soon anyway" -using this complete and utter falsehood as an excuse to continue the looting of the country. The guy interviewed here is a real mixed bag. Some of what he says is nonsense, some of it is accurate. FIRE has basically been a vampire squid on the economy without adding much value. Most of it could be "automated" away by using a magic 8-ball; it doesn't actually provide any value. Except the politicians rice bowls depend on the status quo.
- ericol 8y agoI work for my company as a lone coder in a coworking space, an ocean away from the rest of the company. What gives, you my ask. I sit in one corner of a 6 desks arrangement and the rest of these desks plus another arrangement of 6 desks are occupied by a company in the RPA bussiness. The CEO of the company sits right next to me (When I started working in this space almost a year ago all these desks were empty, so obviously I took the best position) so we have a good couple of chats the days he shows up (They have a "formal" office in the city). They have a couple of whiteboards in the wall next to my right, where they use actual, physical pseudo kanban boards with postits and shit that get reorganized probably weekly, because they can't cope with the amount of new work they are getting. They have doubled the number of programmers since May, in a country (Argentina) where things are not looking that bright for nobody. From my desk, I can _feel_ the pain of all the jobs they are killing. I can't tell you _exactly_ what "large group" of jobs they have killed because the chats are not that intimate, but I see the list of companies they are working with, and all the big ones from the country are there. all of them. They _are_ killing jobs, a plenty, without discrimination. And this is barely starting.
- osrec 8y agoI'm an ex algo trader (currencies and metals) myself, and found this article interesting and mostly accurate. A key takeaway is: "One of the fallacies that people have is the assumption that because the people who are working at certain firms are smart, they must be successful." The number of times a new hotshot hedge fund moved in to office building where I rent, only to be gone without a trace in a month is staggering. Usually, they'll raise money from an unsuspecting family office or two, employ a techy and an admin person to do the heavy lifting, whilst the smooth talking CEO alt-tabs between Bloomberg and reddit all day. Most have terrible risk management strategies, and the majority are simply trend following (sorry, machine learning/AI). Some of these firms are also my clients, and I have often needed to write off money they owe because they've literally gone bankrupt overnight.
- xiphias2 8y agoWhy are you ex algo trader? It still seems like a growing market, so it's interesting that you stopped doing it. I'm right now working on optimizing my long term investment strategy, but the hardest thing is that there's nobody to talk to it about it (unlike when I have other coding problems), and I have no idea when I'm doing something totally stupid in my code base. Getting the right metrics to optimize is really hard, as mostly I'm just looking at numbers and graphs.
- osrec 8y agoIt's a crowded market, but not necessarily a lucrative one. I worked for a large investment bank, was paid well, but was also rather unhappy. I'm not a fan of office politics and also hated the fact that what we were doing was actually quite unsophisticated, yet we dressed it up like the most complicated thing ever to clients. We were using wavelets for analysing market micro-structure and using any generated signals to influence our trades. Wavelets were the topic of my thesis, so I was a good fit for the team. I know it sounds complicated, but if you looked at the signals we generated, they looked awfully similar to a rather basic trend-following style regression model. I believe we found a very complicated way to compute signals that a simple trend follower would compute in far fewer CPU cycles. Anyway, I got a fairly decent bonus one year (because markets were trending nicely), and decided I want to set up my own tech company. I was always a nerdy guy, and I thought I could contribute more in the tech space. With hindsight it was a good move, and financially I believe I am now better off than most of my trading colleagues. My advice to new guys joining our desk (maybe it helps you a bit): focus on your exit, more than your entry, and if you can help it, be really, really, really lucky :P
- ggm 8y agoExploiting microtime moments in information flows is (in my view) parasitical on the real element of information-flow in markets. I don't think profiting in 10us of speed advantage seeing a thing, and acting on a thing, is itself market-informing and so it has to be (kind of definitionally) market-distorting. algorithmic trading is market distorting. its not actually helping "us" with real value for our futures, funds, company value. Its not informed trade on real conditions, its trade on the mechanistic moments in trading message flows. its a really bad feedback of second-order differential.
- floatrock 8y agoThings you can do with that 10us advantage like frontrunning are parasitical, but why is a speed advantage itself a bad thing? Surely waiting a day to act on the information isn't helpful. How about a minute? A few hundred ms, about the speed of a conscious awareness? Getting down to 10us seems like the logical conclusion of all this. I think the more nuanced view was talked 3/4 of the way down: with algo trading, prices are based on the models not the fundamentals. And then you add in models of what the other guys are doing. And pretty soon you don't care where the price should be, you just care about computing in 10us where everyone else thinks it's going to be in 15us. > Another fallacy in the lead-up to the financial crisis was the assumption that financial markets were so efficient that participants didn’t need to do the underlying work to figure out what the securities were actually worth. Because you could rely on the market to efficiently incorporate all available information about the bond. All you need to think about is the price that someone else is willing to buy it from you at or sell it to you at.... if they assume that the price of an instrument already reflects all of the information and analysis that you could possibly do—then they are vulnerable to that assumption being false. That's now flash-crash territory.
- ggm 8y agoI used to believe the only socialised goal of a market I appreciated was setting price. I now realize a significantly higher percentage of trade is not aimed at that outcome but solely at extraction of profit in the movement of price. At the point nobody cares what the price is but only cares about it's Delta and velocity, I'm out. Why are we doing this?
- cryptosteve 8y agoBut isn’t there a strong financial incentive to try to understand why you’re doing what you’re doing, whether it’s an algorithm or a human executing the trades? Otherwise it seems very easy to lose a lot of money. I can't find the article, however a guy once drop a million in bitcoin using a trading bot on a short sale.
- thibautx 8y agoThere's typically multiple layers (at different points between tick to an order hitting an exchange) of risk-management/circuit-breakers that prevent these types of things from happening at most shops that know what they're doing. No one wants a repeat of Knight Capital's 2012 meltdown. I would imagine it may be easy for an unsophisticated/hobby "algo-trader" to make this type of mistake but with over a million in capital, you should probably be a bit more prudent with risk management.
- deleted 8y ago[deleted]
- bdibs 8y agoIn Ray Dalio's book Principles, he attributes his success to modeling the markets with computers, but never relying solely on them. All trades would be looked at by an actual human, making sure that everything made sense and lined up. I have my own doubts about a purely algorithmic approach (for now at least). Computers are great for many tasks, but for something as inherently irrational as the markets I think that should be at least partially moderated by a person with a deep understanding of the markets.
- pc86 8y agoWhat trades are you referring to when you say they "would be looked at" by a person? The interview states quite the opposite: > The level of human oversight varies. Among sophisticated quantitative investors, the process is fairly automatic. The models are being researched and refined almost constantly, but you would rarely intervene in the trading decisions of a live model. A number of hedge funds, mutual funds, and exchange-traded funds (ETFs) run on auto-pilot.
- lordnacho 8y agoAs an algo trading guy myself I see the recent ML -> trading stuff as a bit of a bubble. There's a lot of job ads out at the moment looking for ML people in trading. Quant firms have not done great recently, and they all want to find new things. The question is whether there's anything out there to be found with ML techniques. Also a lot of the well known quant firms that hire loads of phds don't actually do anything particularly interesting or profitable. They are simply benefitting from institutional momentum: there are still pension funds and family offices with a mandate to stick money in trend followers, so the big trend following names hold on to their assets.
- JackPoach 8y agoThe biggest problem with algo trading is that it all works fine and dandy, until it doesn't. You can make money several years in a row and then lose it all. Nassim Taleb makes this point exceedingly clear in his works.
- smileypete 8y agoGood topic, would also recommend reading this guys stuff - Robert Carver, ex AHL: https://qoppac.blogspot.com/ https://qoppac.blogspot.com/ https://twitter.com/investingidiocy https://twitter.com/investingidiocy https://www.systematicmoney.org/ https://www.systematicmoney.org/ Also has a few videos scattered around that are worth a look too.
- JoblessWonder 8y agoIs there an article somewhere that shows what a basic or advanced trading algorithm looks like and explains how they work? I'd be curious to see one walked through (even if it wasn't effective at making money and was just a proof of concept/example.)