16 ms·
Introduction to Zipline: A Trading Library for Python
- chollida1 10y agoI've typed and deleted this post a few times trying to find a way that it doesn't sound kind of pompous but if it helps save one person alot of money then screw it, I'll sound pompous.... I get asked quite a bit on how to start doing algorithmic trading and the first thing I always tell people is don't. I think I've said this many times now but the number of people who come at it with the thinking "I'm a computer scientist. I'll just fire up R or python and apply some machine learning to the markets and watch the money roll in" is staggering. I mean each day 100's of Phd's start with clean market data, more data sources than you could possibly think of and statistical back testing systems that have 1000's of man hours put into them, trying to find a way to make money. After all of that if you really want to I wrote this in response to an Ask Hacker News a little while ago https://news.ycombinator.com/item?id=11352562 https://news.ycombinator.com/item?id=11352562 TL/DR - focus on time periods greater than a day - expect to lose money - expect to take a year to figure out some edge in the market - most decent trading strategies that a normal person can use come from economic/market insights first and technology second. The site: https://www.quantstart.com/ https://www.quantstart.com/ is also decent at bringing you up to speed on the math you'll need to know though I believe that the material there oversells how easy it is to find a decent trading strategy.
- canttestthis 10y agoI never understood this. If its possible to be a profitable independent day trader, and we know it is because many are, then it should be possible to code the rules you follow and become a profitable algo trader.
- someone7x 10y agoIt's a slightly different thing though, being an independent trader isn't the same as becoming an independent trader. A trader that is independent may still have advantages that prevent some random programmer from bootstrapping his/her way to also being an independent trader. Just thinking out loud though, I'm not a trader of any sort.
- machbio 10y agoIts not as simple as converting rules followed by independent day trader into algorithms - most of them just do not rely on just technical analysis but also on fundamentals which is hard problem to decipher with all the hype around any stocks in the news and analyst of institutional investors influencing with the media news through various sources. The best for you to start learning - would be to get a start on how the market works - my suggestion would be - https://www.amazon.com/dp/B000THOD1G/ https://www.amazon.com/dp/B000THOD1G/
- asdfologist 10y agoA lot of day trading is done based on intuition, which is difficult to translate to code. Also, keep in mind your survivorship bias... Most "day traders" lose money and quit the game.
- RussianCow 10y agoThere are many strategies that are difficult to encode into an algorithm. When I briefly did day trading independently, my most profitable strategies played on public perception and (over)reactions to news. Those things are very, very difficult to properly automate (though many companies try, with varying degrees of success).
- semi-extrinsic 10y ago> Those things are very, very difficult to properly automate (though many companies try, with varying degrees of success). Case in point is the fluctuations in Warren Buffett's fund whenever Anne Hathaway is in the news. http://ftalphaville.ft.com//2011/03/28/528481/for-the-bots-anne-hathaway-is-not-warren-buffett/ http://ftalphaville.ft.com//2011/03/28/528481/for-the-bots-a...
- dragonwriter 10y ago> If its possible to be a profitable independent day trader, and we know it is because many are, then it should be possible to code the rules you follow and become a profitable algo trader. There's actually a logical error here: if trading results were essentially random, a certain subset of traders (including day traders) would, at any given time, have profitable records. But you would not be able to derive (and, therefore, implement in code) any set of rules which would make automated trading profitable. You would just either be lucky, or not, as an algorithmic traders just as you would as any other kind of trader. Now, I'm not saying that trading profits are random, but the existence of some profitable traders does not mean that there are rules you can deduce from their behavior that will guarantee profitable trading when implemented by someone else (either in an automated system or otherwise.)
- gedrap 10y agoAgreed. Probably a large fraction of 'profitable independent day traders' mention by parent depend on luck. It's one thing to post a profitable month/year, whole different thing is a profitable decade. It's relatively easy to have a profitable period on a strongly bull market. But it's irrelevant if any 'unexpected' (people call such events unexpected despite the fact that they tend to happen regularly over time) event such as 2008 crisis will completely wipe you out.
- lojack 10y ago> If its possible to be a profitable independent day trader, and we know it is because many are You're forgetting about the many that aren't profitable. You're also forgetting about the majority that are less profitable than the market average. The latter is probably the easiest to overlook. Yes, they are technically profitable, but so are packaged portfolios of stock held for long periods of time. If you can't be more profitable than someone who puts in zero effort, then whats the point of putting in more work?
- roywiggins 10y agoThere are lots of successful software developers too, but writing out a series of rules to automate writing code is not going to happen soon. Unless day trading is substantially easier than programming, an autonomous robotrader that can make money seems unlikely.
- SeanDav 10y agoLeaving aside the question of how many profitable day traders there actually are, there are profitable professional poker players out there but no bot can compete with them. Now scale up the complexity of poker by at least an order of magnitude and you end up with the financial markets. Sure, it is possible to to program very niche behaviour, but we are nowhere near any sort of program that can act as general "day trader".
- bsbechtel 10y agoThe comments here are interesting...Looking at your question from a different angle, yes, of course it's possible to be a profitable algo trader - set your first investment to buy("SPY") and do nothing else for 20 years. What you are asking though depends entirely on your risk tolerance and what you are benchmarking your strategy against.
- hellofunk 10y agoYou vastly underestimate the complexity of a discretionary trader's intuition and experience. You cannot just replicate years of human experience with computer code so easily.
- gaius 10y agoIndeed. Think about how many tens of millions of people drive everyday... But a self-driving car still proved elusive...
- selectron 10y agoThis statement is too general. You could of said the same thing about chess, there are chess Grandmasters who devote their lives to studying the game yet computers play chess at a much higher level than any human.
- semi-extrinsic 10y agoChess is rational, following a easily understood set of rules, and both players have perfect information. The big problem has always been analysing all future possibilities. The stock markets are very far from a rational, perfect information game with simple rules.
- hellofunk 10y agoIf you honestly think that living a real human life, with all the concurrent decisions that are simultaneously and relentlessly made on a micro and macro level throughout every second, every day is the same as a single game of chess, then by all means, go trade the stock market and show us how it's done.
- gaius 10y agoAn organisation at the scale of IBM was able to create, after many attempts, and vast investment, a computer that can beat Grandmasters. That insight isn't useful to an individual trying to do the same.
- abrookewood 10y agoNo idea why are you being downvoted for expressing an opinion and asking for an explanation ...
- deutronium 10y agoI don't know anything about algorithmic trading, but I'm just wondering, are the people who do make money lots of money out of it, those who have servers close to the data source, who do high frequency trades.
- harmegido 10y agoNo. Compare any big banks revenues from trading against the revenues of a public HFT firm like virtu. No contest.
- cloudjacker 10y agoalgorithmic trading doesn't equal high frequency trading. people conflate these a lot even though they know better. high frequency trading requires algorithmic trading. but algorithmic trading can implement warren buffet or suzi orman's style if thats what you wanted.
- sjbase 10y agoAgreed, HFT has become a catch-all term for anyone trying to make a marketing/political statement about trading. Like the term "big data" - it's been used to describe so many things that it no longer describes anything. Another thing that gets ignored is the the difference between a trading strategy and an execution algorithm.
- cloudjacker 10y agoI wish I could get some rest API and websockets into the debt and credit default swaps market, really inefficient markets that reveal a lot about sentiment don't need high frequency at all, just pretty average latency actually (for now anyway) IB probably has this, not sure though. But those data costs are a huge deterrent for me!
- whatok 10y agoI think DTCC has some freely available live CDS data but I forget where and don't know how the APIs are.
- hobo_mark 10y agoCan't find the original comment but somebody wrote here time ago that some HFs let you run on their infrastructure (keeping 85/90% of any profits) if you can prove to have a valid strategy. Just out of curiosity, I could not find any other information about this, is it a thing that actually happens?
- sjbase 10y agoZipline (from OP's article) was created by a company that does something like that: https://www.quantopian.com/faq https://www.quantopian.com/faq
- __d 10y agoThere are places (like Quantopian, but also more professionally-oriented) that will not only provide infrastructure but also capital if you come to them with a trading algorithm that does well on back tests. The next step down are firms that'll give you a platform including hardware and software for the "infrastructure" bits: often called an "algo container", but lots of brokers have an API you can use to avoid needing to write feed handlers, etc. eg. Pico. Then there are heaps of providers who'll rent you servers, connectivity, rackspace, etc. You do all the software. Lucera is a trading-oriented "cloud" provider. OptionsIT or Fixnetics are infrastructure providers. Or you just go straight to the data centers -- any decent finance-oriented datacenter will have a POP for most of the venues, and you can just cross-connect. No relationship with any of the firms named -- just examples off the top of my head.
- brianwawok 10y agoYes it is. Small arrangements like this all over Chicago. Perhaps NY also. The big issue isnt actually infa it's capital. If you want to trade big you need a million bucks or something on hold with the exchange. How do you borrow a mil to park at an exchange? The guys mentioned above.
- lordnacho 10y agoYep. I am currently doing this, but under an NDA, at an HFT firm. 80% of profits seems a bit rich, though. If they're supplying top spec colo servers, a super fast network to connect them, and people to stare at it, I'd say something a bit lower (but still pretty chunky) would make more sense. I suppose the key for them is they have a bunch of existing infrastructure that pays for itself, so any extra cream is good. Also they have capital sitting around, so why not? The strategies have to be more than unusually profitable, though. I don't think I've seen a down day in the 3 months I've been sitting here, so your average trend following sharpe ~1.5 or so ain't gonna impress people.
- CaptainKanuk 10y agoI run Quantopian's Lectures, which are also intended to teach the statistics behind trading. https://www.quantopian.com/lectures https://www.quantopian.com/lectures
- blobbers 10y agoThese have been really great. Thanks for putting them together!
- CaptainKanuk 10y agoNo problem, more on the way.
- BayesStreet 10y agoHi Delaney, you're my favorite quantopian. Just watched all the lectures and I think jupyter notebooks are the best way to learn combined with a lecture for nearly any programming related subject.
- CaptainKanuk 10y agoThanks, happy they've been helpful. I agree that they're a super powerful teaching tool. We actually got the idea when seeing a prof at Harvard using them to teach the entire class: homeworks, lectures, everything.
- lpage 10y agoI mostly agree with this. Treating markets as an exercise purely in data science is a _bad idea_. Risking money without a solid understanding of market mechanics and trading conventions is a recipe for disaster. Taking a Bayesian view won't save you either - this is a land where some new "six sigma" event happens every six months. That said, the same logic that holds for identifying profitable strategies within an institution holds for individuals: unless you have better gear, don't fish a crowded pond. As an individual, your small size is, in some regards, an advantage. Institutions routinely pass on strategies that don't have capacity (there's not enough liquidity to make the strategy's returns worthwhile relative to their trading level) or strategies that aren't quite up to their standards (but might be up to yours). There's also a huge class of strategies that have somewhat choppy but long run consistent returns. Traders and funds worried about MoM track records won't touch those. Fees are the biggest barrier to entry. Institutions enjoy substantial discounts and an ability to amortize costs across a much wider base, making the strategy performance hurdle rate proportionally higher for smaller traders.
- blobbers 10y agoI'm curious what folk's thoughts are on these two platforms: Interactive brokers offers low fee access to their platform (https://www.interactivebrokers.com/en/index.php?f=13869 https://www.interactivebrokers.com/en/index.php?f=13869) Quantopian (www.quantopian.com) gives you the ability to trade through robinhood with long trades at 0 commission. There has been some skepticism in the Q forums on how well robinhood's execution is (possibly the effect of a "you get what you pay for"-attitude). Trading real capital with algorithms is more difficult than it sounds. Mentally if you're algorithm is doing stock selection as well as trade execution, you no longer understand what you own. During a period of extended drawdown, you have to be mentally tough enough to believe that your algorithm has what it takes to dig itself out. In a sense, it is like the start-up game. You have to believe you're doing something well enough that in the end you'll be right (and not bankrupt).
- seagreen 10y agoMy thoughts are that if you use either platform for anything other than entertainment you're a fool. Seriously, here's your competition: https://en.wikipedia.org/wiki/Renaissance_Technologies https://en.wikipedia.org/wiki/Renaissance_Technologies How deluded do you have to be to think you have an edge over that?
- bduerst 10y agoThis was my experience as well. Doing "quant" trading on the side quickly becomes a second job, as you are always adjusting, training, simulating, and even just observing the models you're using. It's not impossible to make money though. When it comes down to it, the trade-off for the hours you put in and the money you make only really pays off if you're trading with someone else's money, and lots of it.
- smnscu 10y agoRandom idea, but what if you could combine market data with news articles, and (obviously only for the past few years) data from blogs/reddit/hn/twitter/etc.? Sometimes there is fascinating insight to be learned in obscure places on the Internet (hn itself being a great example); I imagine a system that would collect and analyze that kind of information would be quite interesting, despite the huge amount of noise. Such a hypothetical system reminds me of this story: https://www.facebook.com/notes/robin-sloan/julie-rubicon/985697811525170 https://www.facebook.com/notes/robin-sloan/julie-rubicon/985...
- chollida1 10y agoGlib answer.... You'd get really mad at Anne Hathaway http://www.theatlantic.com/technology/archive/2011/03/does-anne-hathaway-news-drive-berkshire-hathaways-stock/72661/ http://www.theatlantic.com/technology/archive/2011/03/does-a... Longer answer, this has been done for 10+ years. Bloomberg will sell you a sentiment annotated news feed for 5 figures a month if you'd like to try.
- deleted 10y ago[deleted]
- mikkom 10y agoThis has been made many, many times already. Zipline that we are talking about here is created by Quantopian that actually provides aggregated twitter harvesting datafeed free for this year. https://www.quantopian.com/data/psychsignal/stocktwits https://www.quantopian.com/data/psychsignal/stocktwits
- alfalfasprout 10y agoBeen done tons of times. Some of them make quite a bit of money, others not. The real game-changer is if you're effective at separating out the real signal from the noise (and the noise floor is high in these sorts of feeds).
- hiou 10y agoYou would probably have as much luck correlating features like whether your neighbors kid was at the bus stop or had to run to catch it each day and the amount of times your neighbor complained about someone not cleaning up their dog's poop that week. Bullshit random edge case features don't get any better or worse if they come from the internet. In regular trading(algorithmic or not) you really need some non-trivial insight that for some reason no one else will have.
- ohazi 10y agoIn a similar vein: https://xkcd.com/1570/ https://xkcd.com/1570/
- shogunmike 10y agoDisclaimer(s): QuantStart.com founder here, background as a quant dev at a small fund. I should probably nuance my statement that it is easy to find trading strategies by saying that it is easy to find new trading /ideas/. There are a huge number of freely available trading ideas on forums, pre-print servers (arXiv, SSRN), blogs etc. The trick is knowing how to implement them properly, accounting for any transaction costs and adjusting the parameters of the model. This is often where the stated performance falls down. It takes a lot of time to carry out this sort of research. Long-term profitable strategies are tricky to find, due to the ever-present spectre of "alpha decay". This is where your strategy's edge is "arb'd out" - everyone else knows what you're doing and so there's no tradeable edge anymore. Hence it is necessary to have a portfolio of strategies and gradually phase out the ones that aren't doing well, and bring in new ones over time. That being said there are a large number of trend following funds (known as Commodity Trading Advisors, or CTAs, in the industry) that all broadly do the same thing (follow "trends" in the commodity futures markets) and have great years every now and then. There are some well-known "retail" quant traders who do well by trend following, but it does require quite a bit of capital to trade in futures. The philosophy that I do try to emphasise is to always be learning and researching new ideas. Also, as you mention, I'm pretty keen on discussing the math(s)/statistics aspect because once you have a solid math capability, it is easier to see where potential edges might exist and how to really assess whether it is a true "edge" or just a statistical anomaly. I believe someone else in a grandchild comment below said that there are many areas that bigger quant funds won't touch because of institutional incentives. If you have $10bn assets under management (AUM), then you're not going to care about investing $100-200k, even if the returns are good, because it won't move the needle on your monthly reports. The trick is to niche down into markets that you can spend a lot of time researching to find a distinct edge, that won't likely be touched by bigger funds. One area that is becoming interesting recently, due to the prevalence of satellite data/AI/deep learning-esque VC-backed startups, is building commodity supply/demand models. A good example is forecasting oil supply/demand by analysing large quantities of storage tank heights in global refineries [1]. Also, a small related-to-Zipline plug: I've recently started a free Python-based MIT-licensed open-source backtester [2], predominantly as a learning tool for programming and quant trading. There's about 4-5 of us working on it at the moment and it's in an early alpha stage, but we're always looking for people willing to help. [1] - https://orbitalinsight.com/solutions/ https://orbitalinsight.com/solutions/ [2] - https://github.com/mhallsmoore/qstrader/ https://github.com/mhallsmoore/qstrader/
- voiceclonr 10y agoI agree with you. I have spent my nights and weekends into trying to find an edge and am still looking. As a retail trader, it is very difficult to find anything but random data all over. I hope I will prove myself wrong someday, but as of now, I think a retail trader finding an edge is nearly impossible.
- gnaritas 10y agoI've had the same experience. Lots of people fooled by random returns think they haven't edges that don't exist.
- Sukotto 10y agoThis has been on my mind recently as someone who was in the front office for a few years (but not making trades) but is now on the outside. I mean each day 100's of Phd's start with clean market data, more data sources than you could possibly think of and statistical back testing systems that have 1000's of man hours put into them, trying to find a way to make money. It seems to me that the Tiger Rule applies here. You don't have to outrun the tiger, you just have to outrun your buddy. Is the choice really that one can be in the cohort you mention, or you can buy an index fund (or whatever the equivalent is in the market you're interested in)... and that's it?[1] Is the market so efficient that there is no middle ground where a smart and methodical person can make more money than the index plodders without being obliterated by the big players? That seems really unlikely to me. [1] (I don't think that's what you are saying... but you've given me a chance to try and express something I've been thinking about. Thank you for that)
- bunderbunder 10y agoDon't think in terms of there being one tiger that stops as soon as it gets something to eat. The market's more like a sea full of countless sharks, where all the sharks have to survive by eating other sharks. You're not wrong to think that, if there's any way to consistently make money by trading, it's by exploiting inefficiencies in the market. But remember that those inefficiencies come from people. Are you confident enough in your skills, your education and your resources to be sure that you'll be one of the ones finding and exploiting inefficiencies, rather than one of the ones who's creating inefficiencies for others to find and exploit?
- joeyspn 10y ago> I mean each day 100's of Phd's... Yes, but then... "Finance novice beats hedge fund pros, winning $100k in Quantopian trading contest" https://pando.com/2015/03/05/finance-novice-beats-hedge-fund-pros-wins-100k-in-inqugural-quantopian-trading-contest/ https://pando.com/2015/03/05/finance-novice-beats-hedge-fund... Thousands of brilliant people are out there undiscovered and I think quantopian does a great job leveling the playing field.
- dragonwriter 10y agoNovices sometimes beating pros is something you'd expect to be true the more performance was due to luck rather than strategy. If it was strongly strategy-dependent, best-information would win predictably and consistently.
- lpage 10y agoYup. Incidentally this is exactly the same effect that makes model selection within a portfolio challenging. Two worthwhile reads on that front: the multiple comparison problem [1] and regression towards the mean [2]. Regardless of your objective - hosting a contest like this or allocating to individual institutions and strategies - you'll always face this problem. Good allocation decisions are tactical, taking both the logic of the strategy and the broader portfolio into account. Bad allocation decisions chase returns. [1] https://en.wikipedia.org/wiki/Multiple_comparisons_problem https://en.wikipedia.org/wiki/Multiple_comparisons_problem [2] https://en.wikipedia.org/wiki/Regression_toward_the_mean https://en.wikipedia.org/wiki/Regression_toward_the_mean
- gedrap 10y agoEqually, there are many athletes that play at amateur/semi-professional level and suddenly they are top level starts (e.g. Chris Smalling, Jamie Vardy). So yes, it's possible. It's just that the odds are not in your favor :) should you keep chasing your childhood athlete dream because someone did it? Probably not :) Same rules apply to your example.
- oldmartin 10y ago"Yes, but then... "Finance novice beats hedge fund pros, winning $100k in Quantopian trading contest" Not sure if you know, but that winning algorithm was taken offline within one month for poor performance. It was probably just overoptimised on past data to win the competition and then failed miserably out in the real market.
- meritt 10y agoThe rapid rise in popularity of algorithmic trading among everyday coders, the massive growth of /r/wallstreetbets, and the popularity of Quantopian tells me one thing: Go long on retail brokers (AMTD, ETFC, IBKR)
- rdtsc 10y agoThat was my first thought. If I had the resources, I'd built up that subreddit. Youtube videos. HN posts. Some trading libraries on Github. Some forums. Promotional deals to start out. Previously there were other pyramid schemes, penny stocks, FX trading forums and so on. As those went out of fashion there is a new cohort of young people with great imagination who think they can beat the stock market. Some surely see a great potential there ready to exploit. They'd be silly not to. Or to put it more obviously when everyone goes digging for goal, don't follow them, but start selling shovels. Then maybe take some profits and put some ads out in the saloons with stories about awesome finds of massive amounts of gold.
- pfarnsworth 10y agoMassive growth of wallstreetbets? Come on, I've been subscribed to that subreddit since the beginning and it's largely a joke.
- meritt 10y agoIt's absolutely a joke but that doesn't mean it's not causing a lot of people to trade options
- YZF 10y agoAs long as the target you're optimizing for is different than what those 100 of PhD's (and most of the market) are optimizing for then maybe you've got a chance. You also don't need to pay their salary. So I agree wrt/ to longer time periods. Most of those PhD's are probably trying to predict minute to minute moves, or daily moves. A lot of them would be out of a job if they lose over 90 day periods. Not sure how much of an edge can be gained over buying and holding some reasonably diverse equity portfolio. If anyone is thinking they'll beat some HFT hedge fund they're almost certainly not going to. If they're thinking of beating a reasonably diverse/optimized portfolio over longer terms that's maybe possible IMO but the difference isn't going to be huge. To amplify the difference requires taking on more risk, for example by leveraging, options etc. If I want to decide whether long term investing in the US vs. Turkey, Greece, Brazil, Russia, or the UK, or gold :) it's not clear machine learning can give me useful insight. The historical data has its limits. Predicting geo-political processes and things like interest rates for the long term seems like a very difficult problem. I would suspect the machine answer to this question would be something like P/E is lower so it's a good investment but who knows. So I agree with the economic/market insight comment as well.
- sp527 10y ago> I would suspect the machine answer to this question would be something like P/E is lower so it's a good investment but who knows. So I agree with the economic/market insight comment as well. This is spot on (tried it once :). ML doesn't really help much at the macro level. MPT, perhaps combined with some decent economic insight, is a lot more useful.
- SEJeff 10y agoYou're pretty much spot on sans the "clean market data" part. I work for a very large electronic trading firm (and have been in HFT professionally the past 8.5-9 years of my career) who has a team dedicated to just grooming this data. It is a lot of work and it is noisy. Coming from some exchanges, it is even often wrong. Look at the entire mess the recent leap second did to some exchanges.
- chollida1 10y agoI think you misunderstood, or I wasn't clear:) The reason they have clean data is that big firms have teams of people dedicated to just cleaning data. The little guy has to spend time cleaning data even before he/she starts to compete.
- SEJeff 10y agoRight, I'm saying I work at one of said firms. The data even then isn't clean, just cleaner. Perhaps I'm just too pedantic :)
- algoisgame 10y agoYou can have a good starting point here, if you are interested in Algorithmic Trading http://www.quantinsti.com/epat/ http://www.quantinsti.com/epat/
- wslh 10y agoWhat do you think about cryptocurrencies trading? Do you think the same warning applies or there is an advantage because there are few players?
- unknown_apostle 10y agoEverybody's trading nowadays. How about just investing :-) I.e. focus on periods longer than a year, which so few people/professional market participants do. And on actual businesses instead of the crazy antics of a line. I wonder if you could use something like Zipline/Quantopian to screen huge amounts of consolidated balance sheets for markers of undervaluation. You could reject 1000s of companies and focus your “manual” vetting on the few that remain. If you can find the dollar selling for half a dollar and you can understand why it's selling for that price (e.g. because the entire market is down), you may have identified a winner. Then all you need is a little guts and lots of patience. And a predefined set criteria that you would constantly monitor to decide if your thesis is still valid.
- unknown_apostle 10y agoActually, does anyone know where an amateur could buy/download quarterly balance sheets/income statements for the broad stock market universe?
- sjbase 10y agoMost professionals I've known use Capital IQ, which is expensive. It's surprisingly tough to get broad, machine-readable market data for free but there are some cheaper options. Check this thread: https://www.reddit.com/r/SecurityAnalysis/comments/2ci5du/capital_iq_v_factset_v_bloomberg/ https://www.reddit.com/r/SecurityAnalysis/comments/2ci5du/ca... Or you could always scrape Yahoo Finance :)
- unknown_apostle 10y agoYeah but you don't want to spend most of your time writing and maintaining scraping code :-)
- sjbase 10y agoYep, therein lies the classic problem: if you want something of value, you can either spend your time or spend your money.
- overcast 10y agoI basically implemented this, and a lot of other features for my own personal trading bot against the Cryptsy API written in Python. The idea was to makes tons of small trades on alt coins throughout the day, constantly buying and selling on short crossovers, making fractions of a perfect profit after fees. It turns into an up and down roller coaster, sometimes you're way ahead, and other times you lose it all. The biggest issue was the low volume on most of the alt coins. At the end of the day, like many others have said, it's mostly luck, you will lose money eventually,but it's an interesting learning exercise. The only real way to win, is to have some insight into the market, not machines.
- seibelj 10y agoI use the Magic Formula[0] strategy because index funds are too boring for me. It's a value strategy and you have to hold for a year, but it's fun to see your stocks rise (and fall). [0] https://www.magicformulainvesting.com/ https://www.magicformulainvesting.com/
- edsouza 10y agoI have looked at Zipline before, but it does not handle intraday trades, and does some guesses on when the trade executes during the "day", so you may not get the best price. Running an algorithm for multi-day trades for more than a few months does not make sense on how the markets move, as certain events like "brexit", earnings, M&A, etc... affect stock price. If you are really interested in algorithmic trading, and you have programming experience, it's best to build your own backtesting system with intraday market data (pay for this). This way you will know the ins and outs of a trading system.
- flatM 10y agoFor backtesting intraday trades, LEAN from QuantConnect is a better alternative.
- joejev 10y agoCould you explain why you feel that it is better? What in particular does LEAN support that Zipline does not support?
- jaredbroad 10y agoEquities, FOREX, Futures, Options; tick, second, minute, hour and daily resolutions. Python, C# and F# backtesting. Dozens of models for improving the accuracy of your backtest. Live trading on IB, Tradier, FXCM, Oanda and paper trading. Local charting built in for desktop and backtesting. Lots of tools provided for free data downloads to work with public free data libraries. lean.quantconnect.com (I'm founder of QC :))
- joejev 10y agoIt is a little disingenuous to say that zipline doesn't support python, the short description from github says: "Zipline, a Pythonic Algorithmic Trading Library". Zipline also supports equities at minute and daily frequencies. There is no charting built into zipline itself but tearsheets and graphs can be generated with pyfolio (a project by the same people as zipline). Zipline also comes with the ability to pull pricing and splits data from quandl and yahoo. I realize I can't win you over but I wanted to present a fair comparison for others ;) Also, I work on zipline
- jb1991 10y agoBefore you jump in try to make your millions, a fairly well-known and accepted statistic is that at least 95% of day traders (using any method) lose money. All the very successful day traders I know lost lots of money in the beginning before learning how to do it properly. You need to have a solid source of funds to fuel your learning, and tremendous patience.
- lootsauce 10y agoFor the past year I have been trying to learn more about trading, risk management, etc. There are so many stories about how the markets work and how to make money in them. You could spend your lifetime throwing money down a hole trying each one and probably do worse than random. I can't say enough good things about the perspective I have gained from just listening to good interviews of people that trade and manage funds for a living. Take a look at https://chatwithtraders.com/podcast/ https://chatwithtraders.com/podcast/ and https://realvisiontv.com https://realvisiontv.com
- usefulcat 10y agoNever trust stories about how to make money in the stock markets, unless said stories are told entirely in the past tense. If someone really does have a successful trading strategy, the only way it makes economic sense to publish it is if they believe they can make more money by publishing it now (i.e. selling books, pageviews, whatever) than by using it to trade. Either that, or the algorithm is being described in sufficiently general terms that you're not actually given enough information to use it effectively.
- lootsauce 10y agoCouldn't agree more.
- cponeill 10y agoHave you read Market Wizards by Jack Schwager? Fantastic book and it really goes deep with how these traders think about approaching and exiting a trade/market.
- gnaritas 10y agoThe problem is it's all hogwash due to survivorship bias; every single one of them could simply be lucky. This is literally no different than interviewing lottery winners and asking them how they chose their numbers. People who write books about the successful are conning you for cash, always. There is nothing to be learned from the habits of just successful people; other than misleading yourself with superstition.
- deleted 10y ago[deleted]
- shortstuffsushi 10y agoGeneral question: how do you take this and interact directly with the market? Is there some sort of general, public api that you're making calls agains, where do you get an account for it, etc? Or, is this going through some firm that interfaces with the market?
- shortstuffsushi 10y agoUpvotes, but no comments :( If anyone comes back to this ever, still interested in knowing!
- lifeisstillgood 10y agoi just want to check my understanding of the algorithmic trading "world", so please do jump in. Once upon a time (1986ish) the equities and bond trading world was run by humans talking to humans and agreeing deals, the prices then fed into computer systems and the exchanges passed the prices around to make things mostly fair. Fair of course is relative, the Eco-system was very hierarchical, with major institutions at the top, trading between each other at low fees, with brokers feeding up into them and retail shops feeding into major brokers. The customer got a raw deal, being charged heavy fees per transaction, and getting a poor "spread". Spread was where the major institutions made their money. Human traders effectively bought very low and sold very high - both because they were human and could not easily handle algorithms in their heads and because who was going to stop them? At the top of the hierarchy traders got to see both sides of every trade - they could net trades off one against the other to make deals with little risk. And if it was not visible in a fair exchange they had even more leverage. Spreadsheets took off around now, making it possible for one trader to plan and monitor his trades and look really good to his boss. And then it became obvious that having a human in the spreadsheet-to-trade loop was sub optimal. A human with a spreadsheet still needed to dial a phone, make a decision, go to the toilet. A perl script could out perform him. And at the time the algorithms were simple. If Exxon's share price dropped then pretty obviously other oil companies would drop too, but so would say car company stocks, but maybe coal miner shares would go up. And that's just in LSE - the same goes for Hong Kong and Chicago. Those correlations I could work out in a perl script. (OK, 1980, maybe some Basic :-) And so algo trading was feasible with really tiny hardware - because the correlations in the world markets were simple, and large. And so low latency trading started. Because if I can use my ZX spectrum of my Commodore 64 to beat major traders to the punch, then all you need is a faster computer than the commodore and you beat me to the punch. And so it goes. Fast forward twenty years and - the hierarchy of the past is mostly still in place. Retail shops pull in the customers money, pass it upwards to brokers and they deal with traders at large banks. However the traders are much reduced, the volumes they do are orders of magnitude larger now. - the spread has gone. Major institutions make money on tiny margins and tiny fees and just do vast vast volumes. Major FX desks will make maybe 10 USD on a billion dollars of Eurodollar trades (I think). - the spread has gone for the algo traders. The reason PhD's are needed is because the correlations and arbitrage is all eaten up. The wins are few and far between and mostly need real world events (Brexit) - this is generally good, there is more trade on open exchanges (good for everyone) there is smaller spreads (good for customers). The break neck automation to a good for contractors like me :-) I'm not sure where I am going with this to be honest - but mostly it's that I am sure zip line is a good library, that the core part is written in the way a proprietary engine would look if someone took a year to rewrite it, but the core tech will not give you any edge - that edge has gone. The correlations have gone except in esoteric areas. If you want the edge, you need to be at the top of the tree again.
- ocschwar 10y agoAs a veteran of one algo shop, I have this to say: Play with the data all you like. Don't try to trade on it if you don't really know what you're doing. (Or, just recklessly trade other people's money. It's fun.) What you're seeing here is the "napsterization of finance." (Google it, it will lead you to the article I am almost plagiarizing). Basically, the market at large puts together a pot of money (called "alpha", debatably) The better you are at trading, the more of that pot you get. BUT this is not a zero sum game. It's worse. If the markets are functioning properly, then the better you are a this, the bigger the share of the pot you get, AND the smaller the pot of money gets. It used to be that middlemen like the NYSE stock market specialists made very large amounts of money doing what Homer Simpson automated with a drinky bird. Now, the also shops have already shrunk that pot considerably. Good news for your pension fund. Bad news for you if you try this yourself. So don't.
- cowmoo 10y agoHi, a shameless plug: I went to the Quantopian (the company that is behind Zipline and essentially uses Zipline as the core backend to their cloud platform) algo-trading hackathon two weekends ago and came up with this algo: https://www.quantopian.com/posts/xiv-slash-vxx-pair-trade-1 https://www.quantopian.com/posts/xiv-slash-vxx-pair-trade-1 Pair-trading VXX and XIV based on the StockTwits sentiments of the SPY at market open. The backtest did really well from 2011 to 2014 with 1700-1800% return in 3 years; and flat between 2014 to present-time, I'd really love it if people can improve upon the algo and see what people when they clone the algo and come up with ways to mitigate the drawdown's and improve the performance!
- lordnacho 10y agoI'm sitting at an HFT here, coding. This zipline thing is quite interesting if you're new, but if you can code, I'm not sure what the advantage is. The idea of a backtest is quite simple, and you can easily fire up something like pandas to do it for you. The equity line is simply your positions x returns, minus costs. To determine your positions, you have to make sure you aren't looking at future prices, but apart from that you are flexible in doing whatever you like. And this was a question for me. Suppose I want to code a cross-sectional strategy. How would I do that in zipline? It seems to be the kind of thing that gives you one backtest for one time series. Perhaps I just haven't looked into it enough. When we backtest, often we want to do things across the ensemble. We also take positions in a whole universe of instruments, so the backtest needs to be a matrix, rather than just one column. Incidentally, the example strategy will work quite well for retail traders. You can add a bunch of futures together and get a sharpe well over 1, basically what every CTA does but won't admit to. If you're wondering what all those PhDs do all day, it's adding capacity and researching minor improvements on that MA strategy. A colleague of mine worked at one of these brand names, and another friend owns one. So, does that mean anyone can simply do this? Well, yes. But you'd have a lot of leg work to do, and you might get discouraged before you start. You need an account from someone like Interactive Brokers. You need a fair bit of money, or you'll have increment problems trading the large contracts. And you'll have to set up all the data feeds and look at it each day.
- infinite8s 10y ago2 questions - what kind of leg work is involved, and how much is a fair bit of money? Is it possible with 250k of working capital?
- lordnacho 10y agoLeg work: - Getting the data into a shape that you can use. Normally a total PITA. For futures, you have to either stitch the contracts yourself, or get a pre-stitched series, which you have to take time to understand. Filtering it for weird data points. - Writing the strategy / backtesting code. The fun part. - Connecting to a broker. Gotta read API docs, test the functions, connect it to your code in a way that makes sense, and probably in a way that makes it easy to switch brokers. Test the price feed, write error handling code. - Daily operations code. You'll need a daily process where you can see what's going on. Automated testing of the trade report for correctness. Notifications from brokers need responses, you need to post margin as well. Some kind of SMS or Whatsapp for when something is wrong. Holiday calendar. 250k is not enough. Some of the futures contracts are quite large, and you won't be able to get the full benefit of diversification if you don't have a bunch of instruments to trade (look for a blog called Investment Idiocy, he recently talked about this). Above ~$3-5M, it isn't a problem and you can ignore it.
- ksec 10y agoI think Ruby needs to start broadening its Appeal beyond Rails.