5 ms·

I've been playing with the trading platform Alpaca[1] and wrote some code that essentially pulls trade information (each tick) for a handful of tech companies I

I've been playing with the trading platform Alpaca[1] and wrote some code that essentially pulls trade information (each tick) for a handful of tech companies I am interested in. It's a lot of data, but once I have the data stored, I'm now thinking... Ok now what? I have volume, price, time, and a lot of other metrics. Not sure where to go from here honestly. Should the code evaluate the data in real-time for each trade, or wait to the end/start of the trading day and do some evaluations then make a decision? Obviously, I am not going to be able to compete with the big boys using really sophisticated code (written in a low level language like C), executing in colocation right next to the exchanges, running Apache Hadoop and Spark. Still, I am interested in just tinkering with simple algorithms with the data. [1] https://alpaca.markets/ https://alpaca.markets/

web007 8y ago
I don't know about Alpaca, but Quantopian lets you do the equivalent and then also backtesting on historic data.
- KloudTrader 8y ago
  Shameless plug: We built a hosted Heroku-style platform for algo trading - https://kloudtrader.com/narwhal https://kloudtrader.com/narwhal Created it after not being able to do livetrading on Quantopian. Here's the associated python library and documentation: https://docs.kloudtrader.com/ https://docs.kloudtrader.com/ https://github.com/KloudTrader/libkloudtrader https://github.com/KloudTrader/libkloudtrader
casual_slacker 8y ago
The python library Backtrader is popular and has good intro docs. https://www.backtrader.com/docu/quickstart/quickstart.html https://www.backtrader.com/docu/quickstart/quickstart.html
khold_stare 8y ago
I'd say play around and see if you can predict the price - treat it like any machine learning/optimization problem - if you want to learn and have fun. If your goal is to make money, unfortunately it'll take a lot more. I've worked at a high frequency trading company, and now work at an algorithmic hedge fund, and you really need to scale horizontally to make money overall. There's a lot I can talk about but here are some points: - Most of the time you don't trade instruments in isolation. They are all correlated (even if inversely) one way or another , so movements in one will affect others. Having a myopic view of just one instrument will have too much unexplainable randomness. - The other way to scale is having more data inform your pricing - i.e. don't just look at the prices of one or more instruments. Look at twitter, look at weather reports, news etc. It kind of suggested that in the article. The meaningful movements in the market are most of the time due to truths outside of the price itself. - You have to decide whether you want to trade directionally and long term (like a hedge fund) and take on positions over time, or trade in and out of positions quickly (like high frequency trading firms) to minimize risk. You can be anywhere along that spectrum. The faster you are, the more myopic you can be and just react to current trading activity of other participants. The more long-term you are, the more you have to look at the big picture, more data, more instruments, making sure you have the right balance in your portfolio (whatever that means to you). I'd be happy to expand more :)
- richardw 8y ago
  Very interesting. No obligation but if you're bored feel free to expand!
  - khold_stare 8y ago
    Sure! I'll just ramble a bit about HFTs. I'd say starting a "pure" HFT firm is incredibly hard nowadays. There's a big upfront investment and huge operating costs, and there's a lot of consolidation happening in the industry. In general, most hardware and software is tailored to high-throughput, and a huge part of the effort of HFT is fighting all that to prioritize latency. If you want to compete with the very best firms, that means renting rackspace in a colocation with the exchanges, overclocked CPUs, expensive network cards, FPGAs, paying out the nose for the most accurate market-data feeds, and many order-entry connections. Then you have to tweak the BIOS, kernel settings, isolate your processes on cores, write your own userspace networking stack etc. etc. etc. On top of that if you want to trade between different colocations, you might need a network of microwave towers because fiber is too slow (light travels about 2/3 slower in fiber because it effectively "bounces" along the fiber). At the end of the day though, if your algorithm or strategy doesn't work, you won't make any money - speed is only a part of it. At the previous firm, we had a market making desk that was very profitable, but was quite "slow" by HFT standards (triple digit microseconds vs sub-microsecond response times). You still need to be smart about it. Having said all that, I think HFT gets a bad rap, about being a waste of resources/talent, and somehow "stealing" from the common man. The truth is, they are only "stealing" from each other - they compete to remove the tiniest inefficiencies in pricing. They ensure the cost of trading is actually as low as it can possibly be. By themselves they don't have enough capital to move the market in any meaningful way. In fact, most flash crashes occur when HFT firms step out of the way! They move out of the way of huge market movements because they can't cushion the blow without taking an unnecessary risk as a business. It's like asking your plumber to fix your sink while your house is on fire. Market orders, stop orders and panic are most of the problem when it comes to flash crashes. PSA: Please use limit orders :) I might ruffle some feathers with the comments, but I think overall HFTs aren't the bogeyman most people think them to be. Sure there are some bad apples, but that's true of any industry, and it comes down to the people - not the system itself.
lowdest 8y ago
I've spent a huge amount of time playing with similar stuff. The goal is to predict anything that you can make money on -- this can be obviously be price, but you can also try volatility, or predicting a probability distribution. Try relaxing constraints to find your signal: price ranges instead of exact prices, date ranges instead of exact dates. If you can output a probability, distribution, or confidence interval as well, that information can be used for position sizing (see Kelly criterion.) If you start expanding beyond a small set of specific companies, don't forget to find data for delisted companies to avoid survivorship bias.
- xiphias2 8y ago
  I looked at Kelly criterion, and what's interesting is that I was just thinking yesterday about whether I should optimize E(log(wealth)) - c * std(log(wealth)) or E(wealth) - c * std(wealth) The first one is better for building up wealth over a long time, but when I looked at Modern Portfolio Theory, it is suggesting to use the second formula. I'm also planning to spend about 5% of my wealth every year, and the second number gives me lower risk in the short time frame, so I think something in the middle would be a better criterion.
  - lowdest 8y ago
    Kelly sizing is optimal long term, but is highly volatile. A common practice is to size by a fraction of Kelly, to reduce variance and since risk of ruin increases fast after 1.0 * Kelly. It also allows room for error in the probability predictions, to avoid going over 1.0 * Kelly by accident. I try not to worry about stdev and focus on the soundness of my process, but in practice I have money split into higher-risk where I try to be clever, and low-risk failsafe investments to raise the floor of the worst case scenario.
marketgod 8y ago
This is kind of a backwards philosophy because there are so many things you can test in the market. What you want to try to do is find an edge and then capitalize on it with the tools at your disposal, just my opinion. Try this strategy and see if you can automate it. It's called the wheel strategy. What you want to do is setup a $200K account, and sell puts with certain deltas, and then if assigned, sell calls with certain deltas, until you get called. This will get you a relative return of up to 20% per year if done well. You can use your skills to figure out the best delta position to trade, you can look for high volatility to take advantage of. Once you get this completed, you can look for other trends. I always recommend starting with this because it's the easiest way to make some money in the market. It's a lot of work to do it manually, but automated, it would let you earn income while you are not doing much. You will also want to read and understand what calls and puts are, and what selling/buy calls/puts entails. Probably want to do this with stocks like Disney, GE, and so forth, something with large market caps, small movements, and something you are willing to hold long term and which will not go bankrupt/drop significantly in the near future, disclaimer: I do not like GE at all, however, you also want to try this with 1 contract to start, because 1 contract is 100 shares. You can probably work with a smaller starting capital. If you get this implemented and working, it can generate passive income for you in a sense, or you can blow an account. Edit: You want to settle/close out around earnings because the volatility is insane at that time.
- tomp 8y ago
  Option selling is “picking pennies in front of a train”. Small wins, big losses.
  - marketgod 8y ago
    I only buy calls or puts, however, you have to look at the wheel in a different view. You are making pennies in front of a steam roller, but you are getting paid to buy the stock you want. Most people say, if AAPL goes to $140, I will buy. Well this lets you sell puts until AAPL hits $140, then you get assigned, so you get paid every week or 3 weeks it goes up. Then when you have the stock, you sell a call against it which helps reduce your overall cost basis. This is a fairly easy strategy to automate and you can use a small account to do so.
floatrock 8y ago
From the article: > You’ve already got robo-advisors, which use algorithms to manage assets for retail investors. We’re also probably only a few years away from you being able to log into a brokerage account and run a sophisticated institutional-grade algorithm yourself. > People tend to assume that the diffusion of these technologies is a good thing. I’m more ambivalent. I think it could be a big mistake to have the population at large play around with algorithms. Some people who are very good at it might benefit from having access to this broadened toolset. But most people will just end up paying too much or make bad decisions because they’re being given access to a technology that they aren’t equipped to do anything useful with. They can lose money with it, however. Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges.
- infinite8s 8y ago
  > Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges. Those folks with fiber lines into the exchanges are definitely not running hadoop.
kaolti 8y ago
From my limited experience it seems the game is less about "predicting" price - which seems to be impossible, and more about placing strategic bets that will give you a positive return over a long period of time.

web007 8y ago

I don't know about Alpaca, but Quantopian lets you do the equivalent and then also backtesting on historic data.

KloudTrader 8y ago
Shameless plug: We built a hosted Heroku-style platform for algo trading - https://kloudtrader.com/narwhal https://kloudtrader.com/narwhal Created it after not being able to do livetrading on Quantopian. Here's the associated python library and documentation: https://docs.kloudtrader.com/ https://docs.kloudtrader.com/ https://github.com/KloudTrader/libkloudtrader https://github.com/KloudTrader/libkloudtrader

casual_slacker 8y ago

The python library Backtrader is popular and has good intro docs. https://www.backtrader.com/docu/quickstart/quickstart.html https://www.backtrader.com/docu/quickstart/quickstart.html

khold_stare 8y ago

I'd say play around and see if you can predict the price - treat it like any machine learning/optimization problem - if you want to learn and have fun. If your goal is to make money, unfortunately it'll take a lot more. I've worked at a high frequency trading company, and now work at an algorithmic hedge fund, and you really need to scale horizontally to make money overall. There's a lot I can talk about but here are some points: - Most of the time you don't trade instruments in isolation. They are all correlated (even if inversely) one way or another , so movements in one will affect others. Having a myopic view of just one instrument will have too much unexplainable randomness. - The other way to scale is having more data inform your pricing - i.e. don't just look at the prices of one or more instruments. Look at twitter, look at weather reports, news etc. It kind of suggested that in the article. The meaningful movements in the market are most of the time due to truths outside of the price itself. - You have to decide whether you want to trade directionally and long term (like a hedge fund) and take on positions over time, or trade in and out of positions quickly (like high frequency trading firms) to minimize risk. You can be anywhere along that spectrum. The faster you are, the more myopic you can be and just react to current trading activity of other participants. The more long-term you are, the more you have to look at the big picture, more data, more instruments, making sure you have the right balance in your portfolio (whatever that means to you). I'd be happy to expand more :)

richardw 8y ago
Very interesting. No obligation but if you're bored feel free to expand!
- khold_stare 8y ago
  Sure! I'll just ramble a bit about HFTs. I'd say starting a "pure" HFT firm is incredibly hard nowadays. There's a big upfront investment and huge operating costs, and there's a lot of consolidation happening in the industry. In general, most hardware and software is tailored to high-throughput, and a huge part of the effort of HFT is fighting all that to prioritize latency. If you want to compete with the very best firms, that means renting rackspace in a colocation with the exchanges, overclocked CPUs, expensive network cards, FPGAs, paying out the nose for the most accurate market-data feeds, and many order-entry connections. Then you have to tweak the BIOS, kernel settings, isolate your processes on cores, write your own userspace networking stack etc. etc. etc. On top of that if you want to trade between different colocations, you might need a network of microwave towers because fiber is too slow (light travels about 2/3 slower in fiber because it effectively "bounces" along the fiber). At the end of the day though, if your algorithm or strategy doesn't work, you won't make any money - speed is only a part of it. At the previous firm, we had a market making desk that was very profitable, but was quite "slow" by HFT standards (triple digit microseconds vs sub-microsecond response times). You still need to be smart about it. Having said all that, I think HFT gets a bad rap, about being a waste of resources/talent, and somehow "stealing" from the common man. The truth is, they are only "stealing" from each other - they compete to remove the tiniest inefficiencies in pricing. They ensure the cost of trading is actually as low as it can possibly be. By themselves they don't have enough capital to move the market in any meaningful way. In fact, most flash crashes occur when HFT firms step out of the way! They move out of the way of huge market movements because they can't cushion the blow without taking an unnecessary risk as a business. It's like asking your plumber to fix your sink while your house is on fire. Market orders, stop orders and panic are most of the problem when it comes to flash crashes. PSA: Please use limit orders :) I might ruffle some feathers with the comments, but I think overall HFTs aren't the bogeyman most people think them to be. Sure there are some bad apples, but that's true of any industry, and it comes down to the people - not the system itself.

lowdest 8y ago

I've spent a huge amount of time playing with similar stuff. The goal is to predict anything that you can make money on -- this can be obviously be price, but you can also try volatility, or predicting a probability distribution. Try relaxing constraints to find your signal: price ranges instead of exact prices, date ranges instead of exact dates. If you can output a probability, distribution, or confidence interval as well, that information can be used for position sizing (see Kelly criterion.) If you start expanding beyond a small set of specific companies, don't forget to find data for delisted companies to avoid survivorship bias.

xiphias2 8y ago
I looked at Kelly criterion, and what's interesting is that I was just thinking yesterday about whether I should optimize E(log(wealth)) - c * std(log(wealth)) or E(wealth) - c * std(wealth) The first one is better for building up wealth over a long time, but when I looked at Modern Portfolio Theory, it is suggesting to use the second formula. I'm also planning to spend about 5% of my wealth every year, and the second number gives me lower risk in the short time frame, so I think something in the middle would be a better criterion.
- lowdest 8y ago
  Kelly sizing is optimal long term, but is highly volatile. A common practice is to size by a fraction of Kelly, to reduce variance and since risk of ruin increases fast after 1.0 * Kelly. It also allows room for error in the probability predictions, to avoid going over 1.0 * Kelly by accident. I try not to worry about stdev and focus on the soundness of my process, but in practice I have money split into higher-risk where I try to be clever, and low-risk failsafe investments to raise the floor of the worst case scenario.

marketgod 8y ago

This is kind of a backwards philosophy because there are so many things you can test in the market. What you want to try to do is find an edge and then capitalize on it with the tools at your disposal, just my opinion. Try this strategy and see if you can automate it. It's called the wheel strategy. What you want to do is setup a $200K account, and sell puts with certain deltas, and then if assigned, sell calls with certain deltas, until you get called. This will get you a relative return of up to 20% per year if done well. You can use your skills to figure out the best delta position to trade, you can look for high volatility to take advantage of. Once you get this completed, you can look for other trends. I always recommend starting with this because it's the easiest way to make some money in the market. It's a lot of work to do it manually, but automated, it would let you earn income while you are not doing much. You will also want to read and understand what calls and puts are, and what selling/buy calls/puts entails. Probably want to do this with stocks like Disney, GE, and so forth, something with large market caps, small movements, and something you are willing to hold long term and which will not go bankrupt/drop significantly in the near future, disclaimer: I do not like GE at all, however, you also want to try this with 1 contract to start, because 1 contract is 100 shares. You can probably work with a smaller starting capital. If you get this implemented and working, it can generate passive income for you in a sense, or you can blow an account. Edit: You want to settle/close out around earnings because the volatility is insane at that time.

tomp 8y ago
Option selling is “picking pennies in front of a train”. Small wins, big losses.
- marketgod 8y ago
  I only buy calls or puts, however, you have to look at the wheel in a different view. You are making pennies in front of a steam roller, but you are getting paid to buy the stock you want. Most people say, if AAPL goes to $140, I will buy. Well this lets you sell puts until AAPL hits $140, then you get assigned, so you get paid every week or 3 weeks it goes up. Then when you have the stock, you sell a call against it which helps reduce your overall cost basis. This is a fairly easy strategy to automate and you can use a small account to do so.

floatrock 8y ago

From the article: > You’ve already got robo-advisors, which use algorithms to manage assets for retail investors. We’re also probably only a few years away from you being able to log into a brokerage account and run a sophisticated institutional-grade algorithm yourself. > People tend to assume that the diffusion of these technologies is a good thing. I’m more ambivalent. I think it could be a big mistake to have the population at large play around with algorithms. Some people who are very good at it might benefit from having access to this broadened toolset. But most people will just end up paying too much or make bad decisions because they’re being given access to a technology that they aren’t equipped to do anything useful with. They can lose money with it, however. Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges.

infinite8s 8y ago
> Hobby away, friend, but when you start betting money you can't afford to lose, probably worth just turning that hobby into a resume sent to those folks running hadoop with the shortest fiber lines into the exchanges. Those folks with fiber lines into the exchanges are definitely not running hadoop.

kaolti 8y ago

From my limited experience it seems the game is less about "predicting" price - which seems to be impossible, and more about placing strategic bets that will give you a positive return over a long period of time.