23 ms·

Case study: Algorithmic trading with Go

442 points by _zkyx 3y ago

victorbjorklund 3y ago
Nice. Been wanting to build a trading bot in elixir.
_zkyx 3y ago
Happy to answer any questions about this. It's been a side project that turned into a full blown obsession. There is nothing too secret about the system since it's more about having a solid platform that you can plug your strategies into. I'd probably even open source it but I'd have to clean up all my hacks :)
- czbond 3y ago
  @WestCoastJustin I've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly. Can you give some thoughts with your experimentation on the following from a Go perspective. 1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks) 2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB 3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
  - eshnil 3y ago
    Why not switch to Mojo lang for this? It's Python-compatible with Golang like performance from what I hear.
    - _zkyx 3y ago
      Haven't looked. Honestly, I'm so invested in this now it would be a pain to change anything.
    - czbond 3y ago
      @eshnil (& others familiar with Mojo) I had not looked at Mojo... thanks for the pointer. In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc). Know if Mojo directly uses existing Python ecosystem? I've been using Polars, etc.
      - maest 3y ago
        > Know if Mojo directly uses existing Python ecosystem? They say they do, including libs with C bindings.
    - jjtheblunt 3y ago
      It's not available yet outside of hosted environments provided by Modular.
    - silisili 3y ago
      I'm excited for Mojo, but it'll be years before it's ready for prime time as a general purpose language. It's not even available to download yet AFAIK...
    - bberrry 3y ago
      Hilarious how you suggest someone use a technology that isn't even publicly available, showing how extremely shallow your knowledge is.
  - _zkyx 3y ago
    > Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks) I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing. > Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly. > If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods) Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.
    - iakh 3y ago
      I'd be careful using ChatGPT for this. I tried the same recently to bootstrap spot yields from par, which is a fairly common code problem with hundreds of examples online. ChatGPT couldn't do it. It produced code that looked right, but would fail my tests abysmally. I ended up writing it by hand. I hope you are validating their code with known data.
      - _zkyx 3y ago
        Yeah, I'm heavily testing everything it writes. So, I'm very sure it's correct or get it close enough and then code it. I've seen ChatGPT add imports, functions, etc that don't even exist in Go. What I've found so useful though, is not even coding, but asking it how to solve problems, and then having it code things up. With Google, you need to pretty much already know how to solve the problem then go looking for answers. With ChatGPT you can ask it how it would solve this problem. For algorithms and stuff it have been unreal. Thanks for the warning though. You're totally right.
  - pdimitar 3y ago
    Not a complete answer but I quite liked https://github.com/markcheno/go-talib https://github.com/markcheno/go-talib for technical indicators.
- kirse 3y ago
  Do you have an email? I've been building a similar system for awhile except in F#, would love to connect.
  - deleted 3y ago
    [deleted]
  - programtrader 3y ago
    [dead]
- k9jd883jb 3y ago
  Awesome write up. I have a similar project in Go myself, although I just use minute bar data instead of realtime ticks. Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system. Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?
  - _zkyx 3y ago
    I wish this was better, but honestly, I'm just hard coding them right into the BUY loop. So, I need to restart the app anytime I want to change something. That's why I build this logic to dump and reload the state into a gob file (go memory dump essentially). Ideally, you'd have some type of format to write out your algorithm, and then have a way of hot loading it or something. But, I don't change these enough to really do that yet. For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time (1 file per day). Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this. This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.
    - slashdev 3y ago
      Fascinating. Have you considered open sourcing this without your algorithms ( empty buy/ sell loops)? I’d like to play with something like this but I couldn’t be bothered to build it all from scratch.
      - _zkyx 3y ago
        Yeah, I can do that. Ping me at [redacted] and I'll send you a note when I've done it.
- antiviral 3y ago
  You may be interested in this: https://www.techtrader.ai/#wall https://www.techtrader.ai/#wall Excerpt from his site: "Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist. Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "
  - _zkyx 3y ago
    Thanks, I'll check it out!
  - dist-epoch 3y ago
    No broker API change in 10 years? No bugs? I call bullshit. Reads more like a scam for trading signals.
    - antiviral 3y ago
      I don't run that site myself, but a quick search on sec.gov shows it's a registered hedge fund: https://www.sec.gov/Archives/edgar/data/1653903/000165390318000002/xslFormDX01/primary_doc.xml https://www.sec.gov/Archives/edgar/data/1653903/000165390318... I can't say any more about its validity. Ask your financial planner if techtrader is right for you :)
- theLiminator 3y ago
  Is your system consistently profitable enough for you to live off of? How long did it take you to get to this point?
  - _zkyx 3y ago
    No way. Not even close. That would be the dream though. Taxes are insane in Canada. Like 50% insane. Housing and kids just eat money. Maybe if I was renting and didn't have a family but that's not happening.
    - theLiminator 3y ago
      Curious if you mind sharing your sharpe/annual return? I understand if you don't want to though!
- TacticalCoder 3y ago
  You connect to IB's TWS API... If you execute your trade there only, wouldn't it be an option to fetch the price in real-time from the IB API and not having to use polygon.io at all? I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?) Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?
  - _zkyx 3y ago
    Yeah, I initially tried that actually. They have very low resolution data and do not provide raw trades/quotes (something like a message every 250ms), you cannot watch the entire market so you're stuck with like 100 tickers [1] vs 5500 tickers, and fetching historical data is very cumbersome. You can basically pull down pre-aggregated candlestick data and I wanted to base things off the raw trades/quotes. I actually probably spend a month or two trying to make this work. So, that's why I'm using polygon.io in that you can stream trades/quotes for the entire stock market in real-time, without any caps, which is amazing! [1] https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Feducation%2Fpdfnotes%2FPDF-API.php https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Fe...
- programtrader 3y ago
  Congrats on the post. Very inspiring. I've been going through this journey myself. I started learning on Tradingview, then bought Build Alpha to discover how to test strategies. I chose Portfolio 123 for my automated factor trading but had been working towards creating a program/basket trading system like yours that can act on intraday data. I moved to long-term investment until I could build a simulator capable of verifying the correctness of my investment strategies using fuzzy testing ideas stolen from TiggerBettle. I have almost two years of polygon quotes and trades for the whole market captured with a monotonic timestamp to be able to replay the data –and test the handling of polygon socket glitches. I'm focusing initially on capturing the data in a way that allows fast replay and aggregation, similar to what Kafka can do with topics but in-process using zig and custom memory-mapped data structures. My idea is to be able to generate signals like VIX (once I add options data), ETFs, and indexes and hopefully be faster at doing so than others :), please HN folks, call me out here if I'm being too naive. This has been a three-year learning process for me. I have been a retail investor for +10 years, but over the last three years, I've gone deep into learning algo-trading, drank del Prado Kool-aid, and read numerous trading and investment books. I'm now focusing on my technical chops to build the engine to build order books for individual stocks, baskets, and indexes with realistic market prices. I aim to develop a system that can get as close to the market price in the next dollar bar as possible. This has been a very lonely journey, and after reading the responses to this post, I'd love to connect with others on a similar path. Sending you an email!
- happytiger 3y ago
  How do you relate with tradingview? I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO. I've used wonderbit, zigz, and some of the others. Cool project.
- Shocka1 3y ago
  I've been algo trading as a side project for a few years now, but all through TD Ameritrade. I get all my quotes and whatnot with them, and it's free with my accounts. I also get other data elsewhere, which is free as well. I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in. I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!
sneak 3y ago
Slightly related: when running 3 screens like that, put the middle (primary) one in landscape, so the array is like a big H. It works better for videos and code, IMO. You still have two huge portrait fields for documentation/webpages/etc, or 8 portrait-oriented quarters. I usually run 8 portrait-shaped windows quartered on the sides, then two side by side on the middle, which are each approximately square. I don't like my editor window getting too tall.
- _zkyx 3y ago
  Yeah, personally, it's been a game changer just being able to load so much more into your field of view. Having two side-by-side makes it really easy to look at other parts of your code and make changes. I'm loving it! I'll give the H method a try for a few days and see how it goes.
  - magundu 3y ago
    What is h method?
    - _zkyx 3y ago
      Oh, sorry. I was referencing the OP comment there. > 3 screens like that, put the middle (primary) one in landscape, so the array is like a big H.
- ilyt 3y ago
  Eh, honestly only thing horizontal is good for is media, any documents/code is better in narrower vertical columns. Honestly my perfect screen for work would probably had something like 1:1 ratio, so I can have 2 nice columns of code that are tall enough that can be split horizontally if needed while still being useful.
ghoshbishakh 3y ago
Who noticed the Chat GPT running on the right screen?
- pqdbr 3y ago
  If you read the entire article, chatGPT is mentioned as something that increased OP's productivity 3x.
infamousclyde 3y ago
Fascinating application of the language and a terrific write-up. I would presume a GC language would normally be a disqualifying factor in real-time trading, but I think I'm coughing up some premature optimization, especially with what looks like a pretty beefy rig. Congratulations though, this is spectacular.
- _zkyx 3y ago
  Yeah, I never really looked a GC. Most of my trades take at least 500ms+ round trip. It's more in the latency to the broker and getting confirmations that I've been trying to tighten up. At least that's what I'm seeing right now.
- mhh__ 3y ago
  You say this as if memory allocation in general isn't extremely slow.
  - kasey_junk 3y ago
    For trading systems that are still software based they absolutely do not allocate or reclaim on the hot path for this reason.
    - infamousclyde 3y ago
      "That are still software based" is interesting. Are there hardware-based trading systems?
      - kasey_junk 3y ago
        Yes. The fastest systems are asic or fpga based. They typically never leave the router.
  - infamousclyde 3y ago
    I guess my presumption was that algorithmic trading was a very tight feedback loop, with as many controlled variables (i.e., GC) as possible, so I think it just subverted some of my misplaced expectations.
giis 3y ago
did you use TWS API or Web portal API for IBKR?
- _zkyx 3y ago
  Go app -> TWS API (on the desktop client) -> IB What's cool about this is that you can look at the IB TWS client and see things happening in real-time. So, it acts as sort of a sanity check. I know they have that gateway too but personally I like to look at the client all the time too. My workflow is to run the bot and the TWS client side-by-side and watch it make trades, see how things are moving, etc.
spockz 3y ago
Excellent write-up! Is there somewhere where I can read up on all the trading jargon? Also I wonder how can there be changes in the price of a stock after market if the exchange has closed? Isn’t the whole point that trades need to happen for stocks to get a certain value?
- xyzelement 3y ago
  There's after market trading on venues. But because most of the liquidity has "gone home" you can't transact as much at those prices and those prices are based on relatively few/small transactions.
  - _zkyx 3y ago
    Yeah, this is where a system like this shines. In that you can build a tool, like top, but for the stock market. You have all the real-time data. So, you can look at where the party is happening at. Just by looking at the trade flows. This is where building your own system absolutely kills anything else.
- _zkyx 3y ago
  https://www.investopedia.com/ https://www.investopedia.com/ is probably the best resource and chatgpt. I honestly, just google stock related terms and all roads lead to investopedia.
- mhh__ 3y ago
  Work in finance. Nothing else comes close.
makestuff 3y ago
This is really interesting. Have your strategies out performed buy and holding index funds, or are you mainly just doing this with a small amount of capital to learn how the markets work? I have always wanted to try algorithmic trading to learn about it, but I have always read it is a fools errand to think you will beat just buy and hold.
- nemothekid 3y ago
  1. I think buying and holding is more probable to have a higher return 2. Sometimes you get a cool api and think wow this would be fun, and next thing you know you've lost thousands on boneheaded trades. I did something similar during the pandemic with Rust with the Polygon API (and instead of interactive brokers, I used tradier). Eventually I learned I actually had more fun building the thing than actually trying to beat the market.
  - _zkyx 3y ago
    I have won and lots thousands for sure. Haha. When stocks were on a rip the bot was making lots of money just because everything was going way up. Then, in 2022 when everything went way down, like tons of tech stocks, my bot sucked. So, I really need to add shorting or something. I'm still exploring things on the strategy side.
    - pqdbr 3y ago
      Isn't the trend detection equally applicable to shorting as well, with swapped signals? Or do you believe there are more fundamental changes needed so your app can trade in shorting as well?
      - _zkyx 3y ago
        Yeah, right now I'm only set up to buy stocks. I haven't tried to short anything yet. I want to get to this eventually since it would be nice to make money when the market goes down too.
botdan 3y ago
If anyone from Polygon is reading, there's a typo in the URL: https://polygon.io/blog/case-study-algorithmict-trading-with-go/ https://polygon.io/blog/case-study-algorithmict-trading-with... (algorithmict has an erroneous "t" on the end). Normally I wouldn't mention this but the corrected spelling 404's.
gochi 3y ago
>I went from not knowing how to solve a problem, blindly googling around and reading books, to just telling ChatGPT the problem, and then asking how it would solve it, then asking it to code the solution. This is absolutely insane and has easily 3x my productivity. Very interesting. We can facetiously say that ChatGPT is using you as a medium between setting up algorithmic trading in Go!
blobbers 3y ago
Regarding open sourcing: don't be afraid to show your flaws; it doesn't have to be lonely if it is open source. Obviously whatever trading bot you're running separate from the actual trading engine itself is somewhat proprietary, but it would be great for the community to get more of this type of software in the hands of other hackers. Quantopian / Robinhood tried and failed, and the numerous clones since then have been somewhat sub par.
- JacKTrocinskI 3y ago
  You might find https://www.quantconnect.com/ https://www.quantconnect.com/ interesting! They offer a platform for quickly developing and backtesting trading strategies. They have a good community and overall do a great job. Give it a shot!
marsupialtail_2 3y ago
Hi Justin, you might be interested in my blog: https://github.com/marsupialtail/quokka/blob/master/blog/backtest.md https://github.com/marsupialtail/quokka/blob/master/blog/bac... advocating a cloud based approach. You don't have to use the system I am building, but it's worth thinking about that design.
- _zkyx 3y ago
  Cool, thanks. I'll check it out!
ye-olde-sysrq 3y ago
I worked in HFT for a while. I find the whole space fascinating. I'm glad OP found similar thrills. "This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading." I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy. Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive. But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry. So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it. It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks. So it makes it really hard to even speak the same language to each other when talking in common forums. It's a fun world. I miss it sometimes.
- _zkyx 3y ago
  YES, 100%! I honestly, cannot agree more. I've read pretty much every book that I could get my hands on and no one talks about the platform. This makes so much sense. Thank you for your perspective. It seems like so much of this stuff is silod off into each company and no one is talking about it. HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha. Cheers and thanks for the awesome comment! [1] https://www.researchgate.net/figure/Latencies-in-the-Electronic-Communication-System-of-OSMs-in-Northern-New-Jersey-Measured_fig5_291418411 https://www.researchgate.net/figure/Latencies-in-the-Electro...
joshu 3y ago
also having worked in the space: HFT should not exist. Break up the day into segments and have a single crossing. Do it every five minutes or whatever for sufficient timeliness. the millisecond race does not make anything better for anybody except the people doing the trading.
- alchemist1e9 3y ago
  [flagged]
  - grog454 3y ago
    > but overall it performs a valuable economic function Can you explain what that is? > To understand this requires understanding markets very deeply Apparently you aren't the only person on this forum who does. Are we just supposed to take your word for it?
    - mrcode007 3y ago
      He’s referring to the fact that the HFT “provides liquidity” as in getting a share off your hands and flipping it to someone else that is not you a moment later. This way you didn’t have to wait 3 seconds or 1 minute or whatever to sell your shares because someone knew they could flip them (alpha) on a short notice so they participated in a transaction with you. He’s assuming that without the HFT, the transactions wouldn’t be as frequent and you’d have to either a) wait for the fill on your trade , or b) lower your ask price if selling under time deadline. Analogously for buy orders. The only parties with semi-global visibility are prime brokers by definition; they see every position of everyone who custodies with them. The parties with global visibility of the US market are TRF (trade reporting facility) and those are the only parties who can sort of evaluate the HFT claims without bias or vested interest. Most of the studies in the field have some sort of an angle or vested interest so it’s hard to evaluate the veracity of the claims one way or another. A counterpoint to HFT is that stock markets existed before the the advent of computers and they had runs, panics and blow ups just like regular markets do now.
0xdeadbeefbabe 3y ago
Has the network or the power ever gone out at a bad time?
- _zkyx 3y ago
  Yes, both. Basically, I learned early that I needed a backup power supply. I have a couple that power my computer, monitors, and the internet router. For the network, that one is really tricky. In that, you're screwed. I've only had this happen once and then I needed to manually sell all my positions. Luckily, I didn't lose any money.
noname123 3y ago
QQ for everybody since OP is using IB API which is notoriously bad that many wrappers have been written for it (ib-insync). This is a general question, I'm wondering is there any good framework/wrappers out there that one can learn from to code up a complex trading application? Like dealing with all the asynchronous nature of process/submitting trading and quotes messages.
- yeahwhatever10 3y ago
  I use ib-insync with asyncio and it is surprisingly easy, the most difficult is order management. Jason Brownlee’s SuperFastPython was the most approachable asyncio explanation I found.
- _zkyx 3y ago
  Yeah, I ended up taking https://github.com/gofinance/ib https://github.com/gofinance/ib and rewrote my own wrapper. This took a long time but has been stable since. I'm basically only doing buy lmt, sell lmt, cancel, and updates orders though. So, the logic is pretty simple. Catching all the return messages and structuring them correctly took tons of debugging, trail, and error. Basically, mapping the messages into the correct orders for state tracking.
dataviz1000 3y ago
I've been building a bot with Typescript because it is what I know and Python because of the tools available. It is an overwhelming lonely endeavor. With all my other projects, I've always worked on teams, although they have always been very small teams and most of my work was autonomous -- still there was the occasional meeting and stand-ups. I've been working on this for six months and thought about bringing a friend onboard for no other reason than to not be alone.
- dzink 3y ago
  Count me in the group of solo algo trading developers. Maybe we should connect and chat.
  - Def_Os 3y ago
    There are dozens of us! I'd be open to a chat channel or something.
    - MuffinFlavored 3y ago
      I wonder how many people are actually willing to share "secret" details versus how many actually have anything to share that is worth anything. We all know how to load up OHLCV data and do basic math on it. Where does that gain us any edge, you know?
    - Shocka1 3y ago
      Myself included. One issue I've run into with the public Discords is there are just too many people and therefore too much noise. A smaller non-public group would be ideal.
  - antiviral 3y ago
    This is why I miss Quantopian. They had a great discussion forum, an ongoing competition, and annual conferences. I would be interested in some kind of regular meetup as well to discuss new techniques and to analyze publicly disclosed strategies, etc. I recently posted this article on HN... There is so much more knowledge to share: https://news.ycombinator.com/item?id=36344587 https://news.ycombinator.com/item?id=36344587
nurettin 3y ago
I love IB. They give you a papertrading account as a testbed. The executions are pretty close to reality, even in low liquidity markets.
pdimitar 3y ago
Really funny coincidence that I am seeing Interactive Brokers mentioned in this good article. Story + rant time, feel free to skip if you are not interested how one guy gambled and lost. It's also a tentative call for partnership if somebody is interested. And a call for chat if anyone has any interest in the topic. (It's also kind of off-topic, my apologies for that. To me it seems semi-related but would agree with mods' assessment if it doesn't match mine.) --- I recently "broke up" with some extremely toxic "investors" that wanted me to do a fully parallel trading demo -- meaning it receives ticks for N instruments (I successfully got to little less than 300) and trades with each of them depending on strategy. All in real-time. I got very far but the open-source libraries for Interactive Brokers are quite low quality in general and it was very hard and slow to progress (one of them couldn't even post orders, another used Mutex-es for "parallelism" which was of course not parallel at all, another one seemed to work well but only worked on servers of older versions compared to those I have access to, etc). I also had to gather code from separate places and assemble my own Frankenstein as I went along. Eventually I muscled through but by that time I have drained all my savings, my tax fund and even got a new loan. And the liaison + the investors of course refused to acknowledge the demo was basically 90% done (couldn't do full parallel trading due to defects of the IBKR libraries I have used and I used like 5 of them, and was in the process of repairing 2 of them to unlock the said full parallelism). They refused to send a small pre-funding wire (we're talking something small like $30k - $50k, not millions; for the work that was done, namely months of professional Rust programming work, that's a -75% discount if we look at market rates). They had all the proof and paper trail they needed to see that I was very close but I had to stop because I was literally about to be unable to pay rent and bills. They still did not concede. Obviously I picked a job and dumped them but I still have some regrets because the door is technically still open (they have not cut my access to the IB Gateway servers that they own), but other factors like now-ruined health are seriously getting in the way as well. Not to mention completely shattered trust. They insist "if you just finish the demo we'll give you money" and used every gaslighting technique I knew about (and many I didn't know about, so I learned a lot about gaslighting from them, lol) to try and coerce me to keep working for free with zero guarantees of funding -- but I no longer trust such rich investors to fulfill a promise without a binding and legally enforceable contract; I am in Eastern Europe, they are in the USA, even if they sign contract and violate it I practically cannot do anything to them i.e. I can't afford to travel and sue. Also they insist to get access to the source code but swear to everything that's holy that they will not run away with it and never give me a penny. Which is exactly what I think would happen. I had to draw the line at one point. To me it was red flags all around. --- I made many concessions and I will not make another one until they do. I'll be finding a new job soon again since the previous contract was agreed upon to be for several months, I helped a team accelerate bootstrapping a business-critical product (and we did that successfully). And then maybe, just maybe, after I settle a bit, I might work an hour or two on this again during some evenings. Maybe. And that won't be used to provide the demo to these toxic investors -- I'll use it to have a proven implementation that I can pitch to other people. These guys don't deserve the time of day from me. I have quite a lot of good code (mostly Rust, but also some Elixir and Golang -- I experimented a lot) that interfaces with Interactive Brokers and I have many building blocks of a good trading bot framework. That kind of creative and thorough work needs funding and peace of mind to be finished properly, of course. And I can't afford to invest so much energy on this without a stable income so I am shifting focus to that for however long it might take for me to feel comfortable to invest time and energy in this again. If any business person wants to partner up -- give me a shout. Mail is in profile. I also wouldn't refuse fellow techies (or anyone else really) stopping by and telling me how stupid and naive I was -- I'll agree right away and say that they are right. Because truth is truth. I admit I still can't get over the fact how close I got but I can't afford to count the stars while my livelihood is being endangered the longer I coast on savings. Dreams don't pay bills. And some dreams need more than one person to be fulfilled. Rant + story over. Thanks for reading if you stuck this far.
afterburner 3y ago
I see the new bull market is bringing back these kinds of articles.
- _zkyx 3y ago
  Yeah, I've been messing around with this for 4+ years so. Got to see COVID nose dive, meme stocks, massive build up, market taking a dump, bank collapses, and now a bull run again. I have no idea if it was always like this but it has been in insane.
  - OutThisLife 3y ago
    It's always like that.
tdiff 3y ago
I find this article to be absolutely pointless and clickbaity. It can be boiled down to: "you can use poligon.io for market data but algotrading is difficult anyway so there is not much to share yet".
- _zkyx 3y ago
  Hey, I'm sorry you feel that way and I wrote it. Personally, I wanted to share the high-level structure of how you'd build your own system. I would have loved to have seen something like this when I went down this rabbit hole in that I needed to figure it all out myself. What do you think would have made it better? I'd be happy to roll that back in.
  - silentwanderer 3y ago
    FWIW I thought it was a really good overview of the project structure and I appreciated the lessons learned as well.
  - MuffinFlavored 3y ago
    lol just admit you work for a niche SaaS in Polygon and you wanted exposure
pyrrhotech 3y ago
One of the most misunderstood concepts about algorithmic trading is that speed is not a critical factor for the majority of systems. My systems like https://grizzlybulls.com/models/vix-ta-macro-mp-extreme https://grizzlybulls.com/models/vix-ta-macro-mp-extreme have been crushing the market with live trading for 3+ years and yet trade on a frequency of only once ever 18 trading days on average (some a little more often, some even less), and are set up to only generate signals around hourly breakpoints. The last 18 months have been weaker than the first given the enormous structural shift in the market in this high inflation and rapidly rising interest rates environment, but we've still managed to deliver a return of +14.11% since the site launched in Jan 2022 compared to -7.83% for the SPX. We've managed to do it without any use of leverage and also with lower drawdowns as well of -16.48% vs. -27.57% for the SPX over that time frame.
- Brystephor 3y ago
  How much external data is used for these models or human input/fine tuning on a regular basis?
  - pyrrhotech 3y ago
    The models vary greatly in the number of external data sources they pull from to compute signals. The simplest ones like https://grizzlybulls.com/models/ta-mr-basic https://grizzlybulls.com/models/ta-mr-basic and https://grizzlybulls.com/models/ta-trend-basic https://grizzlybulls.com/models/ta-trend-basic only use a technical analysis calculations on the raw price and volume data of SPX on a few different timeframes. The most complex pulls data from dozens of external sources as it incorporates macroeconomics, monetary policy, various market sentiment sources, yield curve, valuation data, fundamentals, earnings estimates, etc. It's also worth noting that every additional data source adds some risk of that data source being down or publishing inaccurate data during real-time signal calculation which can cause inaccurate signals, so in order to justify that risk, the external source must meaningfully contribute to alpha or better risk-adjusted returns. None of them involve a human element in real-time. However, they are occasionally updated as new data comes in, but any updates only apply going forward so as to preserve the live trading history accurately (live trading start date varies by model from mid 2020 to jan 2022 with 2009 - 2020 being purely backtest for all models).
RomanPushkin 3y ago
Go itself, or any other language won't give you too much of an advantage. What gives you advantage is trading algo, which is always hard to find. I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different. I could use Visual Basic, and it would be better than Go, Rust, or whatever it is out there, given the algo and strategy are flawless. Language is just a tool. It's great you used Go, but I think the title is a bit misleading - people think of it as some kind of advantage. It isn't. And for HFT trading a language with Garbage Collector is not a great choice IMO.
- _zkyx 3y ago
  Hey, I write this. I did program this in Go. The reason is that it's my go to programming language. Using it is actually a disadvantage in that the industry uses C++ and Python but that's just what I know how to program in. I wasn't trying to be misleading. Go does work really well for taking in data, doing something with it, and then calling a remote API. So, it actually works really well but if you wanted to get a job based off this it probably wouldn't help you.
  - hu3 3y ago
    Go is great for network heavy apps like algo trading! I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes: - Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage. - Getting caught in the middle of a pump and dump event (also frequent) - Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %. Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
    - tomcam 3y ago
dzink 3y ago
I love finding other people who enjoy the pure feedback loop of code, strategy, and money that trading provides for me. I’ve been working on timing and correlations between stocks and indexes as a source of alpha and having some success. Now trying to automate the process and absolutely loving the work along the way. Not trying to play market maker and not jumping between stocks. Instead honing in on an accurate model that monetizes a few really well. Some tools I find useful: TradingView and Pinescript indicators and strategies, excel models with exported data for backtesting, Python and Go for the backend machine learning, ChatGPT for faster iteration on new code. Would love to talk shop if you guys are interested: trading @ dianazink.com
oh_sigh 3y ago
Are there any examples out there ofactual algotrading strategies that made money in a real market, say within the past 10 years? I know these are closely guarded secrets, but figured someone may have published a strategy that ceased being profitable.
TechBro8615 3y ago
Nice writeup. I appreciate the concrete discussion and especially the screenshots. I understand not discussing strategies, but can you elaborate on whether your trading is entirely based on technical analysis, or do you use external (or "alt") data feeds? In other words, is this a mostly closed system with inputs from Polygon and outputs to the IB API, or do you have a more extensive setup of custom data feeds of news sites, twitter, reddit, etc? If so, I'd be curious what challenges you face balancing those with historical backtesting, when some data sources might only have partial historical coverage.
- _zkyx 3y ago
  All my data is from polygon.io and nothing external. I have been exploring using lookup tables of pre-computed historical values to do things like anomaly detection (ie, is this normal activity for this stock). A good example would be BXRX today [1]. I have been looking at options trading activity too and trying to use that as a signal. [1] https://www.google.com/search?q=BXRX https://www.google.com/search?q=BXRX
cempaka 3y ago
While I was laid off and looking for work, I connected with a cryptocurrency market making firm that had access to a private broker feed which is not attached to any matching, so would occasionally go crossed and offer a pure arb opportunity for the same pairs. I had done some algo trading with Java in Kospi 200 options in 2011-2012 so I decided to put together a simple bot for them to try and grab the crossed markets when they occur. Even an incredibly simple trade like this requires quite a lot of work to get the risk management in place. I also took it as a chance to catch up on new Java features since I had been out of that ecosystem for awhile. It did successfully grab the arbs but there wasn't enough juice to justify more work on it and I got a job in the meantime, so I open sourced the whole thing: https://github.com/abissell/cempaka https://github.com/abissell/cempaka
- carabiner 3y ago
  Cool. Did you think Java was a good choice for this kind of work? Why not Go like OP or something else?
  - cempaka 3y ago
    I mostly went with Java because it's what I knew best, and I thought was likely to be "fast enough" for the market I was in. I'm sure Go would be similar in this case but didn't want to have to take the time to learn it. I do think that if Java can deliver on the combination of the foreign function/memory interface and value types, it might really start to look competitive for certain strategies which are just a bit too complex for the "do everything in the network card" approach. When the Aeron guys first implemented their protocols in Java, C#, and C++, C# was actually the fastest, which they attributed to the presence of both the optimizing runtime and value types.
mr_o47 3y ago
I’m always fascinated by fintech especially when it comes to technology and trading and the author did a really great job of explaining the concepts
- _zkyx 3y ago
  Thank you!
xgdgsc 3y ago
I've used Go for this before and still use Go for log monitoring. Other parts I've moved to julia/cpp/python mixed workflow.
asavinov 3y ago
When developing an automatic trading system the following aspects are important: - Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format - Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work - Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds. - Real trading and order management as well as coordination of all activities. The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels. I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important
- _zkyx 3y ago
  Hey, thanks. Yeah, I agree with you. That's an oversight on my end. I'll tell you here though. I'm just using go routines and channels to talk between them and then a giant mutex for locking. That's basically it. So, as new data comes in, it builds aggregates (tick based candlesticks) as needed, this then triggers the the BUY logic loop on that new data, if something is detected, that triggers a IB API order. It is dead simple and nothing complex in here. I've had upwards of 100 positions being tracked at anyone time and seems to just work. So, I haven't messed around with complex async logic too much. I'm actually just hard coding the parameters right into the BUY loop. This probably sounds crazy but for a small setup like this they don't change that much. So, I can run some trades, tweak things, restart, and then test some more. I imagine if you were doing that in an enterprise setting you've have some formal language and hot loading and stuff. But, for me hard coding seems to work well enough.
i-use-nixos-btw 3y ago
Nicely done. A couple of pointers. One on data, one on that RAM usage. First I’ll go with the RAM usage because this is hackernews and everyone loves algorithms. —- There are a lot of libraries out there that do technical analysis, and most of them are designed for batch processing. TALib is an example - it works on large data sets but is not appropriate for live trading because it repeats calculations over and over and over again. If you have 10000 datapoints and calculate indicators, it’ll calculate 10000 of them. Add one more bar, now you have a dataset of 10001 items, which TALib will calculate the indicators on from scratch. Or maybe you just feed the last 10000, and still perform that calculation over all of those, but save that 10001st oldest one. Either way, it’s bad. Same goes for every library I’ve seen, presumably because no one would open source a production grade indicator generation system. The production approach to this is somewhat different. Turn your features into state engines. Most features are just running calculations that are very easy to perform once per bar. Moving averages are a perfect example - for a moving average of N bars, store N items in a fixed size array on the stack and keep track of the latest index to be written to. When a new value (X) comes in, increment that index, and grab the value (Y). Modify your old mean by adding (X-Y)/N. Then write X over the old value Y (It’s a ring buffer). For EMAs, it’s even easier, because you just need to keep track of a numerator and a denominator - nothing else is needed. On a new value X with the scaling factor A (such that A^halflife = 0.5), the numerator N becomes NA+X and the denominator D becomes DA+1. Divide the two and you’ve got your new value. If an indicator depends on another, don’t recalculate it. Break the indicators down into fundamental calculations and you’ll often find a lot of redundant calculations being done. Rearrange it all so it fits. Automate that process if you enjoy that kind of thing like I do. Most indicators can be handled this way. The indicators that can’t - are rarely useful. After all, what you’re tracking is the evolving state of the market, and if you’re doing gymnastics over an indefinite number of bars, it probably doesn’t mean much. The end result is that you don’t end up accumulating memory throughout the day. You receive a bar, you throw it through your indicator generators (all of which using a fixed size of memory), then you discard the bar and wait for the next. Save state at the end of the day and load that on the following market day. The result will be a speed up like you couldn’t imagine. I promise. I run an indicator generation engine in a container capped at 40MB ram on one CPU, and it generates hundreds in much less than a millisecond after the bar arrives. —- Now, onto data. I recommend cleaning your data. You have a screenshot of a Tesla chart in there with some funky highs/lows every so often. It has been a long time since I’ve worked with US equities (and gladly so, it’s a mess of a system!) but the following is the best of my recollection. The trades you receive will come from several sources. For US stocks, there are several different venues that operate their own order books. These will operate as typical markets between the open and close of the day. By typical I mean that the bid and ask represent what you’d get if you market order instantly (which you can’t), and the market trades on them have to take from the bid and ask side of the order book (formed by people placing limit orders). However. There’s also the ADF - the Alternative Display Facility. This is the DIY of trade reporting (and quote reporting, but no one does). If someone sells some shares to their grandmother for a low price in exchange for the recipe to her famous apple pie, the ADF is where they can tell other participants about that trade, manually, subject to fat finger errors, prices of weird fractions of cents, and very relaxed constraints on timing. It’s also where dark pools post trades. The problem with this is that this data has no direct relationship to the rest of the market. This is why, every so often, you’ll see those blips. If you don’t clean them, they’ll play havoc with any indicator that uses highs and lows. The other problem is that ADF trades - at least when I last analysed this very issue - are not rare. They make up a large fraction of trades. So my approach was to clean ADF trades more rigorously than the venues by matching them against prior prices from an ADF-free background.
bofadeez 3y ago
This is professional gambling. Embrace (weak form) market efficiency. Why do you think mutual funds have never generated profit in the history of their existence?
- hu3 3y ago
  I'm naive but as far as I know mutual funds are risk-averse and by consequence their profits are lower, if any.
jimmysnuka 3y ago
I caught the algotrading bug recently and I've been wanting to backtest a couple of strategies involving ETFs. Right now I'm only looking at the open, close, daily high/low. The problem however is that you have to account for dividends and splits, which I'm not sure how to do (Admittedly, I haven't looked too deeply into it). Any advice on how to do that? Let's say I want to backtest some strategy involving UPRO; would I be fine just using yahoo finance historical prices as is, or would I have to manipulate it in some way? Also, what would be the ideal backtesting software/service to use?
- TheAlchemist 3y ago
  Harsh answer - if you have to ask this question here, then stop it now. Thing is, you will be fooled by bad data - you will find strategies that works in a backtest but not in production. And the reason will be because you missed some important finance concept (like taxes, dividends, stock splits etc). In this field 99% of success in my opinion is knowing what you don’t know. And only model that small part that you know you know and are fairly sure about it. Ps. By stop it now - I mean stop algo simulations and learn about those concepts, make sure you understand perfectly what data you are putting into your backtest.
  - jimmysnuka 3y ago
    Fair point. Right now I'm just doing it as a hobby. I'm not going to be making any real trades for a while and even when I do I don't plan on making money from it (at least not for some time).
- 1ark 3y ago
  Depending on your provider, but there could be an "Adjusted Close" (Yahoo has it) which includes stock splits and dividends. I tried a few backtesting libraries, thought backtesting.py[1] was easy to work with and straight to the point. 1. https://github.com/kernc/backtesting.py https://github.com/kernc/backtesting.py
  - jimmysnuka 3y ago
jjice 3y ago
As a layman boglehead, what are the returns of doing trades like this as opposed to an index fund? I assume it either works or there enough money to hope it works, but really curious what the situation is.
- _zkyx 3y ago
  This whole project started when I was looking for an investment adviser and didn't know anything. I was looking at all these funds that had like 4% return per year and then looked at stocks like Tesla that were up like 45% return and was like this doesn't make sense. When you dig in you see that these index funds are super super low risk. However, I like extremely high risk so I was willing to explore this a little. I don't really want to say you can get better returns, because you could end up losing all your money, extremely quickly betting on single stocks, it's basically a sure thing. So, that's why I turned to building a tool that basically bets for you. So, it's more about risk. If you put everything you had into TSLA you'd be up like 50% just in the past month but that's risky as hell. So, it's more of a risk question than a return question.
  - stefap2 3y ago
    At the end of the day or better say, at the end of the year you still have to look at average return on investment.
aantix 3y ago
For those that are interested in algorithmic trading - take a look at Collective2. Where engineers make available their buy/sell signals for a subscription fee. It's like the minor leagues for algorithmic trading. It's fascinating. The system keeps track of the gains/losses, so no cheating on the reporting. You can authorize Collective2 to access your Interactive Brokers account, so that the trade signals are managed on your behalf. It's been around for at least a decade, so you can see some longstanding performance numbers, but most systems just don't last that long. https://collective2.com/leader-board https://collective2.com/leader-board None of the leaders have been there for very long. One to two years. Showing that most system's alpha disappears fairly rapidly. The equity curves are sporadic ~a few of the trades accounting for a majority of the gains.
lost_tourist 3y ago
Does anyone know if he beat the market with this and became a multimillionaire?
- _zkyx 3y ago
  Not yet :)
nhggfu 3y ago
great work OP.
jasfi 3y ago
I'm working on an algo crypto trading platform. I have pre-defined rules set up in NoCode fashion, but it seems like many people will want to code their own algos. I'm thinking of supporting a scripting option.
pyrrhotech 3y ago
I've also just started a blog series for getting started with algorithmic trading in NodeJS for complete beginners: https://grizzlybulls.com/blog/building-an-algorithmic-trading-backtester-with-node-js-part-1 https://grizzlybulls.com/blog/building-an-algorithmic-tradin.... A lot of folks would raise an eyebrow at the thought of using NodeJS for algotrading, but I've found it to be a great fit for my non-HFT, more swing-trading style, and I've used it successfully in production for over 3 years.
Terretta 3y ago
Here's a recent piece on a change going on in systematic trading engineering: THE GIG ECONOMY COMES FOR HEDGE FUNDS Platforms that offer money managers the freedom to build a business and maximize their return on performance while removing the hurdles of launching independently could change financial markets. If the trend continues it could have a big effect on financial markets by making it easier for a wider assortment of unconventional managers to rise in the industry and offering investors better and cheaper access to them. … Well-received start-up ClearAlpha Technologies has moved closest to the gig model. Its first offering is a commingled fund apportioned among its managers, but it has the platform to act as an exchange, matching investors to individual managers or customized portfolios of managers, cutting out all the expense of intermediaries. Source: -- https://www.bloomberg.com/opinion/articles/2023-06-09/the-gig-economy-comes-for-hedge-funds https://www.bloomberg.com/opinion/articles/2023-06-09/the-gi... Reprints in case paywalled: -- https://www.washingtonpost.com/business/2023/06/09/the-gig-economy-comes-for-hedge-funds/9c52ab8e-06ae-11ee-b74a-5bdd335d4fa2_story.html https://www.washingtonpost.com/business/2023/06/09/the-gig-e... -- https://www.garp.org/risk-intelligence/technology/brave-new-world-062323 https://www.garp.org/risk-intelligence/technology/brave-new-... You no longer have to have graduated as finance into finance. (In fact, we prefer if you graduated with some other math modeling heavy emphasis, think turbulence and flow, or actuarial modeling.) A bit more about us, although this article is about our first fund mentioned above, not about the firm co-founded by Brian and I that owns the fund and built the platform it runs on: https://www.bloomberg.com/news/articles/2023-06-01/goldman-aqr-alumni-start-niche-multistrat-with-asness-backing https://www.bloomberg.com/news/articles/2023-06-01/goldman-a... If you're into this, we've come out of two years' stealth and are now hiring and remote work friendly.

442 points by _zkyx 3y ago

Nice. Been wanting to build a trading bot in elixir.

Happy to answer any questions about this. It's been a side project that turned into a full blown obsession. There is nothing too secret about the system since it's more about having a solid platform that you can plug your strategies into. I'd probably even open source it but I'd have to clean up all my hacks :)

czbond 3y ago
@WestCoastJustin I've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly. Can you give some thoughts with your experimentation on the following from a Go perspective. 1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks) 2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB 3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
- eshnil 3y ago
  Why not switch to Mojo lang for this? It's Python-compatible with Golang like performance from what I hear.
  - _zkyx 3y ago
    Haven't looked. Honestly, I'm so invested in this now it would be a pain to change anything.
  - czbond 3y ago
    @eshnil (& others familiar with Mojo) I had not looked at Mojo... thanks for the pointer. In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc). Know if Mojo directly uses existing Python ecosystem? I've been using Polars, etc.
    - maest 3y ago
      > Know if Mojo directly uses existing Python ecosystem? They say they do, including libs with C bindings.
  - jjtheblunt 3y ago
    It's not available yet outside of hosted environments provided by Modular.
  - silisili 3y ago
    I'm excited for Mojo, but it'll be years before it's ready for prime time as a general purpose language. It's not even available to download yet AFAIK...
  - bberrry 3y ago
    Hilarious how you suggest someone use a technology that isn't even publicly available, showing how extremely shallow your knowledge is.
- _zkyx 3y ago
  > Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks) I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing. > Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly. > If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods) Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.
  - iakh 3y ago
    I'd be careful using ChatGPT for this. I tried the same recently to bootstrap spot yields from par, which is a fairly common code problem with hundreds of examples online. ChatGPT couldn't do it. It produced code that looked right, but would fail my tests abysmally. I ended up writing it by hand. I hope you are validating their code with known data.
    - _zkyx 3y ago
      Yeah, I'm heavily testing everything it writes. So, I'm very sure it's correct or get it close enough and then code it. I've seen ChatGPT add imports, functions, etc that don't even exist in Go. What I've found so useful though, is not even coding, but asking it how to solve problems, and then having it code things up. With Google, you need to pretty much already know how to solve the problem then go looking for answers. With ChatGPT you can ask it how it would solve this problem. For algorithms and stuff it have been unreal. Thanks for the warning though. You're totally right.
- pdimitar 3y ago
  Not a complete answer but I quite liked https://github.com/markcheno/go-talib https://github.com/markcheno/go-talib for technical indicators.
kirse 3y ago
Do you have an email? I've been building a similar system for awhile except in F#, would love to connect.
- deleted 3y ago
  [deleted]
- programtrader 3y ago
  [dead]
k9jd883jb 3y ago
Awesome write up. I have a similar project in Go myself, although I just use minute bar data instead of realtime ticks. Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system. Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?
- _zkyx 3y ago
  I wish this was better, but honestly, I'm just hard coding them right into the BUY loop. So, I need to restart the app anytime I want to change something. That's why I build this logic to dump and reload the state into a gob file (go memory dump essentially). Ideally, you'd have some type of format to write out your algorithm, and then have a way of hot loading it or something. But, I don't change these enough to really do that yet. For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time (1 file per day). Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this. This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.
  - slashdev 3y ago
    Fascinating. Have you considered open sourcing this without your algorithms ( empty buy/ sell loops)? I’d like to play with something like this but I couldn’t be bothered to build it all from scratch.
    - _zkyx 3y ago
      Yeah, I can do that. Ping me at [redacted] and I'll send you a note when I've done it.
antiviral 3y ago
You may be interested in this: https://www.techtrader.ai/#wall https://www.techtrader.ai/#wall Excerpt from his site: "Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist. Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "
- _zkyx 3y ago
  Thanks, I'll check it out!
- dist-epoch 3y ago
  No broker API change in 10 years? No bugs? I call bullshit. Reads more like a scam for trading signals.
  - antiviral 3y ago
    I don't run that site myself, but a quick search on sec.gov shows it's a registered hedge fund: https://www.sec.gov/Archives/edgar/data/1653903/000165390318000002/xslFormDX01/primary_doc.xml https://www.sec.gov/Archives/edgar/data/1653903/000165390318... I can't say any more about its validity. Ask your financial planner if techtrader is right for you :)
theLiminator 3y ago
Is your system consistently profitable enough for you to live off of? How long did it take you to get to this point?
- _zkyx 3y ago
  No way. Not even close. That would be the dream though. Taxes are insane in Canada. Like 50% insane. Housing and kids just eat money. Maybe if I was renting and didn't have a family but that's not happening.
  - theLiminator 3y ago
    Curious if you mind sharing your sharpe/annual return? I understand if you don't want to though!
TacticalCoder 3y ago
You connect to IB's TWS API... If you execute your trade there only, wouldn't it be an option to fetch the price in real-time from the IB API and not having to use polygon.io at all? I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?) Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?
- _zkyx 3y ago
  Yeah, I initially tried that actually. They have very low resolution data and do not provide raw trades/quotes (something like a message every 250ms), you cannot watch the entire market so you're stuck with like 100 tickers [1] vs 5500 tickers, and fetching historical data is very cumbersome. You can basically pull down pre-aggregated candlestick data and I wanted to base things off the raw trades/quotes. I actually probably spend a month or two trying to make this work. So, that's why I'm using polygon.io in that you can stream trades/quotes for the entire stock market in real-time, without any caps, which is amazing! [1] https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Feducation%2Fpdfnotes%2FPDF-API.php https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Fe...
programtrader 3y ago
Congrats on the post. Very inspiring. I've been going through this journey myself. I started learning on Tradingview, then bought Build Alpha to discover how to test strategies. I chose Portfolio 123 for my automated factor trading but had been working towards creating a program/basket trading system like yours that can act on intraday data. I moved to long-term investment until I could build a simulator capable of verifying the correctness of my investment strategies using fuzzy testing ideas stolen from TiggerBettle. I have almost two years of polygon quotes and trades for the whole market captured with a monotonic timestamp to be able to replay the data –and test the handling of polygon socket glitches. I'm focusing initially on capturing the data in a way that allows fast replay and aggregation, similar to what Kafka can do with topics but in-process using zig and custom memory-mapped data structures. My idea is to be able to generate signals like VIX (once I add options data), ETFs, and indexes and hopefully be faster at doing so than others :), please HN folks, call me out here if I'm being too naive. This has been a three-year learning process for me. I have been a retail investor for +10 years, but over the last three years, I've gone deep into learning algo-trading, drank del Prado Kool-aid, and read numerous trading and investment books. I'm now focusing on my technical chops to build the engine to build order books for individual stocks, baskets, and indexes with realistic market prices. I aim to develop a system that can get as close to the market price in the next dollar bar as possible. This has been a very lonely journey, and after reading the responses to this post, I'd love to connect with others on a similar path. Sending you an email!
happytiger 3y ago
How do you relate with tradingview? I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO. I've used wonderbit, zigz, and some of the others. Cool project.
Shocka1 3y ago
I've been algo trading as a side project for a few years now, but all through TD Ameritrade. I get all my quotes and whatnot with them, and it's free with my accounts. I also get other data elsewhere, which is free as well. I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in. I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!

sneak 3y ago

Slightly related: when running 3 screens like that, put the middle (primary) one in landscape, so the array is like a big H. It works better for videos and code, IMO. You still have two huge portrait fields for documentation/webpages/etc, or 8 portrait-oriented quarters. I usually run 8 portrait-shaped windows quartered on the sides, then two side by side on the middle, which are each approximately square. I don't like my editor window getting too tall.

_zkyx 3y ago
Yeah, personally, it's been a game changer just being able to load so much more into your field of view. Having two side-by-side makes it really easy to look at other parts of your code and make changes. I'm loving it! I'll give the H method a try for a few days and see how it goes.
- magundu 3y ago
  What is h method?
  - _zkyx 3y ago
    Oh, sorry. I was referencing the OP comment there. > 3 screens like that, put the middle (primary) one in landscape, so the array is like a big H.
ilyt 3y ago
Eh, honestly only thing horizontal is good for is media, any documents/code is better in narrower vertical columns. Honestly my perfect screen for work would probably had something like 1:1 ratio, so I can have 2 nice columns of code that are tall enough that can be split horizontally if needed while still being useful.

ghoshbishakh 3y ago

Who noticed the Chat GPT running on the right screen?

pqdbr 3y ago
If you read the entire article, chatGPT is mentioned as something that increased OP's productivity 3x.

infamousclyde 3y ago

Fascinating application of the language and a terrific write-up. I would presume a GC language would normally be a disqualifying factor in real-time trading, but I think I'm coughing up some premature optimization, especially with what looks like a pretty beefy rig. Congratulations though, this is spectacular.

_zkyx 3y ago
Yeah, I never really looked a GC. Most of my trades take at least 500ms+ round trip. It's more in the latency to the broker and getting confirmations that I've been trying to tighten up. At least that's what I'm seeing right now.
mhh__ 3y ago
You say this as if memory allocation in general isn't extremely slow.
- kasey_junk 3y ago
  For trading systems that are still software based they absolutely do not allocate or reclaim on the hot path for this reason.
  - infamousclyde 3y ago
    "That are still software based" is interesting. Are there hardware-based trading systems?
    - kasey_junk 3y ago
      Yes. The fastest systems are asic or fpga based. They typically never leave the router.
- infamousclyde 3y ago
  I guess my presumption was that algorithmic trading was a very tight feedback loop, with as many controlled variables (i.e., GC) as possible, so I think it just subverted some of my misplaced expectations.

giis 3y ago

did you use TWS API or Web portal API for IBKR?

_zkyx 3y ago
Go app -> TWS API (on the desktop client) -> IB What's cool about this is that you can look at the IB TWS client and see things happening in real-time. So, it acts as sort of a sanity check. I know they have that gateway too but personally I like to look at the client all the time too. My workflow is to run the bot and the TWS client side-by-side and watch it make trades, see how things are moving, etc.

spockz 3y ago

Excellent write-up! Is there somewhere where I can read up on all the trading jargon? Also I wonder how can there be changes in the price of a stock after market if the exchange has closed? Isn’t the whole point that trades need to happen for stocks to get a certain value?

xyzelement 3y ago
There's after market trading on venues. But because most of the liquidity has "gone home" you can't transact as much at those prices and those prices are based on relatively few/small transactions.
- _zkyx 3y ago
  Yeah, this is where a system like this shines. In that you can build a tool, like top, but for the stock market. You have all the real-time data. So, you can look at where the party is happening at. Just by looking at the trade flows. This is where building your own system absolutely kills anything else.
_zkyx 3y ago
https://www.investopedia.com/ https://www.investopedia.com/ is probably the best resource and chatgpt. I honestly, just google stock related terms and all roads lead to investopedia.
mhh__ 3y ago
Work in finance. Nothing else comes close.

makestuff 3y ago

This is really interesting. Have your strategies out performed buy and holding index funds, or are you mainly just doing this with a small amount of capital to learn how the markets work? I have always wanted to try algorithmic trading to learn about it, but I have always read it is a fools errand to think you will beat just buy and hold.

nemothekid 3y ago
1. I think buying and holding is more probable to have a higher return 2. Sometimes you get a cool api and think wow this would be fun, and next thing you know you've lost thousands on boneheaded trades. I did something similar during the pandemic with Rust with the Polygon API (and instead of interactive brokers, I used tradier). Eventually I learned I actually had more fun building the thing than actually trying to beat the market.
- _zkyx 3y ago
  I have won and lots thousands for sure. Haha. When stocks were on a rip the bot was making lots of money just because everything was going way up. Then, in 2022 when everything went way down, like tons of tech stocks, my bot sucked. So, I really need to add shorting or something. I'm still exploring things on the strategy side.
  - pqdbr 3y ago
    Isn't the trend detection equally applicable to shorting as well, with swapped signals? Or do you believe there are more fundamental changes needed so your app can trade in shorting as well?
    - _zkyx 3y ago
      Yeah, right now I'm only set up to buy stocks. I haven't tried to short anything yet. I want to get to this eventually since it would be nice to make money when the market goes down too.

botdan 3y ago

If anyone from Polygon is reading, there's a typo in the URL: https://polygon.io/blog/case-study-algorithmict-trading-with-go/ https://polygon.io/blog/case-study-algorithmict-trading-with... (algorithmict has an erroneous "t" on the end). Normally I wouldn't mention this but the corrected spelling 404's.

gochi 3y ago

>I went from not knowing how to solve a problem, blindly googling around and reading books, to just telling ChatGPT the problem, and then asking how it would solve it, then asking it to code the solution. This is absolutely insane and has easily 3x my productivity. Very interesting. We can facetiously say that ChatGPT is using you as a medium between setting up algorithmic trading in Go!

blobbers 3y ago

Regarding open sourcing: don't be afraid to show your flaws; it doesn't have to be lonely if it is open source. Obviously whatever trading bot you're running separate from the actual trading engine itself is somewhat proprietary, but it would be great for the community to get more of this type of software in the hands of other hackers. Quantopian / Robinhood tried and failed, and the numerous clones since then have been somewhat sub par.

JacKTrocinskI 3y ago
You might find https://www.quantconnect.com/ https://www.quantconnect.com/ interesting! They offer a platform for quickly developing and backtesting trading strategies. They have a good community and overall do a great job. Give it a shot!

marsupialtail_2 3y ago

Hi Justin, you might be interested in my blog: https://github.com/marsupialtail/quokka/blob/master/blog/backtest.md https://github.com/marsupialtail/quokka/blob/master/blog/bac... advocating a cloud based approach. You don't have to use the system I am building, but it's worth thinking about that design.

_zkyx 3y ago
Cool, thanks. I'll check it out!

ye-olde-sysrq 3y ago

I worked in HFT for a while. I find the whole space fascinating. I'm glad OP found similar thrills. "This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading." I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy. Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive. But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry. So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it. It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks. So it makes it really hard to even speak the same language to each other when talking in common forums. It's a fun world. I miss it sometimes.

_zkyx 3y ago
YES, 100%! I honestly, cannot agree more. I've read pretty much every book that I could get my hands on and no one talks about the platform. This makes so much sense. Thank you for your perspective. It seems like so much of this stuff is silod off into each company and no one is talking about it. HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha. Cheers and thanks for the awesome comment! [1] https://www.researchgate.net/figure/Latencies-in-the-Electronic-Communication-System-of-OSMs-in-Northern-New-Jersey-Measured_fig5_291418411 https://www.researchgate.net/figure/Latencies-in-the-Electro...

joshu 3y ago

also having worked in the space: HFT should not exist. Break up the day into segments and have a single crossing. Do it every five minutes or whatever for sufficient timeliness. the millisecond race does not make anything better for anybody except the people doing the trading.

alchemist1e9 3y ago
[flagged]
- grog454 3y ago
  > but overall it performs a valuable economic function Can you explain what that is? > To understand this requires understanding markets very deeply Apparently you aren't the only person on this forum who does. Are we just supposed to take your word for it?
  - mrcode007 3y ago
    He’s referring to the fact that the HFT “provides liquidity” as in getting a share off your hands and flipping it to someone else that is not you a moment later. This way you didn’t have to wait 3 seconds or 1 minute or whatever to sell your shares because someone knew they could flip them (alpha) on a short notice so they participated in a transaction with you. He’s assuming that without the HFT, the transactions wouldn’t be as frequent and you’d have to either a) wait for the fill on your trade , or b) lower your ask price if selling under time deadline. Analogously for buy orders. The only parties with semi-global visibility are prime brokers by definition; they see every position of everyone who custodies with them. The parties with global visibility of the US market are TRF (trade reporting facility) and those are the only parties who can sort of evaluate the HFT claims without bias or vested interest. Most of the studies in the field have some sort of an angle or vested interest so it’s hard to evaluate the veracity of the claims one way or another. A counterpoint to HFT is that stock markets existed before the the advent of computers and they had runs, panics and blow ups just like regular markets do now.

0xdeadbeefbabe 3y ago

Has the network or the power ever gone out at a bad time?

_zkyx 3y ago
Yes, both. Basically, I learned early that I needed a backup power supply. I have a couple that power my computer, monitors, and the internet router. For the network, that one is really tricky. In that, you're screwed. I've only had this happen once and then I needed to manually sell all my positions. Luckily, I didn't lose any money.

noname123 3y ago

QQ for everybody since OP is using IB API which is notoriously bad that many wrappers have been written for it (ib-insync). This is a general question, I'm wondering is there any good framework/wrappers out there that one can learn from to code up a complex trading application? Like dealing with all the asynchronous nature of process/submitting trading and quotes messages.

yeahwhatever10 3y ago
I use ib-insync with asyncio and it is surprisingly easy, the most difficult is order management. Jason Brownlee’s SuperFastPython was the most approachable asyncio explanation I found.
_zkyx 3y ago
Yeah, I ended up taking https://github.com/gofinance/ib https://github.com/gofinance/ib and rewrote my own wrapper. This took a long time but has been stable since. I'm basically only doing buy lmt, sell lmt, cancel, and updates orders though. So, the logic is pretty simple. Catching all the return messages and structuring them correctly took tons of debugging, trail, and error. Basically, mapping the messages into the correct orders for state tracking.

dataviz1000 3y ago

I've been building a bot with Typescript because it is what I know and Python because of the tools available. It is an overwhelming lonely endeavor. With all my other projects, I've always worked on teams, although they have always been very small teams and most of my work was autonomous -- still there was the occasional meeting and stand-ups. I've been working on this for six months and thought about bringing a friend onboard for no other reason than to not be alone.

dzink 3y ago
Count me in the group of solo algo trading developers. Maybe we should connect and chat.
- Def_Os 3y ago
  There are dozens of us! I'd be open to a chat channel or something.
  - MuffinFlavored 3y ago
    I wonder how many people are actually willing to share "secret" details versus how many actually have anything to share that is worth anything. We all know how to load up OHLCV data and do basic math on it. Where does that gain us any edge, you know?
  - Shocka1 3y ago
    Myself included. One issue I've run into with the public Discords is there are just too many people and therefore too much noise. A smaller non-public group would be ideal.
- antiviral 3y ago
  This is why I miss Quantopian. They had a great discussion forum, an ongoing competition, and annual conferences. I would be interested in some kind of regular meetup as well to discuss new techniques and to analyze publicly disclosed strategies, etc. I recently posted this article on HN... There is so much more knowledge to share: https://news.ycombinator.com/item?id=36344587 https://news.ycombinator.com/item?id=36344587

nurettin 3y ago

I love IB. They give you a papertrading account as a testbed. The executions are pretty close to reality, even in low liquidity markets.

pdimitar 3y ago

Really funny coincidence that I am seeing Interactive Brokers mentioned in this good article. Story + rant time, feel free to skip if you are not interested how one guy gambled and lost. It's also a tentative call for partnership if somebody is interested. And a call for chat if anyone has any interest in the topic. (It's also kind of off-topic, my apologies for that. To me it seems semi-related but would agree with mods' assessment if it doesn't match mine.) --- I recently "broke up" with some extremely toxic "investors" that wanted me to do a fully parallel trading demo -- meaning it receives ticks for N instruments (I successfully got to little less than 300) and trades with each of them depending on strategy. All in real-time. I got very far but the open-source libraries for Interactive Brokers are quite low quality in general and it was very hard and slow to progress (one of them couldn't even post orders, another used Mutex-es for "parallelism" which was of course not parallel at all, another one seemed to work well but only worked on servers of older versions compared to those I have access to, etc). I also had to gather code from separate places and assemble my own Frankenstein as I went along. Eventually I muscled through but by that time I have drained all my savings, my tax fund and even got a new loan. And the liaison + the investors of course refused to acknowledge the demo was basically 90% done (couldn't do full parallel trading due to defects of the IBKR libraries I have used and I used like 5 of them, and was in the process of repairing 2 of them to unlock the said full parallelism). They refused to send a small pre-funding wire (we're talking something small like $30k - $50k, not millions; for the work that was done, namely months of professional Rust programming work, that's a -75% discount if we look at market rates). They had all the proof and paper trail they needed to see that I was very close but I had to stop because I was literally about to be unable to pay rent and bills. They still did not concede. Obviously I picked a job and dumped them but I still have some regrets because the door is technically still open (they have not cut my access to the IB Gateway servers that they own), but other factors like now-ruined health are seriously getting in the way as well. Not to mention completely shattered trust. They insist "if you just finish the demo we'll give you money" and used every gaslighting technique I knew about (and many I didn't know about, so I learned a lot about gaslighting from them, lol) to try and coerce me to keep working for free with zero guarantees of funding -- but I no longer trust such rich investors to fulfill a promise without a binding and legally enforceable contract; I am in Eastern Europe, they are in the USA, even if they sign contract and violate it I practically cannot do anything to them i.e. I can't afford to travel and sue. Also they insist to get access to the source code but swear to everything that's holy that they will not run away with it and never give me a penny. Which is exactly what I think would happen. I had to draw the line at one point. To me it was red flags all around. --- I made many concessions and I will not make another one until they do. I'll be finding a new job soon again since the previous contract was agreed upon to be for several months, I helped a team accelerate bootstrapping a business-critical product (and we did that successfully). And then maybe, just maybe, after I settle a bit, I might work an hour or two on this again during some evenings. Maybe. And that won't be used to provide the demo to these toxic investors -- I'll use it to have a proven implementation that I can pitch to other people. These guys don't deserve the time of day from me. I have quite a lot of good code (mostly Rust, but also some Elixir and Golang -- I experimented a lot) that interfaces with Interactive Brokers and I have many building blocks of a good trading bot framework. That kind of creative and thorough work needs funding and peace of mind to be finished properly, of course. And I can't afford to invest so much energy on this without a stable income so I am shifting focus to that for however long it might take for me to feel comfortable to invest time and energy in this again. If any business person wants to partner up -- give me a shout. Mail is in profile. I also wouldn't refuse fellow techies (or anyone else really) stopping by and telling me how stupid and naive I was -- I'll agree right away and say that they are right. Because truth is truth. I admit I still can't get over the fact how close I got but I can't afford to count the stars while my livelihood is being endangered the longer I coast on savings. Dreams don't pay bills. And some dreams need more than one person to be fulfilled. Rant + story over. Thanks for reading if you stuck this far.

afterburner 3y ago

I see the new bull market is bringing back these kinds of articles.

_zkyx 3y ago
Yeah, I've been messing around with this for 4+ years so. Got to see COVID nose dive, meme stocks, massive build up, market taking a dump, bank collapses, and now a bull run again. I have no idea if it was always like this but it has been in insane.
- OutThisLife 3y ago
  It's always like that.

tdiff 3y ago

I find this article to be absolutely pointless and clickbaity. It can be boiled down to: "you can use poligon.io for market data but algotrading is difficult anyway so there is not much to share yet".

_zkyx 3y ago
Hey, I'm sorry you feel that way and I wrote it. Personally, I wanted to share the high-level structure of how you'd build your own system. I would have loved to have seen something like this when I went down this rabbit hole in that I needed to figure it all out myself. What do you think would have made it better? I'd be happy to roll that back in.
- silentwanderer 3y ago
  FWIW I thought it was a really good overview of the project structure and I appreciated the lessons learned as well.
- MuffinFlavored 3y ago
  lol just admit you work for a niche SaaS in Polygon and you wanted exposure

pyrrhotech 3y ago

One of the most misunderstood concepts about algorithmic trading is that speed is not a critical factor for the majority of systems. My systems like https://grizzlybulls.com/models/vix-ta-macro-mp-extreme https://grizzlybulls.com/models/vix-ta-macro-mp-extreme have been crushing the market with live trading for 3+ years and yet trade on a frequency of only once ever 18 trading days on average (some a little more often, some even less), and are set up to only generate signals around hourly breakpoints. The last 18 months have been weaker than the first given the enormous structural shift in the market in this high inflation and rapidly rising interest rates environment, but we've still managed to deliver a return of +14.11% since the site launched in Jan 2022 compared to -7.83% for the SPX. We've managed to do it without any use of leverage and also with lower drawdowns as well of -16.48% vs. -27.57% for the SPX over that time frame.

Brystephor 3y ago
How much external data is used for these models or human input/fine tuning on a regular basis?
- pyrrhotech 3y ago
  The models vary greatly in the number of external data sources they pull from to compute signals. The simplest ones like https://grizzlybulls.com/models/ta-mr-basic https://grizzlybulls.com/models/ta-mr-basic and https://grizzlybulls.com/models/ta-trend-basic https://grizzlybulls.com/models/ta-trend-basic only use a technical analysis calculations on the raw price and volume data of SPX on a few different timeframes. The most complex pulls data from dozens of external sources as it incorporates macroeconomics, monetary policy, various market sentiment sources, yield curve, valuation data, fundamentals, earnings estimates, etc. It's also worth noting that every additional data source adds some risk of that data source being down or publishing inaccurate data during real-time signal calculation which can cause inaccurate signals, so in order to justify that risk, the external source must meaningfully contribute to alpha or better risk-adjusted returns. None of them involve a human element in real-time. However, they are occasionally updated as new data comes in, but any updates only apply going forward so as to preserve the live trading history accurately (live trading start date varies by model from mid 2020 to jan 2022 with 2009 - 2020 being purely backtest for all models).

RomanPushkin 3y ago

Go itself, or any other language won't give you too much of an advantage. What gives you advantage is trading algo, which is always hard to find. I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different. I could use Visual Basic, and it would be better than Go, Rust, or whatever it is out there, given the algo and strategy are flawless. Language is just a tool. It's great you used Go, but I think the title is a bit misleading - people think of it as some kind of advantage. It isn't. And for HFT trading a language with Garbage Collector is not a great choice IMO.

_zkyx 3y ago
Hey, I write this. I did program this in Go. The reason is that it's my go to programming language. Using it is actually a disadvantage in that the industry uses C++ and Python but that's just what I know how to program in. I wasn't trying to be misleading. Go does work really well for taking in data, doing something with it, and then calling a remote API. So, it actually works really well but if you wanted to get a job based off this it probably wouldn't help you.
- hu3 3y ago
  Go is great for network heavy apps like algo trading! I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes: - Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage. - Getting caught in the middle of a pump and dump event (also frequent) - Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %. Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
  - tomcam 3y ago

dzink 3y ago

I love finding other people who enjoy the pure feedback loop of code, strategy, and money that trading provides for me. I’ve been working on timing and correlations between stocks and indexes as a source of alpha and having some success. Now trying to automate the process and absolutely loving the work along the way. Not trying to play market maker and not jumping between stocks. Instead honing in on an accurate model that monetizes a few really well. Some tools I find useful: TradingView and Pinescript indicators and strategies, excel models with exported data for backtesting, Python and Go for the backend machine learning, ChatGPT for faster iteration on new code. Would love to talk shop if you guys are interested: trading @ dianazink.com

oh_sigh 3y ago

Are there any examples out there ofactual algotrading strategies that made money in a real market, say within the past 10 years? I know these are closely guarded secrets, but figured someone may have published a strategy that ceased being profitable.

TechBro8615 3y ago

Nice writeup. I appreciate the concrete discussion and especially the screenshots. I understand not discussing strategies, but can you elaborate on whether your trading is entirely based on technical analysis, or do you use external (or "alt") data feeds? In other words, is this a mostly closed system with inputs from Polygon and outputs to the IB API, or do you have a more extensive setup of custom data feeds of news sites, twitter, reddit, etc? If so, I'd be curious what challenges you face balancing those with historical backtesting, when some data sources might only have partial historical coverage.

_zkyx 3y ago
All my data is from polygon.io and nothing external. I have been exploring using lookup tables of pre-computed historical values to do things like anomaly detection (ie, is this normal activity for this stock). A good example would be BXRX today [1]. I have been looking at options trading activity too and trying to use that as a signal. [1] https://www.google.com/search?q=BXRX https://www.google.com/search?q=BXRX

cempaka 3y ago

While I was laid off and looking for work, I connected with a cryptocurrency market making firm that had access to a private broker feed which is not attached to any matching, so would occasionally go crossed and offer a pure arb opportunity for the same pairs. I had done some algo trading with Java in Kospi 200 options in 2011-2012 so I decided to put together a simple bot for them to try and grab the crossed markets when they occur. Even an incredibly simple trade like this requires quite a lot of work to get the risk management in place. I also took it as a chance to catch up on new Java features since I had been out of that ecosystem for awhile. It did successfully grab the arbs but there wasn't enough juice to justify more work on it and I got a job in the meantime, so I open sourced the whole thing: https://github.com/abissell/cempaka https://github.com/abissell/cempaka

carabiner 3y ago
Cool. Did you think Java was a good choice for this kind of work? Why not Go like OP or something else?
- cempaka 3y ago
  I mostly went with Java because it's what I knew best, and I thought was likely to be "fast enough" for the market I was in. I'm sure Go would be similar in this case but didn't want to have to take the time to learn it. I do think that if Java can deliver on the combination of the foreign function/memory interface and value types, it might really start to look competitive for certain strategies which are just a bit too complex for the "do everything in the network card" approach. When the Aeron guys first implemented their protocols in Java, C#, and C++, C# was actually the fastest, which they attributed to the presence of both the optimizing runtime and value types.

mr_o47 3y ago

I’m always fascinated by fintech especially when it comes to technology and trading and the author did a really great job of explaining the concepts

_zkyx 3y ago
Thank you!

xgdgsc 3y ago

I've used Go for this before and still use Go for log monitoring. Other parts I've moved to julia/cpp/python mixed workflow.

asavinov 3y ago

When developing an automatic trading system the following aspects are important: - Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format - Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work - Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds. - Real trading and order management as well as coordination of all activities. The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels. I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important

_zkyx 3y ago
Hey, thanks. Yeah, I agree with you. That's an oversight on my end. I'll tell you here though. I'm just using go routines and channels to talk between them and then a giant mutex for locking. That's basically it. So, as new data comes in, it builds aggregates (tick based candlesticks) as needed, this then triggers the the BUY logic loop on that new data, if something is detected, that triggers a IB API order. It is dead simple and nothing complex in here. I've had upwards of 100 positions being tracked at anyone time and seems to just work. So, I haven't messed around with complex async logic too much. I'm actually just hard coding the parameters right into the BUY loop. This probably sounds crazy but for a small setup like this they don't change that much. So, I can run some trades, tweak things, restart, and then test some more. I imagine if you were doing that in an enterprise setting you've have some formal language and hot loading and stuff. But, for me hard coding seems to work well enough.

i-use-nixos-btw 3y ago

Nicely done. A couple of pointers. One on data, one on that RAM usage. First I’ll go with the RAM usage because this is hackernews and everyone loves algorithms. —- There are a lot of libraries out there that do technical analysis, and most of them are designed for batch processing. TALib is an example - it works on large data sets but is not appropriate for live trading because it repeats calculations over and over and over again. If you have 10000 datapoints and calculate indicators, it’ll calculate 10000 of them. Add one more bar, now you have a dataset of 10001 items, which TALib will calculate the indicators on from scratch. Or maybe you just feed the last 10000, and still perform that calculation over all of those, but save that 10001st oldest one. Either way, it’s bad. Same goes for every library I’ve seen, presumably because no one would open source a production grade indicator generation system. The production approach to this is somewhat different. Turn your features into state engines. Most features are just running calculations that are very easy to perform once per bar. Moving averages are a perfect example - for a moving average of N bars, store N items in a fixed size array on the stack and keep track of the latest index to be written to. When a new value (X) comes in, increment that index, and grab the value (Y). Modify your old mean by adding (X-Y)/N. Then write X over the old value Y (It’s a ring buffer). For EMAs, it’s even easier, because you just need to keep track of a numerator and a denominator - nothing else is needed. On a new value X with the scaling factor A (such that A^halflife = 0.5), the numerator N becomes NA+X and the denominator D becomes DA+1. Divide the two and you’ve got your new value. If an indicator depends on another, don’t recalculate it. Break the indicators down into fundamental calculations and you’ll often find a lot of redundant calculations being done. Rearrange it all so it fits. Automate that process if you enjoy that kind of thing like I do. Most indicators can be handled this way. The indicators that can’t - are rarely useful. After all, what you’re tracking is the evolving state of the market, and if you’re doing gymnastics over an indefinite number of bars, it probably doesn’t mean much. The end result is that you don’t end up accumulating memory throughout the day. You receive a bar, you throw it through your indicator generators (all of which using a fixed size of memory), then you discard the bar and wait for the next. Save state at the end of the day and load that on the following market day. The result will be a speed up like you couldn’t imagine. I promise. I run an indicator generation engine in a container capped at 40MB ram on one CPU, and it generates hundreds in much less than a millisecond after the bar arrives. —- Now, onto data. I recommend cleaning your data. You have a screenshot of a Tesla chart in there with some funky highs/lows every so often. It has been a long time since I’ve worked with US equities (and gladly so, it’s a mess of a system!) but the following is the best of my recollection. The trades you receive will come from several sources. For US stocks, there are several different venues that operate their own order books. These will operate as typical markets between the open and close of the day. By typical I mean that the bid and ask represent what you’d get if you market order instantly (which you can’t), and the market trades on them have to take from the bid and ask side of the order book (formed by people placing limit orders). However. There’s also the ADF - the Alternative Display Facility. This is the DIY of trade reporting (and quote reporting, but no one does). If someone sells some shares to their grandmother for a low price in exchange for the recipe to her famous apple pie, the ADF is where they can tell other participants about that trade, manually, subject to fat finger errors, prices of weird fractions of cents, and very relaxed constraints on timing. It’s also where dark pools post trades. The problem with this is that this data has no direct relationship to the rest of the market. This is why, every so often, you’ll see those blips. If you don’t clean them, they’ll play havoc with any indicator that uses highs and lows. The other problem is that ADF trades - at least when I last analysed this very issue - are not rare. They make up a large fraction of trades. So my approach was to clean ADF trades more rigorously than the venues by matching them against prior prices from an ADF-free background.

bofadeez 3y ago

This is professional gambling. Embrace (weak form) market efficiency. Why do you think mutual funds have never generated profit in the history of their existence?

hu3 3y ago
I'm naive but as far as I know mutual funds are risk-averse and by consequence their profits are lower, if any.

jimmysnuka 3y ago

I caught the algotrading bug recently and I've been wanting to backtest a couple of strategies involving ETFs. Right now I'm only looking at the open, close, daily high/low. The problem however is that you have to account for dividends and splits, which I'm not sure how to do (Admittedly, I haven't looked too deeply into it). Any advice on how to do that? Let's say I want to backtest some strategy involving UPRO; would I be fine just using yahoo finance historical prices as is, or would I have to manipulate it in some way? Also, what would be the ideal backtesting software/service to use?

TheAlchemist 3y ago
Harsh answer - if you have to ask this question here, then stop it now. Thing is, you will be fooled by bad data - you will find strategies that works in a backtest but not in production. And the reason will be because you missed some important finance concept (like taxes, dividends, stock splits etc). In this field 99% of success in my opinion is knowing what you don’t know. And only model that small part that you know you know and are fairly sure about it. Ps. By stop it now - I mean stop algo simulations and learn about those concepts, make sure you understand perfectly what data you are putting into your backtest.
- jimmysnuka 3y ago
  Fair point. Right now I'm just doing it as a hobby. I'm not going to be making any real trades for a while and even when I do I don't plan on making money from it (at least not for some time).
1ark 3y ago
Depending on your provider, but there could be an "Adjusted Close" (Yahoo has it) which includes stock splits and dividends. I tried a few backtesting libraries, thought backtesting.py[1] was easy to work with and straight to the point. 1. https://github.com/kernc/backtesting.py https://github.com/kernc/backtesting.py
- jimmysnuka 3y ago

jjice 3y ago

As a layman boglehead, what are the returns of doing trades like this as opposed to an index fund? I assume it either works or there enough money to hope it works, but really curious what the situation is.

_zkyx 3y ago
This whole project started when I was looking for an investment adviser and didn't know anything. I was looking at all these funds that had like 4% return per year and then looked at stocks like Tesla that were up like 45% return and was like this doesn't make sense. When you dig in you see that these index funds are super super low risk. However, I like extremely high risk so I was willing to explore this a little. I don't really want to say you can get better returns, because you could end up losing all your money, extremely quickly betting on single stocks, it's basically a sure thing. So, that's why I turned to building a tool that basically bets for you. So, it's more about risk. If you put everything you had into TSLA you'd be up like 50% just in the past month but that's risky as hell. So, it's more of a risk question than a return question.
- stefap2 3y ago
  At the end of the day or better say, at the end of the year you still have to look at average return on investment.

aantix 3y ago

For those that are interested in algorithmic trading - take a look at Collective2. Where engineers make available their buy/sell signals for a subscription fee. It's like the minor leagues for algorithmic trading. It's fascinating. The system keeps track of the gains/losses, so no cheating on the reporting. You can authorize Collective2 to access your Interactive Brokers account, so that the trade signals are managed on your behalf. It's been around for at least a decade, so you can see some longstanding performance numbers, but most systems just don't last that long. https://collective2.com/leader-board https://collective2.com/leader-board None of the leaders have been there for very long. One to two years. Showing that most system's alpha disappears fairly rapidly. The equity curves are sporadic ~a few of the trades accounting for a majority of the gains.

lost_tourist 3y ago

Does anyone know if he beat the market with this and became a multimillionaire?

_zkyx 3y ago
Not yet :)

nhggfu 3y ago

great work OP.

jasfi 3y ago

I'm working on an algo crypto trading platform. I have pre-defined rules set up in NoCode fashion, but it seems like many people will want to code their own algos. I'm thinking of supporting a scripting option.

pyrrhotech 3y ago

I've also just started a blog series for getting started with algorithmic trading in NodeJS for complete beginners: https://grizzlybulls.com/blog/building-an-algorithmic-trading-backtester-with-node-js-part-1 https://grizzlybulls.com/blog/building-an-algorithmic-tradin.... A lot of folks would raise an eyebrow at the thought of using NodeJS for algotrading, but I've found it to be a great fit for my non-HFT, more swing-trading style, and I've used it successfully in production for over 3 years.

Terretta 3y ago

Here's a recent piece on a change going on in systematic trading engineering: THE GIG ECONOMY COMES FOR HEDGE FUNDS Platforms that offer money managers the freedom to build a business and maximize their return on performance while removing the hurdles of launching independently could change financial markets. If the trend continues it could have a big effect on financial markets by making it easier for a wider assortment of unconventional managers to rise in the industry and offering investors better and cheaper access to them. … Well-received start-up ClearAlpha Technologies has moved closest to the gig model. Its first offering is a commingled fund apportioned among its managers, but it has the platform to act as an exchange, matching investors to individual managers or customized portfolios of managers, cutting out all the expense of intermediaries. Source: -- https://www.bloomberg.com/opinion/articles/2023-06-09/the-gig-economy-comes-for-hedge-funds https://www.bloomberg.com/opinion/articles/2023-06-09/the-gi... Reprints in case paywalled: -- https://www.washingtonpost.com/business/2023/06/09/the-gig-economy-comes-for-hedge-funds/9c52ab8e-06ae-11ee-b74a-5bdd335d4fa2_story.html https://www.washingtonpost.com/business/2023/06/09/the-gig-e... -- https://www.garp.org/risk-intelligence/technology/brave-new-world-062323 https://www.garp.org/risk-intelligence/technology/brave-new-... You no longer have to have graduated as finance into finance. (In fact, we prefer if you graduated with some other math modeling heavy emphasis, think turbulence and flow, or actuarial modeling.) A bit more about us, although this article is about our first fund mentioned above, not about the firm co-founded by Brian and I that owns the fund and built the platform it runs on: https://www.bloomberg.com/news/articles/2023-06-01/goldman-aqr-alumni-start-niche-multistrat-with-asness-backing https://www.bloomberg.com/news/articles/2023-06-01/goldman-a... If you're into this, we've come out of two years' stealth and are now hiring and remote work friendly.