13 ms·
(Disclaimer that I'm working on an open source FIX engine in Lisp.) This is a really cool project! But alas I couldn't use it as I'm not in the US, and I don't
by srazzaque 6y ago
(Disclaimer that I'm working on an open source FIX engine in Lisp.)
This is a really cool project! But alas I couldn't use it as I'm not in the US, and I don't trade with Alpaca. And I feel like this plays into a negative trend I'm seeing.
The retail brokerage market right now is treading on similar 'mistakes' in institutional markets 15ish years ago. Proprietary APIs were prolific, and therefore increased switching costs and lock-in (looking at you, Reuters and Bloomberg).
On the one hand, retail brokers need to support popular programming languages and paradigms to gain traction. So, they are releasing http-based APIs, or popular language-specific libraries/bindings for their proprietary APIs. But, they are doing this instead of pushing well-publicised battle-tested standards such as FIX[1], that would normalise the playing field across brokers. So my gut feel is that they are banking on users coding against their APIs and having some lock-in effects. Notable exception is that any broker that supports cTrader will have a FIX API, because cTrader provides that for them.
The problem is: if I write a bunch of code to work against Alpaca, TD Ameritrade, or whoever, that code will likely need to be revisited for some other broker like Robinhood. Granted, market connectivity code _should_ be well abstracted from my model if I'm following good design principles. But at least with FIX, the core concepts, fields, and messages remain the same across most FIX endpoints (NOS, ExecRpt, etc), allowing for a more uniform code holistically.
[1] https://www.fixtrading.org https://www.fixtrading.org
- Random_ernest 6y agoProbably not the question you had in mind, but what Lisp and what made you choose it?
- srazzaque 6y agoCommon Lisp on SBCL. I lived on the JVM for quite some time (and still do), and I wanted to explore other ecosystems out of sheer curiosity. Got to playing with Lisp/SBCL, some initial tests showed it was pretty quick, and was impressed that I could disassemble functions in the REPL and iterate rapidly. Then a while later... I noticed there wasn't a FIX library at around the same time I was exploring creating my own trading models. So I can't say there was a lot of analysis across the different Lisps, etc. Had the coin flipped a little differently a while back this might have been in Racket. Also the choice of language in the book Professional Automated Trading[1] did influence me somewhat. [1] https://www.wiley.com/en-us/Professional+Automated+Trading%3A+Theory+and+Practice-p-9781118129852 https://www.wiley.com/en-us/Professional+Automated+Trading%3...
- thidr0 6y agoDo you recommend that book and/or any others? Interested in automated trading + functional languages
- srazzaque 6y agoI read it quite a while back, so memory is a little hazy. It's one of the few books (perhaps the only one I've found) that approaches this subject matter from the perspective of an engineer wanting to architect their own system from the ground up, and potentially work for themselves or in a firm. But, going in, you should know that it focusses less on trading techniques and more on software architecture. What data structures you might use, and one potential approach that's laid out in the book. Whilst some areas in the book are light (for example, the performance section), I'd recommend it if you're interested in ground-up building a trading system. If you work in the space already, you'll know the parts that are misaligned with majority of systems out there (choice of language being one). If you don't, and are looking to enter the space, it'll increase familiarity with core concepts (for instance, before I read this book, I had no distinction between "model" and "algo"). It's also refreshing that it stands out from the pack of books that aim to make you zillions of dollars just by using "these 10 key trading secrets!".
- thidr0 6y agoExcellent. That sounds like something I'd really enjoy. I don't work in the space, but I'm more interested in the trading systems than the algos, primarily. After all, I'm a little skeptical about my ability to beat the PhDs with their institutional nukes while I'm over here sharpening a dagger.
- paladfn 6y agoI would recommend "Professional Automated Trading: Theory and Practice" [0]. The code is here [1]. [0]: https://www.amazon.com/Professional-Automated-Trading-Theory-Practice/dp/1118129857 https://www.amazon.com/Professional-Automated-Trading-Theory... [1]: https://github.com/wzrdsappr/trading-core https://github.com/wzrdsappr/trading-core
- klohto 6y agoSince you’re not from US, what trading platforms do you use? I haven’t found any with free commissions and public APIs in the EU. Well, except for Interactive Brokers.
- hnarn 6y agoNot sure about API but there’s Degiro.
- nelaboras 6y agoIB doesn't seem to be commission free though.
- srazzaque 6y agoI'm in Australia, so the choices here are also a bit limited. Also, I'm less interested in equities. In the CFD space, my shortlisting technique now is basically "do you offer cTrader", because as mentioned above, this means they have a FIX endpoint. Broker offerings here seem a few "paradigm shifts" behind the US. Zero commissions are unheard of last I checked. Also, if you approach some brokers here and ask for an API or a FIX endpoint, you're either told to go away, or you're effectively ushered into a luxurious room, poured some exquisite scotch, and asked about how many millions of AUD you're looking to trade per month. And _then_ told to go away. Edit: disambiguate "Aus" :-)
- pedalpete 6y agoIs IB "Interative Brokers"? I'm trying to figure out if Aus is Australia or Austria. I'm in Australia, and have been looking at who to go with. Almost signed up with CBA last week, but definitely more expensive to what I'm used to in North America.
- zelly 6y agoThe proprietary API is guaranteed to be slower. The brokers must send order flow as FIX to the exchange. This adds latency of however long it takes the broker's software to translate to FIX. I don't think there are any successful quant strategies that don't rely on speed, so basically anything with a proprietary API is solely for amateurs.
- oarsinsync 6y ago> I don't think there are any successful quant strategies that don't rely on speed There are plenty of successful quantitative algorithms that are in use by non HFT firms > so basically anything with a proprietary API If you’re not a direct market participant, colocated in the exchange, you’re not in a position to be talking about speed at all. Also most exchanges optimising for speed are not using FIX. > is solely for amateurs You’re not necessarily an amateur, you’re just leaving money on the table for someone else as you’ve not optimised that area of your trading infra yet.
- getlawgdon 6y agoActually the majority of quant trading isn't dependent on speed.
- smabie 6y agoI don't think you understand what quant finance is about. Almost all quant strategies don't rely on speed. In fact, HFT is usually considered a totally different space than investing (and therefore quant investing).
- egwor 6y ago> The proprietary API is guaranteed to be slower. Yeah, most of the time that's not the case. Binary encoding/decoding is often much faster than FIX. Look at ITCH/OUCH, SBE and FastFix.
- nurettin 6y agoFIX is just a protocol template. Vendors will make non-standard extensions to it. For example: OCO you send to CQG won't work with StoneX.
- srazzaque 6y agoOf course, and in practice you basically always need to tweak your FIX layer when connecting to multiple brokers. Whilst there will be differences, you can rely on a surprising amount being uniform. The general session/authentication flow and serialisation mechanisms will be largely the same. Or the use of standard fields ClOrdID, or OrderQty. Or the "general" meaning of the messages NewOrderSingle or ExecutionReport. It's somewhat predictable where the differences are going to be: identifying securities (different codes, etc), expressing order types and algo parameters, and subtle differences to order lifecycles. Again this is a gut feel, but I doubt that if you take 2 proprietary APIs, you'll get that level of uniformity.
- nurettin 6y agoExecution report is another thing that is different between the two. You can't even list your running orders without resorting to custom tags. You can't send multiple orders without providing their custom delimiters. You have to modify login messages and insert custom tags before a message is sent. The only uniformity I am left with is the fix template of key=value\0x01 repeating N times. Nobody conforms to fix 4.2 or 5.0xxx they say they derive from it, then you go through their conformance test of 100 messages and see for yourself what changes they made.
- srazzaque 6y agoI hear your pain... There's definitely brokers doing some strange things out there, or being overly sensitive to the order of fields. Let's not even talk about repeating groups. > custom delimiters Surely you mean custom _message_ delimiters, not field delimiters? In any case I think it's a foolish endeavour to think that if a broker says "we conform to FIX4.4" to ever take that at face value. Agree that the conformance test always reveals the true extent of conformity. But zooming out a little, despite these challenges, I'd say you're still in a better position than having N endpoints where 1 gives you a Java library with its own ridiculous threading model, another gives you a Python library that calls a HTTP endpoint (that you need to provide an OAuth token for), and yet another that only has a hosted C# interface. That's where we're headed towards.
- Amorymeltzer 6y agoFor the US and Robinhood folks: there's a fairly effective project that allows you to do something similar: https://github.com/bcwik9/robinhood-on-rails https://github.com/bcwik9/robinhood-on-rails I used it a bit for fun, seemed legit and worked fine.
- loblollyboy 6y agoRobinhood has a website tho
- mellosouls 6y agoHow is it similar; the whole point of the parent comment is independence from that walled-garden approach - it sounds like precisely the opposite to me?
- daniel-levin 6y agoI have implemented FIX for a living for a few years. It’s fragmented beyond belief. You can have different standards for different parts of the same brokerage. You have to do separate conformance tests for different asset classes at the same (big, well known, reputable) broker. The proprietary APIs like Bloomberg’s provide additional value rather than being alternatives to FIX. Make no mistake: you cannot interchange a FIX endpoint for another without making changes. The core problem is that you can impose arbitrary semantics on top of FIX messages. For example, a broker might require you to send timestamps in UTC for equities, and local time for FX, on the same FIX connection. Another might have a limit on the number of characters you can put in a ClOrdId - so GUIDs as ClOrdIds go out the window. Another big screw up is extensive use of non-ISO currency codes and the ambiguity of major and minor denomination. Is “GBp” pence? Sometimes yes! Sometimes not. Then - the most colossal screw up of them all - the reverse proxy people. They are companies that will convince management that they have a “network” and “access to liquidity”. It’s actually just a reverse proxy server that _manipulates messages in flight_! It sounds ideal, as if they will standardize your message flow so you don’t have to worry about broker specifics. In practice, it never works, and they opaquely mess with things you don’t want them to. Imagine sending an order denominated in cents, and the reverse proxy forwards it having changed the denomination to dollars. This really happened to me. BTW, I love working on this stuff and swapping notes with other people. Drop me a line (email in profile) if you want to talk anything related to FIX!
- misja111 6y agoI can confirm. When I was implementing FIX to communicate with some big banks, it was common practice to put everything specific for the interface as XML in a single FIX field. FIX was only used because many banks had existing infrastructure built around it, such as reverse proxies on application level that would scan the FIX messages for some standard fields and route the incoming message to the appropriate server.
- srazzaque 6y agoThanks for the insights, I'll certainly hit you up!
- offlineguy 6y agoI wrote a FIX engine in Common Lisp once, working for a Norwegian retail broker. Worked like a charm for many years, connected to multiple market places and brokerages. Adapting to new counterparties was certainly a bit of work, but generally far less than implementing their binary protocols. (We tried that too from time to time, usually if FIX was not an option from the exchange. Admittedly rare i later years.) Unfortunately I wasn’t allowed to open source it, and it’s now lost in the void after a takeover. :( Good luck to you. It’s not an impossible project. :)
- srazzaque 6y agoThanks! Until your reply, I was pretty sure the intersection of lispers and markets devs was close to nil. It's been mainly Java/C++ from what I've seen. It's a shame that you couldn't open source it. If you're open to it, I'd appreciate any feedback, thoughts or code review you could offer. You can get to it (and my email) with the second link in my profile. Edit: typo, and point about Java/C++ dominance.
- jetti 6y agoHas the FIX spec been free and open for a long time? I was going to create my own FIX engine a couple years back but for some reason I remember having to pay for the spec and it wasn't cheap.