6 ms·
Advice on the implementation of a real-time trading system [closed]
- brutuscat 14y agoInteresting feedback I already got: https://groups.google.com/d/topic/celluloid-ruby/9RHPmYkxGtU/discussion https://groups.google.com/d/topic/celluloid-ruby/9RHPmYkxGtU... https://groups.google.com/d/topic/rubysur/ZJoZxDVcRYU/discussion https://groups.google.com/d/topic/rubysur/ZJoZxDVcRYU/discus...
- aidos 14y agoLmax was the first thing I thought of too but I don't really see how it applies here. Celluloid looks interesting have you played with it yet?
- RossM 14y agoWhile it's certainly an interesting question, I presume you're posting it here to garner support for it being reopened (as far as the rules go, I agree with the closure, it's very open-ended). It might fit better on http://programmers.stackexchange.com http://programmers.stackexchange.com. I know little about stock markets but I quite like the idea of spawning workers to monitor orders after being placed. These could then update a page live to the customer.
- brutuscat 14y agoHi there, I'm the SO owner of the question. It's Ok that the questions was closed I guess... But I couldn't find a way of make it more "to the point". But now that it's here it would be great to have some comments from HN though ;)
- aidos 14y agoI too think this is an interesting question and I'd love to hear how people would approach it. How long do the orders need to be monitored for? Indefinitely? If so, is it really feasible to keep them all running? How much data do they need to analyse / aggregate to make their decisions? Very interesting question.
- brutuscat 14y agoFor instance orders would be open from week into week if the user enables it (each week). In my case "decision making" is only based on some fixed price threshold in order to provide some risk management. So the amount of data to aggregate would be minimal. But the data to analyse came directly (as a huge stream) from the markets.
- dschiptsov 14y agoUse Java, of course.
- brutuscat 14y agoWhy "of course"?
- meaty 14y agoOff the top of my head, the op is probably talking about the off the shelf stuff available for mq stuff and integration which is unsurpassed in java, plus the very low latency IO. TBH when I write java, I usually write virtually no code: I just glue other shit together. I don't get why people have so much against Java - it gets shit done in incredibly short amounts of time.
- chewxy 14y agoUntil you have to come up with your own algorithm. Then you're stuck with ohgods-so-many-.java-files
- meaty 14y agoDepends. I wrote a rather complicated corridor based routing algorithm for a courier company in the UK which worked on real time traffic data. It was only 11 files and 12,000 loc and that was their entire business model.
- scorpioxy 14y agoI find that people who "hate" java or any other language for that matter, haven't even used it in any meaningful way. I find its a great language for integration in general and talking to "legacy enterprise" systems. I do find it verbose, and have other remarks about it, but it does the job well enough.
- brutuscat 14y agoYeah, I actually kind-of think like that. So I use JRuby to integrate things with Apache Camel for example... Thanks
- rhapsodyv 14y agoI have made one in ruby for my trades. The key point of the architecture was to do all asynchronous with queues and workers. It worked for me because the network time was big and I don't do high frequency trading. I have used rabbitmq and ruby with event machine.
- yesbabyyes 14y agoYesterday I learned about Postgres' LISTEN/NOTIFY functionality. I'm not sure about your architecture, or how many listeners/watchers Posgres supports, but it might be that you can just listen to changes in your database -- that might be the most lightweight solution. Edit: Here is a small tutorial: http://bjorngylling.com/2011-04-13/postgres-listen-notify-with-node-js.html http://bjorngylling.com/2011-04-13/postgres-listen-notify-wi...
- meaty 14y agoIf you're trading, keep it all out of the database. It won't scale and latency is terrible and unpredictable regardless of the engine. Keep the database for storing stuff that you don't need in ram. Keep the rest in ram and message queuing/pubsub bus architecture. Plus events in a database engine are a pain in the butt to write tests against.
- BUGHUNTER 14y agoWouldn´t it be much more interesting to spend that precious brainpower into something that actually solves real problems like war, disease, crime, militarism, xenophopia, lack of education, etc. instead of building another greed-engine that will trigger even more poverty, exploitation and repression? Of course everybody is free to act on his own karma (in a few countries) - but you will have a much better life by doing some more useful work that will benefit all human beeings, not only a few criminals that build weapons of mass murder just for the mentally poor reason of making more and more money. Programmers haz powers to push human evolution, don´t waste your life on golden lies, that will turn into deadly dust in the end!
- maeon3 14y agoA "Quantitative Analyst Salary" starts at 100k/year. Trying to build a trading system is like trying to invent the next 10x battery technology could end up golden dust in the end as well. You're distracting people from proper cost benefit analysis and assessment of risk/reward. http://www.glassdoor.com/Salaries/quantitative-analyst-salary-SRCH_KO0,20.htm http://www.glassdoor.com/Salaries/quantitative-analyst-salar... Programming getting into high frequency trading is "high risk, higher reward". It would be better to use our Neural Networks to find risk factors for prostate cancer you'll get in 25 years, but the money just isn't there. AKA high risk, low (possibly no) reward.
- PoliticsJunkie 14y agocommunist
- davidmr 14y agoThis kind of sanctimony is tiring. Every time one of these threads comes up here, it inevitably devolves to a point where someone brings up something like the above. There are those of us who actually enjoy these kinds of problems in and of themselves. I don't consider what I do particularly noble, but I'm paid to solve very interesting and unusual problems in a very fast-paced environment, and I don't feel the need to apologize for it. I'm under no illusions that I'm doing God's Work, but if the bar for useful work is something that will end "war, disease, crime, militarism, xenophopia, lack of education", I think >99% of us fall quite short of it. Maybe I fall shorter than you do (you don't mention what it is that you do), but I still enjoy my job.
- eduardordm 14y agoHe needs to get in touch with the stock exchange (and brokers) before jumping into any implementation conclusions. He is not a broker, you can't really interact with any exchange directly. He needs to use some bank (which will already do 99% of what he is implementing). Without knowing exactly how the market systems work, this is a toy, he should have talked to someone BEFORE writing any code. Things are way more complex and simpler than it looks. It's extremely simple to use bank services to do that for you and close to impossible for a developer to just jumpt into writing trading software without a bank backing it.
- eduardordm 14y agoConsidering that users in this thread might be interested in trading software, let me add this link I posted yesterday: http://news.ycombinator.com/item?id=5041040 http://news.ycombinator.com/item?id=5041040 "Markets too complex for high frequency trading software"
- RaphiePS 14y agoSomeone ought to create a "stripe for stock trades" to make building stock bots easy-peasy for any developer looking for a challenge. I'd certainly use it.
- eduardordm 14y agoSounds like an idea PG would fund. The problem here is not even money, is having enough influence to get people and banks to trust you - and get a deal sweet enough to be profitable.
- brutuscat 14y agoAnd that's exactly why is not that easy to get in touch with banks and brokers too. "Financials" sucks and they hide every inner workings...
- blcArmadillo 14y agoThis isn't quite accurate. There are brokerages such as Interactive Brokers that provide an API so that you can write automated trading systems. And automated trading isn't limited to just high frequency trading. It's true that most banks have the simple features he mentions in the post but as soon as you want to get a bit more complex what the bank offers just isn't going to cut it.
- pencilcode 14y agoYou could use one of the realtime api's, like pusher or realtime.co. They can handle the scale. Otherwise erlang/otp looks like a good fit
- pknerd 14y agoSome of my similar architecture questions were closed by Mods. I don't know where one can ask such questions?
- eduardordm 14y agoyou could try quora
- brutuscat 14y agoYes, I was asking the same thing. Too bad that we can't use SO to discuss things. HN is good, be the feedback here is not that technical. What I ended up doing is to cross-post the questions in some mailing list/groups that are slightly related to the topic (celluloid, any active ruby/my-lang group).
- gtt 14y agoI often see well-thought and well formulated architecture questions to be closed on SO. Why do mods do such thing?
- DanBC 14y agoThe SE sites have a strict rule - only questions of a certain type are allowed. That's a good rule. It means SE can focus on that thing. But it's a great shame that the huge knowledge of the SE sites is locked in that format. There's quite a lot of heat generated around the fuzzy borders of acceptable and unacceptable borders, and having a question closed or deleted doesn't feel good. Meta is death. I kind of agree, yet SE does have active meta. SE also has chat stuff. So, would it really kill them to have "freeform SE question site"? It could be a paid feature (although recommending people buy a sub to post a question would cause all kinds of furore). Thus, you'd have: http://gamedev.stackexchange.com/ http://gamedev.stackexchange.com/ http://meta.gamedev.stackexchange.com/ http://meta.gamedev.stackexchange.com/ *http://freeform.gamedev.stackexchange.com/ http://freeform.gamedev.stackexchange.com/ It would still have strict rules, and mods, and etc. But it would allow more open form discussion. They might need to have a different structure to allow threading and better replies?
- boothead 14y agoIt's not the orders you need to monitor from the sound of it (at least once the trade is complete), it's the position that you hold in the market vs the current price. * Actually the previous statement could be wrong: You might also want to do this on a per order basis, although they are on some sense functions of each other. I would suggest the following: For any market which you have a position in, subscribe to that market's price feed. I'd use zeromq for this, but there are plently of pub/sub libraries and products. The previous point implies that you are republishing your market data feed into specific channels. How they are mapped depends on many factors, but basically you want to de-multiplex the market data so that you are able to pick out one market's price data as a stream of updates from the whole. Once you have a feed of the market with a position you can use it to transform the stream of market data to multiple streams of values. You might have new steam of PnL values for each order you have on the market, or a stream of price differences or whatever, you then re-publish those streams. Depending on how many of these derivative streams you have and the rate of data, you might want to do some downsampling too. What I've described above is pretty similar to the market data architecture in hedge funds that I've worked at. Have a look at the ideas behind function reactive programming http://en.wikipedia.org/wiki/Functional_reactive_programming http://en.wikipedia.org/wiki/Functional_reactive_programming for more info on stream processing. For the front end, you'll bridge the particular stream of data to a WebSocket (probably translating it to json or a binary data format that javascript can understand at the same time). I don't know how you do that in RoR, I'd use gevent or eventlet in the python world. You'll then need to route this data in some way to a representation of the order/position in the front end. I'd suggest looking at knockout.js, ember.js, angular or something similar as those declarative style libraries will massively reduce the complexity of your client side coding. Last bit is visualization and there's really only one sane choice for this imho: D3 and libraries built on top of it. Look out for http://square.github.com/cubism/ http://square.github.com/cubism/ and http://square.github.com/crossfilter/ http://square.github.com/crossfilter/ and http://code.shutterstock.com/rickshaw/ http://code.shutterstock.com/rickshaw/ for those bits. Bit of a whirlwind here, hope it helps!
- brutuscat 14y agoGreat! I think you nailed it. Function reactive programming was being developed in my head somehow. I really really appreciate your comment. Thanks!
- dunster 14y agoYou might be interested in zipline, an open source backtester written in Python. Zipline, Quantopian's open source backtester (yes, I work at Quantopian). The backtester is designed to also be a trading engine. You feed zipline data in "events" where each event is a point in time. Zipline doesn't care if the events are live or replayed from a database - it processes the data and executes any trades in the order book. The tricky part here is that zipline is more aimed at implementing trading algorithms. You'd have to write a simple algorithm to implement the buy/sell orders coming in. https://github.com/quantopian/zipline https://github.com/quantopian/zipline
- OldSchool 14y agoFrom some of the comments I see, let's not read too much into this. He's not asking how to build an -automated- trading system, just one that's realtime and all analysis and decisions are made by the user. The best I've seen that's commonly available is "Trader Workstation" and it's free with your account at Interactive Brokers. It's written in Java.