4 ms·
I currently run a batch of trading servers solo. The trading system is a C++ process with an asynchronous logger that prints log levels and times. One of the
by CoolGuySteve 7y ago
I currently run a batch of trading servers solo. The trading system is a C++ process with an asynchronous logger that prints log levels and times. One of the issues with trading is that you're dependent on your datafeed and exchange connections working which is out of your control.
I use a python monitoring script that tails logs watching for ALERT level log lines and constant order activity combined with a cron watchjob to ensure the process is alive during trading hours. The exception handler in the monitoring script sends alerts if the script itself dies.
If there are any issues I use twilio to text me the exception text/log line. I also use AWS SES to email myself but getting gmail to permanently not block SES is a pain in the ass. By design Twilio + AWS SES are the only external dependencies I have for the monitoring system (too bad SES sucks).
On my phone I have Termius SSH setup so I can log in and check/fix things. I have a bunch of short aliases in my .profile on the trading server to do the most common stuff so that I can type them easily from my phone.
I also do all my work through a compressed SSH tmux including editing and compiling code. So if things get hairy I can pair my phone with my laptop, attach to the tmux right where I left off, and fix things over even a 3G connection.
This compressed SSH trick is a huge quality of life improvement compared to previous finance jobs I've worked where they use Windows + Citrix/RDP just to launch a Putty session into a Linux machine. It's almost like finance IT has never actually had to fix anything while away from work.
- tebbers 7y agoIf it helps at all, I found that buying a dedicated IP for SES helped our deliverability enormously.