4 ms·
I've used Lua in the realm of quantitative finance for years. Early on, it was primarily a flexible configuration tool, embedded in a complex event processing
by neomantra 13y ago
I've used Lua in the realm of quantitative finance for years. Early on, it was primarily a flexible configuration tool, embedded in a complex event processing system.
With the advent of LuaJIT 2, I've used it standalone for analyzing many many millions of messages per day in real-time, then feeding resultant datasets to other more batteries-included environments which aren't near as fast (e.g. Python, Matlab) or to datastores like Redis and Mysql
I also use OpenResty for web applications (risk management apps, charting, and other visualizations), often pulling from those same datastores. OpenResty is a joy to work with and is crazy fast.
I make heavy use of the FFI both for library bindings (including system calls) and general C structure use. The messages mentioned above are represented as C-structures and are efficiently processed by LuaJIT. I find myself often writing Lua scripts to do networking programs rather than C. Check out `ljsyscall`.
Dealing with memory has definitely been a stumbling point, but once I built enough scaffolding to deal with it (including using jemalloc), it has been clear sailing. Now I store millions of objects taking many GB of memory without significant GC pressure using FFI-based HashMaps and Vectors.
Also understanding how to keep the code in the JIT (versus the interpreter) has taken some artistry -- e.g. examinging the output of -jv and -jdump.
Although this thread is about LuaJIT 2.0.3 release, I use the LuaJIT 2.1 branch which has many enhancements; I especially appreciate the string improvements and trace stitching.
- minikomi 13y agoThat sounds pretty cool! It's mainly as a normalization step before hard crunching?
- neomantra 13y agoDefinitely for normalization, e.g. take a UTDF message, convert it to some representation (either a C-blob, CSV, or JSON, depending on final use), then push it to a Redis list named utdf.MSFT. But beyond normalization, there is building books (be they level-1, 2, or 3), maintaining state, and extracting information from that. Random example (i.e. not a real query I've done): emit all the trade executions on any venue that happen within the first two depth levels of NASDAQ. LuaJIT could do the hard crunching, like regression analysis, quite well, and there are some interesting projects like GSL-shell... but Matlab and numpy/matplotlib are quite expressive and more complete environments.
- lukego 13y agoCool. I found the hash maps and jemalloc binding on Github via your profile. That's something I imagine we will need in Snabb Switch eventually. Now we know where to find it! Thanks.
- neomantra 13y agojemalloc really improved my situation. Before it, I would run a study and then randomly get an out-of-memory error despite having relatively low number/size of GC objects... my FFI objects were snagging space from them. Switching allocations to jemalloc (which LDS makes easy to do) alleviated the problem entirely. I've been eyeing SnabbSwitch for a while -- I use OpenOnload extensively, so appreciate what you are doing.
- egeozcan 13y agoIf you are using or are interested in using OpenResty, you may also like Lapis: http://leafo.net/lapis/ http://leafo.net/lapis/
- neomantra 13y agoI haven't found a need for Lapis, although I think it is an interesting project and enjoy all his work. I use leafo's template engine: https://github.com/leafo/etlua https://github.com/leafo/etlua