3 ms·

I know many people don't consider Common Lisp as particularly fast language, but actually in my experience you can make it quite fast depending on what you need

I know many people don't consider Common Lisp as particularly fast language, but actually in my experience you can make it quite fast depending on what you need. When using SBCL, for example, you get your application to be compiled to native instruction set. Moreover, you get control over optimization level for each piece of the code separately. Even more, you get complete control over the resulting native assembly. Something you don't get to do with other higher level languages. One of the production applications I did in Common Lisp was parsing a stream of XDP messages (https://www.nyse.com/publicdocs/nyse/data/XDP_Common_Client_Specification_v2.0.pdf https://www.nyse.com/publicdocs/nyse/data/XDP_Common_Client_...) with a requirement for very low latency. I made the parser in Common Lisp so that it generates optimal binary code from XML specification of message types, fields, field types, etc. using a bunch of macros. The goal of application was to proxy messages to the actual client of the stream. The proxy was there so that it was possible to introduce changes to the stream in real time without requiring to restart any components. Using REPL I was able to "deploy" any arbitrary transformation on the messages. The actual client of the messages was a black box application that we had no control over but we sometimes had problems with when it received something it did not like. I liked Common Lisp in particular because it does not force you to make your performance decision upfront. You can develop your application using very high level constructs and then you get the option to focus on parts that are critical for your performance. Macros allow you to present DSL to your application but then have full control over code that is actually working beneath the DSL. If everything fails, calling C code is a breeze in Common Lisp compared to other languages.

logicchains 6y ago
I explored Common Lisp for high performance code once (I work in HFT), the biggest issue was that it didn't have native support for "arrays of structs" (where all the structs would be values stored "unboxed" adjacent in memory). I know it'd probably be possible to write a library to do that, but that'd be a huge amount of work compared to just using a language with existing support for unboxed arrays.
- lmilcin 6y ago
  Well, I did some algorithmic trading (not really HFT but still monitoring the market and responding within 5 microseconds of incoming messages). I would not use Common Lisp on the critical path because I would end up basically rewriting everything to ensure I have control over what is happening so that some kind of lazy logic does not suddenly interrupt the flow to do some lazy thing. A large part of the application was basically about controlling the memory layout, cache usage, messaging between different cores, ensuring branch predictor is happy, etc which would be really awkward in Common Lisp (technically possible, practically you would have to redo almost everything). We have also experimented with Java with the end result being that the code looked like C but was much more awkward. I have, however, successfully used Common Lisp to build the higher layer of the application that was orchestrating a bunch of compiled C code and also did things like optimizing and compiling decision trees to machine code or giving us REPL to interact with the application during trading session.
  - logicchains 6y ago
    >giving us REPL to interact with the application during trading session This to me is what the big appeal of Common Lisp for a trading system could be, particularly if it allowed live recovery from errors (dropping somebody into the debugger, rather than just core-dumping), which could save a lot of money by reducing downtime. But as you say it would require redoing everything to make the code fit latency constraints and be cache friendly, which would be a lot of work.

logicchains 6y ago

I explored Common Lisp for high performance code once (I work in HFT), the biggest issue was that it didn't have native support for "arrays of structs" (where all the structs would be values stored "unboxed" adjacent in memory). I know it'd probably be possible to write a library to do that, but that'd be a huge amount of work compared to just using a language with existing support for unboxed arrays.

lmilcin 6y ago
Well, I did some algorithmic trading (not really HFT but still monitoring the market and responding within 5 microseconds of incoming messages). I would not use Common Lisp on the critical path because I would end up basically rewriting everything to ensure I have control over what is happening so that some kind of lazy logic does not suddenly interrupt the flow to do some lazy thing. A large part of the application was basically about controlling the memory layout, cache usage, messaging between different cores, ensuring branch predictor is happy, etc which would be really awkward in Common Lisp (technically possible, practically you would have to redo almost everything). We have also experimented with Java with the end result being that the code looked like C but was much more awkward. I have, however, successfully used Common Lisp to build the higher layer of the application that was orchestrating a bunch of compiled C code and also did things like optimizing and compiling decision trees to machine code or giving us REPL to interact with the application during trading session.
- logicchains 6y ago
  >giving us REPL to interact with the application during trading session This to me is what the big appeal of Common Lisp for a trading system could be, particularly if it allowed live recovery from errors (dropping somebody into the debugger, rather than just core-dumping), which could save a lot of money by reducing downtime. But as you say it would require redoing everything to make the code fit latency constraints and be cache friendly, which would be a lot of work.