5 ms·

I wrote my first trading system in Clojure because a few of the libraries we relied on were in java. It was 2011 and I'd had about 3 years of lisp experience.

I wrote my first trading system in Clojure because a few of the libraries we relied on were in java. It was 2011 and I'd had about 3 years of lisp experience. I got a bit of side eye from people when I told them I was using a relatively new programming language but the fact that it was based on the JVM, which alot of HFT firms were using helped make the case. We didn't use if for more than a few years before it was retired and rewritten, though that was due to new requirements that included C++ interop. In the end tracking memory usage and allocations got too hard and if you've ever written something that is time sensitive you'll know just how slow memory allocation is, so you could argue I made a poor choice but for the rewrite was easy to reuse the java libraries when we moved to java. The harder part was porting the algorithm implementation code. The reason was that I chose a common lisp trick of first writing a DSL in Clojure that was then used to write the algos. That was the one part that seemed worse in the rewrite as it became far more verbose and clunky. But man was it fun, I learned more in a month than I often do in a year now. Thanks Rich!! It's not often you get to experiment with cool tech and make lots of money doing it. I appreciate your work.

behnamoh 3y ago
> The harder part was porting the algorithm implementation code. The reason was that I chose a common lisp trick of first writing a DSL in Clojure that was then used to write the algos. Doesn't it mean that Lisp macros should really be avoided at all costs?
- chollida1 3y ago
  Can't say for sure, given that my sample size is one. The upsides of DSL(Domain specific languages) and macros in general is that they cut down on code duplication and give everyone using them a common language and set of tools on which to build an app. That part is awesome and I think really helped to speed up development of the algos that were built on top of the DSL. The downside is, that you need to learn the DSL but that's a very small impact in my experience. THe larger downside as you and I point out is that those tools aren't really there for other languages so if you do plan on porting the code base to another language then maybe steer clear of macros. But again, I wouldn't say I have anywhere near enough experience to claim to be an expert in that.
uxcolumbo 3y ago
So you had to switch to C++ because Clojure / Java wasn't performant enough?
- chollida1 3y ago
  > So you had to switch to C++ because Clojure / Java wasn't performant enough? Clojure yes, java no. Many HFT firms build their systems in java and run on the jvm(specialized hardware aside), its plenty fast if you avoid memory allocations. It is also possible Clojure is more than up to the task now and its also reasonable to believe that I wasn't a strong enough clojure developer to fix those issues at the time and a more competent developer could have made it work. But like almost all things if you chose a very out of the mainstream language and start to have issues, most developers will point fingers at the language first. Side note, its a miracle that Jane Street kept using OCaml after the first few years and points to how strong their core tech team was. C++ was an external dependency brought on by other systems that needed to be integrated and is a far safer language to use. I'd imagine that if you say HFT to most developers, their first thought would be C++ even if they haven't worked in the domain before. It's like choosing Julia for your machine learning platform and then switching to python.
  - Jach 3y ago
    It's a shame you couldn't have pitched Common Lisp. Maybe could have ported your DSL pretty quickly and wound up having SBCL output nice compiled assembly instructions and shown x% speed improvement without much effort by using a "sister language" of sorts? ABCL in 2011 would have been a harder sell, it only had its 1.0 release that October. A few years ago another commenter noted they had developed a trading system in Lisp and C https://news.ycombinator.com/item?id=25222297 https://news.ycombinator.com/item?id=25222297 (further comments elaborate a bit) before being asked to rewrite it in Java. What keeps less popular languages alive at companies, when there's no forcing reason to get rid of it like really unacceptable performance, is a conviction (borne out as far as I can tell) that it's ok to hire people who don't know the language -- they'll get up to speed more than fast enough.
  - roenxi 3y ago
nerdponx 3y ago
I suppose you couldn't convince your employer to let you use an embeddable Scheme interpreter for the DSL?
- chollida1 3y ago
  Well the performance hit alone would be a non starter.
kaba0 3y ago
> you'll know just how slow memory allocation is Just a small nitpick, but allocation is definitely not slow in case of the JVM, it is often faster than manually managed languages. It is a pointer bump only. All the necessary mechanisms of a GC does have an overhead, so your point stands, but not for the mentioned reason.
- deleted 3y ago
  [deleted]
- chollida1 3y ago
  Ok sure that’s fair. Typically in HFT We just want to avoid any full sweep garbage collection between 8am and 6pm

behnamoh 3y ago

> The harder part was porting the algorithm implementation code. The reason was that I chose a common lisp trick of first writing a DSL in Clojure that was then used to write the algos. Doesn't it mean that Lisp macros should really be avoided at all costs?

chollida1 3y ago
Can't say for sure, given that my sample size is one. The upsides of DSL(Domain specific languages) and macros in general is that they cut down on code duplication and give everyone using them a common language and set of tools on which to build an app. That part is awesome and I think really helped to speed up development of the algos that were built on top of the DSL. The downside is, that you need to learn the DSL but that's a very small impact in my experience. THe larger downside as you and I point out is that those tools aren't really there for other languages so if you do plan on porting the code base to another language then maybe steer clear of macros. But again, I wouldn't say I have anywhere near enough experience to claim to be an expert in that.

uxcolumbo 3y ago

So you had to switch to C++ because Clojure / Java wasn't performant enough?

chollida1 3y ago
> So you had to switch to C++ because Clojure / Java wasn't performant enough? Clojure yes, java no. Many HFT firms build their systems in java and run on the jvm(specialized hardware aside), its plenty fast if you avoid memory allocations. It is also possible Clojure is more than up to the task now and its also reasonable to believe that I wasn't a strong enough clojure developer to fix those issues at the time and a more competent developer could have made it work. But like almost all things if you chose a very out of the mainstream language and start to have issues, most developers will point fingers at the language first. Side note, its a miracle that Jane Street kept using OCaml after the first few years and points to how strong their core tech team was. C++ was an external dependency brought on by other systems that needed to be integrated and is a far safer language to use. I'd imagine that if you say HFT to most developers, their first thought would be C++ even if they haven't worked in the domain before. It's like choosing Julia for your machine learning platform and then switching to python.
- Jach 3y ago
  It's a shame you couldn't have pitched Common Lisp. Maybe could have ported your DSL pretty quickly and wound up having SBCL output nice compiled assembly instructions and shown x% speed improvement without much effort by using a "sister language" of sorts? ABCL in 2011 would have been a harder sell, it only had its 1.0 release that October. A few years ago another commenter noted they had developed a trading system in Lisp and C https://news.ycombinator.com/item?id=25222297 https://news.ycombinator.com/item?id=25222297 (further comments elaborate a bit) before being asked to rewrite it in Java. What keeps less popular languages alive at companies, when there's no forcing reason to get rid of it like really unacceptable performance, is a conviction (borne out as far as I can tell) that it's ok to hire people who don't know the language -- they'll get up to speed more than fast enough.
- roenxi 3y ago

nerdponx 3y ago

I suppose you couldn't convince your employer to let you use an embeddable Scheme interpreter for the DSL?

chollida1 3y ago
Well the performance hit alone would be a non starter.

kaba0 3y ago

> you'll know just how slow memory allocation is Just a small nitpick, but allocation is definitely not slow in case of the JVM, it is often faster than manually managed languages. It is a pointer bump only. All the necessary mechanisms of a GC does have an overhead, so your point stands, but not for the mentioned reason.

deleted 3y ago
[deleted]
chollida1 3y ago
Ok sure that’s fair. Typically in HFT We just want to avoid any full sweep garbage collection between 8am and 6pm