22 ms·

I wrote a trading system where the strategies are written in Lua. It has been a delight, fast and simple. The traders have the expressivity of a full programmin

I wrote a trading system where the strategies are written in Lua. It has been a delight, fast and simple. The traders have the expressivity of a full programming language and I can add any function to that language and provide any data as well so that their programming (and execution) sandbox is extremely ergonomic and suited to their specific task. In other words, a trading DSL. Other code outside the sandbox pulls in up to date price data and if certain safety rules are violated will automatically close out positions. So even if the traders code their way into an infinite loop or make other mistakes, the supervisor can step in and automatically prevent disaster. Using Lua to make a language for others has been a wonderful experience. FYI, it was approx 11K lines of Lua).

nattaylor 2y ago
A friend built a system where the ad campaigns with all their targeting rules were Lua scripts and it was also fast and simple in a glorious way!
- knighthack 2y ago
  Why wasn't Python used? I imagine for such a use case it wouldn't be that much heavier to use Python vs Lua.
  - qwertox 2y ago
    Speed, probably. Lua is much faster.
    - nattaylor 2y ago
      Yes, speed. 1,000,000+ QPS and 1,000s of ads to evaluate is common. At that scale it's very distributed, so replication is another challenge. In a way, its a search problem where bid≈relevance and targeting_match≈recall, so I've seen Solr used here too.
  - catlifeonmars 2y ago
    Integration complexity could be a factor. I’m making an assumption about the architecture, but embedding Lua in a program is dead simple and you can do so without introducing external dependencies. Python IIRC requires you to ship and package the standard library or have it already installed, the Lua interpreter can be statically embedded in a program.
- ing33k 2y ago
  I tried a similar approach with a team I was working with. We were building it on top of redis and after some basic benchmarks gave up the idea and figured out from the docs that that eval script is blocking.
  - btilly 2y ago
    When I did that on a project I solved that problem by replicating redis to the same container that the ads were being served from. Replication was very fast, and all that was blocked was the local redis. I worked to make the matching rules in the script efficient, and the blockage was truly not a problem.
    - catlifeonmars 2y ago
      Doesn’t local redis kind of miss the point of using redis in the first place? On the surface, that’s a big chunk of additional complexity for something that could be done internal to an application. Was this meant to be an incremental step in a larger refactor?
      - ing33k 2y ago
        well, redis manages the replication without fuss. Unless the framework / language one uses supports it out of the box, there is no point in re implementing replication. https://www.erlang.org/docs/17/reference_manual/distributed https://www.erlang.org/docs/17/reference_manual/distributed
        catlifeonmars 2y ago
        Oh so this would effectively serve the same purpose as a centralized cluster.
        ing33k 2y ago
        Yeah with the added advantage of reduced latency. The system I worked on had a CRUD application which was in Django , ultimately the rules was stored in a Redis.. The ad serving application was in golang that connected to this local redis instances that were replicas of the central redis.. https://redis.io/docs/latest/commands/replicaof/ https://redis.io/docs/latest/commands/replicaof/
rhelz 2y ago
Man, I tried at Bloomberg to get my managers to let me incorporate lua into Tradebook's trading system. They just simply couldn't get it. I'd try to explain and they would look at me like I was from mars or something. Eventually they got tired of me pitching for it and fired me. How did you ever get managerial buy-in for something like this?
- chipdart 2y ago
  > Eventually they got tired of me pitching for it and fired me. I'm sorry but your anectode is simply not believable. No one gets fired for suggesting a harmless feature.
  - cess11 2y ago
    If you repeatedly contact management with the same proposal you risk coming across as dissatisfied in the organisation and sometimes there's very little tolerance for this, might be because it's just a tyrannical place, but could also reflect the type of business and customers.
    - theplatman 2y ago
      If you’re the type of person who won’t just move on from an idea when you’ve been repeatedly told no that indicates you’re probably a difficult person to work with in other ways. That kind of behavior is less welcome in enterprise environments where they want you to get in line with the top level goals. From my experience working at Bloomberg, there was a limit to how much you could rock the boat as a dev and in many ways the technology/language choices were limited to the approved stack.
      - jnordwick 2y ago
        bloomberg has some great teams though. i had lunch with the bond pool team about 5 years ago, and oh my gd they were good. plus the snack floor is awesome in the main office.
theturtletalks 2y ago
Would love to check it out if you’re accepting new users.
jasfi 2y ago
Which language did you write the engine in?
dist1ll 2y ago
What latency range are your strategies aimed at? Are you executing strategies on CPU, or lowering the DSL to some kind of HDL or ASIC config?
edem 2y ago
where can i try this system? I've been working on my own but if there is already one out there i might use that instead

nattaylor 2y ago

A friend built a system where the ad campaigns with all their targeting rules were Lua scripts and it was also fast and simple in a glorious way!

knighthack 2y ago
Why wasn't Python used? I imagine for such a use case it wouldn't be that much heavier to use Python vs Lua.
- qwertox 2y ago
  Speed, probably. Lua is much faster.
  - nattaylor 2y ago
    Yes, speed. 1,000,000+ QPS and 1,000s of ads to evaluate is common. At that scale it's very distributed, so replication is another challenge. In a way, its a search problem where bid≈relevance and targeting_match≈recall, so I've seen Solr used here too.
- catlifeonmars 2y ago
  Integration complexity could be a factor. I’m making an assumption about the architecture, but embedding Lua in a program is dead simple and you can do so without introducing external dependencies. Python IIRC requires you to ship and package the standard library or have it already installed, the Lua interpreter can be statically embedded in a program.
ing33k 2y ago
I tried a similar approach with a team I was working with. We were building it on top of redis and after some basic benchmarks gave up the idea and figured out from the docs that that eval script is blocking.
- btilly 2y ago
  When I did that on a project I solved that problem by replicating redis to the same container that the ads were being served from. Replication was very fast, and all that was blocked was the local redis. I worked to make the matching rules in the script efficient, and the blockage was truly not a problem.
  - catlifeonmars 2y ago
    Doesn’t local redis kind of miss the point of using redis in the first place? On the surface, that’s a big chunk of additional complexity for something that could be done internal to an application. Was this meant to be an incremental step in a larger refactor?
    - ing33k 2y ago
      well, redis manages the replication without fuss. Unless the framework / language one uses supports it out of the box, there is no point in re implementing replication. https://www.erlang.org/docs/17/reference_manual/distributed https://www.erlang.org/docs/17/reference_manual/distributed
      - catlifeonmars 2y ago
        Oh so this would effectively serve the same purpose as a centralized cluster.
        ing33k 2y ago
        Yeah with the added advantage of reduced latency. The system I worked on had a CRUD application which was in Django , ultimately the rules was stored in a Redis.. The ad serving application was in golang that connected to this local redis instances that were replicas of the central redis.. https://redis.io/docs/latest/commands/replicaof/ https://redis.io/docs/latest/commands/replicaof/

rhelz 2y ago

Man, I tried at Bloomberg to get my managers to let me incorporate lua into Tradebook's trading system. They just simply couldn't get it. I'd try to explain and they would look at me like I was from mars or something. Eventually they got tired of me pitching for it and fired me. How did you ever get managerial buy-in for something like this?

chipdart 2y ago
> Eventually they got tired of me pitching for it and fired me. I'm sorry but your anectode is simply not believable. No one gets fired for suggesting a harmless feature.
- cess11 2y ago
  If you repeatedly contact management with the same proposal you risk coming across as dissatisfied in the organisation and sometimes there's very little tolerance for this, might be because it's just a tyrannical place, but could also reflect the type of business and customers.
  - theplatman 2y ago
    If you’re the type of person who won’t just move on from an idea when you’ve been repeatedly told no that indicates you’re probably a difficult person to work with in other ways. That kind of behavior is less welcome in enterprise environments where they want you to get in line with the top level goals. From my experience working at Bloomberg, there was a limit to how much you could rock the boat as a dev and in many ways the technology/language choices were limited to the approved stack.
    - jnordwick 2y ago
      bloomberg has some great teams though. i had lunch with the bond pool team about 5 years ago, and oh my gd they were good. plus the snack floor is awesome in the main office.

theturtletalks 2y ago

Would love to check it out if you’re accepting new users.

jasfi 2y ago

Which language did you write the engine in?

dist1ll 2y ago

What latency range are your strategies aimed at? Are you executing strategies on CPU, or lowering the DSL to some kind of HDL or ASIC config?

edem 2y ago

where can i try this system? I've been working on my own but if there is already one out there i might use that instead