3 ms·

I spent a total of 11 months consulting with a company that built a large (50kloc) financial system in erlang. They had terrible performance problems that were

by jamii 12y ago

I spent a total of 11 months consulting with a company that built a large (50kloc) financial system in erlang. They had terrible performance problems that were caused entirely by erlang. Imagine you have a large amount of data (order books, accounts, etc). You could put it all in on erlang process, but the gc does not cope well with large heaps (multi-second pauses). You could store the data outside the heap (eg ets) but then you pay the cost of copying on every access and have to tradeoff ease of use (more data per key) vs performance (less data per key). You could split the data up into many processes and then all your simple calculations become asynchronous protocols. Have fun debugging the math or rolling back changes on errors. I went into that contract with a fondness for erlang. Now I wouldn't touch it ever again. A naive single-threaded blocking server achieved 10x less code, 40x better throughput and 100x better latency. I used clojure, but any sane platform would have worked just as well with that design.

Toine 12y ago
50kloc is considered "large financial system" ? Just curious.
- sz4kerto 12y ago
  It depends. I worked on a trading platform that >1M LOC. The overall complexity was lower than another quantitative modelling engine that only had couple of tens of thousands LOC. Complexity, work and LOC are correlated, but the correlation is not 1. In the case of the former system, 95% of the code was connectivity, GUI, persistence, etc., the actual business logic was maybe 5%. And I'm not even talking about stuff written in kdb/q, where I produced maybe 500 LOC in a month. :)
polskibus 12y ago
I'm curious whether you brought this problem up to the attention of erlang maintainers, what where their reactions and whether it will be addressed in the future. GCing large objects on the heap was (perhaps still is despite improvements) a significant problem also on .NET platform. Not sure whether this is the case, but have you considered patching the GC yourself or perhaps moving the problem to BIFs and managing that memory manually (without copying to ETS) ?
- jamii 12y ago
  The problem was not a few large objects, but large numbers of small objects eg thousands of orders per market. I considered using manual memory management but it would have required rewriting most of the code (it was almost all business logic). I sketched out a prototype in clojure to see whether a better GC would help and found that even with hastily written code and much abuse of persistent data structures, the G1 collector could keep the pauses under 100ms. I expect that carefully tuned code would reduce those much further.

by jamii 12y ago

Toine 12y ago

50kloc is considered "large financial system" ? Just curious.

sz4kerto 12y ago
It depends. I worked on a trading platform that >1M LOC. The overall complexity was lower than another quantitative modelling engine that only had couple of tens of thousands LOC. Complexity, work and LOC are correlated, but the correlation is not 1. In the case of the former system, 95% of the code was connectivity, GUI, persistence, etc., the actual business logic was maybe 5%. And I'm not even talking about stuff written in kdb/q, where I produced maybe 500 LOC in a month. :)

polskibus 12y ago

I'm curious whether you brought this problem up to the attention of erlang maintainers, what where their reactions and whether it will be addressed in the future. GCing large objects on the heap was (perhaps still is despite improvements) a significant problem also on .NET platform. Not sure whether this is the case, but have you considered patching the GC yourself or perhaps moving the problem to BIFs and managing that memory manually (without copying to ETS) ?

jamii 12y ago
The problem was not a few large objects, but large numbers of small objects eg thousands of orders per market. I considered using manual memory management but it would have required rewriting most of the code (it was almost all business logic). I sketched out a prototype in clojure to see whether a better GC would help and found that even with hastily written code and much abuse of persistent data structures, the G1 collector could keep the pauses under 100ms. I expect that carefully tuned code would reduce those much further.