7 ms·

Languages for online betting: an investigation of Go, Erlang, and Elixir

79 points by iamd3vil 9y ago

tybit 9y ago
Would be curious to see Pony added to the mix. It has the best of both worlds for my tastes, from Go: statically typed and native code, and Erlang: actor model, GCd at the actor level. Plus more features that neither has like capabilities.
- digitalzombie 9y ago
  Pony isn't even version 1 yet iirc and instable API. I mean would you like to bet on that stack on top of your betting platform? >___> Pony promise great things and the language have a proof and all that but it still needs to be battle tested and get to some sort of stability. https://www.ponylang.org/discover/#what-is-pony https://www.ponylang.org/discover/#what-is-pony (under why not pony section, API unstable) The degree of tech hype and adoption of bleeding edge is crazy. The thought of programmers just betting on these thing without thoughts on the schmuck who would inherit these technical debt still boggle my mind.
  - holydude 9y ago
    That just means most of the software is not that critical or people are more careless i do not know.
mbertschler 9y ago
I am really interested in some numbers to show how much the Go's GC harms program performance. I wonder if this is really an issue for an online service, given the stop-the-world phases are usually <1ms. Does anyone have some material on that?
- fcbrooklyn 9y ago
  Sasa Juric, and the folks at pusher wrote a couple of blog posts on this: http://theerlangelist.com/article/reducing_maximum_latency http://theerlangelist.com/article/reducing_maximum_latency
yupyup 9y ago
"Erlang Solutions" says Erlang is the bee's knees... I really like when someone that doesn't have a horse in the race reviews some language (better if it has been used in anger). You gotta value the perspective of a pragmatic engineer that has misapplied a tool. There's a lot to learn about where a language is best used, but when someone pay is somewhat tied to a language I don't trust the analysis as much.
- batrat 9y ago
  The 1st thing I thought when I so the article. A "Solutions" company advertising their product? Use the tools that get your job done and stop bragging about it.
misja111 9y ago
The article briefly mentions the similarity of online betting to stock markets, when it is about the requirements for stability and real time processing without long GC delays. This makes me wonder why Erlang/ Elixir are used so little in the financial industry? I have been working for a couple of financial firms and even a HFT company but I have never seen Erlang being used, despite its apparent advantages. Does anybody know why this is? Or is Erlang actually being used a lot in finance and have I just been working in the wrong places?
- sargun 9y ago
  Erlang is pretty poor when it comes to P50 latency compared to C++ and Java. Part of this is due to BEAM being a pretty unsophisticated VM, and part of this is due to the language and VM promoting inefficient patterns, (FP, immutability—these aren’t “bad”, just bad for our intel CPUs). Hiring for Erlang can be difficult too. That being said, Erlang is used heavily in FSI outside of HFT, or algorithmic trading. Several large FSI institutions are known users of Erlang (Visa, Banks).
  - tankenmate 9y ago
    One thing I have noticed that is mentioned less about immutable systems is the trade off of GC storm pause vs copy storm pause. Obviously system that use immutability either exclusively or primarily tend to use lazy evaluation to avoid copy storms sometimes it can't be helped. Mutability does have it upsides as well as down sides; the trade off isn't always obvious though.
  - deleted 9y ago
    [deleted]
- user5994461 9y ago
  A betting exchange is a stock exchange. Same thing. Let's say that there is a football game Liverpool vs Manchester. One client puts £5 that liverpool will win. The exchange takes the money and adds it to the order book. Another client puts £5 that manchester will win. The exchange takes the money. It's matched with the opposite order and frozen. The money will be redistributed to the winner at the end of the game, minus a fee. It's exactly like buying and selling in a normal exchange. You can do real time trading during the game and arbitrage between exchanges. Disclaimer: I worked in betting before I moved to finance. Erlang is super weird and hard to recruit for. When used right, it's not faster or more productive than C++ or Java. I wouldn't advise anyone to use Erlang for anything.
ilitirit 9y ago
As someone who works in this industry, the biggest issues are (besides like stuff like match-fixing and illegal internal trading/leaks): 1. Highly domain-specific legacy pricing model running on... Excel spreadsheets. These present huge performance bottle-necks and aren't easy to migrate. 2. Reliance on 3rd Party data. Did a scout just send incorrect data? Did their device disconnect? Did they rollback the wrong information? Is the source lagging behind the TV/online feed? GGs. Suspend the game and refund bets. The tooling really isn't that important. Scalable software can be written in any software, and while some languages do make things simpler, often you can just also find a suitable paradigm (or middleware) that can provide similar advantages. Having said that, we do make extensive use of RabbitMQ (which is written in Erlang) in our environment. I suppose this is also just an example of my previous point though...
- user5994461 9y ago
  1. I don't know what you are talking about. The major betting exchanges all support real time trading, they have market makers and are arbitraged against one another. The only issue is with low popularity events. Noone cares about them, there is low liquidity and volume, it's hard to price them.
  - twic 9y ago
    I think ilitirit is talking about bookmakers, not exchanges.
    - ilitirit 9y ago
      There are companies that do a combination of things: Real-time trading (manual, semi, auto), pricing (model-driven, or sourced), risk-management, etc. They can either provide their own prices using models, or apply prices from partners. The business is usually separated into discrete units which feed into each other, and outward to partners/customers. Yes, exchanges support real-time trading, but not every company has all their models running in modern paradigms. It's not uncommon to find companies that run multiple VMs for the sole purpose of... running Excel calculations. Yes, this is exactly what it sounds like. A server that has a several, multi-meg Excel files in memory with external APIs that plug values into cells and read outputs. As I said, within the grand scheme of things, for sports who still run on legacy models, this is a major bottleneck. Remember, many of these companies were started in the 90s, and that's when these models were created. The people who created them are usually swallowed up by the financial industry and now come with a hefty price tag. So the question for the business is "Do we spend millions recreating these proven models and run the risk of them failing? Or do we simply just add another VM?" At my company things have slowly started changing, but the new models (written in F#) still need a lot of work.
jstimpfle 9y ago
I'm currently writing a GUI for industrial automation, where we have to talk to a number of machine control threads and image processing threads (all through a procedural interface that connects to other processes). I'm writing in C and decided to stay in a single OS thread and to instead write my own simplistic user thread scheduler. User thread ("green threads") work by simply switching call stacks and saving registers. In my case this is as simple as making a few Windows API calls like CreateFiber() or SwitchToFiber(). It's one of the best things I've ever done. Maybe 1 day of work to get the threads rolling, no bugs, and still in a familiar C environment. No bindings required to interface with the libraries that I need to interface with (like Qt) and perfect deterministic control over what happens (just not the other processes of course). I think I can understand people are not using C doing web stuff (lots of strings) but for what I do I'm convinced it's the easiest, fastest to develop, most performant, most robust thing you can do.
- lucidguppy 9y ago
  How can you be convinced if you're only running the control in an experiment?
  - jstimpfle 9y ago
    It's not an experiment.
    - todd8 9y ago
      I don't believe that lucidguppy was saying that your system was an experiment. Instead, it would be interesting to try implementing your system in a second language so you could compare and contrast the differences. (Of course, in the real-world one often doesn't have the luxury of doing this.)
      - jstimpfle 9y ago
        Ok, so my conviction comes from the fact that the code is super straight-forward and I don't have to deal with the complexities of a managed language's run-time system (like tuning the garbage collector or scheduler, using synchronisation primitives, depending on the ecosystem/using terrible bindings, etc). All investment is just a page of code using the Windows Fiber API, and the scheduler ist just 10 lines of code doing round-robin scheduling. Which is totally fine right now, and there's the added benefit of being in total control and being able to make adjustments if they are needed in the future. Since the scheduling is non-preemptive, threads are not allowed to block. But only a synchronous, blocking coding style is maintainable, so threads use wrapper functions to execute slow operations. The wrapper functions have a synchronous interface, but internally they call asynchronous functions, and unless the request can be served immediately, they call the scheduler. The wrapper functions make another 50 lines (so far) of really simple bullet-proof maintainable code. It's really like Javascript technically (non-preemptive scheduling), but you don't have to do callback chaining. You just write your code in an ordinary synchronous blocking style. Think about that. You write thread-heavy interactive C code that is much more maintainable than Javascript code. And of course, much more maintainable than clumsy OS threads - no management, no synchronisation primitives and no excessive copying between threads. On top of that, the scheduling is deterministic. All with 100 lines of straightforward infrastructure code. The only boilerplate in the project comes from using Qt/Qml. (If you're on POSIX and can't use the Windows Fiber API, you can depend on the deprecated makecontext/getcontext/setcontext API, or write a little inline assembly. It's not a big deal either way, since it's really just a few lines of code that are neatly separated from the rest of the system).
eternalban 9y ago
That was poorly written. And amazingly, the author neglected to shed light as to why Erlang has a more a 'complete' story in regards to distributed f/t systems ("reliability", "concurrency"). Erlang has OTP, Go does not. Erlang is FP, Go is not. These are the critical differences. Silver lining (in context of OTP) for Go is that the du jour 'cloud' based architectures will provide the OTP like capabilities ("elastic", "fail-over", etc.) for deployed (micro-)services. It seems that in the (future) long-term, Go is better positioned than Erlang for cloud based systems.
Thaxll 9y ago
What a garbage article, Go is faster that Erlang and not by a small margin, same for the GC. Erlang has definitely its place in some scenarios but the article reflects it poorly by trying too hard to dismiss Go.
- iamd3vil 9y ago
  Go maybe faster than Erlang in raw throughput but Erlang can compete with Go in Latency (mainly predictable latency). As you said it all depends on the scenario. The article is mainly in the scenario of betting infrastructure where I think erlang is better due to OTP (saying this as a person who works on building trading systems and daily user of both).
  - Thaxll 9y ago
    One example: "When you pit Go’s requests per seconds against the likes of Ruby on Rails or Django, Go returns some impressive benchmarks performing 3x better. Go can scale to hundreds of thousands with relative ease, in much the same way that Erlang and Elixir scale to millions." He's talking about raw performance where Go is much faster than Erlang and still talking about different order of magnitude to diminish Go.
gaius 9y ago
I spent a few years working at the UK's largest betting exchange, I could tell you what language we used for bet matching and so on but you wouldn't believe me...
- ac2u 9y ago
  No experience in that area, but are we talking something like COBOL? Wouldn't be surprised if software architectures used to manage order books in exchanges from the early days got re-purposed.
- PaulRobinson 9y ago
  I remember when Betfair seemed to be a bunch of shonky crap stitched together. I can easily imagine a load of Perl scripts before they invested in Oracle in a big way.
  - gaius 9y ago
    before they invested in Oracle in a big way Yep, got it: it was all done in PL/SQL. Clicks on the website went through a simple Java layer and called stored procs. At the time we were doing more transactions/sec than all the European stock exchanges combined, and often the Japanese too. It makes me chuckle when people say to me now "but SQL databases don't scale" when they are running 1/10th of the workload we were running 10 yeara ago... No idea what they're using now, mind, I left after the IPO but before the merger.
- posterboy 9y ago
  Another comment mentioned excel, and I have no hard time believing it because the reactive interface is intuitive and the algorithms involved are likely not too technical, for lack of a better word.
- fceccon 9y ago
  Did you use TCL?
  - gaius 9y ago
PaulRobinson 9y ago
What a terrible article. It's spam designed to get hits for bookmaker CTOs scouting around. No benchmarks, no numbers, no real detail, just empty praise for Erlang from a company called "Erlang Solutions".

79 points by iamd3vil 9y ago

Would be curious to see Pony added to the mix. It has the best of both worlds for my tastes, from Go: statically typed and native code, and Erlang: actor model, GCd at the actor level. Plus more features that neither has like capabilities.

digitalzombie 9y ago
Pony isn't even version 1 yet iirc and instable API. I mean would you like to bet on that stack on top of your betting platform? >___> Pony promise great things and the language have a proof and all that but it still needs to be battle tested and get to some sort of stability. https://www.ponylang.org/discover/#what-is-pony https://www.ponylang.org/discover/#what-is-pony (under why not pony section, API unstable) The degree of tech hype and adoption of bleeding edge is crazy. The thought of programmers just betting on these thing without thoughts on the schmuck who would inherit these technical debt still boggle my mind.
- holydude 9y ago
  That just means most of the software is not that critical or people are more careless i do not know.

mbertschler 9y ago

I am really interested in some numbers to show how much the Go's GC harms program performance. I wonder if this is really an issue for an online service, given the stop-the-world phases are usually <1ms. Does anyone have some material on that?

fcbrooklyn 9y ago
Sasa Juric, and the folks at pusher wrote a couple of blog posts on this: http://theerlangelist.com/article/reducing_maximum_latency http://theerlangelist.com/article/reducing_maximum_latency

yupyup 9y ago

"Erlang Solutions" says Erlang is the bee's knees... I really like when someone that doesn't have a horse in the race reviews some language (better if it has been used in anger). You gotta value the perspective of a pragmatic engineer that has misapplied a tool. There's a lot to learn about where a language is best used, but when someone pay is somewhat tied to a language I don't trust the analysis as much.

batrat 9y ago
The 1st thing I thought when I so the article. A "Solutions" company advertising their product? Use the tools that get your job done and stop bragging about it.

misja111 9y ago

The article briefly mentions the similarity of online betting to stock markets, when it is about the requirements for stability and real time processing without long GC delays. This makes me wonder why Erlang/ Elixir are used so little in the financial industry? I have been working for a couple of financial firms and even a HFT company but I have never seen Erlang being used, despite its apparent advantages. Does anybody know why this is? Or is Erlang actually being used a lot in finance and have I just been working in the wrong places?

sargun 9y ago
Erlang is pretty poor when it comes to P50 latency compared to C++ and Java. Part of this is due to BEAM being a pretty unsophisticated VM, and part of this is due to the language and VM promoting inefficient patterns, (FP, immutability—these aren’t “bad”, just bad for our intel CPUs). Hiring for Erlang can be difficult too. That being said, Erlang is used heavily in FSI outside of HFT, or algorithmic trading. Several large FSI institutions are known users of Erlang (Visa, Banks).
- tankenmate 9y ago
  One thing I have noticed that is mentioned less about immutable systems is the trade off of GC storm pause vs copy storm pause. Obviously system that use immutability either exclusively or primarily tend to use lazy evaluation to avoid copy storms sometimes it can't be helped. Mutability does have it upsides as well as down sides; the trade off isn't always obvious though.
- deleted 9y ago
  [deleted]
user5994461 9y ago
A betting exchange is a stock exchange. Same thing. Let's say that there is a football game Liverpool vs Manchester. One client puts £5 that liverpool will win. The exchange takes the money and adds it to the order book. Another client puts £5 that manchester will win. The exchange takes the money. It's matched with the opposite order and frozen. The money will be redistributed to the winner at the end of the game, minus a fee. It's exactly like buying and selling in a normal exchange. You can do real time trading during the game and arbitrage between exchanges. Disclaimer: I worked in betting before I moved to finance. Erlang is super weird and hard to recruit for. When used right, it's not faster or more productive than C++ or Java. I wouldn't advise anyone to use Erlang for anything.

ilitirit 9y ago

As someone who works in this industry, the biggest issues are (besides like stuff like match-fixing and illegal internal trading/leaks): 1. Highly domain-specific legacy pricing model running on... Excel spreadsheets. These present huge performance bottle-necks and aren't easy to migrate. 2. Reliance on 3rd Party data. Did a scout just send incorrect data? Did their device disconnect? Did they rollback the wrong information? Is the source lagging behind the TV/online feed? GGs. Suspend the game and refund bets. The tooling really isn't that important. Scalable software can be written in any software, and while some languages do make things simpler, often you can just also find a suitable paradigm (or middleware) that can provide similar advantages. Having said that, we do make extensive use of RabbitMQ (which is written in Erlang) in our environment. I suppose this is also just an example of my previous point though...

user5994461 9y ago
1. I don't know what you are talking about. The major betting exchanges all support real time trading, they have market makers and are arbitraged against one another. The only issue is with low popularity events. Noone cares about them, there is low liquidity and volume, it's hard to price them.
- twic 9y ago
  I think ilitirit is talking about bookmakers, not exchanges.
  - ilitirit 9y ago
    There are companies that do a combination of things: Real-time trading (manual, semi, auto), pricing (model-driven, or sourced), risk-management, etc. They can either provide their own prices using models, or apply prices from partners. The business is usually separated into discrete units which feed into each other, and outward to partners/customers. Yes, exchanges support real-time trading, but not every company has all their models running in modern paradigms. It's not uncommon to find companies that run multiple VMs for the sole purpose of... running Excel calculations. Yes, this is exactly what it sounds like. A server that has a several, multi-meg Excel files in memory with external APIs that plug values into cells and read outputs. As I said, within the grand scheme of things, for sports who still run on legacy models, this is a major bottleneck. Remember, many of these companies were started in the 90s, and that's when these models were created. The people who created them are usually swallowed up by the financial industry and now come with a hefty price tag. So the question for the business is "Do we spend millions recreating these proven models and run the risk of them failing? Or do we simply just add another VM?" At my company things have slowly started changing, but the new models (written in F#) still need a lot of work.

jstimpfle 9y ago

I'm currently writing a GUI for industrial automation, where we have to talk to a number of machine control threads and image processing threads (all through a procedural interface that connects to other processes). I'm writing in C and decided to stay in a single OS thread and to instead write my own simplistic user thread scheduler. User thread ("green threads") work by simply switching call stacks and saving registers. In my case this is as simple as making a few Windows API calls like CreateFiber() or SwitchToFiber(). It's one of the best things I've ever done. Maybe 1 day of work to get the threads rolling, no bugs, and still in a familiar C environment. No bindings required to interface with the libraries that I need to interface with (like Qt) and perfect deterministic control over what happens (just not the other processes of course). I think I can understand people are not using C doing web stuff (lots of strings) but for what I do I'm convinced it's the easiest, fastest to develop, most performant, most robust thing you can do.

lucidguppy 9y ago
How can you be convinced if you're only running the control in an experiment?
- jstimpfle 9y ago
  It's not an experiment.
  - todd8 9y ago
    I don't believe that lucidguppy was saying that your system was an experiment. Instead, it would be interesting to try implementing your system in a second language so you could compare and contrast the differences. (Of course, in the real-world one often doesn't have the luxury of doing this.)
    - jstimpfle 9y ago
      Ok, so my conviction comes from the fact that the code is super straight-forward and I don't have to deal with the complexities of a managed language's run-time system (like tuning the garbage collector or scheduler, using synchronisation primitives, depending on the ecosystem/using terrible bindings, etc). All investment is just a page of code using the Windows Fiber API, and the scheduler ist just 10 lines of code doing round-robin scheduling. Which is totally fine right now, and there's the added benefit of being in total control and being able to make adjustments if they are needed in the future. Since the scheduling is non-preemptive, threads are not allowed to block. But only a synchronous, blocking coding style is maintainable, so threads use wrapper functions to execute slow operations. The wrapper functions have a synchronous interface, but internally they call asynchronous functions, and unless the request can be served immediately, they call the scheduler. The wrapper functions make another 50 lines (so far) of really simple bullet-proof maintainable code. It's really like Javascript technically (non-preemptive scheduling), but you don't have to do callback chaining. You just write your code in an ordinary synchronous blocking style. Think about that. You write thread-heavy interactive C code that is much more maintainable than Javascript code. And of course, much more maintainable than clumsy OS threads - no management, no synchronisation primitives and no excessive copying between threads. On top of that, the scheduling is deterministic. All with 100 lines of straightforward infrastructure code. The only boilerplate in the project comes from using Qt/Qml. (If you're on POSIX and can't use the Windows Fiber API, you can depend on the deprecated makecontext/getcontext/setcontext API, or write a little inline assembly. It's not a big deal either way, since it's really just a few lines of code that are neatly separated from the rest of the system).

eternalban 9y ago

That was poorly written. And amazingly, the author neglected to shed light as to why Erlang has a more a 'complete' story in regards to distributed f/t systems ("reliability", "concurrency"). Erlang has OTP, Go does not. Erlang is FP, Go is not. These are the critical differences. Silver lining (in context of OTP) for Go is that the du jour 'cloud' based architectures will provide the OTP like capabilities ("elastic", "fail-over", etc.) for deployed (micro-)services. It seems that in the (future) long-term, Go is better positioned than Erlang for cloud based systems.

Thaxll 9y ago

What a garbage article, Go is faster that Erlang and not by a small margin, same for the GC. Erlang has definitely its place in some scenarios but the article reflects it poorly by trying too hard to dismiss Go.

iamd3vil 9y ago
Go maybe faster than Erlang in raw throughput but Erlang can compete with Go in Latency (mainly predictable latency). As you said it all depends on the scenario. The article is mainly in the scenario of betting infrastructure where I think erlang is better due to OTP (saying this as a person who works on building trading systems and daily user of both).
- Thaxll 9y ago
  One example: "When you pit Go’s requests per seconds against the likes of Ruby on Rails or Django, Go returns some impressive benchmarks performing 3x better. Go can scale to hundreds of thousands with relative ease, in much the same way that Erlang and Elixir scale to millions." He's talking about raw performance where Go is much faster than Erlang and still talking about different order of magnitude to diminish Go.

gaius 9y ago

I spent a few years working at the UK's largest betting exchange, I could tell you what language we used for bet matching and so on but you wouldn't believe me...

ac2u 9y ago
No experience in that area, but are we talking something like COBOL? Wouldn't be surprised if software architectures used to manage order books in exchanges from the early days got re-purposed.
PaulRobinson 9y ago
I remember when Betfair seemed to be a bunch of shonky crap stitched together. I can easily imagine a load of Perl scripts before they invested in Oracle in a big way.
- gaius 9y ago
  before they invested in Oracle in a big way Yep, got it: it was all done in PL/SQL. Clicks on the website went through a simple Java layer and called stored procs. At the time we were doing more transactions/sec than all the European stock exchanges combined, and often the Japanese too. It makes me chuckle when people say to me now "but SQL databases don't scale" when they are running 1/10th of the workload we were running 10 yeara ago... No idea what they're using now, mind, I left after the IPO but before the merger.
posterboy 9y ago
Another comment mentioned excel, and I have no hard time believing it because the reactive interface is intuitive and the algorithms involved are likely not too technical, for lack of a better word.
fceccon 9y ago
Did you use TCL?
- gaius 9y ago

PaulRobinson 9y ago

What a terrible article. It's spam designed to get hits for bookmaker CTOs scouting around. No benchmarks, no numbers, no real detail, just empty praise for Erlang from a company called "Erlang Solutions".