5 ms·

My company replaced the integration server I had originally written in Python/Django/Celery/RabbitMQ with one built in Elixir/OTP. The results were impressive.

by teilo 9y ago

My company replaced the integration server I had originally written in Python/Django/Celery/RabbitMQ with one built in Elixir/OTP. The results were impressive. Our task load was getting out of hand, and during peak load throwing memory and CPU at it was not helping. Task monitoring was also a pain. It needed a complete rewrite, because the original design, while robust for what it targeted at the time, could not keep up with our growth. The integration developer I hired to take over the project looked into using Go at first, but then discovered Elixir. I started learning it with him (he quickly surpassed me), and after a few trial task implementations, I gave him the go-ahead. We couldn't be happier with the result. Thanks to OTP and the existing tools built around it, we have great monitoring, and the ability to easily and quickly scale up workers at peak times.

sillysaurus3 9y ago
Are you sure this isn't more of a case of "a motivated developer writes great software" than "a silver bullet language"?
- simplify 9y ago
  Did he say "silver bullet language"? It seemed like a good fit for his use case; I don't see him claiming anything more than that.
  - teilo 9y ago
    Correct. Not saying anything more than that. The BeamVM/OTP is a better fit for long-running high-concurrency fault-tollerant systems. That's what it was designed for. I have not switched to Elixir/Phoenix for my other web development. I still use Django for that.
Xeoncross 9y ago
Anything specific you liked about Elixir more than Go?
- tsukaisute 9y ago
  As someone who writes both, it's even difficult to compare these as they are apples and oranges. Elixir (due to Erlang base) is great for long-running or distributed services and includes tooling to, for example, easily introspect the state/memory usage of your services down to a single mutable data structure, to heal from failures, and to hot-swap upgrade/code while the server is still running, and to perform node-to-node RPC. Go would be a replacement for something that was previously written in C/C++, that may need concurrency, and that should compile to native code. You get very fine-grained control of everything from network to file operations and excellent concurrency constructs. A web crawler, a device driver, or writing an Nginx/HAProxy-type daemon, all may be great example use cases. One could write their next social network/app backend using Go (just as one could do it in C), but it would probably take more work than just using a higher-level language.
  - oregontechninja 9y ago
    Go is much more suited for web apps than any driver or real time performance application; although, I will agree that using Go for command-line tools is great. I haven't used Elixir but as somebody who uses Go, I highly recommend not replacing C/C++ with it unless it isn't performance critical. PS: Go's memory usage is pretty up there some times, another reason you should probably stick with C/C++ in the context mentioned above.
- teilo 9y ago
  I'm the wrong person to answer that, perhaps, but with OTP you get a task broker / monitor for free. With Go we would be back to piecing together a system using other tools such as AMPQ or similar. It also isn't as easy to write it for high-concurrency monitored tasks. But I have little experience with Go, so correct me if I am wrong.
relyio 9y ago
It sounds like you hired a great engineer to be honest. Elixir, and the OTP environment is well designed. But it shouldn't downplay the achievement of the guy who re-implemented it (or yours, for letting him do so and provide him with the adequate mentorship). Props to you both, it's an impressive feat.
- teilo 9y ago
  Despite being a $120M company, we still have and foster an entrepreneurial spirit. I love working here.
  - relyio 9y ago
    Are you hiring?
anko 9y ago
I had a simple rabbitmq -> elasticsearch logger I wrote in scala, which had to handle peaks of about 8000 messages/s on a single t2.large instance (with elasticsearch on the same box). It took about a week and a half to write, and would have taken a bunch more time to fix to apply backpressure properly, so it suffered from getting OOMed. Even it's base memory usage was around 300MB. I rewrote it in elixir + genstage in about 8 hours, now it takes 60MB of ram for basically as many messages as I can throw at it, because 1) beam uses a lot less base memory 2) backpressure concerns are sorted out. In scala, I would have to use akka streams or similar, and it would take me weeks and still I don't think I could get it to fit in such a small footprint. I plan to add websockets to view a live stream, but I have a little trouble finding the right genstage patterns (as it's still new). If I were doing high frequency numerical calculations, maybe scala would be a better fit, but for most high speed messaging elixir is pretty amazing.

by teilo 9y ago

sillysaurus3 9y ago

Are you sure this isn't more of a case of "a motivated developer writes great software" than "a silver bullet language"?

simplify 9y ago
Did he say "silver bullet language"? It seemed like a good fit for his use case; I don't see him claiming anything more than that.
- teilo 9y ago
  Correct. Not saying anything more than that. The BeamVM/OTP is a better fit for long-running high-concurrency fault-tollerant systems. That's what it was designed for. I have not switched to Elixir/Phoenix for my other web development. I still use Django for that.

Xeoncross 9y ago

Anything specific you liked about Elixir more than Go?

tsukaisute 9y ago
As someone who writes both, it's even difficult to compare these as they are apples and oranges. Elixir (due to Erlang base) is great for long-running or distributed services and includes tooling to, for example, easily introspect the state/memory usage of your services down to a single mutable data structure, to heal from failures, and to hot-swap upgrade/code while the server is still running, and to perform node-to-node RPC. Go would be a replacement for something that was previously written in C/C++, that may need concurrency, and that should compile to native code. You get very fine-grained control of everything from network to file operations and excellent concurrency constructs. A web crawler, a device driver, or writing an Nginx/HAProxy-type daemon, all may be great example use cases. One could write their next social network/app backend using Go (just as one could do it in C), but it would probably take more work than just using a higher-level language.
- oregontechninja 9y ago
  Go is much more suited for web apps than any driver or real time performance application; although, I will agree that using Go for command-line tools is great. I haven't used Elixir but as somebody who uses Go, I highly recommend not replacing C/C++ with it unless it isn't performance critical. PS: Go's memory usage is pretty up there some times, another reason you should probably stick with C/C++ in the context mentioned above.
teilo 9y ago
I'm the wrong person to answer that, perhaps, but with OTP you get a task broker / monitor for free. With Go we would be back to piecing together a system using other tools such as AMPQ or similar. It also isn't as easy to write it for high-concurrency monitored tasks. But I have little experience with Go, so correct me if I am wrong.

relyio 9y ago

It sounds like you hired a great engineer to be honest. Elixir, and the OTP environment is well designed. But it shouldn't downplay the achievement of the guy who re-implemented it (or yours, for letting him do so and provide him with the adequate mentorship). Props to you both, it's an impressive feat.

teilo 9y ago
Despite being a $120M company, we still have and foster an entrepreneurial spirit. I love working here.
- relyio 9y ago
  Are you hiring?

anko 9y ago

I had a simple rabbitmq -> elasticsearch logger I wrote in scala, which had to handle peaks of about 8000 messages/s on a single t2.large instance (with elasticsearch on the same box). It took about a week and a half to write, and would have taken a bunch more time to fix to apply backpressure properly, so it suffered from getting OOMed. Even it's base memory usage was around 300MB. I rewrote it in elixir + genstage in about 8 hours, now it takes 60MB of ram for basically as many messages as I can throw at it, because 1) beam uses a lot less base memory 2) backpressure concerns are sorted out. In scala, I would have to use akka streams or similar, and it would take me weeks and still I don't think I could get it to fit in such a small footprint. I plan to add websockets to view a live stream, but I have a little trouble finding the right genstage patterns (as it's still new). If I were doing high frequency numerical calculations, maybe scala would be a better fit, but for most high speed messaging elixir is pretty amazing.