10 ms·

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get rig

As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. Erlang requires a lot of TLC to get right, it's super slow, and it's hard to burst. Like, super hard to burst. Erlang nodes are meant to cluster as a k graph and never go down. Modern ops, especially container ops, does availability through ephemerality of services. The BEAM just doesn't like to be treated like cattle. Also Erlang has notoriously bad error messages, very little abstraction, and converting between binary strings and lists is a pain. Gaining Erlang operational knowledge also takes a while. We eventually had to rewrite things like gen_server, ditch mnesia etc. as we scaled. So why Rust? Like Erlang, it's damn good at concurrency and enables functional programming. It also enables event driven programming through tokio, which is a better fit for web servers than green threads (you're mostly waiting on the network). Unlike Erlang, it's super fast (even at math), has a great type system, amazing error messages, low memory usage, and the community is already quite a bit bigger.

dev_dull 8y ago
The world of ephemeral containers really has changed the dynamic. Erlang solves a lot of problems with good solutions, but writing good software is only part of the battle. Lifecycle management, monitoring, etc make up a huge operational burden and should always be thought of as first-class problems in language design. Rust makes that better in many ways but not uniquely better than alternative languages.
- ilovecaching 8y ago
  In my experience any compiled language is easier to manage. There is no disconnect between your runtime and the system. You're investing in systems knowledge rather than knowledge of a particular VM's bytecode/abstractions as well. Erlang does provide good solutions to lifecycle management. The problem is that whenever someone picks a solution for you, you're now locked it and it can be difficult to migrate to a new paradigm like containers.
  - dev_dull 8y ago
    Totally agree regarding compiled languages. I’m surprised python et all don’t offer a way to check for file/script “correctness” without execution. It’s so bad that I will often add a “-h” flag to scripts I encounter just so I can run it and make sure no one introduce a silly bug (such as typo, import, etc).
    - ilovecaching 8y ago
      Yep, I've seen my fair share of Python stack traces. Completely sympathize, and it is one of the reasons I believe in statically types and compiled as well.
abledon 8y ago
What is TLC?
- ilovecaching 8y ago
  Tender loving care
kauboy 8y ago
What do you think of Elixir?
- ilovecaching 8y ago
  Elixir doesn't affect most of the operational aspects of Erlang, and it certainly in no way affects performance. You're still running on the BEAM, except now you actually have more apps running (The Elixir runtime runs as an app). Syntactically it cleans some things up, adds some niceties, and annoyingly makes atoms require a prefix of ":" (annoying because atoms are central to Erlang's readability). Personally, I don't find the changes to make a big difference when using the BEAM, and I prefer the familiarity of Erlang's syntax. It's also the language all of the documentation for the BEAM will use. If you absolutely can't get your coworkers to ditch dynamic/gradual typing, Elixir or Erlang are still great choices and way better than Python/Ruby. Otherwise, try Rust.
  - pera 8y ago
    Are you actually using Rust for professional web development? I've tried it earlier this year and my experience wasn't that great, so I'm quite surprised to read this comment.
    - zinclozenge 8y ago
      What did you end up using for Rust? The actix ecosystem is pretty good, and quite fast.
      - pera 8y ago
        I did like Rust (as a language), but I'm using Elixir, Phoenix and Ecto right now.
    - ilovecaching 8y ago
      What issues did you have? We haven't had very many issues. We rebuilt some internals tools, but for the most part it's been smooth sailing. We do use hyper and h2.
enraged_camel 8y ago
Your complaints about Erlang give me the impression that you may have picked the wrong language for your problem domain, and paid for that mistake by having to do things like rewriting gen_server. Erlang was designed to be reliable and scalable. It was not designed to be fast. If fast is a hard prerequisite for you, you're right, you should go with another language. But Erlang also does a ton of things correctly (again, in the domains it was designed and optimized for), and is battle-tested far beyond Rust, since it has been around for much, much longer. That's not to say Rust is a bad language, but one should probably not judge its merits by its current popularity. That holds true for any language or tool. Anyways, if you want abstraction (the lack of which was one of your complaints about Erlang), you should take a look at Elixir. Specifically, Phoenix is excellent for web programming, especially if you need real-time messaging at scale.
- ilovecaching 8y ago
  I'm aware of all of these claims. Any Erlang developer has heard this shtick a thousand times. The reality is that you do eventually have to twist the BEAM once you reach a certain level of scale, and there are plenty of bugs. Like I said, it's also difficult to burst, so it's not always as black and white as you say. It's also missing a lot of the introspection you need to operationalize at scale. We've had to patch the Beam many times to keep moving forward, and work with super dangerous nifs to talk to other systems. The issues I've had with Erlang had nothing to do with the products I've worked with. Erlang was a 'good fit' for them. Elixir and Phoenix doesn't really solve any of these issues, and neither of them are particularly 'battle tested'.
  - bthornbury 8y ago
    As an Erlang fan, who hasn't used it in production services, I'm wondering if you can let us know some specific scaling issues you encountered.
    - ilovecaching 8y ago
      I can't really go into it in detail, but at a high level Erlang default methods of distribution and security don't scale very well. There are people working on better mechanisms for this, and I know of several companies that have custom solutions for clustering nodes. One big issue is you cannot easily burst Erlang nodes to handle peak traffic. The number of nodes is usually relatively static in a deployment.
plainOldText 8y ago
Rust has too big of a learning curve, and it's quite complex for writing service oriented systems. Granted, having a static compiler eliminates many types of bugs found in other systems. In my opinion a more sensible approach is Elixir + Rust (via rustler). Your get the elegance and productivity of Elixir, the concurrency and fault tolerance of Erlang/OTP, while still being able to writing super fast, low level code in Rust. Plus, you shouldn't use a single language across your whole stack. When the only tool you have available is a hammer, everything looks like a nail.
- ilovecaching 8y ago
  I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -specs, because dialyzer is frustrating. I was able to learn Rust in about week by reading the O'reilly book. Compared to C++ it's almost tiny (I have many Bjarn's C++ tome and the Meyers books right next to my Rust books actually), and unlike C++, the compiler will essentially teach you the language if you didn't learn enough from the book to write code that compiles. Once you've put in the initial investment, Rust code is way easier to maintain, read, and scale. It's also easier to onboard people. Onboarding people for Erlang is hard, and it's hard to hire for. Rust on the other hand is familiar to all of our FP people who like Haskell and Erlang, and our Java and C++ devs. I've also found it extremely suitable for writing services. We have many services in Rust running in production right now. We have a service toolkit for our cross cutting concerns. We can run our IDL through a compiler and get a client and server that are on the order of 20x faster than their Erlang counterparts.
  - plainOldText 8y ago
    For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read. At the end of the day, use whatever language you prefer, just keep in mind that software needs to be (1) shipped, (2) maintained (by multiple people who read each other's code) and (3) evolved. I would also add (0) experimentation; before one even ships any code, one ought to easily experiment with various ideas.
andy_ppp 8y ago
You’re comparing the out of the box, network transparent scaling solution that is fully meshed and doesn’t claim to solve everyone’s problems forever with a solution in rust that probably involves a completely hand rolled implementation of lots and lots of different things OTP is doing. You have made me want to learn Rust though, how does it compare with Golang?
- ilovecaching 8y ago
  We move the orchestration from the Erlang runtime to our container orchestrator though. We have a better separation of concerns, and we can scale any language. The registry, process communication, are just services and protocols running in containers. So compared to Go, the Rust folks seemed to have paid close attention to the things people liked about Go. We have most of those things. We have great tooling, auto formatting, concurrency primitives (although we have freedom of choice. Green threads are a third party library, and aren't part of a runtime). We even have an animal mascot. From a language standpoint, Go isn't anything remarkable (our Haskell colleague calls it a step backwards). If you listen to the Gotime podcast, Brad Fitzpatrick, one of the Go authors even says as much. Go's big contribution was essentially web programming and go routines. Rust, on the other hand, has a lot. It has typeclasses from Haskell. It has lifetimes from the PL research community. It has immutability and pattern matching just like in Erlang. It has all the familiar control structure and enums/structs from C++. Performance of Go in terms of CPU and memory pressure is usually somewhere near Java, if not slightly better. That's pretty damn good. The JVM has improved a lot over the years. Rust performance is a hair shy of C/C++. That's amazing. All those abstractions, and the speed of C++?
Immortalin 8y ago
There's always Pony.
- qaq 8y ago
  Pony has a lot of promise but it needs a few key missing libs. If it had a quality http and postgres libs it prob would be perfect for web dev. but it does not have either yet.
deleted 8y ago
[deleted]
dxhdr 8y ago
> As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. I'm surprised to read this, Erlang and Rust have almost nothing in common. It's understandable that BEAM doesn't suit your needs but suggesting that everyone should write Rust instead is grossly misleading. They are suited to solving different problems!
- ilovecaching 8y ago
  They have a lot in common syntactically, semantically, and philosophically. They are both languages designed to promate safety and robustness. They're both suitable for writing highly concurrent, massively scale-able applications. The future is lots of cores, and the Rust community is on board with this. I'd also like to point out that at both of my jobs, we used Erlang for uses cases that are most heavily associated with the language, and for which popular open source alternatives in Elixir/Erlang already exist. We have had better success scaling, operating, and hiring for Rust. We've saved man hours, time, and money. So I am saying that Rust is a replacement for Erlang. It's not a tool for a different problem.
  - steveklabnik 8y ago
    Early Rust was much closer to Erlang, but it lost more and more of it over time.
namelos 8y ago
I also find erlang does not play well with modern ops work flow like docker and kubernetes. But as for Erlang, a lot of people uses Erlang for building in-memory massive stateful services, and a very fault tolerant system on top of it. I'm interested in both language, but haven't start playing with Rust yet. I tried Akka and find it's pretty good but the virtual machine is not as good as BEAM, but Scala is a powerful language. I'm curious about Rust. Could you please share some insights about if Rust is capable of such stateful services as Erlang does, or there could be other advantages in terms of stateful/fault tolerance?
tormeh 8y ago
Isn't the whole point of the BEAM languages to keep distributed state reasonable to work with through the actor model. If you can put a service in a docker container and spin it up or down without losing data, then it doesn't sound like you need any of the BEAMs.
sargun 8y ago
I'm going to disagree with you on Rust. It's a very different, very verbose language. It doesn't have any of the stories around immutability that Erlang does. You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. A common error in event-driven languages is that you end up writing code that gets slow, and blocks the entire event loop, and everything falls apart, and you get paged at 2 AM, until you add another event loop. One of the best parts of BEAM is that since processes are isolated and preemptively scheduled, you don't have to manage your own call-backs by hand, and although things may get slow, they'll typically only get slow for that one given process. In addition to this, the GC in Erlang is great, compared the lack of GC in Rust. I think most of us can agree that unburdening yourself of having to memory management code is a good thing. Of course, BEAM isn't perfect, after all, it's had no more as much investment as the JVM, and CLR, but I believe its semantics are right for writing predictable, low-latency code. Also, containers have nothing to do with ephemerality. Cluster management systems which dynamically schedule containers may result in scheduling. Erlang isn't really a dataplane runtime. Often times, you implement your control plane in Erlang, and farm out your dataplane to something NIFs, ports, or something else entirely. You're right, disterl is a fucking mess. But, it's better than nothing, and having to write your own IPC. I suggest you read Joe Armstrong's thesis, or a History of Erlang for more.
- ilovecaching 8y ago
  > I'm going to disagree with you on Rust. It's a very different, very verbose language. Erlang is incredibly verbose. It has very few abstractions. Rust has a lot of abstractions. I've rewritten a few Erlang projects now and Rust and I've been able to come out with close to or under the same LOC (I always use specs though). > It doesn't have any of the stories around immutability that Erlang does. let expressions are immutable by default. You specifically have to ask for mutability and even then, the Rust borrow checker will always enforce a single writer. Rust definitely has a story around immutability. > You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Which is good, because Mnesia is crap and I've had to deal with chucking it out the window several times now. Rust has good library support for STM that is completely optional. > Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. They're both models of concurrency, and you can achieve parallelism through either. Oftentimes one is better than the other for a given task (usually determined by the bottleneck), such as serving web requests bottlenecked by IO. > A common error in event-driven languages is that you end up writing code that gets slow, and blocks the entire event loop, and everything falls apart, and you get paged at 2 AM, until you add another event loop. Most high performance web servers use event loops. See the paper "An Architecture for Highly Concurrent, Well-Conditioned Internet Services" for an overview. There are lots of issues with green thread models. See some of the work done by Brian Cantrill for examples, and why it may be a bad idea to bake them into a language. > One of the best parts of BEAM is that since processes are isolated and preemptively scheduled, you don't have to manage your own call-backs by hand, and although things may get slow, they'll typically only get slow for that one given process. I think you're caught on the idea that Rust is Javascript or Python. In Rust you can parallelize an iter chain by changing one method call. You can also use multiple event loop to dispatch to handlers. There is no one size fits all solution to concurrency. > In addition to this, the GC in Erlang is great, compared the lack of GC in Rust. I think most of us can agree that unburdening yourself of having to memory management code is a good thing. Disagree strongly. See Steve Klabnik's latest posts on static garbage collection in Rust for an enlightening take. Rust does have a GC, and it has no runtime performance hit. > Of course, BEAM isn't perfect, after all, it's had no more as much investment as the JVM, and CLR, but I believe its semantics are right for writing predictable, low-latency code. Erlang isn't low latency. It has predictable latency. > Also, containers have nothing to do with ephemerality. Cluster management systems which dynamically schedule containers may result in scheduling. This is pedantic. Any non-trivial container deployment will have to deal with ephemerality. If you're replacing an Erlang cluster, it will be even moreso an issue because you'll need some level of fault tolerance from the orchestrator. > Erlang isn't really a dataplane runtime. Often times, you implement your control plane in Erlang, and farm out your dataplane to something NIFs, ports, or something else entirely. NIFs are extremely dangerous. We've had critical bugs that have taken down entire clusters thanks to NIFs. The architecture your describing is also exceedingly rare. Most Erlang deployments are handling soft real time workloads like routing chat, queue messages, web requests, that have no language separation between control and data. > You're right, disterl is a fucking mess. But, it's better than nothing, and having to write your own IPC. In many cases it is better than nothing. It will take a huge amount of wasted effort to fix some of the scaling issues I'm currently having with our Erlang cluster. > I suggest you read Joe Armstrong's thesis, or a History of Erlang for more. I've read Joe's thesis. I'm assuming your point is that I somehow don't know anything about Erlang despite having worked on it for years professionally and attended multiple Erlang Factorys/given talks on the subject.
quaunaut 8y ago
One thing that's pretty frustrating about your comments, is how you both have an answer for everything but don't actually give any specifics, which makes it nearly impossible to actually address or consider your advice. Also, fundamentally, I think you're way out of the norm in terms of system time to build. Getting the kinds of reliability and business value guarantees out of Rust is enormously harder than what you get out of the box with the BEAM. Having to hand-roll everything would take literally months, where I can be done and providing business value in hours in the BEAM ecosystem. Now, could Rust get to this place in another year or two? Completely. But it isn't there now, and, it still ignores my second issue... ...which is difficulty. I've been programming for just under 7 years, and while I've not learned C++, Rust is not only by far the most difficult language I've encountered thus far, it's exponentially more difficult. Months to grok the basics, likely years to be effective in it. You might've had an easy time hiring for this, but that might say more about your social group and hiring channels than the actual availability of talent.
- ilovecaching 8y ago
  > One thing that's pretty frustrating about your comments, is how you both have an answer for everything but don't actually give any specifics, which makes it nearly impossible to actually address or consider your advice. What is it you want to know? I'm giving you my personal experience, not an essay. It's hard to find people who have professional experience with Erlang, so I thought it would be useful to other people who are evaluating it to hear from someone who has been there. I'm more than happy to dig into things more if there's a pointed question. > Getting the kinds of reliability and business value guarantees out of Rust is enormously harder than what you get out of the box with the BEAM. Having to hand-roll everything would take literally months. It depends on your needs. My issue is that it is enormously hard to fix the BEAM once you have built something successful with it. It's hard to hire for. Working with it involves lots of esoteric knowledge. We also know that people are adopting highly reliable and scaleable systems. It's just not Erlang. Cloud enables this. Containerization enables this. Instead of Erlang's IPC, you can use an IDL and an RPC compiler. Instead of pids, you can have a service registry using etcd or DNS. You don't have to hand roll any of that. What Rust gives you is the ability to quickly build correct, fast, and maintainable systems that are easy to scale because concurrency is a central theme of the language. > I've been programming for just under 7 years, and while I've not learned C++, Rust is not only by far the most difficult language I've encountered thus far, it's exponentially more difficult. What languages have you learned? Did you learn about manual memory management in school/camp/on your own? What's been difficult about it for you? I'll admit, I know a lot of languages. Most of the things in Rust are familiar to me outside of lifetimes, which I don't consider to be to difficult to learn. It's just making something you consider implicitly explicit. We have hired three new grads who we started on Rust, and we got them up in running in a few weeks. They did have the books, mentorship, and assigned work involving small tasks to ramp up on.
pmontra 8y ago
> The BEAM just doesn't like to be treated like cattle. So don't. Deploy using edeliver and give up docker for the Erlang / Elixir parts of the project. If you really can't, you might want to switch to a language with a runtime that plays more nicely with the container idea of the world, but I know companies that deploy Erlang with docker and they are happy with that. Purists frown at the idea but it works.
- ilovecaching 8y ago
  It's a major pain to run it in containers, and that extends beyond Docker. That's why I do advocate switching away from Erlang. Giving up on containerization isn't an option unless we want to manage two completely separate infrastructures, which we definitely do not want to do.
  - pmontra 8y ago
    I think that those people I wrote about are running BEAM in containers as if it were PHP. No connections between nodes and a load balancer in front of the containers. That should leave them with the niceties of the language, most but not all of OTP, and almost none of the deployment woes people have when unfamiliar with BEAM.
ecthiender 8y ago
This is a post on `X` by example. I don't think it's constructive to the discussion here to say X has drawbacks (and then go on at it), use Y (and then go on at it). I have not used Erlang or Rust, and when I read this comment, it seemed flamewar-ish to me.
jeremyjh 8y ago
> It also enables event driven programming through tokio, which is a better fit for web servers than green threads (you're mostly waiting on the network) This is a really strange thing to say. Erlang is still an event-driven I/O runtime, it just doesn't inflict the user with the burden of continuation passing like tokio does. What is the benefit of doing that yourself? Even with futures its far more verbose and awkward than it needs to be. And since the entire ecosystem is not built on this io system, you will always be finding libraries - even very popular ones, such as diesel - that are incompatible with it and forcing you into threadpools. Every library in use with BEAM uses async I/O in exactly the same way.
- ilovecaching 8y ago
  This is an implementation detail that you have no control over. From the point of a view of a process, io is synchronous. This is very limiting. It's better to keep decisions of concurrency in libraries rather than bake it into the language.
  - jeremyjh 8y ago
    Can you elaborate on the limits this imposes? What is it that you can do with direct access to an event loop that you cannot do in Erlang?
  - deleted 8y ago
    [deleted]
stplsd 8y ago
>and it's hard to burst. Like, super hard to burst What exactly does 'burst' mean in this context?
- ilovecaching 8y ago
  Bursting is when you need to add additional capacity to handle expected but higher than normal load for a brief period of time.

dev_dull 8y ago

The world of ephemeral containers really has changed the dynamic. Erlang solves a lot of problems with good solutions, but writing good software is only part of the battle. Lifecycle management, monitoring, etc make up a huge operational burden and should always be thought of as first-class problems in language design. Rust makes that better in many ways but not uniquely better than alternative languages.

ilovecaching 8y ago
In my experience any compiled language is easier to manage. There is no disconnect between your runtime and the system. You're investing in systems knowledge rather than knowledge of a particular VM's bytecode/abstractions as well. Erlang does provide good solutions to lifecycle management. The problem is that whenever someone picks a solution for you, you're now locked it and it can be difficult to migrate to a new paradigm like containers.
- dev_dull 8y ago
  Totally agree regarding compiled languages. I’m surprised python et all don’t offer a way to check for file/script “correctness” without execution. It’s so bad that I will often add a “-h” flag to scripts I encounter just so I can run it and make sure no one introduce a silly bug (such as typo, import, etc).
  - ilovecaching 8y ago
    Yep, I've seen my fair share of Python stack traces. Completely sympathize, and it is one of the reasons I believe in statically types and compiled as well.

abledon 8y ago

What is TLC?

ilovecaching 8y ago
Tender loving care

kauboy 8y ago

What do you think of Elixir?

ilovecaching 8y ago
Elixir doesn't affect most of the operational aspects of Erlang, and it certainly in no way affects performance. You're still running on the BEAM, except now you actually have more apps running (The Elixir runtime runs as an app). Syntactically it cleans some things up, adds some niceties, and annoyingly makes atoms require a prefix of ":" (annoying because atoms are central to Erlang's readability). Personally, I don't find the changes to make a big difference when using the BEAM, and I prefer the familiarity of Erlang's syntax. It's also the language all of the documentation for the BEAM will use. If you absolutely can't get your coworkers to ditch dynamic/gradual typing, Elixir or Erlang are still great choices and way better than Python/Ruby. Otherwise, try Rust.
- pera 8y ago
  Are you actually using Rust for professional web development? I've tried it earlier this year and my experience wasn't that great, so I'm quite surprised to read this comment.
  - zinclozenge 8y ago
    What did you end up using for Rust? The actix ecosystem is pretty good, and quite fast.
    - pera 8y ago
      I did like Rust (as a language), but I'm using Elixir, Phoenix and Ecto right now.
  - ilovecaching 8y ago
    What issues did you have? We haven't had very many issues. We rebuilt some internals tools, but for the most part it's been smooth sailing. We do use hyper and h2.

enraged_camel 8y ago

Your complaints about Erlang give me the impression that you may have picked the wrong language for your problem domain, and paid for that mistake by having to do things like rewriting gen_server. Erlang was designed to be reliable and scalable. It was not designed to be fast. If fast is a hard prerequisite for you, you're right, you should go with another language. But Erlang also does a ton of things correctly (again, in the domains it was designed and optimized for), and is battle-tested far beyond Rust, since it has been around for much, much longer. That's not to say Rust is a bad language, but one should probably not judge its merits by its current popularity. That holds true for any language or tool. Anyways, if you want abstraction (the lack of which was one of your complaints about Erlang), you should take a look at Elixir. Specifically, Phoenix is excellent for web programming, especially if you need real-time messaging at scale.

ilovecaching 8y ago
I'm aware of all of these claims. Any Erlang developer has heard this shtick a thousand times. The reality is that you do eventually have to twist the BEAM once you reach a certain level of scale, and there are plenty of bugs. Like I said, it's also difficult to burst, so it's not always as black and white as you say. It's also missing a lot of the introspection you need to operationalize at scale. We've had to patch the Beam many times to keep moving forward, and work with super dangerous nifs to talk to other systems. The issues I've had with Erlang had nothing to do with the products I've worked with. Erlang was a 'good fit' for them. Elixir and Phoenix doesn't really solve any of these issues, and neither of them are particularly 'battle tested'.
- bthornbury 8y ago
  As an Erlang fan, who hasn't used it in production services, I'm wondering if you can let us know some specific scaling issues you encountered.
  - ilovecaching 8y ago
    I can't really go into it in detail, but at a high level Erlang default methods of distribution and security don't scale very well. There are people working on better mechanisms for this, and I know of several companies that have custom solutions for clustering nodes. One big issue is you cannot easily burst Erlang nodes to handle peak traffic. The number of nodes is usually relatively static in a deployment.

plainOldText 8y ago

Rust has too big of a learning curve, and it's quite complex for writing service oriented systems. Granted, having a static compiler eliminates many types of bugs found in other systems. In my opinion a more sensible approach is Elixir + Rust (via rustler). Your get the elegance and productivity of Elixir, the concurrency and fault tolerance of Erlang/OTP, while still being able to writing super fast, low level code in Rust. Plus, you shouldn't use a single language across your whole stack. When the only tool you have available is a hammer, everything looks like a nail.

ilovecaching 8y ago
I don't buy this argument. Erlang does have a steep learning curve, but it isn't the semantics of the language, it's understanding how to build and deploy OTP applications. Understanding apps, releases, clustering, mnesia, process registries, circuit breaking etc. all take time. It also will cost you hours, and hours, and hours down the road when you have a large sprawling app with very little abstraction and no -specs, because dialyzer is frustrating. I was able to learn Rust in about week by reading the O'reilly book. Compared to C++ it's almost tiny (I have many Bjarn's C++ tome and the Meyers books right next to my Rust books actually), and unlike C++, the compiler will essentially teach you the language if you didn't learn enough from the book to write code that compiles. Once you've put in the initial investment, Rust code is way easier to maintain, read, and scale. It's also easier to onboard people. Onboarding people for Erlang is hard, and it's hard to hire for. Rust on the other hand is familiar to all of our FP people who like Haskell and Erlang, and our Java and C++ devs. I've also found it extremely suitable for writing services. We have many services in Rust running in production right now. We have a service toolkit for our cross cutting concerns. We can run our IDL through a compiler and get a client and server that are on the order of 20x faster than their Erlang counterparts.
- plainOldText 8y ago
  For the record, I would prefer not to get into a programming language flame war. Having said that, I will add the following: It's much harder to reason in Rust, as there are many concepts one needs to keep in their head (borrowing, lifetime, pointers, etc). Plus Rust is much more difficult to read. Whereas Elixir, and I emphasize Elixir over Erlang here, is much easier to reason in, concise, and simpler to read. At the end of the day, use whatever language you prefer, just keep in mind that software needs to be (1) shipped, (2) maintained (by multiple people who read each other's code) and (3) evolved. I would also add (0) experimentation; before one even ships any code, one ought to easily experiment with various ideas.

andy_ppp 8y ago

You’re comparing the out of the box, network transparent scaling solution that is fully meshed and doesn’t claim to solve everyone’s problems forever with a solution in rust that probably involves a completely hand rolled implementation of lots and lots of different things OTP is doing. You have made me want to learn Rust though, how does it compare with Golang?

ilovecaching 8y ago
We move the orchestration from the Erlang runtime to our container orchestrator though. We have a better separation of concerns, and we can scale any language. The registry, process communication, are just services and protocols running in containers. So compared to Go, the Rust folks seemed to have paid close attention to the things people liked about Go. We have most of those things. We have great tooling, auto formatting, concurrency primitives (although we have freedom of choice. Green threads are a third party library, and aren't part of a runtime). We even have an animal mascot. From a language standpoint, Go isn't anything remarkable (our Haskell colleague calls it a step backwards). If you listen to the Gotime podcast, Brad Fitzpatrick, one of the Go authors even says as much. Go's big contribution was essentially web programming and go routines. Rust, on the other hand, has a lot. It has typeclasses from Haskell. It has lifetimes from the PL research community. It has immutability and pattern matching just like in Erlang. It has all the familiar control structure and enums/structs from C++. Performance of Go in terms of CPU and memory pressure is usually somewhere near Java, if not slightly better. That's pretty damn good. The JVM has improved a lot over the years. Rust performance is a hair shy of C/C++. That's amazing. All those abstractions, and the speed of C++?

Immortalin 8y ago

There's always Pony.

qaq 8y ago
Pony has a lot of promise but it needs a few key missing libs. If it had a quality http and postgres libs it prob would be perfect for web dev. but it does not have either yet.

deleted 8y ago

[deleted]

dxhdr 8y ago

> As someone who has worked two jobs now writing, deploying, and operating Erlang clusters, I recommend switching to Rust. I'm surprised to read this, Erlang and Rust have almost nothing in common. It's understandable that BEAM doesn't suit your needs but suggesting that everyone should write Rust instead is grossly misleading. They are suited to solving different problems!

ilovecaching 8y ago
They have a lot in common syntactically, semantically, and philosophically. They are both languages designed to promate safety and robustness. They're both suitable for writing highly concurrent, massively scale-able applications. The future is lots of cores, and the Rust community is on board with this. I'd also like to point out that at both of my jobs, we used Erlang for uses cases that are most heavily associated with the language, and for which popular open source alternatives in Elixir/Erlang already exist. We have had better success scaling, operating, and hiring for Rust. We've saved man hours, time, and money. So I am saying that Rust is a replacement for Erlang. It's not a tool for a different problem.
- steveklabnik 8y ago
  Early Rust was much closer to Erlang, but it lost more and more of it over time.

namelos 8y ago

I also find erlang does not play well with modern ops work flow like docker and kubernetes. But as for Erlang, a lot of people uses Erlang for building in-memory massive stateful services, and a very fault tolerant system on top of it. I'm interested in both language, but haven't start playing with Rust yet. I tried Akka and find it's pretty good but the virtual machine is not as good as BEAM, but Scala is a powerful language. I'm curious about Rust. Could you please share some insights about if Rust is capable of such stateful services as Erlang does, or there could be other advantages in terms of stateful/fault tolerance?

tormeh 8y ago

Isn't the whole point of the BEAM languages to keep distributed state reasonable to work with through the actor model. If you can put a service in a docker container and spin it up or down without losing data, then it doesn't sound like you need any of the BEAMs.

sargun 8y ago

I'm going to disagree with you on Rust. It's a very different, very verbose language. It doesn't have any of the stories around immutability that Erlang does. You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. A common error in event-driven languages is that you end up writing code that gets slow, and blocks the entire event loop, and everything falls apart, and you get paged at 2 AM, until you add another event loop. One of the best parts of BEAM is that since processes are isolated and preemptively scheduled, you don't have to manage your own call-backs by hand, and although things may get slow, they'll typically only get slow for that one given process. In addition to this, the GC in Erlang is great, compared the lack of GC in Rust. I think most of us can agree that unburdening yourself of having to memory management code is a good thing. Of course, BEAM isn't perfect, after all, it's had no more as much investment as the JVM, and CLR, but I believe its semantics are right for writing predictable, low-latency code. Also, containers have nothing to do with ephemerality. Cluster management systems which dynamically schedule containers may result in scheduling. Erlang isn't really a dataplane runtime. Often times, you implement your control plane in Erlang, and farm out your dataplane to something NIFs, ports, or something else entirely. You're right, disterl is a fucking mess. But, it's better than nothing, and having to write your own IPC. I suggest you read Joe Armstrong's thesis, or a History of Erlang for more.

ilovecaching 8y ago
> I'm going to disagree with you on Rust. It's a very different, very verbose language. Erlang is incredibly verbose. It has very few abstractions. Rust has a lot of abstractions. I've rewritten a few Erlang projects now and Rust and I've been able to come out with close to or under the same LOC (I always use specs though). > It doesn't have any of the stories around immutability that Erlang does. let expressions are immutable by default. You specifically have to ask for mutability and even then, the Rust borrow checker will always enforce a single writer. Rust definitely has a story around immutability. > You say you had to rewrite Mnesia, but Rust doesn't even have transactional memory to start with. Which is good, because Mnesia is crap and I've had to deal with chucking it out the window several times now. Rust has good library support for STM that is completely optional. > Suggesting that event-driven programming is anything like language-level threading like Erlang and Go is crazypants. They're both models of concurrency, and you can achieve parallelism through either. Oftentimes one is better than the other for a given task (usually determined by the bottleneck), such as serving web requests bottlenecked by IO. > A common error in event-driven languages is that you end up writing code that gets slow, and blocks the entire event loop, and everything falls apart, and you get paged at 2 AM, until you add another event loop. Most high performance web servers use event loops. See the paper "An Architecture for Highly Concurrent, Well-Conditioned Internet Services" for an overview. There are lots of issues with green thread models. See some of the work done by Brian Cantrill for examples, and why it may be a bad idea to bake them into a language. > One of the best parts of BEAM is that since processes are isolated and preemptively scheduled, you don't have to manage your own call-backs by hand, and although things may get slow, they'll typically only get slow for that one given process. I think you're caught on the idea that Rust is Javascript or Python. In Rust you can parallelize an iter chain by changing one method call. You can also use multiple event loop to dispatch to handlers. There is no one size fits all solution to concurrency. > In addition to this, the GC in Erlang is great, compared the lack of GC in Rust. I think most of us can agree that unburdening yourself of having to memory management code is a good thing. Disagree strongly. See Steve Klabnik's latest posts on static garbage collection in Rust for an enlightening take. Rust does have a GC, and it has no runtime performance hit. > Of course, BEAM isn't perfect, after all, it's had no more as much investment as the JVM, and CLR, but I believe its semantics are right for writing predictable, low-latency code. Erlang isn't low latency. It has predictable latency. > Also, containers have nothing to do with ephemerality. Cluster management systems which dynamically schedule containers may result in scheduling. This is pedantic. Any non-trivial container deployment will have to deal with ephemerality. If you're replacing an Erlang cluster, it will be even moreso an issue because you'll need some level of fault tolerance from the orchestrator. > Erlang isn't really a dataplane runtime. Often times, you implement your control plane in Erlang, and farm out your dataplane to something NIFs, ports, or something else entirely. NIFs are extremely dangerous. We've had critical bugs that have taken down entire clusters thanks to NIFs. The architecture your describing is also exceedingly rare. Most Erlang deployments are handling soft real time workloads like routing chat, queue messages, web requests, that have no language separation between control and data. > You're right, disterl is a fucking mess. But, it's better than nothing, and having to write your own IPC. In many cases it is better than nothing. It will take a huge amount of wasted effort to fix some of the scaling issues I'm currently having with our Erlang cluster. > I suggest you read Joe Armstrong's thesis, or a History of Erlang for more. I've read Joe's thesis. I'm assuming your point is that I somehow don't know anything about Erlang despite having worked on it for years professionally and attended multiple Erlang Factorys/given talks on the subject.

quaunaut 8y ago

One thing that's pretty frustrating about your comments, is how you both have an answer for everything but don't actually give any specifics, which makes it nearly impossible to actually address or consider your advice. Also, fundamentally, I think you're way out of the norm in terms of system time to build. Getting the kinds of reliability and business value guarantees out of Rust is enormously harder than what you get out of the box with the BEAM. Having to hand-roll everything would take literally months, where I can be done and providing business value in hours in the BEAM ecosystem. Now, could Rust get to this place in another year or two? Completely. But it isn't there now, and, it still ignores my second issue... ...which is difficulty. I've been programming for just under 7 years, and while I've not learned C++, Rust is not only by far the most difficult language I've encountered thus far, it's exponentially more difficult. Months to grok the basics, likely years to be effective in it. You might've had an easy time hiring for this, but that might say more about your social group and hiring channels than the actual availability of talent.

ilovecaching 8y ago
> One thing that's pretty frustrating about your comments, is how you both have an answer for everything but don't actually give any specifics, which makes it nearly impossible to actually address or consider your advice. What is it you want to know? I'm giving you my personal experience, not an essay. It's hard to find people who have professional experience with Erlang, so I thought it would be useful to other people who are evaluating it to hear from someone who has been there. I'm more than happy to dig into things more if there's a pointed question. > Getting the kinds of reliability and business value guarantees out of Rust is enormously harder than what you get out of the box with the BEAM. Having to hand-roll everything would take literally months. It depends on your needs. My issue is that it is enormously hard to fix the BEAM once you have built something successful with it. It's hard to hire for. Working with it involves lots of esoteric knowledge. We also know that people are adopting highly reliable and scaleable systems. It's just not Erlang. Cloud enables this. Containerization enables this. Instead of Erlang's IPC, you can use an IDL and an RPC compiler. Instead of pids, you can have a service registry using etcd or DNS. You don't have to hand roll any of that. What Rust gives you is the ability to quickly build correct, fast, and maintainable systems that are easy to scale because concurrency is a central theme of the language. > I've been programming for just under 7 years, and while I've not learned C++, Rust is not only by far the most difficult language I've encountered thus far, it's exponentially more difficult. What languages have you learned? Did you learn about manual memory management in school/camp/on your own? What's been difficult about it for you? I'll admit, I know a lot of languages. Most of the things in Rust are familiar to me outside of lifetimes, which I don't consider to be to difficult to learn. It's just making something you consider implicitly explicit. We have hired three new grads who we started on Rust, and we got them up in running in a few weeks. They did have the books, mentorship, and assigned work involving small tasks to ramp up on.

pmontra 8y ago

> The BEAM just doesn't like to be treated like cattle. So don't. Deploy using edeliver and give up docker for the Erlang / Elixir parts of the project. If you really can't, you might want to switch to a language with a runtime that plays more nicely with the container idea of the world, but I know companies that deploy Erlang with docker and they are happy with that. Purists frown at the idea but it works.

ilovecaching 8y ago
It's a major pain to run it in containers, and that extends beyond Docker. That's why I do advocate switching away from Erlang. Giving up on containerization isn't an option unless we want to manage two completely separate infrastructures, which we definitely do not want to do.
- pmontra 8y ago
  I think that those people I wrote about are running BEAM in containers as if it were PHP. No connections between nodes and a load balancer in front of the containers. That should leave them with the niceties of the language, most but not all of OTP, and almost none of the deployment woes people have when unfamiliar with BEAM.

ecthiender 8y ago

This is a post on `X` by example. I don't think it's constructive to the discussion here to say X has drawbacks (and then go on at it), use Y (and then go on at it). I have not used Erlang or Rust, and when I read this comment, it seemed flamewar-ish to me.

jeremyjh 8y ago

> It also enables event driven programming through tokio, which is a better fit for web servers than green threads (you're mostly waiting on the network) This is a really strange thing to say. Erlang is still an event-driven I/O runtime, it just doesn't inflict the user with the burden of continuation passing like tokio does. What is the benefit of doing that yourself? Even with futures its far more verbose and awkward than it needs to be. And since the entire ecosystem is not built on this io system, you will always be finding libraries - even very popular ones, such as diesel - that are incompatible with it and forcing you into threadpools. Every library in use with BEAM uses async I/O in exactly the same way.

ilovecaching 8y ago
This is an implementation detail that you have no control over. From the point of a view of a process, io is synchronous. This is very limiting. It's better to keep decisions of concurrency in libraries rather than bake it into the language.
- jeremyjh 8y ago
  Can you elaborate on the limits this imposes? What is it that you can do with direct access to an event loop that you cannot do in Erlang?
- deleted 8y ago
  [deleted]

stplsd 8y ago

>and it's hard to burst. Like, super hard to burst What exactly does 'burst' mean in this context?

ilovecaching 8y ago
Bursting is when you need to add additional capacity to handle expected but higher than normal load for a brief period of time.