9 ms·
We developed Online Charging System in Erlang that served couple million subscribers for close to three years. I found the whole experience fairly terrible. Er
by ihuk 7y ago
We developed Online Charging System in Erlang that served couple million subscribers for close to three years. I found the whole experience fairly terrible.
Erlang is nice enough language and I don't mind the syntax but it's also kinda cumbersome and verbose at times. For example, adding `true -> ok` to every if statement gets old fast. Similarly, Erlang/OPT is a nice platform but some parts are fairly bad. Looking at you `inets/httpc`.
But the really big problem was the whole ecosystem. Build process was needlessly complicated. Surprisingly enough multicore support was not great. Performance was not that great. It seemed like a lot of libraries were abandoned after 2013. I could go on.
We ended up rewriting the whole thing in Java and, so far, it worked out great. And after Java 8, Java the language is not so bad. I still have somewhat found memories of Erlang but I don't miss it.
- mapcars 7y agoInteresting, do you think Elixir would be better in your case? Also what was the amount of requests per second? Looks like you didn't really need Erlang's scalability.
- ihuk 7y agoNo; as I understand it Elixir's only advantage is, arguably, better syntax. I don't mind Erlang's syntax that much. We were handling, in peek hours, over 100k requests per second spread over 3 servers. Funny you should mention scalability as something that somehow justifies Erlang's other shortcomings. I don't think scalability is something Erlang/OTP does out of the bag. One thing we learned the hard way is that Erlang does not do overload well at all. Erlang mailboxes are unbounded and it can get very messy very fast. I always found that design choice odd given Erlang's origins in telecommunications. Now this is not to say you can't do massively scalable systems in Erlang, it's just that there's no silver bullet when it comes to scalability.
- mac01021 7y ago> We were handling, in peek hours, over 100k requests per second spread over 3 servers. How big was one server and what does one request entail?
- dajonker 7y agoScalability isn't something any language does out of the box. The design and implementation of the application determines whether it scales or not. In that sense, Erlang scales about as well as any other language. It does have some features that might make scalability easier to implement compared to some other languages, but you have to know how to use it properly.
- ihuk 7y agoExactly my point. It scales about as well as any other language so why not use a language that's nicer to work with?
- defined 7y ago> It scales about as well as any other language You can get just about anything to "scale" given architectural compensations (like scads of containers), so this, ipso facto, doesn't mean much. > so why not use a language that's nicer to work with? Well, why not indeed? But here are some thoughts about that. Nothing is a panacea, Erlang included. Some of the ecosystem is ugly and hard to work with, but improving. Erlang is in a vicious cycle: * Erlang ecosystem sucks * Fewer people want to use it * Fewer people are available to improve the ecosystem * Fewer people are available to fill jobs for Erlang * Less demand for Erlang * Loop Erlang gives you certain things that no other language I know of does, and these things make it a more robust, easier to program, highly concurrent solution without needing architectural compensations for those things. That doesn't mean that you can magically throw 1 million TPS at Erlang running on N boxes and your internal queues/mailboxes won't overflow. You, along with the rest of the world, need backpressure handling mechanisms. What Erlang gives you that Java doesn't is totally decoupled processes that cannot directly affect each other. No mutexes or semaphores, no shared global memory. It also gives you ultra-lightweight processes that garbage-collect independently of each other, that are cheap enough to start and stop that you can create one per incoming connection and make it completely unaffected by any trouble that any other process gets into. This is not true of threads in Java. It gives you a supervisory framework that, if used judiciously (yeah, yeah, no true Scotsman, but that doesn't make this any less valid) gives you an ultra-robust system that degrades gracefully under error or overload conditions. It gives you linked processes that detect a broken process and brings down the others (if you so choose) to avoid leaving orphaned processes cluttering up the system, plus automatic restart, baked in. These do come free of charge when using OTP supervisory frameworks in the recommended fashion. I think that in some cases, Erlang has been overhyped and people come into it expecting it to magically solve all the hard problems. People that overhype Erlang do it a disservice. For example, the stupid "nine nines" thing was an almost once-off, limited situation in a very constrained environment and I am sure many Erlangists wish it had never been mentioned. And I wish nobody had ever said the words "let it crash", because they have been misunderstood and taken out of context and again done Erlang a disservice. Just like the phrase "premature optimization is the root of all evil" has caused untold damage when it was (and is often) taken out of context. The whole "let it crash" thing was shorthand for "if a process cannot reasonably handle and error condition itself, let it die and be reincarnated by the supervisory process". It does not mean "just let everything crash". For example, if you have a system that has long-lived connections, maybe (say) http/2 connections, or XMPP, or Apple push, you most definitely do not want to let it crash without doing everything possible to recover from errors such that the connection stays up under as many error conditions as possible. But for the most part, a process should not be written defensively and try to recover from errors where it has no business doing so - it should leave it to the supervisor. I have written Erlang systems, heavily used in production that ran reliably, without crashing, for many months. Usually they were only stopped to do an OS upgrade. They weren't error-free - we found out about some persistent errors by checking the logs occasionally, then fix them and often hot-patch the fix into the production systems. Again, Erlang is not the world's best programming language or ecosystem. It is, within its design envelope, one of the finest soft real-time distributed multiprocessing environments around when used, like anything, with skill and careful architectural design. In my humble opinion.
- mapcars 7y agoThe point about Elixir is it does better in ecosystem, libraries, tooling and removes a lot of complexities you named.
- elcritch 7y agoEvery time I wade into Erlang projects, I’m struck by the cultural and overall ecosystem differences.
- specialist 7y agoI call that "handling back pressure", because I don't know what to call it. I don't know of any system, stack that handles it directly, natively. Though my sample size is pretty small.
- e_proxus 7y agoNot to refute your experience, but "adding true -> ok to every if statement" sounds like an anti-pattern. A couple of notes to elaborate: 1. If-statementes should rarely be used in Erlang because case is preferred 2. If you're returning ok everywhere, you could also just let it crash by doing something akin to true = function(). Always go for let it crash first, unless you really know you can handle the error sensibly (and then use a case-statement) 3. httpc is kind of known to not be the most formidable HTTP client. Hackney is a more modern and better choice. The good thing with httpc is that it is included with OTP, otherwise there are better choices (scalability- and feature-wise) 4. When someone says "multicore support" and "performance" support is not great it's usually one of two things: (a) they're developing a use case that is not fit for Erlang (e.g. compute heavy) or (b) they don't know how to use Erlang properly (e.g. too complex process setups) Now, many of these things are not obvious to people new to Erlang and takes some experience to know about. This I would say is the bigger problem with the Erlang eco-system.
- ihuk 7y agoAgreed. Couple of clarifications/observations. (1) there are situations like this: if DeductCreditInRatingGroup -> log("deducting credit..."); end; unfortunately you cannot write it this way. You have to write it as: if DeductCreditInRatingGroup -> log("deducting credit..."); true -> ok end; Not a huge deal but it's annoying. (2) Let it crash attitude is something I never understood. It's just not something you can do most of the time. If subscriber consumed 2MB and you let it crash in the middle of rating function you just "lost" 2MB. With couple of million subscribers on LTE that turns into very expensive error handling very fast. (3) We actually ended up using hackney. (4) We were not doing anything compute heavy. I think our application was well architected and we had no problem fitting it into OTP "framework". Comparing CPU and memory usage to our Java implementation, Java is more performant. edit: fixed code formatting.
- e_proxus 7y ago1. Here I would do something like this: log("deducting credit...", DeductCreditInRatingGroup), ... log(Msg, true) -> do_the_logging(); log(Msg, false) -> ok. Nowadays though, you should really use the new logger introduced in Erlang 21. There you can do run-time filtering on many different parameters, or even write your own handlers. Using if-statements is usually considered an anti-pattern in Erlang, and I think very common coming from languages where only if-statements exist. Problems are almost always better solved by using case, or even better, pattern matching in function heads. 2. Sounds like one of the rare cases where you care about the error and can also handle it. In the use case you mentioned, I would buffer the data outside of the loop that produces it and wrap the loop in a try-catch statement. That way, you don't loose the data if one iteration crashes. 4. Erlang usually ends up being good at latency and concurrency (scalability over cores), together with a smaller and easier to read code base. I'm curious about your case, and what would have made Java a better fit. If you share a lot of global objects, Java might be faster for some things (at the cost of concurrency usually). Erlang has tools for that as well though (e.g. ETS and recently persistent_term).
- namelosw 7y agoIf you use Java I'm assuming you are not using process. If so, I believe Phoenix is no worse than Java applications in terms of ease of use and scalability etc. And it in terms of performance it indeed require some design experience on Erlang process. But it shouldn't be bad. After all WhatsApp and Discord have run on top of it with very few engineers to support massive customers before. I doubt online charging system could be more chatty.
- cryptos 7y agoWhat Java framework (for concurrency) do you use?
- sandGorgon 7y agodid you use Vert.x ? IMHO that has the closest cognitive match to Erlang. it regularly tops performance benchmarks - https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=json https://www.techempower.com/benchmarks/#section=data-r18&hw=... and super simple heroku deployments - https://github.com/vert-x3/vertx-examples/tree/master/heroku-example https://github.com/vert-x3/vertx-examples/tree/master/heroku... Also Kotlin !! https://vertx.io/docs/vertx-core/kotlin/ https://vertx.io/docs/vertx-core/kotlin/ (kotlin vertx on heroku: https://github.com/vert-x3/vertx-examples/tree/master/kotlin-examples/web#heroku https://github.com/vert-x3/vertx-examples/tree/master/kotlin...)
- defined 7y agoI can't refute your personal experience, but 10 years of Erlang work left me longing for more. It sounds as if perhaps Erlang was not used optimally, or, alternatively, best practices may not have been followed. 10 years ago the ecosystem was horrendous, and is much better now. Granted, the build experience could be better but I didn't find it that bad. I find that people's experiences are relative to what they are accustomed to. Early years for me included compiling and linking C in Windows (segmented) and that was truly horrible!
- ihuk 7y agoIt's funny how "you're holding it wrong" is the go to response whenever somebody criticizes Erlang. Why is that?
- lostcolony 7y agoReplace "Erlang" with any other language and the statement still holds true. If you want the obvious explanation, it's because we all have different experiences, and no one tool will be the biggest win for all teams across all domains.
- defined 7y agoProbably because they really are holding it wrong?
- enraged_camel 7y agoI mean, what you said about if statements (which should be very very rare in Erlang) kind of betrays it: you most likely _are_ holding it wrong.
- lelf 7y ago> Surprisingly enough multicore support was not great Considering that it’s state of the art, you must have been doing it terribly wrong. > adding `true -> ok` to every if statement That confirms it.
- ihuk 7y agoThat's just not true. Before you would run into all kinds of bottlenecks after 2-4 cores. I believe it was addressed in R13A[1]. [1] http://erlang.org/download/otp_src_R13A.readme http://erlang.org/download/otp_src_R13A.readme