8 ms·

So, I know that Elixir is powerful and good at functional programming. The one thing I can't quite understand is why we would want to program everything as a co

So, I know that Elixir is powerful and good at functional programming. The one thing I can't quite understand is why we would want to program everything as a composition of individual programs and applications, with supervisors and application trees and message passing and the like. It seems like a lot of overhead to accomplish something. Making everything asynchronous and detached makes things more complicated, not less. I hear a lot about OTP and the "let it crash" mantra, but I just don't quite understand what's so great about it. Maybe it's just due to my problem domain (web development), but it doesn't seem like as big a draw to Elixir and Erlang as pattern matching and FP are.

tim333 10y ago
The main argument why seems to be for reliability so it can run for months without going down.
- rpazyaquian 10y ago
  I understand the need for reliability, i.e. "come back up if you get bad data so it doesn't just die on you", but this seems like something that dates back from when any kind of raised exception would bring the entire system down. Don't we have modern frameworks and systems now that can tolerate individual failures instead of completely dying just because someone tried to POST some bad data? Plus, having reliability and coming back up if it goes down doesn't solve the root problem, which is that there's a bug in the code, or some sort of unhandled edge case. 99.9999999% uptime doesn't help if your code doesn't work right.
  - dottedmag 10y ago
    Imagine there is a bug in the code or unhandled edge case, only exhibited on 0.01% transactions. Elixir/OTP handler dies on this edge case, supervisor spawns another worker, 99.99% of user transactions continue to be served. Programmer wakes up next morning, comes to work, reads the exception log, fixes this edge case, optionally hot-upgrades the code in production if it's easier than doing a full restart, problem solved. Other frameworks have to carefully implement the same semantics (that is, prevention of propagation of bad state/cascading failures). Elixir/OTP/Erlang come with it by default, and coding style encourages it.
  - rdtsc 10y ago
    > Plus, having reliability and coming back up if it goes down doesn't solve the root problem, which is that there's a bug in the code, or some sort of unhandled edge case. 99.9999999% uptime doesn't help if your code doesn't work right. Yeah but you don't have to wake up at 4am in the morning to fix the problem if the system stays up and still works (even if a bit slower or some feature is degraded). I've seen systems that had parts crashing and restarting for a weeks and months. Yeah it is bad nobody noticed, but it was secondary service that didn't affect most users. Having the backend with an exception or segfault because if it would not make sense. > Don't we have modern frameworks and systems now that can tolerate individual failures instead of completely dying just because someone tried to POST some bad data? Yeah we do. Nothing Elixir does is magic or absolutely impossible with other systems. You can spawn an OS process to handle a request. Can have a load balancer or something to check the health of backend nodes, and redirect. Or wrap everything in exceptions and try to restart and so on. It is kind of doable. But now there are multiple libraries, frameworks, services to also maintain and look after, maybe that thread just crashed made a mess on the heap and now maybe restarting don't really fix the problem. So doable, but more awkward. > Plus, having reliability and coming back up if it goes down doesn't solve the root problem, Obviously, nothing will write the code to fix the problems except the programmer. Is it worth to completely crash the system because some bug in one thread or some unimportant component failed? It is a bit like having a human drop dead the first time they scratched their finger. Yeah they can't write as fast for a while, but they can still largely function. I mentioned Rust in another comment. Perhaps you'd prefer to have the compiler check correctness, that's very reasonable, something like Rust or Haskell might be what you'd like to use to sort of ensure you have less bugs when the system runs. Higher level of proof of correctness are possible, that is done for avionics software, crypto and other life-critical systems. So it is a continuum there. But there usually dollar signs attached to it.
  - biot 10y ago
    The supervisor is what is responsible for restarting the worker or not. If you have a process that saves files to local disk and the disk is full, causing the worker to crash, the supervisor can recognize that reason and conclude that the worker process should not be restarted. Additionally, the supervisor can take other actions like informing the rest of the system and/or potentially even spawning a different worker process that stores files to a slower, more expensive cloud storage option.
  - ramchip 10y ago
    My personal anecdote to illustrate the difference: I've worked on a python app before which had a bug that sometimes made the websocket closed callback throw an exception. The exception was still being caught by the framework, so the system kept working fine. ...or did it? Eventually, we noticed instances of the app locking up, not responding to any request. There was no obvious error, it just kept running but did not log or do anything. It turned out to be due to the callback bug: the exception was being caught alright, but we were leaking file descriptors like a sieve and eventually running out. Due to this a minor bug affecting only some requests was able to take down the entire app. That's the kind of problem Erlang solves well. Every request runs in its own lightweight process. The process can open (and thus own) files, sockets, OS processes, in-memory DB tables, etc. If the process crashes or dies for any reason, its memory and any other resource it owned gets cleaned up by the VM. If the process tries to hog the CPU or even enters an infinite loop, the preemptive scheduling means other requests will still get a chance to run. In short: having strong isolation between processes means it's unlikely for a bug in a minor feature somewhere to affect the overall application health.
  - hosh 10y ago
    I think you are misunderstanding something. "Let it crash" doesn't mean crash on everything. It's about handling the _unexpected_. Someone posting bad data may or may not be expected. The primitives are such that you can _choose_ how much or how little supervision you want, for a given complexity or maturity of a project, or what you learned as users put the system through the wringer. This is a _choice_ you make as an engineer, and OTP makes it easier for you to make such a tradeoff. Other systems and framework do not necessarily let you make such a tradeoff as gracefully. This not only allows the software to scale gracefully with the number of users, this allows the software to evolve gracefully in complexity and maturity.
tazjin 10y ago
> Making everything asynchronous and detached makes things more complicated, not less It does in languages like JS because they're not built for it. Elixir programs on average don't look very complicated.
- rpazyaquian 10y ago
  I guess I'm just not used to the Erlang/BEAM message-passing/infinite-servers mental model. I'm used to single threaded programs and importing libraries and the like, so I've never had the need to pull off complicated networks.
  - jimbokun 10y ago
    "...so I've never had the need to pull off complicated networks." In other words, you haven't run into one of the main problems Erlang and Elixir aim to solve. So it makes sense you don't see the appeal. Just keep it in the back of your mind, when you someday do need to pull off a complicated network, Erlang and Elixir are excellent tools for solving that problem.
    - brightball 10y ago
      This is actually how I've explained it to a lot of people. Elixir is one of those languages that you won't appreciate it until you suffer the issues of others first. This is how it makes fanboys of people in their 30s, 40s and 50s instead of their 20s.
themartorana 10y ago
I think part of it is that it fits well into the microservices paradigm - where there is a separation of work even within the same program. But that said, for where Erlang shines - backends and distributed services - first-class concurrency is damn near a must-have these days (see: Go, Erlang, etc. recent surge in use and popularity). I ended up with Go, but had I more time to learn Erlang (or Elixir was 1.0 when I started looking) it would have been a solid contender. Also, don't discount the incredibly solid and robust BEAM VM. If you are writing distributed services or servers (see: Riak, RabbitMQ) the overhead for spinning off "instances" or "processes" within the VM is virtually zero-cost, and in production systems has like sixty-four 9s of uptime. [0] [0] ok, nine, but still: https://pragprog.com/articles/erlang https://pragprog.com/articles/erlang
- rpazyaquian 10y ago
  Microservices makes sense, I suppose, but the cases in which distributed services and isolated processes are critical just aren't comment enough in my experience to require sticking specifically to a SOA-like architecture for everything. Especially in a web app like something you'd make in Rails, which already handles exceptions pretty well and rarely require concurrency. The idea of composing individual applications and programs sounds very powerful, and I like the idea of using it as a way to compose programs written in different languages together (a Rails server talking to a Clojure ML service that uses Rust bindings to do hardcore number crunching, for example). That could be interesting. I don't know what I'd use it for, though.
  - andruby 10y ago
    > Especially in a web app like something you'd make in Rails, which [..] rarely requires concurrency. What? If you don't need concurrency in a web app that means that your app can only serve one client at a time and other clients have to wait in line. That won't scale to more than a couple concurrent users. You're right that microservices don't make sense for every web app. Especially when you're still figuring out product-market fit. But once you achieve a certain scale (both in number of users and number of developers) microservices/SOA starts to make a lot more sense. And that's an area where the Erlang VM really shines. Erlang's concurrency model is awesome and fits really well with the hardware evolution to more cores: lightweight processes, immutability, message passing, location transparency, etc.
andy_ppp 10y ago
Immutability stops you from experiencing random state changes. The code you read takes in data structures and spits out data structures making it easy to test things in isolation. OTP is actually a lot lot faster than you'd expect and allows you to use all CPUs very simply and also provides network transparency (on top of supervisors and "let it crash").
rdtsc 10y ago
> is why we would want to program everything as a composition of individual programs and applications, Because it matches the reality better. Concurrent applications built out of individual isolated processes often map well to real world problems. Not all problems, but quite a few of them. Say if you just multiply a matrix or maybe calculate an average, you don't need concurrency and supervision. If you want to sort a list of files, you don't need etc. Not all systems have to be reliable and fault tolerant. > Making everything asynchronous and detached makes things more complicated, not less. The real world is asynchronous and detached. You as an agent are detached from other co-workers or friends. Before you take a breath you don't have to wait for them to finish taking a breath. A car driving down the street is detached from others. They come to an intersection and have to work together in order not to run into each other. But after that they keep going on their way. Also they are fault tolerant -- because your neighbor's car engine crashed, doesn't mean yours should stop working. Seems a bit like a silly example but that maps fairly well to distributed computing -- requests, user connections, work processes, items in a processing pipeline, database shards and so on. So given that there are these types of problem what framework language should you pick to solve it? I would say first of all one which reduces the impedance mismatch and lets your write clearest, simplest code. And also that one that's most fault tolerant. You could spawn threads (or green threads) in some languages -- but you'd probably be sharing memory and have some bugs in there. You could spawn OS processes and/or have multiple containers/vms in the cloud, but that gets expensive. You could use Rust for example to achieve some of those goal though, because it will guarantee to a certain degree during compile time that your application will have less memory and concurrency errors. Or you could use something based on the BEAM VM (Erlang or Elixir) for example, and you get a different approach to fault tolerance there.
weixiyen 10y ago
I think if you are starting from nothing, message passing is actually a more intuitive way to think about and design large distributed programs, because you don't need to deal with a lot of the complexities of OO languages such as state and handling distributed communication / disaster recovery. Running processes in parallel is cheap in Erlang, so you can run hundreds of thousands of them in parallel on a commodity servers. Generally, if you are running any sort of high-traffic service with a lot of dynamic stuff happening, there are probably parts of it that Erlang/OTP can help make more efficient and easier to maintain.
nine_k 10y ago
The key is the ease of writing the target program correctly. You can write any algorithm using any Turing-complete machine, including the original tape-based Turing machine or a language like Malbolge. But it is going to take an inordinate amount of time to write and debug. What FP and message passing buy you is the ease of reasoning. No shared state -> no problem of concurrent updates. No state at all -> no questions like "but does this method update that instance member?". All your state lives in message queues and call stacks, and both provde very stringent and easy-to-understand disciplines. This also gives you composability: you have easy time putting things together and rearranging them as you see fit, without caring of any hidden data dependencies. Everything is explicit. (Combined with a static type checker and a right type system, it gives you the effect of "if it compiles, it runs correctly 99% of times", seen e.g. in Haskell or Rust. Elixir is not a statically-typed language, though.) Yes, it does sometimes come at a cost. Not all immutable data structures can be made as efficient as mutable ones. Usually they are as fast as mutable ones, but pay some space cost; IIRC it's theoretically proven that the cost is no more than a logarithmic one. With RAM being so much cheaper than developers' time, it's a rather good tradeoff. At least, using JIT in Java or Javascript, or just using Python and Ruby, one usually pays a similar or higher RAM cost, for a fraction of benefits. JS does move towards immutability and FP-ness, though, because the above applies to it, too, and it's not so OOP-heavy as for make FP approaches largely impractical.
- hellofunk 10y ago
  > Not all immutable data structures can be made as efficient as mutable ones. Usually they are as fast as mutable ones, I don't think this is true. Clojure uses immutable persistent data structure under the hood, just as most functional languages do, and devs often jump through hoops to circumvent them for performance reasons, opting instead for bit bashing like most other languages.
  - nine_k 10y ago
    An immutable tree is often about as fast as a mutable tree, it's just pointer manipulation in either case. An immutable data structure representing an array is definitely much slower than a mutable array (a contiguous piece of memory). Usually != always, unfortunately :(
mbucc 10y ago
> I hear a lot about OTP and the "let it crash" mantra, but > I just don't quite understand what's so great about it. The top two for me are: (1) you start to just program the sunny path. It's faster and more fun to code in this mode. And with pattern matching of function arguments and return values (as well as guards) and you get pre- and post-condition asserts all while programming the sunny path. (2) What are you supposed to do when your system raises an run-time error anyway? I think in practice, in the majority of cases simply restarting your process in some known state is the best you can do. And that "start in some known state" is non-trivial for a system that handles concurrent inputs. OTP helps you do that correctly. Edit: Formating. Add note pattern matching of function args and guards.
rogerbraun 10y ago
It depends on your use case. The overhead is much smaller than you probably think and the Erlang / Elixir programming model works extremely well for some cases, networking for example. I've been writing pieces for a Bittorrent client in Elixir. Using a process for every connection to another client or for every torrent makes structuring your code really easy.
- pdimitar 10y ago
  Is your project's code public? I'd gladly use it as an educational source since I am learning Elixir right now.
omginternets 10y ago
I don't know if you've read this[0] yet, but I think it does a good job of explaining how the "let it crash" mantra is useful in effect. [0] http://ferd.ca/the-zen-of-erlang.html http://ferd.ca/the-zen-of-erlang.html
- alanning 10y ago
  This exactly. I've read two books about erlang and otp, including Fred's book and it wasn't until reading this that I felt I really "got it". I think this should be the first thing people point to when explaining the let it crash philosophy.
- andrewflnr 10y ago
  That was wonderful. I'd never made the connection between restarting and fixing heisenbugs: bugs that are fixed by restarting are precisely the ones that are hardest to handle otherwise, and vice versa.
- omginternets 10y ago
  I should add that while I've never written a line of Erlang in my life, this article has gotten me to structure my Go programs in a relatively Erlang-esque manner, and I've been quite pleased with the results so far. There are a lot of very good, very simple ideas in the Erlang ecosystem, it seems.
desireco42 10y ago
I can offer you my understanding. I think this way we allow multiple light processes to be created and this should work well with multicore processors. Overhead you are talking about can only be mental as processes in Elixir are very light. I think compared to other similar languages, at least for me Elixir doesn't require as much overthinking. Now, you could also say that I wasn't working on complex apps with it, so that might be why. Anyhow, this is my view.
akeating 10y ago
Elixir and Erlang actually map remarkably well to the _web development_ use case. When every request gets it's own process and shared services are abstracted away, the mental model is quite simple. Use Elixir/ Erlang and you also get to use server push architectures with ease. Want server state to write your own device-sync? Need to write web-hooks? Want to count requests? Need to guarantee uniqueness of a combination of variables in a distributed environment? Want to do server some requests requiring heavy-lifting while also serving short-lived requests with low latency? Choose Elixir / Erlang for web development and as your needs evolve (and they will), you're likely to be surprised and delighted by how much you can achieve before you need to move to a polygot or micro-service-based architecture.
- rpazyaquian 10y ago
  I suppose this means that I've been working on the uninteresting problems in web development (basic CRUD stuff for Rails), and none of the fun networking-related problems, or otherwise. That's pretty disappointing.
  - mpeg 10y ago
    I find elixir incredibly good for CRUD-type applications, macros and pattern matching alone get rid of most of the duplication and hard to follow logic that plagues most web frameworks. Phoenix is quite popular but if you look at the source, it's a testament to Elixir's power as a language how light it actually is, the meatier bits are mostly around working with real-time connections, the cookie-cutter web stuff you can achieve with plug with very little work.
  - bratsche 10y ago
    Even in these cases Erlang/Elixir processes can be a useful way of designing your app. Off the top of my head, here are a couple obvious use cases I can think of: Basic state storage in between requests. Maybe you're building a simple shopping cart application, or maybe something more complex like a SurveyMonkey-style form builder. Each time someone makes a web request that modifies this state, you're storing that state somewhere.. but it's usually somewhere outside of Ruby. Maybe postgresql, maybe redis. If you were using Elixir or Erlang that would still be an option, but another option would be to use a process to store this temporary state. When the user adds things to the cart you send a message to the right process and let it know. This can keep all your code in one system (BEAM), so fewer moving pieces, easy to unit-test the entire thing in ExUnit, and potentially more efficient since there's no I/O to some other database. Then you can continue to save your truly long-term data in postgresql, so for example user accounts and sales made. Another obvious use-case is not needing to have background workers like resque or Sidekiq. When I was doing Rails stuff I found myself doing tons of stuff in Sidekiq workers when I wanted to get the work off the request-response cycle. Sidekiq was great, and it improved things hugely.. but in Elixir I get all that for free with processes. Want to send an email? Have an email-sender process running and just send it a message. Want to do some fancy image processing, or even just thumbnail generation? Just have another process do it asynchronously. And once again, it's all inside the same code base and it's all super easy to unit test. And once again it makes your system simpler because you don't have another system process to deploy and manage.
mavelikara 10y ago
I am currently in the process of learning Erlang, and here are my impressions. In the paper A Note on Distributed Computing, Waldo et al argue that having a distributed object system is bound to fail no matter how hard one tries. Differences in latency, memory access, partial failure, and concurrency make merging of the computational models of local and distributed computing both unwise to attempt and unable to succeed. ... A better approach is to accept that there are irreconcilable differences between local and distributed computing, and to be conscious of those differences at all stages of the design and implementation of distributed applications. Rather than trying to merge local and remote objects, engineers need to be constantly reminded of the differences between the two, and know when it is appropriate to use each kind of object. Erlang works around this problem by making the local object model identical to the remote one in all aspects except latency. The semantic model is copy-on-send asynchronous message-passing. This is inconvenient when working with local components, as you rightly point out. But putting up with this inconvenience up-front makes it easier to refactor towards a distributed system later much easier. The language attempts to work around the inconvenience of asynchronous message-passing using functional abstractions - which is largely what OTP seem to be. BTW, the language can be used in a synchronous context. It works just like another functional language with all code executing from start-to-finish on a single thread of execution. [1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7...
methehack 10y ago
You're right -- there is a conceptual/complexity cost to BEAM applications, distribution, concurrency, supervision trees, etc. I think the idea is that if you start with phoenix you don't have pay that cost unless you need it. If you end up needing to scale, the features are there. I think Uncle Bob said once that "architecture is the art of delaying decisions". Phoenix / Elixir let you do this. Another part of the story is channels. If you're doing websocket work, you can't find a better platform than phoenix / elixir. Once you start digging in, you realize phoenix / elixir are awesome even if you're not using all the concurrency / distribution stuff. The functional nature / immutability posture of elixir helps you with a large code base and/or lots of devs. There are other features -- pattern matching and macros come to mind -- that make Elixir a great place to code. All that said, if you have a client, say, with modest needs that is never going to have much traffic, all that might not be worth going with something you're not familiar with. Similarly, if you're talking about a 1-2 dev project, it might not worth learning something new. In those cases, maybe it's rails or django or whatever ftw.
- rpazyaquian 10y ago
  I am a huge fan of Elixir and Phoenix even without OTP, don't get me wrong. It's my go-to framework now, replacing Rails. I just ended up hitting the OTP chapters in Programming Elixir and Programming Phoenix, and just kind of wondered why I was doing what I was doing, and why anyone would ever have or want to do this. That's been covered pretty well by other commenters, but it seems like it's something you have to experience yourself in order to really understand.
- ams6110 10y ago
  What's the advantage of Elixr/Phoenix over straight Erlang and something like Nitrogen or N2O? I understand that some people (especially those with a Ruby/Rails background) find Elixr/Phoenix more familiar. But is there more going on here than syntactic sugar? If it's ultimately a BEAM application, I'm guessing that Elixr can't do anything that you can't do in Erlang, is that wrong?
  - Tenhundfeld 10y ago
SwellJoe 10y ago
"Maybe it's just due to my problem domain (web development), but it doesn't seem like as big a draw to Elixir and Erlang as pattern matching and FP are." That seems like the perfect domain for this, to me. HTTP is (traditionally) stateless. So, in the past CGI apps were independent entities. Every new hit was a new world in which the app ran. And, every interaction between each CGI app happened via a standard communication medium (a database or writing to files rather than messages, but if you squint and don't look too closely, you might see the analogy there). There are negatives to that...but, there are real benefits, too. That provided a tremendous level of flexibility in how apps were built and in how they interacted with each other. I'd argue that in some regards (though certainly not all), going to an always-on appserver model was a step backward. So, to me, in some ways Elixir looks like CGI, only without most of the negatives of CGI. A lot of the mental overhead, like parsing parameters and setting up database connections, can be abstracted away with a modern/fast language designed just for this kind of work. And, the end result is a robust system without a lot of mental overhead other than learning the abstractions and how they interact. You call out supervisors and message passing and the like as being negatives; complexity you don't want to have to deal with. But, you aren't really comparing apples to apples. Node.js is also asynchronous, and in its bare state is a mess; callbacks are vastly more difficult to interact with and reason about than a system like Erlang/Elixir, and a lot of other languages use the callback model for asynchronous programming. So, I guess what I'm trying to say is that if you're doing asynchronous programming, Elixir is going to be easier than most (though ES6 has concurrency primitives that are starting to make sense, and Python 3 and Ruby have started getting them, Perl 6 has good options, Perl 5 has somewhat clumsy modules for it, etc.). All that said, I'm mostly spending my time learning JavaScript/ES6/Node lately; despite its warts, the ecosystem is huge, and there's so many smart people working on the language that by the time of ES7, there will be a very convincing concurrency story. Maybe still not as convincing as Erlang/Elixir and the OTP. But, probably good enough for web apps and services. I looked into Elixir and Phoenix briefly; watched a few videos, read a few tutorials, never wrote any code. And, came away with a distinct impression of a tiny, tiny, ecosystem. I'm accustomed to going to CPAN or Ruby Gems or PyPI or npm, and finding not just one, but several options for whatever task I want to accomplish, even relatively obscure stuff. It'll be a while before Elixir comes close to even CPAN (which is smaller than all the others these days, though still manages to have modules for nearly every problem I tackle).
pmarreck 10y ago
Let's try a different tack. https://speakerdeck.com/mosic/elixir-at-evercam?slide=6 https://speakerdeck.com/mosic/elixir-at-evercam?slide=6 Make more sense now? :) (in addition to the other responses) tl;dr Evercam replaced Node.js, Sidekiq, Pusher and Upstart with Elixir, huge net gain in stack simplicity which is arguably a strong plus from a long-term maintenance perspective.
- pdimitar 10y ago
  It'd be immensely more helpful if you give Elixir library/framework names as alternatives to the technology names you enumerated. In textual form. In fact, Mike Perham himself (the creator of Sidekiq) is asking the question how do you do scheduled future background processing in Elixir, right in this thread: https://news.ycombinator.com/item?id=12072359 https://news.ycombinator.com/item?id=12072359
  - pmarreck 10y ago
    I am not the originator of the slides; perhaps someone from Evercam will respond. I will try to hail them on Twitter.
  - shadeless 10y ago
    There was a post about their usage of Elixir on the ElixirForum recently: https://elixirforum.com/t/complex-open-source-phoenix-apps-to-learn-from/999/2 https://elixirforum.com/t/complex-open-source-phoenix-apps-t...
hosh 10y ago
You don't need to start with supervisors, application trees and message passing for the initial MVP. However, let's take Ruby on Rails as an example. I have gotten MVP-quality software out in production. We can quickly validate whether this is something customers want. However, when the app starts getting traction, I have found that I need to reach for things outside a RoR monolith. Usually, the first thing is getting some kind of background processing job going, with Sidekiq and Resque. At this point, we have shifted from a synchronous, no-shared-state architecture to one that requires background processing. This is where we start thinking in terms of concurrency and how to deal with it. Since Rails is built on top of Ruby, Ruby does not have a sufficiently robust concurrency primitives that work well at scale. To use something like Sidekiq, we rely on using Redis. I have had projects where I need to pass data through a series of background jobs, like a pipeline. Now I am implementing what are effectively mutexes as a database column. I'm processing data that are effectively being streamed from third parties, so each individual Sidekiq job now has to have guards so that it can be idempotent. I have to be mindful of queues, analyzing where the bottlenecks, and how to structure the queues and tune how many of what kind of jobs should get resources. I have had to rely on outside monitoring solutions to make sure these processes stay up and running, and write things in a way so they can still recover on restart. I had to fork Sidekiq and rewrite core parts of it in order to create a different set of guarantees to fit a use-case -- a big deal to every Rubyist I had talked to, and yet, seems normal in the Erlang world. In other words, in the Rails world, concurrency is handled in a very coarse-grained way, often relying on software outside of Ruby (such as Redis). I found myself reinventing, in a very crude ways at a coarse-grained level, what OTP already offers. I adopted subversion and later git because I was crudely reinventing version control (shell script and tar); I dropped PHP and Perl CGI scripts in favor of Rails back when Rails was version 1.1 because I knew what a project that doesn't use what Rails offers would look like (a messy puddle). And likewise, now that I've found myself badly reinventing ideas OTP already has, it's the tool I'll reach for now. With Erlang and Elixir, it is easier to reason with concurrency because the primitives are baked in deep, and it costs little to spin up something asynchronously. Reasoning with concurrency in the Rails world is treated more like black magic -- obscure, non-obvious, forbidden, potentially dangerous, and socially unacceptable.
- brightball 10y ago

tim333 10y ago

The main argument why seems to be for reliability so it can run for months without going down.

rpazyaquian 10y ago
I understand the need for reliability, i.e. "come back up if you get bad data so it doesn't just die on you", but this seems like something that dates back from when any kind of raised exception would bring the entire system down. Don't we have modern frameworks and systems now that can tolerate individual failures instead of completely dying just because someone tried to POST some bad data? Plus, having reliability and coming back up if it goes down doesn't solve the root problem, which is that there's a bug in the code, or some sort of unhandled edge case. 99.9999999% uptime doesn't help if your code doesn't work right.
- dottedmag 10y ago
  Imagine there is a bug in the code or unhandled edge case, only exhibited on 0.01% transactions. Elixir/OTP handler dies on this edge case, supervisor spawns another worker, 99.99% of user transactions continue to be served. Programmer wakes up next morning, comes to work, reads the exception log, fixes this edge case, optionally hot-upgrades the code in production if it's easier than doing a full restart, problem solved. Other frameworks have to carefully implement the same semantics (that is, prevention of propagation of bad state/cascading failures). Elixir/OTP/Erlang come with it by default, and coding style encourages it.
- rdtsc 10y ago
  > Plus, having reliability and coming back up if it goes down doesn't solve the root problem, which is that there's a bug in the code, or some sort of unhandled edge case. 99.9999999% uptime doesn't help if your code doesn't work right. Yeah but you don't have to wake up at 4am in the morning to fix the problem if the system stays up and still works (even if a bit slower or some feature is degraded). I've seen systems that had parts crashing and restarting for a weeks and months. Yeah it is bad nobody noticed, but it was secondary service that didn't affect most users. Having the backend with an exception or segfault because if it would not make sense. > Don't we have modern frameworks and systems now that can tolerate individual failures instead of completely dying just because someone tried to POST some bad data? Yeah we do. Nothing Elixir does is magic or absolutely impossible with other systems. You can spawn an OS process to handle a request. Can have a load balancer or something to check the health of backend nodes, and redirect. Or wrap everything in exceptions and try to restart and so on. It is kind of doable. But now there are multiple libraries, frameworks, services to also maintain and look after, maybe that thread just crashed made a mess on the heap and now maybe restarting don't really fix the problem. So doable, but more awkward. > Plus, having reliability and coming back up if it goes down doesn't solve the root problem, Obviously, nothing will write the code to fix the problems except the programmer. Is it worth to completely crash the system because some bug in one thread or some unimportant component failed? It is a bit like having a human drop dead the first time they scratched their finger. Yeah they can't write as fast for a while, but they can still largely function. I mentioned Rust in another comment. Perhaps you'd prefer to have the compiler check correctness, that's very reasonable, something like Rust or Haskell might be what you'd like to use to sort of ensure you have less bugs when the system runs. Higher level of proof of correctness are possible, that is done for avionics software, crypto and other life-critical systems. So it is a continuum there. But there usually dollar signs attached to it.
- biot 10y ago
  The supervisor is what is responsible for restarting the worker or not. If you have a process that saves files to local disk and the disk is full, causing the worker to crash, the supervisor can recognize that reason and conclude that the worker process should not be restarted. Additionally, the supervisor can take other actions like informing the rest of the system and/or potentially even spawning a different worker process that stores files to a slower, more expensive cloud storage option.
- ramchip 10y ago
  My personal anecdote to illustrate the difference: I've worked on a python app before which had a bug that sometimes made the websocket closed callback throw an exception. The exception was still being caught by the framework, so the system kept working fine. ...or did it? Eventually, we noticed instances of the app locking up, not responding to any request. There was no obvious error, it just kept running but did not log or do anything. It turned out to be due to the callback bug: the exception was being caught alright, but we were leaking file descriptors like a sieve and eventually running out. Due to this a minor bug affecting only some requests was able to take down the entire app. That's the kind of problem Erlang solves well. Every request runs in its own lightweight process. The process can open (and thus own) files, sockets, OS processes, in-memory DB tables, etc. If the process crashes or dies for any reason, its memory and any other resource it owned gets cleaned up by the VM. If the process tries to hog the CPU or even enters an infinite loop, the preemptive scheduling means other requests will still get a chance to run. In short: having strong isolation between processes means it's unlikely for a bug in a minor feature somewhere to affect the overall application health.
- hosh 10y ago
  I think you are misunderstanding something. "Let it crash" doesn't mean crash on everything. It's about handling the _unexpected_. Someone posting bad data may or may not be expected. The primitives are such that you can _choose_ how much or how little supervision you want, for a given complexity or maturity of a project, or what you learned as users put the system through the wringer. This is a _choice_ you make as an engineer, and OTP makes it easier for you to make such a tradeoff. Other systems and framework do not necessarily let you make such a tradeoff as gracefully. This not only allows the software to scale gracefully with the number of users, this allows the software to evolve gracefully in complexity and maturity.

tazjin 10y ago

> Making everything asynchronous and detached makes things more complicated, not less It does in languages like JS because they're not built for it. Elixir programs on average don't look very complicated.

rpazyaquian 10y ago
I guess I'm just not used to the Erlang/BEAM message-passing/infinite-servers mental model. I'm used to single threaded programs and importing libraries and the like, so I've never had the need to pull off complicated networks.
- jimbokun 10y ago
  "...so I've never had the need to pull off complicated networks." In other words, you haven't run into one of the main problems Erlang and Elixir aim to solve. So it makes sense you don't see the appeal. Just keep it in the back of your mind, when you someday do need to pull off a complicated network, Erlang and Elixir are excellent tools for solving that problem.
  - brightball 10y ago
    This is actually how I've explained it to a lot of people. Elixir is one of those languages that you won't appreciate it until you suffer the issues of others first. This is how it makes fanboys of people in their 30s, 40s and 50s instead of their 20s.

themartorana 10y ago

I think part of it is that it fits well into the microservices paradigm - where there is a separation of work even within the same program. But that said, for where Erlang shines - backends and distributed services - first-class concurrency is damn near a must-have these days (see: Go, Erlang, etc. recent surge in use and popularity). I ended up with Go, but had I more time to learn Erlang (or Elixir was 1.0 when I started looking) it would have been a solid contender. Also, don't discount the incredibly solid and robust BEAM VM. If you are writing distributed services or servers (see: Riak, RabbitMQ) the overhead for spinning off "instances" or "processes" within the VM is virtually zero-cost, and in production systems has like sixty-four 9s of uptime. [0] [0] ok, nine, but still: https://pragprog.com/articles/erlang https://pragprog.com/articles/erlang

rpazyaquian 10y ago
Microservices makes sense, I suppose, but the cases in which distributed services and isolated processes are critical just aren't comment enough in my experience to require sticking specifically to a SOA-like architecture for everything. Especially in a web app like something you'd make in Rails, which already handles exceptions pretty well and rarely require concurrency. The idea of composing individual applications and programs sounds very powerful, and I like the idea of using it as a way to compose programs written in different languages together (a Rails server talking to a Clojure ML service that uses Rust bindings to do hardcore number crunching, for example). That could be interesting. I don't know what I'd use it for, though.
- andruby 10y ago
  > Especially in a web app like something you'd make in Rails, which [..] rarely requires concurrency. What? If you don't need concurrency in a web app that means that your app can only serve one client at a time and other clients have to wait in line. That won't scale to more than a couple concurrent users. You're right that microservices don't make sense for every web app. Especially when you're still figuring out product-market fit. But once you achieve a certain scale (both in number of users and number of developers) microservices/SOA starts to make a lot more sense. And that's an area where the Erlang VM really shines. Erlang's concurrency model is awesome and fits really well with the hardware evolution to more cores: lightweight processes, immutability, message passing, location transparency, etc.

andy_ppp 10y ago

Immutability stops you from experiencing random state changes. The code you read takes in data structures and spits out data structures making it easy to test things in isolation. OTP is actually a lot lot faster than you'd expect and allows you to use all CPUs very simply and also provides network transparency (on top of supervisors and "let it crash").

rdtsc 10y ago

> is why we would want to program everything as a composition of individual programs and applications, Because it matches the reality better. Concurrent applications built out of individual isolated processes often map well to real world problems. Not all problems, but quite a few of them. Say if you just multiply a matrix or maybe calculate an average, you don't need concurrency and supervision. If you want to sort a list of files, you don't need etc. Not all systems have to be reliable and fault tolerant. > Making everything asynchronous and detached makes things more complicated, not less. The real world is asynchronous and detached. You as an agent are detached from other co-workers or friends. Before you take a breath you don't have to wait for them to finish taking a breath. A car driving down the street is detached from others. They come to an intersection and have to work together in order not to run into each other. But after that they keep going on their way. Also they are fault tolerant -- because your neighbor's car engine crashed, doesn't mean yours should stop working. Seems a bit like a silly example but that maps fairly well to distributed computing -- requests, user connections, work processes, items in a processing pipeline, database shards and so on. So given that there are these types of problem what framework language should you pick to solve it? I would say first of all one which reduces the impedance mismatch and lets your write clearest, simplest code. And also that one that's most fault tolerant. You could spawn threads (or green threads) in some languages -- but you'd probably be sharing memory and have some bugs in there. You could spawn OS processes and/or have multiple containers/vms in the cloud, but that gets expensive. You could use Rust for example to achieve some of those goal though, because it will guarantee to a certain degree during compile time that your application will have less memory and concurrency errors. Or you could use something based on the BEAM VM (Erlang or Elixir) for example, and you get a different approach to fault tolerance there.

weixiyen 10y ago

I think if you are starting from nothing, message passing is actually a more intuitive way to think about and design large distributed programs, because you don't need to deal with a lot of the complexities of OO languages such as state and handling distributed communication / disaster recovery. Running processes in parallel is cheap in Erlang, so you can run hundreds of thousands of them in parallel on a commodity servers. Generally, if you are running any sort of high-traffic service with a lot of dynamic stuff happening, there are probably parts of it that Erlang/OTP can help make more efficient and easier to maintain.

nine_k 10y ago

The key is the ease of writing the target program correctly. You can write any algorithm using any Turing-complete machine, including the original tape-based Turing machine or a language like Malbolge. But it is going to take an inordinate amount of time to write and debug. What FP and message passing buy you is the ease of reasoning. No shared state -> no problem of concurrent updates. No state at all -> no questions like "but does this method update that instance member?". All your state lives in message queues and call stacks, and both provde very stringent and easy-to-understand disciplines. This also gives you composability: you have easy time putting things together and rearranging them as you see fit, without caring of any hidden data dependencies. Everything is explicit. (Combined with a static type checker and a right type system, it gives you the effect of "if it compiles, it runs correctly 99% of times", seen e.g. in Haskell or Rust. Elixir is not a statically-typed language, though.) Yes, it does sometimes come at a cost. Not all immutable data structures can be made as efficient as mutable ones. Usually they are as fast as mutable ones, but pay some space cost; IIRC it's theoretically proven that the cost is no more than a logarithmic one. With RAM being so much cheaper than developers' time, it's a rather good tradeoff. At least, using JIT in Java or Javascript, or just using Python and Ruby, one usually pays a similar or higher RAM cost, for a fraction of benefits. JS does move towards immutability and FP-ness, though, because the above applies to it, too, and it's not so OOP-heavy as for make FP approaches largely impractical.

hellofunk 10y ago
> Not all immutable data structures can be made as efficient as mutable ones. Usually they are as fast as mutable ones, I don't think this is true. Clojure uses immutable persistent data structure under the hood, just as most functional languages do, and devs often jump through hoops to circumvent them for performance reasons, opting instead for bit bashing like most other languages.
- nine_k 10y ago
  An immutable tree is often about as fast as a mutable tree, it's just pointer manipulation in either case. An immutable data structure representing an array is definitely much slower than a mutable array (a contiguous piece of memory). Usually != always, unfortunately :(

mbucc 10y ago

> I hear a lot about OTP and the "let it crash" mantra, but > I just don't quite understand what's so great about it. The top two for me are: (1) you start to just program the sunny path. It's faster and more fun to code in this mode. And with pattern matching of function arguments and return values (as well as guards) and you get pre- and post-condition asserts all while programming the sunny path. (2) What are you supposed to do when your system raises an run-time error anyway? I think in practice, in the majority of cases simply restarting your process in some known state is the best you can do. And that "start in some known state" is non-trivial for a system that handles concurrent inputs. OTP helps you do that correctly. Edit: Formating. Add note pattern matching of function args and guards.

rogerbraun 10y ago

It depends on your use case. The overhead is much smaller than you probably think and the Erlang / Elixir programming model works extremely well for some cases, networking for example. I've been writing pieces for a Bittorrent client in Elixir. Using a process for every connection to another client or for every torrent makes structuring your code really easy.

pdimitar 10y ago
Is your project's code public? I'd gladly use it as an educational source since I am learning Elixir right now.

omginternets 10y ago

I don't know if you've read this[0] yet, but I think it does a good job of explaining how the "let it crash" mantra is useful in effect. [0] http://ferd.ca/the-zen-of-erlang.html http://ferd.ca/the-zen-of-erlang.html

alanning 10y ago
This exactly. I've read two books about erlang and otp, including Fred's book and it wasn't until reading this that I felt I really "got it". I think this should be the first thing people point to when explaining the let it crash philosophy.
andrewflnr 10y ago
That was wonderful. I'd never made the connection between restarting and fixing heisenbugs: bugs that are fixed by restarting are precisely the ones that are hardest to handle otherwise, and vice versa.
omginternets 10y ago
I should add that while I've never written a line of Erlang in my life, this article has gotten me to structure my Go programs in a relatively Erlang-esque manner, and I've been quite pleased with the results so far. There are a lot of very good, very simple ideas in the Erlang ecosystem, it seems.

desireco42 10y ago

I can offer you my understanding. I think this way we allow multiple light processes to be created and this should work well with multicore processors. Overhead you are talking about can only be mental as processes in Elixir are very light. I think compared to other similar languages, at least for me Elixir doesn't require as much overthinking. Now, you could also say that I wasn't working on complex apps with it, so that might be why. Anyhow, this is my view.

akeating 10y ago

Elixir and Erlang actually map remarkably well to the _web development_ use case. When every request gets it's own process and shared services are abstracted away, the mental model is quite simple. Use Elixir/ Erlang and you also get to use server push architectures with ease. Want server state to write your own device-sync? Need to write web-hooks? Want to count requests? Need to guarantee uniqueness of a combination of variables in a distributed environment? Want to do server some requests requiring heavy-lifting while also serving short-lived requests with low latency? Choose Elixir / Erlang for web development and as your needs evolve (and they will), you're likely to be surprised and delighted by how much you can achieve before you need to move to a polygot or micro-service-based architecture.

rpazyaquian 10y ago
I suppose this means that I've been working on the uninteresting problems in web development (basic CRUD stuff for Rails), and none of the fun networking-related problems, or otherwise. That's pretty disappointing.
- mpeg 10y ago
  I find elixir incredibly good for CRUD-type applications, macros and pattern matching alone get rid of most of the duplication and hard to follow logic that plagues most web frameworks. Phoenix is quite popular but if you look at the source, it's a testament to Elixir's power as a language how light it actually is, the meatier bits are mostly around working with real-time connections, the cookie-cutter web stuff you can achieve with plug with very little work.
- bratsche 10y ago
  Even in these cases Erlang/Elixir processes can be a useful way of designing your app. Off the top of my head, here are a couple obvious use cases I can think of: Basic state storage in between requests. Maybe you're building a simple shopping cart application, or maybe something more complex like a SurveyMonkey-style form builder. Each time someone makes a web request that modifies this state, you're storing that state somewhere.. but it's usually somewhere outside of Ruby. Maybe postgresql, maybe redis. If you were using Elixir or Erlang that would still be an option, but another option would be to use a process to store this temporary state. When the user adds things to the cart you send a message to the right process and let it know. This can keep all your code in one system (BEAM), so fewer moving pieces, easy to unit-test the entire thing in ExUnit, and potentially more efficient since there's no I/O to some other database. Then you can continue to save your truly long-term data in postgresql, so for example user accounts and sales made. Another obvious use-case is not needing to have background workers like resque or Sidekiq. When I was doing Rails stuff I found myself doing tons of stuff in Sidekiq workers when I wanted to get the work off the request-response cycle. Sidekiq was great, and it improved things hugely.. but in Elixir I get all that for free with processes. Want to send an email? Have an email-sender process running and just send it a message. Want to do some fancy image processing, or even just thumbnail generation? Just have another process do it asynchronously. And once again, it's all inside the same code base and it's all super easy to unit test. And once again it makes your system simpler because you don't have another system process to deploy and manage.

mavelikara 10y ago

I am currently in the process of learning Erlang, and here are my impressions. In the paper A Note on Distributed Computing, Waldo et al argue that having a distributed object system is bound to fail no matter how hard one tries. Differences in latency, memory access, partial failure, and concurrency make merging of the computational models of local and distributed computing both unwise to attempt and unable to succeed. ... A better approach is to accept that there are irreconcilable differences between local and distributed computing, and to be conscious of those differences at all stages of the design and implementation of distributed applications. Rather than trying to merge local and remote objects, engineers need to be constantly reminded of the differences between the two, and know when it is appropriate to use each kind of object. Erlang works around this problem by making the local object model identical to the remote one in all aspects except latency. The semantic model is copy-on-send asynchronous message-passing. This is inconvenient when working with local components, as you rightly point out. But putting up with this inconvenience up-front makes it easier to refactor towards a distributed system later much easier. The language attempts to work around the inconvenience of asynchronous message-passing using functional abstractions - which is largely what OTP seem to be. BTW, the language can be used in a synchronous context. It works just like another functional language with all code executing from start-to-finish on a single thread of execution. [1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7...

methehack 10y ago

You're right -- there is a conceptual/complexity cost to BEAM applications, distribution, concurrency, supervision trees, etc. I think the idea is that if you start with phoenix you don't have pay that cost unless you need it. If you end up needing to scale, the features are there. I think Uncle Bob said once that "architecture is the art of delaying decisions". Phoenix / Elixir let you do this. Another part of the story is channels. If you're doing websocket work, you can't find a better platform than phoenix / elixir. Once you start digging in, you realize phoenix / elixir are awesome even if you're not using all the concurrency / distribution stuff. The functional nature / immutability posture of elixir helps you with a large code base and/or lots of devs. There are other features -- pattern matching and macros come to mind -- that make Elixir a great place to code. All that said, if you have a client, say, with modest needs that is never going to have much traffic, all that might not be worth going with something you're not familiar with. Similarly, if you're talking about a 1-2 dev project, it might not worth learning something new. In those cases, maybe it's rails or django or whatever ftw.

rpazyaquian 10y ago
I am a huge fan of Elixir and Phoenix even without OTP, don't get me wrong. It's my go-to framework now, replacing Rails. I just ended up hitting the OTP chapters in Programming Elixir and Programming Phoenix, and just kind of wondered why I was doing what I was doing, and why anyone would ever have or want to do this. That's been covered pretty well by other commenters, but it seems like it's something you have to experience yourself in order to really understand.
ams6110 10y ago
What's the advantage of Elixr/Phoenix over straight Erlang and something like Nitrogen or N2O? I understand that some people (especially those with a Ruby/Rails background) find Elixr/Phoenix more familiar. But is there more going on here than syntactic sugar? If it's ultimately a BEAM application, I'm guessing that Elixr can't do anything that you can't do in Erlang, is that wrong?
- Tenhundfeld 10y ago

SwellJoe 10y ago

"Maybe it's just due to my problem domain (web development), but it doesn't seem like as big a draw to Elixir and Erlang as pattern matching and FP are." That seems like the perfect domain for this, to me. HTTP is (traditionally) stateless. So, in the past CGI apps were independent entities. Every new hit was a new world in which the app ran. And, every interaction between each CGI app happened via a standard communication medium (a database or writing to files rather than messages, but if you squint and don't look too closely, you might see the analogy there). There are negatives to that...but, there are real benefits, too. That provided a tremendous level of flexibility in how apps were built and in how they interacted with each other. I'd argue that in some regards (though certainly not all), going to an always-on appserver model was a step backward. So, to me, in some ways Elixir looks like CGI, only without most of the negatives of CGI. A lot of the mental overhead, like parsing parameters and setting up database connections, can be abstracted away with a modern/fast language designed just for this kind of work. And, the end result is a robust system without a lot of mental overhead other than learning the abstractions and how they interact. You call out supervisors and message passing and the like as being negatives; complexity you don't want to have to deal with. But, you aren't really comparing apples to apples. Node.js is also asynchronous, and in its bare state is a mess; callbacks are vastly more difficult to interact with and reason about than a system like Erlang/Elixir, and a lot of other languages use the callback model for asynchronous programming. So, I guess what I'm trying to say is that if you're doing asynchronous programming, Elixir is going to be easier than most (though ES6 has concurrency primitives that are starting to make sense, and Python 3 and Ruby have started getting them, Perl 6 has good options, Perl 5 has somewhat clumsy modules for it, etc.). All that said, I'm mostly spending my time learning JavaScript/ES6/Node lately; despite its warts, the ecosystem is huge, and there's so many smart people working on the language that by the time of ES7, there will be a very convincing concurrency story. Maybe still not as convincing as Erlang/Elixir and the OTP. But, probably good enough for web apps and services. I looked into Elixir and Phoenix briefly; watched a few videos, read a few tutorials, never wrote any code. And, came away with a distinct impression of a tiny, tiny, ecosystem. I'm accustomed to going to CPAN or Ruby Gems or PyPI or npm, and finding not just one, but several options for whatever task I want to accomplish, even relatively obscure stuff. It'll be a while before Elixir comes close to even CPAN (which is smaller than all the others these days, though still manages to have modules for nearly every problem I tackle).

pmarreck 10y ago

Let's try a different tack. https://speakerdeck.com/mosic/elixir-at-evercam?slide=6 https://speakerdeck.com/mosic/elixir-at-evercam?slide=6 Make more sense now? :) (in addition to the other responses) tl;dr Evercam replaced Node.js, Sidekiq, Pusher and Upstart with Elixir, huge net gain in stack simplicity which is arguably a strong plus from a long-term maintenance perspective.

pdimitar 10y ago
It'd be immensely more helpful if you give Elixir library/framework names as alternatives to the technology names you enumerated. In textual form. In fact, Mike Perham himself (the creator of Sidekiq) is asking the question how do you do scheduled future background processing in Elixir, right in this thread: https://news.ycombinator.com/item?id=12072359 https://news.ycombinator.com/item?id=12072359
- pmarreck 10y ago
  I am not the originator of the slides; perhaps someone from Evercam will respond. I will try to hail them on Twitter.
- shadeless 10y ago
  There was a post about their usage of Elixir on the ElixirForum recently: https://elixirforum.com/t/complex-open-source-phoenix-apps-to-learn-from/999/2 https://elixirforum.com/t/complex-open-source-phoenix-apps-t...

hosh 10y ago

You don't need to start with supervisors, application trees and message passing for the initial MVP. However, let's take Ruby on Rails as an example. I have gotten MVP-quality software out in production. We can quickly validate whether this is something customers want. However, when the app starts getting traction, I have found that I need to reach for things outside a RoR monolith. Usually, the first thing is getting some kind of background processing job going, with Sidekiq and Resque. At this point, we have shifted from a synchronous, no-shared-state architecture to one that requires background processing. This is where we start thinking in terms of concurrency and how to deal with it. Since Rails is built on top of Ruby, Ruby does not have a sufficiently robust concurrency primitives that work well at scale. To use something like Sidekiq, we rely on using Redis. I have had projects where I need to pass data through a series of background jobs, like a pipeline. Now I am implementing what are effectively mutexes as a database column. I'm processing data that are effectively being streamed from third parties, so each individual Sidekiq job now has to have guards so that it can be idempotent. I have to be mindful of queues, analyzing where the bottlenecks, and how to structure the queues and tune how many of what kind of jobs should get resources. I have had to rely on outside monitoring solutions to make sure these processes stay up and running, and write things in a way so they can still recover on restart. I had to fork Sidekiq and rewrite core parts of it in order to create a different set of guarantees to fit a use-case -- a big deal to every Rubyist I had talked to, and yet, seems normal in the Erlang world. In other words, in the Rails world, concurrency is handled in a very coarse-grained way, often relying on software outside of Ruby (such as Redis). I found myself reinventing, in a very crude ways at a coarse-grained level, what OTP already offers. I adopted subversion and later git because I was crudely reinventing version control (shell script and tar); I dropped PHP and Perl CGI scripts in favor of Rails back when Rails was version 1.1 because I knew what a project that doesn't use what Rails offers would look like (a messy puddle). And likewise, now that I've found myself badly reinventing ideas OTP already has, it's the tool I'll reach for now. With Erlang and Elixir, it is easier to reason with concurrency because the primitives are baked in deep, and it costs little to spin up something asynchronously. Reasoning with concurrency in the Rails world is treated more like black magic -- obscure, non-obvious, forbidden, potentially dangerous, and socially unacceptable.

brightball 10y ago