6 ms·

From the author of this post, here's one of the best demos of Elixir, Erlang and modern distributed computing in general: "The Soul of Erlang and Elixir • Saša

by sph 3y ago

From the author of this post, here's one of the best demos of Elixir, Erlang and modern distributed computing in general: "The Soul of Erlang and Elixir • Saša Jurić • GOTO 2019" - https://www.youtube.com/watch?v=JvBT4XBdoUE https://www.youtube.com/watch?v=JvBT4XBdoUE When you feel sufficiently amazed by this demo, I also recommend the Elixir in Action book, by this same author, to get started with this incredible ecosystem and paradigm. --- It's hard to imagine a better platform than Erlang's for writing distributed and networked systems of any kind. I fell in love with Elixir in 2016 and have been using it full time since, first adopting as CTO in my previous company, now powering my new solo business. It's such a good language, with great community and stewardship, running on a rock solid platform that's so advanced compared to most popular languages today. Erlang was created 38 years ago, and it's good to see the ideas of Armstrong, Virding and Williams to be vindicated today. They simply were too far ahead of their time.

arrowsmith 3y ago
I'm working through Elixir in Action now - it's fantastic. I could have saved a lot of time and effort if I'd read it years ago.
cproctor 3y ago
A great talk, but that cameraperson got a workout!
jefc1111 3y ago
I read Elixir in Action which was a great intro, but it was Concurrent Data Processing in Elixir [1] that really made the penny drop for me. I have only finished one Elixir app. It was a lot fun and the app has proven rock solid. For me it felt a bit like using microservices but with lots of the footguns and ops overhead removed. I would love to use it for more than a hobby project. [1] https://pragprog.com/titles/sgdpelixir/concurrent-data-processing-in-elixir/ https://pragprog.com/titles/sgdpelixir/concurrent-data-proce...
cgopalan 3y ago
Thanks for the talk link, it completely kept me engrossed and was a pleasure to watch! Addendum question: what is the state of database drivers for Elixir? Does it have mature libraries for Postgres for eg? I imagine if it's targeting web development, it will need those.
- mike1o1 3y ago
  Elixir has Ecto, which is not-quite an ORM but is the de-facto standard for database access in Elixir/Phoenix. https://hexdocs.pm/ecto/Ecto.html https://hexdocs.pm/ecto/Ecto.html
  - cgopalan 3y ago
    Thank you!
bhupesh 3y ago
Soul of erlang got me hooked to Elixir recently, trying to get my hands dirty as well. Other than distributed/concurrent system use-cases, could you share what kind of products are best when built with elixir/erlang compared to easier to write languages like Go, for example.
- sph 3y ago
  Anything that spends most of its time waiting on IO, and doesn't do a great deal of number crunching: i.e. any kind of server talking over a network socket. Go, and any other language, lowers the barrier to running concurrent code, but there is much more to concurrent servers than concurrency: fault tolerance, isolation, shared state management, instrumentation, introspection, clustering, process migration. The BEAM and its ecosystem gives you all of that out of the box. Also, the BEAM offers an immutable, functional environment. Data races are impossible, which are the biggest pain and source of heisenbugs in any kind of system with > 1 concurrent thread. This is huge. You can model your entire system as concurrent processes without ever having to deal with concurrency issues. You only ever have to think in "single-threaded" mode.
  - dpflan 3y ago
    What language and programming style did you use prior to using Elixir the most? Have you been able to build systems of similar or even larger size as previous projects in other languages, and what was that experience like (i.e. how did Elixir make things better)?
    - sph 3y ago
      I've been doing this job for 20 years. I am fluent in C, Go, Rust, Python, as well as Elixir, and for my clients I have written concurrent systems running in production in all of these languages. I know how painful and complex the problem space is. My opinion is not that rare, so instead of listing my credentials, you can search and find many others that have found this platform to be a great fit, simply because it has been designed to solve this very problem since 1986 when everybody else was focusing on single core, isolated systems.
williamdclt 3y ago
I’m an elixir noob, but I’m actually surprised that the distributed story is not _better_. For example, a big thing about OTP is that it allows running many processes that communicate by message passing: but all these processes are _colocated_ to a given node, i would have expected that OTP (or genserver or whatever) handles clustering of machines, scheduling of workloads and routing of messages for me so that I can start processes and call them without caring about where they run. What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)
- davydog187 3y ago
  I think you need to go deeper, because it does exactly all the things you’re describing out of the box. A process can be anywhere in the cluster, and you can interact with it as if it was on your local node. I use all of these facilities in production on my product with ease. I’m happy to answer your specific questions.
- jorams 3y ago
  Erlang/Elixir nodes can actually be rather trivially clustered, you can spawn processes on other nodes and transparently pass messages to processes on other nodes, including things like closures. It does not magically schedule workloads across the cluster, but there are libraries that can do so.
- sb8244 3y ago
  You can message pass across a cluster. In fact, it's identical code to message a remote vs local process. Erlang provides "process groups" that allow you to structure groups of processes across a cluster. It doesn't try to be magic. It's not a workload scheduler or "automatic distributed machine". You structure things the way you want, on top of it's primitives. It's just that all of the primitives you need are built in and work really well.
- brightball 3y ago
  It does all of that. The structure of the system is such that you can call any function on any node in the cluster. In order to do it well, you need to understand your own abstractions. If we have a cluster are we better off passing every function around the cluster or running some locally? Is the network overhead worth it in every case or only in some? It's up to your code to make those determinations about your application, but everything you describe is fairly simple to implement on the BEAM.

by sph 3y ago

arrowsmith 3y ago

I'm working through Elixir in Action now - it's fantastic. I could have saved a lot of time and effort if I'd read it years ago.

cproctor 3y ago

A great talk, but that cameraperson got a workout!

jefc1111 3y ago

I read Elixir in Action which was a great intro, but it was Concurrent Data Processing in Elixir [1] that really made the penny drop for me. I have only finished one Elixir app. It was a lot fun and the app has proven rock solid. For me it felt a bit like using microservices but with lots of the footguns and ops overhead removed. I would love to use it for more than a hobby project. [1] https://pragprog.com/titles/sgdpelixir/concurrent-data-processing-in-elixir/ https://pragprog.com/titles/sgdpelixir/concurrent-data-proce...

cgopalan 3y ago

Thanks for the talk link, it completely kept me engrossed and was a pleasure to watch! Addendum question: what is the state of database drivers for Elixir? Does it have mature libraries for Postgres for eg? I imagine if it's targeting web development, it will need those.

mike1o1 3y ago
Elixir has Ecto, which is not-quite an ORM but is the de-facto standard for database access in Elixir/Phoenix. https://hexdocs.pm/ecto/Ecto.html https://hexdocs.pm/ecto/Ecto.html
- cgopalan 3y ago
  Thank you!

bhupesh 3y ago

Soul of erlang got me hooked to Elixir recently, trying to get my hands dirty as well. Other than distributed/concurrent system use-cases, could you share what kind of products are best when built with elixir/erlang compared to easier to write languages like Go, for example.

sph 3y ago
Anything that spends most of its time waiting on IO, and doesn't do a great deal of number crunching: i.e. any kind of server talking over a network socket. Go, and any other language, lowers the barrier to running concurrent code, but there is much more to concurrent servers than concurrency: fault tolerance, isolation, shared state management, instrumentation, introspection, clustering, process migration. The BEAM and its ecosystem gives you all of that out of the box. Also, the BEAM offers an immutable, functional environment. Data races are impossible, which are the biggest pain and source of heisenbugs in any kind of system with > 1 concurrent thread. This is huge. You can model your entire system as concurrent processes without ever having to deal with concurrency issues. You only ever have to think in "single-threaded" mode.
- dpflan 3y ago
  What language and programming style did you use prior to using Elixir the most? Have you been able to build systems of similar or even larger size as previous projects in other languages, and what was that experience like (i.e. how did Elixir make things better)?
  - sph 3y ago
    I've been doing this job for 20 years. I am fluent in C, Go, Rust, Python, as well as Elixir, and for my clients I have written concurrent systems running in production in all of these languages. I know how painful and complex the problem space is. My opinion is not that rare, so instead of listing my credentials, you can search and find many others that have found this platform to be a great fit, simply because it has been designed to solve this very problem since 1986 when everybody else was focusing on single core, isolated systems.

williamdclt 3y ago

I’m an elixir noob, but I’m actually surprised that the distributed story is not _better_. For example, a big thing about OTP is that it allows running many processes that communicate by message passing: but all these processes are _colocated_ to a given node, i would have expected that OTP (or genserver or whatever) handles clustering of machines, scheduling of workloads and routing of messages for me so that I can start processes and call them without caring about where they run. What about “distributed systems” is elixir actually shining at? From my reading, it seems rather like a good option to _avoid_ distributing (thanks to the light processes and concurrency handling)

davydog187 3y ago
I think you need to go deeper, because it does exactly all the things you’re describing out of the box. A process can be anywhere in the cluster, and you can interact with it as if it was on your local node. I use all of these facilities in production on my product with ease. I’m happy to answer your specific questions.
jorams 3y ago
Erlang/Elixir nodes can actually be rather trivially clustered, you can spawn processes on other nodes and transparently pass messages to processes on other nodes, including things like closures. It does not magically schedule workloads across the cluster, but there are libraries that can do so.
sb8244 3y ago
You can message pass across a cluster. In fact, it's identical code to message a remote vs local process. Erlang provides "process groups" that allow you to structure groups of processes across a cluster. It doesn't try to be magic. It's not a workload scheduler or "automatic distributed machine". You structure things the way you want, on top of it's primitives. It's just that all of the primitives you need are built in and work really well.
brightball 3y ago
It does all of that. The structure of the system is such that you can call any function on any node in the cluster. In order to do it well, you need to understand your own abstractions. If we have a cluster are we better off passing every function around the cluster or running some locally? Is the network overhead worth it in every case or only in some? It's up to your code to make those determinations about your application, but everything you describe is fairly simple to implement on the BEAM.