7 ms·

I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp had before

by atupis 7y ago

I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages.

ailideex 7y ago
> It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages. Like what ?
- quaunaut 7y ago
  The fact that the entire ecosystem is based around the actor model is one of the primary ones. It's allowed for code that isn't simply a library, but effectively an entire application, to be used as open source to be dumped into your system, because it has tools like Erlang Term Storage, Mnesia, and other devices to make it easy.
  - hota_mazi 7y ago
    You're still not explaining why this allows to do things that can't be done in other languages. Probably because that claim makes no sense.
    - nathanappere 7y ago
      Reductions. How do you ensure in other languages (or in other VMs) that every actor will get the same cpu time and that none can block the others? That's a tricky problem to solve as a lib.
- atupis 7y ago
  I think Whatsapp served "at peak" like 450 million users with only 32 engineers.
  - dmitriid 7y ago
    Whatsapp also had multiple patches to Erlang VM, backported FreeBSD drivers and code etc. etc. Do not assume you'll be able to build your own Whatsapp just because of Erlang.
    - fouc 7y ago
      The story of freebsd & erlang "needing" to be patched seems to be greatly exaggerated. Especially when it turns out that elixir/phoenix also achieved the same "2 million connections on single server" without needing those optimizations.
      - dmitriid 7y ago
        > The story of freebsd & erlang "needing" to be patched seems to be greatly exaggerated. The story is greatly underreported and all focus is only on "they run Whatsapp on Erlang with just ~50 engineers". Highscalability lists just some of the patches and optimisations they have here [1] and here [2] Here's an incomplete list of patches only. There's also tuning and optimisation: Erlang: Fixed head-of-line blocking in async file IO by patching BEAM, added round-robin scheduling for async file IO, added multiple instrumentation patches. Instrumented scheduler to get utilization information, statistics for message queues, number of sleeps, send rates, message counts, etc. Made lock counting work for larger async thread counts. Patched to dial down spin counts so the scheduler wouldn’t spin. BSD: Backported a TSE time counter. Backported igp network driver. More Mnesia (Erlang) patches discussed here: [3] Are you ready to do this for your Whatsapp? > elixir/phoenix also achieved the same "2 million connections on single server" without needing those optimizations. There's more needed to run a chat server than just "2 million empty connections". --- [1] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi... [2] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... [3] https://www.infoq.com/presentations/whatsapp-scalability/ https://www.infoq.com/presentations/whatsapp-scalability/
- rapsey 7y ago
  Erlang's advantage is being able to build a featureful, secure, scalable and performant enough communication backend in very little time and effort. This is a niche in which it can not be beaten.
  - 0xFACEFEED 7y ago
    Time/effort/security/performance, really? IIRC the reasons to consider Erlang is purely to do with building a fault tolerant highly distributed system.
    - rapsey 7y ago
      You are missing the forest for the trees.
    - vim-guru 7y ago
      This. And it's ability to be introspected in realtime. It has a REPL-experience very much like lisp's do
      - ailideex 7y ago
        Not sure what you mean by introspection in real time ... but if you mean runtime introspection this is not that much of a niche feature. If you mean remote debugging, again not that much of a niche feature. JVM and Javascript provides both.
    - dnautics 7y ago
      I would say that at least the elixir ecosystem puts a very high priority on making sane security very easy, especially Phoenix and nerves
  - otabdeveloper4 7y ago
    > ...in very little time and effort. You have a very naive view of large-scale distributed systems. For companies that truly need them, "time" and "effort" are never a consideration. Furthermore, the challenges facing these companies in battling project delays and cost overruns are the 100% organizational and political, not technical.
- toast0 7y ago
  You can absolutely do anything you can do in Erlang in another language. Some tasks will be easier, or harder, however. In my mind, Erlang excels in a few ways: Less often remarked, binary parsing in Erlang is rather nice. Parsing out an IPv4 packet looks like this: <<4:4, IHL:4, _DSCP:6, _ECN:2, _TotalLength:16, _Id:16, 0:1, _DF:1, MF:1, FragmentOffset:13, _TTL:8, Protocol:8, _IPChksum:16, SrcIP:4/binary, DestIP:4/binary, IPRest/binary>> = Payload, OptsSize = (IHL - 5) * 4, <<_IPOptions:OptsSize/binary, IPPayload/binary>> = IPRest, And then you've got everything. As long as formats are reasonably documented, it's easy to parse them. Sometimes, you can even parse a size and use the size in the same match. More often remarked; because of the language constraints, most notably a lack of shared memory between processes and immutable variables, most programming ideas end up expressed in a way that's amenable to massive concurrency, while being comprehensible. Most Actors have easy to understand behavior --- they may accept messages, leading to them sending messages and changing their internal state. From there, you may need to puzzle out the overall system behavior, but often times, getting each individual Actor's behavior correct, leads to correct (if hard to verify) system behavior. Hot loading code reduces deployment time, which increases developer productivity. When you've got a million users connected to a machine, it takes a lot of time to move them to other machines so you can do a traditional stop / start cycle; hot loading means you can fix things in seconds (and, of course, it means you can break things in seconds too). You can, of course, hotload in C, and probably other languages, but very few people do it. It's comparable to pushing PHP files though. And, of course, the most important thing is ejabberd is in Erlang, and it looks like it has what we need for a chat server, and I heard some other people scaled it really far. ;)
  - samvher 7y ago
    To be fair though many Erlang programmers also avoid hot loading code. For the rest, nice summary.
    - fouc 7y ago
      Why would they avoid that? (hopefully it's not just because of the ease of containerization/docker that is leading to reduced hot loading)
dokem 7y ago
My understanding is that is has an advantage in concurrency domain. It also has a relatively unique 'hot patch' support but i'm not sure how valuable that actually is.
atemerev 7y ago
Erlang is highly optimized to build reliable network communication systems (application-level routers, message brokers, stream aggregators, perhaps MMO game engines, chat backends, etc.) The secret weapons are actor model and process separation (like microservice architecture, but within a single app), supervision (isolated processes are restarted on failure, there is no difference between known "application" errors and exceptions), and beyond state of the art protocol parsing and construction facilities, for both binary and text protocols. For some tasks (scientific computing, UIs, offline analytics) Erlang might not be the best choice. However, within its domain of applications, it really shines.
wildduck 7y ago
My main interest in Erlang is mnesia. A relational/object hybrid data model that is suitable for telecommunications applications. A DBMS query language, Query List Comprehension (QLC) as an add-on library. Persistence. Tables can be coherently kept on disc and in the main memory. Replication. Tables can be replicated at several nodes. Atomic transactions. A series of table manipulation operations can be grouped into a single atomic transaction. Location transparency. Programs can be written without knowledge of the actual data location. Extremely fast real-time data searches. Schema manipulation routines. The DBMS can be reconfigured at runtime without stopping the system.
- samvher 7y ago
  Yeah mnesia is extremely clever and very fun to learn. Also being able to store basically any term without having to do conversions is very useful. I've found some practical aspects a bit painful though. It took me a while to figure out how to make backups, migrate schemas to different sets of nodes, that sort of thing. Also there are size limits on disk-based tables which are a bit limiting, and while you can use table fragmentation to get around that to some extent it doesn't seem straightforward to use (I haven't dared so far). I also don't like the way it deals with netsplits - when nodes reconnect it tends to require manual action to resolve.
  - marci 7y ago
    > Also there are size limits on disk-based tables which are a bit limiting There's now mnesia on leveldb https://github.com/klarna/mnesia_eleveldb https://github.com/klarna/mnesia_eleveldb https://www.youtube.com/watch?v=KcJw3RYFWUM https://www.youtube.com/watch?v=KcJw3RYFWUM
  - lostcolony 7y ago
    'I also don't like the way it deals with netsplits' I actually prefer it. Too often systems out there don't tell you what they do in the event of a netsplit. They may have marketing copy somewhere that tells you what they try to do, but then you see the Jepsen tests and realize that's a lie, or is naive, or whatever. Mnesia makes no secrets about it. In the event of a partition it stays partitioned, operating as separate nodes. It's up to you to figure out what to do about that. You can grab a Raft implementation to perform leader election, with the tradeoffs that entails. You want it to self-heal and deconflict based on arbitrary logic, for an eventually consistent system? You can do that too. You want to pretend it doesn't happen, stick your fingers in your ears, "LALALALA" until it does and requires manual intervention? That's fine too! What it DOESN'T do is give you a false sense of security while handwaving away the decisions and implementation concerns that were made to determine CAP behavior, which I find pretty much every other distributed data store does. It definitely has pain points in learning it, and it also has some very definite limitations, but as a baked in, minimally biased distributed store, it's a battery I really loved having included.
abledon 7y ago
> I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp It has a good marketable brand and 'appearance' for sure. A good Aura and level 3 magicks, but once you start leveling up your spell tree, like other comments in the thread have alluded to, the cracks start to show and you bleed out mana trying to scale what you initially thought would be an effortless process.
MisterTea 7y ago
Its strengths lie in its concurrency, simple message passing, distributed design, and robust VM. The really interesting part is message passing and easy peasy baked in communication that allows to to send messages to other erlang nodes. So building applications that talk to each other over networks is just a normal function of the language and not a higher level dependency such as a library. The concurrency is also pretty neat as erlang treats all threads of execution as an erlang process. So merging that with the distribution and simple message passing you can easily build stuff that scales without much external tooling. It's almost an OS unto itself. There are two interesting IoT systems which use it to such length such as GRiSP which is erlang beam ported to bare metal using RTEMS and Nerves which runs beam directly on top of a Linux kernel making beam the serspace.
segmondy 7y ago
Tons of language are secret weapons if you use it for what they were created. Here are some more outside of Erlang, that I know of and I'm sure there's more. APL/J/K - mathematical algorithms Prolog - logic, tons of business rules Forth/Lisp - bottom up programming, when you have an idea of the primitives you need to tackle a problem, but not exactly sure how to put it together. Assembly - When you must absolutely run impossibly fast especially on very small CPUs AWK/Perl - slice and dice text files For these languages, I don't substitute for any language. I'll never slice and dice with Java, Python, Go. I don't care. Awk or Perl. I'll never implement tons of rules in any other language, I don't care what logic library they implement without first doing so in Prolog. For the solo programmer, the above languages hold very true. I have played around with Erlang since Prolog influenced it's syntax, but I'm yet to have to build a large scale fault tolerant system that needs it, but the knowledge is tucked away at the back of my memory should I ever. I feel that Python, Javascript and Go can also be some sort of secret weapon if used in the right place.

by atupis 7y ago

ailideex 7y ago

> It has similar aura than Lisp had before that with Erlang you can do stuff beyond "normal" languages. Like what ?

quaunaut 7y ago
The fact that the entire ecosystem is based around the actor model is one of the primary ones. It's allowed for code that isn't simply a library, but effectively an entire application, to be used as open source to be dumped into your system, because it has tools like Erlang Term Storage, Mnesia, and other devices to make it easy.
- hota_mazi 7y ago
  You're still not explaining why this allows to do things that can't be done in other languages. Probably because that claim makes no sense.
  - nathanappere 7y ago
    Reductions. How do you ensure in other languages (or in other VMs) that every actor will get the same cpu time and that none can block the others? That's a tricky problem to solve as a lib.
atupis 7y ago
I think Whatsapp served "at peak" like 450 million users with only 32 engineers.
- dmitriid 7y ago
  Whatsapp also had multiple patches to Erlang VM, backported FreeBSD drivers and code etc. etc. Do not assume you'll be able to build your own Whatsapp just because of Erlang.
  - fouc 7y ago
    The story of freebsd & erlang "needing" to be patched seems to be greatly exaggerated. Especially when it turns out that elixir/phoenix also achieved the same "2 million connections on single server" without needing those optimizations.
    - dmitriid 7y ago
      > The story of freebsd & erlang "needing" to be patched seems to be greatly exaggerated. The story is greatly underreported and all focus is only on "they run Whatsapp on Erlang with just ~50 engineers". Highscalability lists just some of the patches and optimisations they have here [1] and here [2] Here's an incomplete list of patches only. There's also tuning and optimisation: Erlang: Fixed head-of-line blocking in async file IO by patching BEAM, added round-robin scheduling for async file IO, added multiple instrumentation patches. Instrumented scheduler to get utilization information, statistics for message queues, number of sleeps, send rates, message counts, etc. Made lock counting work for larger async thread counts. Patched to dial down spin counts so the scheduler wouldn’t spin. BSD: Backported a TSE time counter. Backported igp network driver. More Mnesia (Erlang) patches discussed here: [3] Are you ready to do this for your Whatsapp? > elixir/phoenix also achieved the same "2 million connections on single server" without needing those optimizations. There's more needed to run a chat server than just "2 million empty connections". --- [1] http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html http://highscalability.com/blog/2014/2/26/the-whatsapp-archi... [2] http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an.html http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... [3] https://www.infoq.com/presentations/whatsapp-scalability/ https://www.infoq.com/presentations/whatsapp-scalability/
rapsey 7y ago
Erlang's advantage is being able to build a featureful, secure, scalable and performant enough communication backend in very little time and effort. This is a niche in which it can not be beaten.
- 0xFACEFEED 7y ago
  Time/effort/security/performance, really? IIRC the reasons to consider Erlang is purely to do with building a fault tolerant highly distributed system.
  - rapsey 7y ago
    You are missing the forest for the trees.
  - vim-guru 7y ago
    This. And it's ability to be introspected in realtime. It has a REPL-experience very much like lisp's do
    - ailideex 7y ago
      Not sure what you mean by introspection in real time ... but if you mean runtime introspection this is not that much of a niche feature. If you mean remote debugging, again not that much of a niche feature. JVM and Javascript provides both.
  - dnautics 7y ago
    I would say that at least the elixir ecosystem puts a very high priority on making sane security very easy, especially Phoenix and nerves
- otabdeveloper4 7y ago
  > ...in very little time and effort. You have a very naive view of large-scale distributed systems. For companies that truly need them, "time" and "effort" are never a consideration. Furthermore, the challenges facing these companies in battling project delays and cost overruns are the 100% organizational and political, not technical.
toast0 7y ago
You can absolutely do anything you can do in Erlang in another language. Some tasks will be easier, or harder, however. In my mind, Erlang excels in a few ways: Less often remarked, binary parsing in Erlang is rather nice. Parsing out an IPv4 packet looks like this: <<4:4, IHL:4, _DSCP:6, _ECN:2, _TotalLength:16, _Id:16, 0:1, _DF:1, MF:1, FragmentOffset:13, _TTL:8, Protocol:8, _IPChksum:16, SrcIP:4/binary, DestIP:4/binary, IPRest/binary>> = Payload, OptsSize = (IHL - 5) * 4, <<_IPOptions:OptsSize/binary, IPPayload/binary>> = IPRest, And then you've got everything. As long as formats are reasonably documented, it's easy to parse them. Sometimes, you can even parse a size and use the size in the same match. More often remarked; because of the language constraints, most notably a lack of shared memory between processes and immutable variables, most programming ideas end up expressed in a way that's amenable to massive concurrency, while being comprehensible. Most Actors have easy to understand behavior --- they may accept messages, leading to them sending messages and changing their internal state. From there, you may need to puzzle out the overall system behavior, but often times, getting each individual Actor's behavior correct, leads to correct (if hard to verify) system behavior. Hot loading code reduces deployment time, which increases developer productivity. When you've got a million users connected to a machine, it takes a lot of time to move them to other machines so you can do a traditional stop / start cycle; hot loading means you can fix things in seconds (and, of course, it means you can break things in seconds too). You can, of course, hotload in C, and probably other languages, but very few people do it. It's comparable to pushing PHP files though. And, of course, the most important thing is ejabberd is in Erlang, and it looks like it has what we need for a chat server, and I heard some other people scaled it really far. ;)
- samvher 7y ago
  To be fair though many Erlang programmers also avoid hot loading code. For the rest, nice summary.
  - fouc 7y ago
    Why would they avoid that? (hopefully it's not just because of the ease of containerization/docker that is leading to reduced hot loading)

dokem 7y ago

My understanding is that is has an advantage in concurrency domain. It also has a relatively unique 'hot patch' support but i'm not sure how valuable that actually is.

atemerev 7y ago

Erlang is highly optimized to build reliable network communication systems (application-level routers, message brokers, stream aggregators, perhaps MMO game engines, chat backends, etc.) The secret weapons are actor model and process separation (like microservice architecture, but within a single app), supervision (isolated processes are restarted on failure, there is no difference between known "application" errors and exceptions), and beyond state of the art protocol parsing and construction facilities, for both binary and text protocols. For some tasks (scientific computing, UIs, offline analytics) Erlang might not be the best choice. However, within its domain of applications, it really shines.

wildduck 7y ago

My main interest in Erlang is mnesia. A relational/object hybrid data model that is suitable for telecommunications applications. A DBMS query language, Query List Comprehension (QLC) as an add-on library. Persistence. Tables can be coherently kept on disc and in the main memory. Replication. Tables can be replicated at several nodes. Atomic transactions. A series of table manipulation operations can be grouped into a single atomic transaction. Location transparency. Programs can be written without knowledge of the actual data location. Extremely fast real-time data searches. Schema manipulation routines. The DBMS can be reconfigured at runtime without stopping the system.

samvher 7y ago
Yeah mnesia is extremely clever and very fun to learn. Also being able to store basically any term without having to do conversions is very useful. I've found some practical aspects a bit painful though. It took me a while to figure out how to make backups, migrate schemas to different sets of nodes, that sort of thing. Also there are size limits on disk-based tables which are a bit limiting, and while you can use table fragmentation to get around that to some extent it doesn't seem straightforward to use (I haven't dared so far). I also don't like the way it deals with netsplits - when nodes reconnect it tends to require manual action to resolve.
- marci 7y ago
  > Also there are size limits on disk-based tables which are a bit limiting There's now mnesia on leveldb https://github.com/klarna/mnesia_eleveldb https://github.com/klarna/mnesia_eleveldb https://www.youtube.com/watch?v=KcJw3RYFWUM https://www.youtube.com/watch?v=KcJw3RYFWUM
- lostcolony 7y ago
  'I also don't like the way it deals with netsplits' I actually prefer it. Too often systems out there don't tell you what they do in the event of a netsplit. They may have marketing copy somewhere that tells you what they try to do, but then you see the Jepsen tests and realize that's a lie, or is naive, or whatever. Mnesia makes no secrets about it. In the event of a partition it stays partitioned, operating as separate nodes. It's up to you to figure out what to do about that. You can grab a Raft implementation to perform leader election, with the tradeoffs that entails. You want it to self-heal and deconflict based on arbitrary logic, for an eventually consistent system? You can do that too. You want to pretend it doesn't happen, stick your fingers in your ears, "LALALALA" until it does and requires manual intervention? That's fine too! What it DOESN'T do is give you a false sense of security while handwaving away the decisions and implementation concerns that were made to determine CAP behavior, which I find pretty much every other distributed data store does. It definitely has pain points in learning it, and it also has some very definite limitations, but as a baked in, minimally biased distributed store, it's a battery I really loved having included.

abledon 7y ago

> I have never programmed Erlang but it feels like it is the currently only language that is some kind of secret weapon. It has similar aura than Lisp It has a good marketable brand and 'appearance' for sure. A good Aura and level 3 magicks, but once you start leveling up your spell tree, like other comments in the thread have alluded to, the cracks start to show and you bleed out mana trying to scale what you initially thought would be an effortless process.

MisterTea 7y ago

Its strengths lie in its concurrency, simple message passing, distributed design, and robust VM. The really interesting part is message passing and easy peasy baked in communication that allows to to send messages to other erlang nodes. So building applications that talk to each other over networks is just a normal function of the language and not a higher level dependency such as a library. The concurrency is also pretty neat as erlang treats all threads of execution as an erlang process. So merging that with the distribution and simple message passing you can easily build stuff that scales without much external tooling. It's almost an OS unto itself. There are two interesting IoT systems which use it to such length such as GRiSP which is erlang beam ported to bare metal using RTEMS and Nerves which runs beam directly on top of a Linux kernel making beam the serspace.

segmondy 7y ago

Tons of language are secret weapons if you use it for what they were created. Here are some more outside of Erlang, that I know of and I'm sure there's more. APL/J/K - mathematical algorithms Prolog - logic, tons of business rules Forth/Lisp - bottom up programming, when you have an idea of the primitives you need to tackle a problem, but not exactly sure how to put it together. Assembly - When you must absolutely run impossibly fast especially on very small CPUs AWK/Perl - slice and dice text files For these languages, I don't substitute for any language. I'll never slice and dice with Java, Python, Go. I don't care. Awk or Perl. I'll never implement tons of rules in any other language, I don't care what logic library they implement without first doing so in Prolog. For the solo programmer, the above languages hold very true. I have played around with Erlang since Prolog influenced it's syntax, but I'm yet to have to build a large scale fault tolerant system that needs it, but the knowledge is tucked away at the back of my memory should I ever. I feel that Python, Javascript and Go can also be some sort of secret weapon if used in the right place.