4 ms·

@toast0, would you still use Erlang/OTP today? If so, for what kind of apps. If not anymore, what would you use instead.

toast0 3y ago
Yes. Certainly. In my mind, there's two really good fits (and some cross over between them). a) binary matching syntax is really nice for dealing with bit-packed things; although it's not pretty if dealing with little endian values where there's a couple bits in one byte and a couple more in the neighboring byte --- you've got to get the pieces and put them together if it's not just whole bytes. Big endian bit packed structures are easy, but little endian is dominant these days. I don't know if performance is good, but it's easy to read and write for developers. b) anything with a large number of connections and significant state per connection. A chat server, video conference, etc. This is why you see 'everyone' build chat from ejabberd. Erlang makes code for this kind of service fairly simple, and hotloading means you can fix bugs without kicking everyone off to restart. Observability features make it reasonable to see what's going on in your system and where bottlenecks are.
- alberth 3y ago
  Thanks for your insights as always. Related to (b), any thoughts on Rustler? My understanding it’s used extensively at Discord to eliminate Erlang/OTP performance bottlenecks. https://github.com/rusterlium/rustler https://github.com/rusterlium/rustler
  - toast0 3y ago
    I haven't used it. I've written and used NIFs, but mostly they're simple enough that C is fine, and Rust isn't really needed. I think discord ends up with complex data structures in their native layer; Rust seems reasonable for that, but I never experienced that need. That said, I did have a concurrency issue with a NIF I used, but I don't know if Rust would have been helpful or just made it really hard to express.
  - asa400 3y ago
    I like it a lot. The big advantage of Rustler vs. say, writing C or using Zig's equivalent, is that you can just use any Rust libraries. For example, I wrote Rust<->Elixir bindings to xxhash and CANBus for our industrial control/embedded applications, and it was quick to do. Very little ceremony. Rustler provides these Encoder and Decoder traits (and implementations for most stdlib data structures) that make it easy to pass data back and forth across the FFI boundary without you having to manually do it, and they've done all of the work of writing/generating the bindings to Erlang's C API. It also includes a generator to easily create a new NIF that integrates into your Elixir project. Of course, you need to know both Elixir and Rust to a high enough level, but if you do, it's really pretty straightforward to just either wrap an existing Rust library, or to rewrite some hot path in Rust and schedule it on a dirty CPU or IO thread.
- takeda 3y ago
  > This is why you see 'everyone' build chat from ejabberd. Erlang makes code for this kind of service fairly simple, and hotloading means you can fix bugs without kicking everyone off to restart. Sadly in practice I saw people avoid hotloading. I saw this behavior with Riak data store, in my company's product and also ejabberd in FreeBSD ports (this one is especially annoying as the port demands uninstalling the package before it even starts building it[1] which is even worse experience than with alternatives). Why almost everyone (from my experience) is doing this? [1] https://github.com/freebsd/freebsd-ports/blob/9d74858de3b5e458c52662107d2294a6f92100e9/net-im/ejabberd/Makefile#L158-L164 https://github.com/freebsd/freebsd-ports/blob/9d74858de3b5e4...
  - toast0 3y ago
    Well, hotloading is very powerful, and can really mess things up in a hurry. You've got to plan for it. I wouldn't expect ejabberd to be built to hotload between releases (unless the docs say you can) But for hotfixes, it's great. And for code you write, it's often possible to structure even large changes for hotloading. A lot of people have built deployment around the limitations of other technology, and it's a hard thing to get them to hotload. But if you can, it really reduces the cycle time between when you detect an issue and when its fixed and deployed. Short cycles are great because for many issues you can pick the issue up, fix it, deploy it, verify the fix, and be done without having to switch tasks. Additionally, a short cycle reduces the duration of impact for simple bugs, which means you need to spend less effort on finding them before deploy (probably not no effort though). Complex bugs that persist broken state are still difficult to remedy, so you should still spend significant effort on finding those before deployment. (Also, I'm really not sure why having the port installed would interfere with building a new one, but I've always built Erlang with make and erlc which is maybe a bit unusual)
    - takeda 3y ago
      > (Also, I'm really not sure why having the port installed would interfere with building a new one, but I've always built Erlang with make and erlc which is maybe a bit unusual) It looks like this happens because the ejabberd is installed into system's erlang lib directory and that impacts the build process. I'm wondering why author of the port opted to force un-installation instead of just install it in another directory? Is there a benefit of doing it this way? I was thinking to suggest that, but I don't know enough about erlang if that even makes sense.
- rwmj 3y ago
  (a) has been copied in quite a few languages now. Here's my OCaml copy: https://bitstring.software/documentation/ https://bitstring.software/documentation/ (sorry about the bad website cert, it's not under my control) https://github.com/xguerin/bitstring https://github.com/xguerin/bitstring We handle endianness very naturally. Performance of the OCaml version depends on whether everything is byte-aligned. It will compile-time optimize byte-aligned stuff to be fast, and do slow bit shifting for everything else. I believe this is an advantage over the Erlang original.
  - toast0 3y ago
    This is pretty interesting. Thanks! I didn't have time to dig into it [1], but something that's a bit ugly is something like a PCI BAR, defined as Bits 31-4: 16-Byte Aligned Base Address Bit 3: Prefetchable (flag) Bits 2-1: Type Bit 0: Always zero In Erlang, this gets parsed with something like: <<BaseLS:4, Prefetch:1, Type:2, 0:1, BaseMS:24/little>> = BAR and then you have to put the Most Significant and Least Significant parts back together somehow, and there's the zero filling too, like maybe <<Base:32>> = <<BaseMS:24, BaseLS:4, 0:4>> If the structure were big endian, you'd probably have bits 0-28 be the address and bits 29-31 be the flags, and you could parse it as <<ShiftedBase:28, Prefetch:1, Type:2, 0:1>> = BAR, Base = ShiftedBase bsl 4 You could also just do <<BaseAndFlags:32/little>> = BAR and then mask out the flags and stuff, but then you lose the joy of pattern matching, or you defer the pattern matching which means you'd lose out on matching in function heads or have to have an intermediate parsing function that calls the real function with matching in its heads. [1] I looked a bit more and the example seems inline with Erlang syntax https://bitstring.software/documentation/#constructing-bitstrings https://bitstring.software/documentation/#constructing-bitst... shows something similar to what I would expect with Version = 1, Data = 10, <<Version:4, Data:12>>. You could say Data:12/little, and that does flip bits around, but a little endian C developer isn't going to be happy with what results. It's not natural for them to combine the lower bits of byte 0 with the bits of byte 1.
datetime 3y ago
We are running Erlang/OTP for telemetry backend in the industrial automation field. Basically, we have hundreds of thousands sensors connected through gateways that keep open TCP/IP connections to the Erlang/OTP distributed backend. We do bidirectional communication as we have many control functions, OTA firmware updates, etc. There are frequent failures which we handle with supervision trees and “let it fail” design principles. Failures are due to: * Gateways which are connected through cellular networks with varying signal strength conditions. * Sensors failing and providing incorrect data (eg. invalid float binaries). * Buggy firmware in some 3rd party gateways / sensors. * Buggy firmware in our own gateways and sensors ;-) I highly recommend Elrang/OTP due to: * Fault tolerance - processes fail, nodes carry on. * Concurrency model (mailboxes, linking processes via supervision trees, monitors, trapping, etc) * Built-in distribution and related modules in standard library * Pattern matching which makes processing binary data super convenient * Mnesia database (if used for right things)

toast0 3y ago

Yes. Certainly. In my mind, there's two really good fits (and some cross over between them). a) binary matching syntax is really nice for dealing with bit-packed things; although it's not pretty if dealing with little endian values where there's a couple bits in one byte and a couple more in the neighboring byte --- you've got to get the pieces and put them together if it's not just whole bytes. Big endian bit packed structures are easy, but little endian is dominant these days. I don't know if performance is good, but it's easy to read and write for developers. b) anything with a large number of connections and significant state per connection. A chat server, video conference, etc. This is why you see 'everyone' build chat from ejabberd. Erlang makes code for this kind of service fairly simple, and hotloading means you can fix bugs without kicking everyone off to restart. Observability features make it reasonable to see what's going on in your system and where bottlenecks are.

alberth 3y ago
Thanks for your insights as always. Related to (b), any thoughts on Rustler? My understanding it’s used extensively at Discord to eliminate Erlang/OTP performance bottlenecks. https://github.com/rusterlium/rustler https://github.com/rusterlium/rustler
- toast0 3y ago
  I haven't used it. I've written and used NIFs, but mostly they're simple enough that C is fine, and Rust isn't really needed. I think discord ends up with complex data structures in their native layer; Rust seems reasonable for that, but I never experienced that need. That said, I did have a concurrency issue with a NIF I used, but I don't know if Rust would have been helpful or just made it really hard to express.
- asa400 3y ago
  I like it a lot. The big advantage of Rustler vs. say, writing C or using Zig's equivalent, is that you can just use any Rust libraries. For example, I wrote Rust<->Elixir bindings to xxhash and CANBus for our industrial control/embedded applications, and it was quick to do. Very little ceremony. Rustler provides these Encoder and Decoder traits (and implementations for most stdlib data structures) that make it easy to pass data back and forth across the FFI boundary without you having to manually do it, and they've done all of the work of writing/generating the bindings to Erlang's C API. It also includes a generator to easily create a new NIF that integrates into your Elixir project. Of course, you need to know both Elixir and Rust to a high enough level, but if you do, it's really pretty straightforward to just either wrap an existing Rust library, or to rewrite some hot path in Rust and schedule it on a dirty CPU or IO thread.
takeda 3y ago
> This is why you see 'everyone' build chat from ejabberd. Erlang makes code for this kind of service fairly simple, and hotloading means you can fix bugs without kicking everyone off to restart. Sadly in practice I saw people avoid hotloading. I saw this behavior with Riak data store, in my company's product and also ejabberd in FreeBSD ports (this one is especially annoying as the port demands uninstalling the package before it even starts building it[1] which is even worse experience than with alternatives). Why almost everyone (from my experience) is doing this? [1] https://github.com/freebsd/freebsd-ports/blob/9d74858de3b5e458c52662107d2294a6f92100e9/net-im/ejabberd/Makefile#L158-L164 https://github.com/freebsd/freebsd-ports/blob/9d74858de3b5e4...
- toast0 3y ago
  Well, hotloading is very powerful, and can really mess things up in a hurry. You've got to plan for it. I wouldn't expect ejabberd to be built to hotload between releases (unless the docs say you can) But for hotfixes, it's great. And for code you write, it's often possible to structure even large changes for hotloading. A lot of people have built deployment around the limitations of other technology, and it's a hard thing to get them to hotload. But if you can, it really reduces the cycle time between when you detect an issue and when its fixed and deployed. Short cycles are great because for many issues you can pick the issue up, fix it, deploy it, verify the fix, and be done without having to switch tasks. Additionally, a short cycle reduces the duration of impact for simple bugs, which means you need to spend less effort on finding them before deploy (probably not no effort though). Complex bugs that persist broken state are still difficult to remedy, so you should still spend significant effort on finding those before deployment. (Also, I'm really not sure why having the port installed would interfere with building a new one, but I've always built Erlang with make and erlc which is maybe a bit unusual)
  - takeda 3y ago
    > (Also, I'm really not sure why having the port installed would interfere with building a new one, but I've always built Erlang with make and erlc which is maybe a bit unusual) It looks like this happens because the ejabberd is installed into system's erlang lib directory and that impacts the build process. I'm wondering why author of the port opted to force un-installation instead of just install it in another directory? Is there a benefit of doing it this way? I was thinking to suggest that, but I don't know enough about erlang if that even makes sense.
rwmj 3y ago
(a) has been copied in quite a few languages now. Here's my OCaml copy: https://bitstring.software/documentation/ https://bitstring.software/documentation/ (sorry about the bad website cert, it's not under my control) https://github.com/xguerin/bitstring https://github.com/xguerin/bitstring We handle endianness very naturally. Performance of the OCaml version depends on whether everything is byte-aligned. It will compile-time optimize byte-aligned stuff to be fast, and do slow bit shifting for everything else. I believe this is an advantage over the Erlang original.
- toast0 3y ago
  This is pretty interesting. Thanks! I didn't have time to dig into it [1], but something that's a bit ugly is something like a PCI BAR, defined as Bits 31-4: 16-Byte Aligned Base Address Bit 3: Prefetchable (flag) Bits 2-1: Type Bit 0: Always zero In Erlang, this gets parsed with something like: <<BaseLS:4, Prefetch:1, Type:2, 0:1, BaseMS:24/little>> = BAR and then you have to put the Most Significant and Least Significant parts back together somehow, and there's the zero filling too, like maybe <<Base:32>> = <<BaseMS:24, BaseLS:4, 0:4>> If the structure were big endian, you'd probably have bits 0-28 be the address and bits 29-31 be the flags, and you could parse it as <<ShiftedBase:28, Prefetch:1, Type:2, 0:1>> = BAR, Base = ShiftedBase bsl 4 You could also just do <<BaseAndFlags:32/little>> = BAR and then mask out the flags and stuff, but then you lose the joy of pattern matching, or you defer the pattern matching which means you'd lose out on matching in function heads or have to have an intermediate parsing function that calls the real function with matching in its heads. [1] I looked a bit more and the example seems inline with Erlang syntax https://bitstring.software/documentation/#constructing-bitstrings https://bitstring.software/documentation/#constructing-bitst... shows something similar to what I would expect with Version = 1, Data = 10, <<Version:4, Data:12>>. You could say Data:12/little, and that does flip bits around, but a little endian C developer isn't going to be happy with what results. It's not natural for them to combine the lower bits of byte 0 with the bits of byte 1.

datetime 3y ago

We are running Erlang/OTP for telemetry backend in the industrial automation field. Basically, we have hundreds of thousands sensors connected through gateways that keep open TCP/IP connections to the Erlang/OTP distributed backend. We do bidirectional communication as we have many control functions, OTA firmware updates, etc. There are frequent failures which we handle with supervision trees and “let it fail” design principles. Failures are due to: * Gateways which are connected through cellular networks with varying signal strength conditions. * Sensors failing and providing incorrect data (eg. invalid float binaries). * Buggy firmware in some 3rd party gateways / sensors. * Buggy firmware in our own gateways and sensors ;-) I highly recommend Elrang/OTP due to: * Fault tolerance - processes fail, nodes carry on. * Concurrency model (mailboxes, linking processes via supervision trees, monitors, trapping, etc) * Built-in distribution and related modules in standard library * Pattern matching which makes processing binary data super convenient * Mnesia database (if used for right things)