5 ms·

This looks very good and will be interesting to see how it scales in a large cluster - should be possible with it being written in Elixir to scale this out quit

This looks very good and will be interesting to see how it scales in a large cluster - should be possible with it being written in Elixir to scale this out quite well. I'll also be interested to see how stable the NIFs are, I'd feel much more confident they weren't going to crash the VM if they were written in Rust.

sidkshatriya 8y ago
Good point. Whats the point of using the Erlang VM if your NIF could bring down the whole VM. How is this different from using any other language like Go then to implement this platform? The approach confuses me because there is so much praise for the Erlang VM but to get all its benefits of the Erlang VM they need to use ports. (Which they confess is going to be slower).
- nailer 8y ago
  What's a NIF in this context?
  - andy_ppp 8y ago
    Native Implementation Functions - http://erlang.org/doc/tutorial/nif.html http://erlang.org/doc/tutorial/nif.html - Erlang/Elixir would not be performant enough to generate realtime video/audio encoding. It's possible to make them crash the VM if they leak memory or don't prempt when asked. In Rust you can make better guarantees about your code and there is this that allows you to write them in Rust: https://github.com/hansihe/rustler https://github.com/hansihe/rustler
    - mspanc 8y ago
      Nice hint, we'll definitely take this under consideration!
- mspanc 8y ago
  We're thinking about adding an abstraction layer over C code that will allow to switch between NIF and port backends. Ports have their limitations so you won't be able to use them e.g. for passing GPU buffers but they will do the job for many other cases. Anyway even with NIFs we delegate to them only code that cannot be written in Elixir or it really makes no sense to keep it in Elixir e.g. in performance reasons. Even if 30% of the framework will be in C it's way easier to manage 30% of C code than 100% of C code. Moreover, all the hard stuff, such as parallelism, being able to inspect running state of the VM, ability to spread app over cluster, I/O scalability, being multiplatform etc. comes out of the box with BEAM VM. There are many things that are give the point of using Erlang VM even if part of the code is in C and is potentially dangerous.
  - sidkshatriya 8y ago
    Thanks for your reply. The BEAM VM is amazing (and I'm a fan of Elixir) but I'm trying to see what benefit you're getting out of BEAM if you're using NIFs. Because the moment you use NIFs then your VM can crash (without recourse to the various recovery mechanisms of the BEAM) just like, say, a Golang executable. If you used Golang you would also get parallelism, I/O scalablity, multiplaform capability... all the things you list as advantages for BEAM. The only thing that BEAM is giving you "extra" vs, say, Golang is the ability to spread your app over a cluster easily. But again, I'm sure you can achieve that with some sort of a load distributer if you used Golang on multiple machines. Also with Golang it is easy to do CPU intensive tasks because its "closer" to the metal. So I guess what I'm saying is that for your specific use case (that involves a lot of media related CPU intensive work like video/audio processing) I wonder if the BEAM VM will give you that many advantages. With Golang in this case I think you can get the scalability, reliability as your current setup with one big advantage: better performance. But this is just my ~5 min view of the subject...
    - elcritch 8y ago
      Not the OP, but it strikes me that at some level it becomes a personal preference alongside the technical benefits. Though regarding error handling and NIFs, Erlang/OTP manages high reliability not just through local OTP mechanisms, but also through the clustering. It’s a fundamental part of the OTP ethos and something that would need to be “bolted on” in Go. That and the introspection tools are pretty great!
sjtgraham 8y ago
Solution: NIFs on remote nodes.
- dqv 8y ago
  Like a C node? Or a node on another VM?
mathek 8y ago
Concerning NIFs, it is entirely possible to write Membrane elements using NIFs in Rust ;) The reason why current ones are written in pure C is that they mostly consist of calls to existing libraries, also in C. We've experimented with using Rust, but we ended up with most part of our code being unsafe calls to library functions, providing more boilerplate than profits. If we are to create a considerable big native part of Membrane, we will definitely consider using Rust.

sidkshatriya 8y ago

Good point. Whats the point of using the Erlang VM if your NIF could bring down the whole VM. How is this different from using any other language like Go then to implement this platform? The approach confuses me because there is so much praise for the Erlang VM but to get all its benefits of the Erlang VM they need to use ports. (Which they confess is going to be slower).

nailer 8y ago
What's a NIF in this context?
- andy_ppp 8y ago
  Native Implementation Functions - http://erlang.org/doc/tutorial/nif.html http://erlang.org/doc/tutorial/nif.html - Erlang/Elixir would not be performant enough to generate realtime video/audio encoding. It's possible to make them crash the VM if they leak memory or don't prempt when asked. In Rust you can make better guarantees about your code and there is this that allows you to write them in Rust: https://github.com/hansihe/rustler https://github.com/hansihe/rustler
  - mspanc 8y ago
    Nice hint, we'll definitely take this under consideration!
mspanc 8y ago
We're thinking about adding an abstraction layer over C code that will allow to switch between NIF and port backends. Ports have their limitations so you won't be able to use them e.g. for passing GPU buffers but they will do the job for many other cases. Anyway even with NIFs we delegate to them only code that cannot be written in Elixir or it really makes no sense to keep it in Elixir e.g. in performance reasons. Even if 30% of the framework will be in C it's way easier to manage 30% of C code than 100% of C code. Moreover, all the hard stuff, such as parallelism, being able to inspect running state of the VM, ability to spread app over cluster, I/O scalability, being multiplatform etc. comes out of the box with BEAM VM. There are many things that are give the point of using Erlang VM even if part of the code is in C and is potentially dangerous.
- sidkshatriya 8y ago
  Thanks for your reply. The BEAM VM is amazing (and I'm a fan of Elixir) but I'm trying to see what benefit you're getting out of BEAM if you're using NIFs. Because the moment you use NIFs then your VM can crash (without recourse to the various recovery mechanisms of the BEAM) just like, say, a Golang executable. If you used Golang you would also get parallelism, I/O scalablity, multiplaform capability... all the things you list as advantages for BEAM. The only thing that BEAM is giving you "extra" vs, say, Golang is the ability to spread your app over a cluster easily. But again, I'm sure you can achieve that with some sort of a load distributer if you used Golang on multiple machines. Also with Golang it is easy to do CPU intensive tasks because its "closer" to the metal. So I guess what I'm saying is that for your specific use case (that involves a lot of media related CPU intensive work like video/audio processing) I wonder if the BEAM VM will give you that many advantages. With Golang in this case I think you can get the scalability, reliability as your current setup with one big advantage: better performance. But this is just my ~5 min view of the subject...
  - elcritch 8y ago
    Not the OP, but it strikes me that at some level it becomes a personal preference alongside the technical benefits. Though regarding error handling and NIFs, Erlang/OTP manages high reliability not just through local OTP mechanisms, but also through the clustering. It’s a fundamental part of the OTP ethos and something that would need to be “bolted on” in Go. That and the introspection tools are pretty great!

sjtgraham 8y ago

Solution: NIFs on remote nodes.

dqv 8y ago
Like a C node? Or a node on another VM?

mathek 8y ago

Concerning NIFs, it is entirely possible to write Membrane elements using NIFs in Rust ;) The reason why current ones are written in pure C is that they mostly consist of calls to existing libraries, also in C. We've experimented with using Rust, but we ended up with most part of our code being unsafe calls to library functions, providing more boilerplate than profits. If we are to create a considerable big native part of Membrane, we will definitely consider using Rust.