4 ms·

> * Dirty schedulers: This allows easy integration of blocking C-based libraries. So for example can wrap something like RocksDb and make it available to the re

> * Dirty schedulers: This allows easy integration of blocking C-based libraries. So for example can wrap something like RocksDb and make it available to the rest of the VM easier. Or libcurl and others. Silly question, but shouldn't elang have some good & highly concurrent http libraries?

tomjakubowski 9y ago
libcurl supports some other protocols, too. Maybe Erlang lacks good and highly concurrent Gopher libraries. :-)
micmus 9y ago
Dirty schedules are about functions implemented in C - called NIFs (Native Implemented Functions) - a sort of FFI you'd find in most languages. Because of preemptive nature of Erlang, doing lengthy work in those functions can destabilise the system (the "magic value" is said to be around 1ms). That is because C functions can't be preempted in the middle of execution like Erlang ones can. Using dirty schedulers lifts this time limitation, but gives a higher constant overhead when calling a function on a dirty scheduler, since it means switching OS thread. This tradeoff, however, is perfectly acceptable for a lot of cases. And yes, Erlang has good & highly concurrent HTTP libraries implemented in Erlang.
- eugene_pirogov 9y ago
  A question regarding dirty schedulers: does it still mean that when called C code crashes during execution, entire Erlang VM will crash as well?
  - micmus 9y ago
    Yes. That's not changed. All NIFs (regular or dirty ones) are executed directly in the context of the VM. A safe option would be a port - a regular program where you communicate through stdin/stdout. Ports allow representing such program (running in a separate OS process) as something equivalent to a native Erlang process. There's also work on supporting writing NIFs in Rust, which gives some degree of additional safety. The relevant project would be: https://github.com/hansihe/rustler https://github.com/hansihe/rustler
  - mononcqc 9y ago
    Yes. For such cases, you'll want to instead use a C-Node or a Port program to interface with the C code.
rdtsc 9y ago
Oh it does there are already a good number of decent ones. I just used libcurl as an example. And as someone suggested one reason to wrap libcurl could be is because it supports a lot of the corner cases and protocols.
- puzza007 9y ago
  I wrapped libcurl as a port https://github.com/puzza007/katipo https://github.com/puzza007/katipo. It would be interesting to see how much faster it could be made as a NIF.
lpgauth 9y ago
Yes, you can see some HTTP client benchmarks here. Disclaimer: I wrote the Buoy (https://github.com/lpgauth/buoy https://github.com/lpgauth/buoy)

tomjakubowski 9y ago

libcurl supports some other protocols, too. Maybe Erlang lacks good and highly concurrent Gopher libraries. :-)

micmus 9y ago

Dirty schedules are about functions implemented in C - called NIFs (Native Implemented Functions) - a sort of FFI you'd find in most languages. Because of preemptive nature of Erlang, doing lengthy work in those functions can destabilise the system (the "magic value" is said to be around 1ms). That is because C functions can't be preempted in the middle of execution like Erlang ones can. Using dirty schedulers lifts this time limitation, but gives a higher constant overhead when calling a function on a dirty scheduler, since it means switching OS thread. This tradeoff, however, is perfectly acceptable for a lot of cases. And yes, Erlang has good & highly concurrent HTTP libraries implemented in Erlang.

eugene_pirogov 9y ago
A question regarding dirty schedulers: does it still mean that when called C code crashes during execution, entire Erlang VM will crash as well?
- micmus 9y ago
  Yes. That's not changed. All NIFs (regular or dirty ones) are executed directly in the context of the VM. A safe option would be a port - a regular program where you communicate through stdin/stdout. Ports allow representing such program (running in a separate OS process) as something equivalent to a native Erlang process. There's also work on supporting writing NIFs in Rust, which gives some degree of additional safety. The relevant project would be: https://github.com/hansihe/rustler https://github.com/hansihe/rustler
- mononcqc 9y ago
  Yes. For such cases, you'll want to instead use a C-Node or a Port program to interface with the C code.

rdtsc 9y ago

Oh it does there are already a good number of decent ones. I just used libcurl as an example. And as someone suggested one reason to wrap libcurl could be is because it supports a lot of the corner cases and protocols.

puzza007 9y ago
I wrapped libcurl as a port https://github.com/puzza007/katipo https://github.com/puzza007/katipo. It would be interesting to see how much faster it could be made as a NIF.

lpgauth 9y ago

Yes, you can see some HTTP client benchmarks here. Disclaimer: I wrote the Buoy (https://github.com/lpgauth/buoy https://github.com/lpgauth/buoy)