4 ms·

If I'm not wrong, Go added uncooperative task scheduling that behaves similarly?

finder83 4y ago
Interesting, at least when I was using it the goroutines themselves were cooperative, and of course the OS threads were preemptive when GO_MAXPROCS > 1. Not finding much with a search, there was a proposal to make them preemptive. Curious if others will chime in that it's now preemptive. Even so I like Elixir better as a language, but the processing speed of Go with a preemptive scheduler would be tempting for some use cases.
ksbrooksjr 4y ago
I'm not a Go expert but preemptive scheduling of goroutines is supposed to be one of the primary benefits of Go as a language [1]. Although I'm not sure if there's any runtime that adopts preemptive scheduling as fervently as the BEAM. Even the regular expression engine is preemptive in Erlang, which prevents regular expression denial of service attacks [2]. [1] https://go.dev/src/runtime/preempt.go https://go.dev/src/runtime/preempt.go [2] https://www.erlang.org/doc/man/re.html#:~:text=run/3%20always%20gives%20control%20back%20to%20the%20scheduler%20of%20Erlang%20processes%20at%20intervals%20that%20ensures%20the%20real%2Dtime%20properties%20of%20the%20Erlang%20system https://www.erlang.org/doc/man/re.html#:~:text=run/3%20alway....
- Fire-Dragon-DoL 4y ago
  It wasn't non-cooperative pre-emptive scheduling for a while, which is why I kept pushing Elixir, until Go implemented it. Now Go has the same benefit as Elixir, but it's very, very fast.
  - ksbrooksjr 4y ago
    Yeah with preemptive scheduling, static typing, and better cpu utilization Go is a pretty compelling option for Erlang and Elixir users.
    - cztomsik 4y ago
      It's not immutable, there is no messaging, no hot-swap, no supervisors, it's not FP, and it doesn't have transparent support for clustering. Also, it's not memory-safe, 30ys old, and battle-tested pretty much everywhere.
      - ksbrooksjr 4y ago
        There are definitely trade offs when switching from Elixir or Erlang to Go. If you're a functional programmer who can't live without immutability, or you plan on running a cluster of machines that can communicate with each other and hot-swap code into the running system, then Elixir and Erlang are good choices. If you have some extremely CPU intensive code, or you like cross compiling to a lightweight binary, or you need static typing (without giving up preemptive scheduling), Go is a decent choice. Elixir and Erlang are not slow by any metric, but there are faster languages. Discord famously had to augment their Elixir code with Rust for example to scale to 11 million users [1]. [1] https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...
        cztomsik 4y ago
        that is much better point, but erlang gives you reliable system by default and you have to try really hard to make it crash, whereas go is just regular compiled language and such programs typically crash more easily/often.
        ksbrooksjr 4y ago
        Erlang is definitely more fault tolerant than most languages, but I've found that static typing tends to catch a lot of errors in development that would otherwise crash an application in production. The compiler won't catch every bug, and you'll still typically have to restart a crashed service periodically (via systemd, or a container orchestrator, or whatever process manager you use), but it definitely helps. Gleam is a pretty good choice if you need type safety and you want to run on the BEAM https://gleam.run/ https://gleam.run/. It still has the same performance characteristics as Erlang (which it compiles to), but at least it gives you type safety.
- ogogmad 4y ago
  Regex DoSes can be made impossible without using any concurrency. See Go's and Rust's implementation. Also see https://swtch.com/~rsc/regexp/ https://swtch.com/~rsc/regexp/
  - ksbrooksjr 4y ago
    I wasn't trying to imply that concurrency is the only way to mitigate ReDos attacks, I was just pointing out that the BEAM applies preemption seemingly everywhere, which I found interesting.
    - toast0 4y ago
      The BEAM misbehaves when a scheduler gets stuck in a process (or other activity) for too long. As a result, anything that can take a long time needs to have yield points. This is a human process; when I started using Erlang, garbage collection didn't yield, List1 ++ List2 took forever with large lists and didn't yield, we didn't have line numbers in backtraces, and we had to hotload our code uphill in the snow both ways. It's not unusual for a new OTP release to have added new yield points in BIFs or NIFs that could run long or even in core VM workings.
    - ogogmad 4y ago
      There was someone recently on HN who didn't know about the possibility of non-backtracking implementations of regex.
Fire-Dragon-DoL 4y ago
https://stackoverflow.com/a/73932230/312907 https://stackoverflow.com/a/73932230/312907 I found this answer. I can't find the original announcement, but I found the accepted Go proposal: https://github.com/golang/go/issues/24543 https://github.com/golang/go/issues/24543 I remember that being the straw that made me drop Elixir (which I love, to be clear). Go excels in many, many places where Elixir does, but it's way faster. I do think Elixir has immutability on their side, which is huge for new developers, but there are way less developers in Elixir than Go, so the end result doesn't change unfortunately.
- finder83 4y ago
  Thanks for sharing! That's really exciting, going to have to make sure we're on the latest Go version everywhere.
deleted 4y ago
[deleted]

Interesting, at least when I was using it the goroutines themselves were cooperative, and of course the OS threads were preemptive when GO_MAXPROCS > 1. Not finding much with a search, there was a proposal to make them preemptive. Curious if others will chime in that it's now preemptive. Even so I like Elixir better as a language, but the processing speed of Go with a preemptive scheduler would be tempting for some use cases.

ksbrooksjr 4y ago

I'm not a Go expert but preemptive scheduling of goroutines is supposed to be one of the primary benefits of Go as a language [1]. Although I'm not sure if there's any runtime that adopts preemptive scheduling as fervently as the BEAM. Even the regular expression engine is preemptive in Erlang, which prevents regular expression denial of service attacks [2]. [1] https://go.dev/src/runtime/preempt.go https://go.dev/src/runtime/preempt.go [2] https://www.erlang.org/doc/man/re.html#:~:text=run/3%20always%20gives%20control%20back%20to%20the%20scheduler%20of%20Erlang%20processes%20at%20intervals%20that%20ensures%20the%20real%2Dtime%20properties%20of%20the%20Erlang%20system https://www.erlang.org/doc/man/re.html#:~:text=run/3%20alway....

Fire-Dragon-DoL 4y ago
It wasn't non-cooperative pre-emptive scheduling for a while, which is why I kept pushing Elixir, until Go implemented it. Now Go has the same benefit as Elixir, but it's very, very fast.
- ksbrooksjr 4y ago
  Yeah with preemptive scheduling, static typing, and better cpu utilization Go is a pretty compelling option for Erlang and Elixir users.
  - cztomsik 4y ago
    It's not immutable, there is no messaging, no hot-swap, no supervisors, it's not FP, and it doesn't have transparent support for clustering. Also, it's not memory-safe, 30ys old, and battle-tested pretty much everywhere.
    - ksbrooksjr 4y ago
      There are definitely trade offs when switching from Elixir or Erlang to Go. If you're a functional programmer who can't live without immutability, or you plan on running a cluster of machines that can communicate with each other and hot-swap code into the running system, then Elixir and Erlang are good choices. If you have some extremely CPU intensive code, or you like cross compiling to a lightweight binary, or you need static typing (without giving up preemptive scheduling), Go is a decent choice. Elixir and Erlang are not slow by any metric, but there are faster languages. Discord famously had to augment their Elixir code with Rust for example to scale to 11 million users [1]. [1] https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...
      - cztomsik 4y ago
        that is much better point, but erlang gives you reliable system by default and you have to try really hard to make it crash, whereas go is just regular compiled language and such programs typically crash more easily/often.
        ksbrooksjr 4y ago
        Erlang is definitely more fault tolerant than most languages, but I've found that static typing tends to catch a lot of errors in development that would otherwise crash an application in production. The compiler won't catch every bug, and you'll still typically have to restart a crashed service periodically (via systemd, or a container orchestrator, or whatever process manager you use), but it definitely helps. Gleam is a pretty good choice if you need type safety and you want to run on the BEAM https://gleam.run/ https://gleam.run/. It still has the same performance characteristics as Erlang (which it compiles to), but at least it gives you type safety.
ogogmad 4y ago
Regex DoSes can be made impossible without using any concurrency. See Go's and Rust's implementation. Also see https://swtch.com/~rsc/regexp/ https://swtch.com/~rsc/regexp/
- ksbrooksjr 4y ago
  I wasn't trying to imply that concurrency is the only way to mitigate ReDos attacks, I was just pointing out that the BEAM applies preemption seemingly everywhere, which I found interesting.
  - toast0 4y ago
    The BEAM misbehaves when a scheduler gets stuck in a process (or other activity) for too long. As a result, anything that can take a long time needs to have yield points. This is a human process; when I started using Erlang, garbage collection didn't yield, List1 ++ List2 took forever with large lists and didn't yield, we didn't have line numbers in backtraces, and we had to hotload our code uphill in the snow both ways. It's not unusual for a new OTP release to have added new yield points in BIFs or NIFs that could run long or even in core VM workings.
  - ogogmad 4y ago
    There was someone recently on HN who didn't know about the possibility of non-backtracking implementations of regex.

Fire-Dragon-DoL 4y ago

https://stackoverflow.com/a/73932230/312907 https://stackoverflow.com/a/73932230/312907 I found this answer. I can't find the original announcement, but I found the accepted Go proposal: https://github.com/golang/go/issues/24543 https://github.com/golang/go/issues/24543 I remember that being the straw that made me drop Elixir (which I love, to be clear). Go excels in many, many places where Elixir does, but it's way faster. I do think Elixir has immutability on their side, which is huge for new developers, but there are way less developers in Elixir than Go, so the end result doesn't change unfortunately.

finder83 4y ago
Thanks for sharing! That's really exciting, going to have to make sure we're on the latest Go version everywhere.

deleted 4y ago

[deleted]