5 ms·

> Network requests without timeouts are the top silent killer of distributed systems. YES 1000% (I mean maybe not 'top' but it's up there) languages must move

> Network requests without timeouts are the top silent killer of distributed systems. YES 1000% (I mean maybe not 'top' but it's up there) languages must move connection pooling, timeouts, and retry semantics into the stdlib API client libraries have to do a better job of documenting what happens when a request fails systems need to do a better job of centralizing how timeouts are configured; this can't be left to chance

dnautics 6y ago
May I introduce you to erlang/BEAM? These issues were solved 30 years ago, and the ecosystem has had plenty of time to solidify best practices around exactly what you're asking for.
- Thaxll 6y ago
  What actually BEAM protect you against resources consumption ? If you set o timeout all you process will be up and idle.
  - dnautics 6y ago
    Default BEAM timeout is usually 5s (probably too long, in some cases), if you miss it the default is unhandled exception, which crashes the process that made the call (and only that process, no others). The VM will then recover all of the resources (file descriptors, sockets, data to be GC'd) associated with that process. All in zero lines of code. Also you can have millions of processes per core, with minimal performance regression, do you're likely to notice it in monitoring before it becomes a problem.
- awinter-py 6y ago
  hmm -- this is a compelling sales pitch. What should I read to learn about how erlang API clients manage pooling + timeout logic?
  - dnautics 6y ago
    To be honest I use elixir. Docs are better anyways. For timeouts that information is in the standard library, for example https ://hexdocs.pm/elixir/GenServer.html#call/3 Note that call is even more sophisticated, it incorporates a liveness check on its counterparty to quit out before timeout if there's been a catastrophe. Note this is effectively a simple form of backpressure management that degrades availability gracefully under stress in favor of ensuring the integrity of existing connections. Because fallibility is so baked into the runtime, every good library incorporates timeouts where it's sensible. For connection pools, it's not explicitly part of the standard library but basically everyone (maybe not whatsapp) uses poolboy: https://elixirschool.com/en/lessons/libraries/poolboy/ https://elixirschool.com/en/lessons/libraries/poolboy/ I have never used it myself, yet, but I trust more experienced devs have incorporated it successfully in ecto (rdbms) and Phoenix (web framework).

dnautics 6y ago

May I introduce you to erlang/BEAM? These issues were solved 30 years ago, and the ecosystem has had plenty of time to solidify best practices around exactly what you're asking for.

Thaxll 6y ago
What actually BEAM protect you against resources consumption ? If you set o timeout all you process will be up and idle.
- dnautics 6y ago
  Default BEAM timeout is usually 5s (probably too long, in some cases), if you miss it the default is unhandled exception, which crashes the process that made the call (and only that process, no others). The VM will then recover all of the resources (file descriptors, sockets, data to be GC'd) associated with that process. All in zero lines of code. Also you can have millions of processes per core, with minimal performance regression, do you're likely to notice it in monitoring before it becomes a problem.
awinter-py 6y ago
hmm -- this is a compelling sales pitch. What should I read to learn about how erlang API clients manage pooling + timeout logic?
- dnautics 6y ago
  To be honest I use elixir. Docs are better anyways. For timeouts that information is in the standard library, for example https ://hexdocs.pm/elixir/GenServer.html#call/3 Note that call is even more sophisticated, it incorporates a liveness check on its counterparty to quit out before timeout if there's been a catastrophe. Note this is effectively a simple form of backpressure management that degrades availability gracefully under stress in favor of ensuring the integrity of existing connections. Because fallibility is so baked into the runtime, every good library incorporates timeouts where it's sensible. For connection pools, it's not explicitly part of the standard library but basically everyone (maybe not whatsapp) uses poolboy: https://elixirschool.com/en/lessons/libraries/poolboy/ https://elixirschool.com/en/lessons/libraries/poolboy/ I have never used it myself, yet, but I trust more experienced devs have incorporated it successfully in ecto (rdbms) and Phoenix (web framework).