6 ms·

Unpacking Elixir: Observability

176 points by lawik 3y ago

nesarkvechnep 3y ago
Observability on the BEAM is not an afterthought.
- giraffe_lady 3y ago
  BEAM is actually as good as every new technology tries to pretend to be.
  - hmmokidk 3y ago
    Seriously. It’s an incredible tool and deserves way more praise.
sieabahlpark 3y ago
[dead]
roblh 3y ago
Love this. Great series of articles too. I’m just getting started with Elixir and I had no idea about a ton of this stuff.
WaffleIronMaker 3y ago
Small typo > their shape very mnuch in compilation
- lawik 3y ago
  Ah yes. Will correct it to "munch" ;)
thibaut_barrere 3y ago
The Erlang "telemetry" module can even be used "applicatively" with good uses. An example of that is a "caching reverse proxy" in use at https://transport.data.gouv.fr/ https://transport.data.gouv.fr/, which helps protect data producers servers from excessive load from data consumers (which we manage for them). "External" requests come to us (via proxy.transport.data.gouv.fr) and if the response is not in cache (for a configured TTL), we forward the request as an "internal" request to the target (3rd party) server. The proxy uses "telemetry" to generate events for these internal/external queries, without knowing how they will be consumed (https://github.com/etalab/transport-site/blob/master/apps/unlock/lib/controller.ex https://github.com/etalab/transport-site/blob/master/apps/un...). The main app receives those events and does what it wants, in this case upserting metrics in the database (https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport/telemetry.ex https://github.com/etalab/transport-site/blob/master/apps/tr...). As a result, it's easy to use LiveView to create a real-time dashboard to show these metrics (https://jumpshare.com/s/rdGtTa47CzlcgOz4jsOI https://jumpshare.com/s/rdGtTa47CzlcgOz4jsOI - low traffic at time of posting) or generate stats for the data producers which are behind the proxy (EDIT: LiveView source code at https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport_web/live/backoffice/proxy_config_live.ex https://github.com/etalab/transport-site/blob/master/apps/tr... and https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport_web/live/backoffice/proxy_config_live.html.heex https://github.com/etalab/transport-site/blob/master/apps/tr...). It's all quite natural to do because it's really baked into the runtime!
- chasers 3y ago
  We use :telemetry to collect usage data per tenant for Supabase Realtime. We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/telemetry/logger.ex https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.
  - thibaut_barrere 3y ago
    Thanks for sharing!
chasers 3y ago
We just had some orphaned database connections (our bug) to a tenant and were able to find and kill them because of Erlang's observability: https://gist.github.com/chasers/e4aa33625e6ea39ecb89f03383dd9e38 https://gist.github.com/chasers/e4aa33625e6ea39ecb89f03383dd... edit: link to gist
- pawelduda 3y ago
  Not possible in your typical Rails setup where remote console spawns a separate process so you can't inspect what's going on with your actual servers. Even if so, the amount of observability is not remotely (pun intended) close. You don't appreciate it until your production puma workers start going haywire and you have no idea why
  - byroot 3y ago
    You can attach to a running Ruby process with rbtrace easily.
innocentoldguy 3y ago
I love this article. I’m about 10 years away from retiring and Elixir/Erlang have been the highlight of my software engineering career because of things like this.
- lawik 3y ago
  Thanks. Glad it resonated :)
jdporter 3y ago
> Your compiled C or C++ program will be very different from the code you wrote Sorry, you just lost beaucoup credibility points. C is just "high-level assembly language". The output looks exactly like the input, modulo some optimizations by the compiler. Every computer programmer knows this. Or should.

176 points by lawik 3y ago

nesarkvechnep 3y ago

Observability on the BEAM is not an afterthought.

giraffe_lady 3y ago
BEAM is actually as good as every new technology tries to pretend to be.
- hmmokidk 3y ago
  Seriously. It’s an incredible tool and deserves way more praise.

sieabahlpark 3y ago

[dead]

roblh 3y ago

Love this. Great series of articles too. I’m just getting started with Elixir and I had no idea about a ton of this stuff.

WaffleIronMaker 3y ago

Small typo > their shape very mnuch in compilation

lawik 3y ago
Ah yes. Will correct it to "munch" ;)

thibaut_barrere 3y ago

The Erlang "telemetry" module can even be used "applicatively" with good uses. An example of that is a "caching reverse proxy" in use at https://transport.data.gouv.fr/ https://transport.data.gouv.fr/, which helps protect data producers servers from excessive load from data consumers (which we manage for them). "External" requests come to us (via proxy.transport.data.gouv.fr) and if the response is not in cache (for a configured TTL), we forward the request as an "internal" request to the target (3rd party) server. The proxy uses "telemetry" to generate events for these internal/external queries, without knowing how they will be consumed (https://github.com/etalab/transport-site/blob/master/apps/unlock/lib/controller.ex https://github.com/etalab/transport-site/blob/master/apps/un...). The main app receives those events and does what it wants, in this case upserting metrics in the database (https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport/telemetry.ex https://github.com/etalab/transport-site/blob/master/apps/tr...). As a result, it's easy to use LiveView to create a real-time dashboard to show these metrics (https://jumpshare.com/s/rdGtTa47CzlcgOz4jsOI https://jumpshare.com/s/rdGtTa47CzlcgOz4jsOI - low traffic at time of posting) or generate stats for the data producers which are behind the proxy (EDIT: LiveView source code at https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport_web/live/backoffice/proxy_config_live.ex https://github.com/etalab/transport-site/blob/master/apps/tr... and https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport_web/live/backoffice/proxy_config_live.html.heex https://github.com/etalab/transport-site/blob/master/apps/tr...). It's all quite natural to do because it's really baked into the runtime!

chasers 3y ago
We use :telemetry to collect usage data per tenant for Supabase Realtime. We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/telemetry/logger.ex https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.
- thibaut_barrere 3y ago
  Thanks for sharing!

chasers 3y ago

We just had some orphaned database connections (our bug) to a tenant and were able to find and kill them because of Erlang's observability: https://gist.github.com/chasers/e4aa33625e6ea39ecb89f03383dd9e38 https://gist.github.com/chasers/e4aa33625e6ea39ecb89f03383dd... edit: link to gist

pawelduda 3y ago
Not possible in your typical Rails setup where remote console spawns a separate process so you can't inspect what's going on with your actual servers. Even if so, the amount of observability is not remotely (pun intended) close. You don't appreciate it until your production puma workers start going haywire and you have no idea why
- byroot 3y ago
  You can attach to a running Ruby process with rbtrace easily.

innocentoldguy 3y ago

I love this article. I’m about 10 years away from retiring and Elixir/Erlang have been the highlight of my software engineering career because of things like this.

lawik 3y ago
Thanks. Glad it resonated :)

jdporter 3y ago

> Your compiled C or C++ program will be very different from the code you wrote Sorry, you just lost beaucoup credibility points. C is just "high-level assembly language". The output looks exactly like the input, modulo some optimizations by the compiler. Every computer programmer knows this. Or should.