5 ms·

I really am in awe of Elixir. I think it's an amazing language, and it introduced me to Erlang and OTP. I've come from using scala actors, and I can do more i

by anko 9y ago

I really am in awe of Elixir. I think it's an amazing language, and it introduced me to Erlang and OTP. I've come from using scala actors, and I can do more in way less time, less memory, and similar performance if I use Elixir. That said, there are some things that suck so hard about it. The log messages suck. If I want to use them in production and actually ship them to ELK or similar, the format really sucks. Getting information on what happens at compile time vs run time is hard. How do you handle operating system signals? like when your autoscale group decides to scala down and you get a SIGTERM with some time to handle it, what do you do? wrapper scripts suck. Not many great design resources.. I mean I've got lots of books on OTP. But if i'm building a daemon which reads off a queue and talks to some databases - where are some patterns I can follow? How do I handle config? What if I want to hotload it? What command do I run? when do I run an app vs compiling it? What If I want command line args? Where's a good example of writing an app, which is a daemon, and using an escript to reconfigure it at runtime? The error messages. Holy hell. What actually caused a pattern match to not work? what is it expecting? especially a few function calls deep - like interaction between tuples and arrays and keyword lists. Flow / Genstage and getting them into a supervision tree? what's the best practices. I want to introduce it to my workplace but without some of these common patterns easy to find, it's kind of a risk to the business.

plainOldText 9y ago
I know it's probably not the answer you're looking for, but you can learn a lot by just reading the code of popular Elixir libraries, such as Ecto, Phoenix, etc. Error messages have gotten better in Elixir 1.5 The whole ecosystem is actually getting better with each release. For shipping to production I use 'distillery'. You create a 'release' and you can even instruct it to read some arguments from the env vars. But yeah, more resources would definitely be welcome. Hopefully as more people start using the language, more resources will be created.
- anko 9y ago
  I appreciate the response, at any rate. I will read more libraries; hopefully I can find some which use flow. I have used distillery but I find myself going down a pretty big rabbit hole just to release something which in most languages is pretty simple. I will look at it again.
  - andruby 9y ago
    If it's any help: I use docker multistage to build a barebones alpine docker image. It uses distillery. I based my dockerfile on https://gist.github.com/bsedat/16cb74ebc8ab0ed61ac598a129b0a7ea https://gist.github.com/bsedat/16cb74ebc8ab0ed61ac598a129b0a... Blogpost: https://zorbash.com/post/docker-multi-stage-elixir-distillery-releases/ https://zorbash.com/post/docker-multi-stage-elixir-distiller... (I'm not the author)
    - thejosh 9y ago
      I just got the same setup this week. Docker multistage builds are amazing and make builds super small and awesome. You can ship releases without having to install Erlang or elixir on the imag, and just use what distillery has. Distillery really is amazing.
  - tmat 9y ago
    Releases and deployments with Elixir are still the area that needs most work, but it's nothing that would ever prevent me from using it. I have several in production and once we figured it out, it was easy peasy from there on. We have some on DO and some on Heroku for clients. Both work well enough.
- dudul 9y ago
  > You create a 'release' and you can even instruct it to read some arguments from the env vars. That's my main complain about Elixir. Releasing and configuring at runtime is a huge pain. I honestly don't see any value in this whole "compile time configuration". If it's at compile time, I can put it in the code. What I call configuration is things I can change at runtime. The fact that something that is a given in any language (config via env vars) requires an extra dependency in Elixir is kind of sad. Otherwise I love the language, the paradigm and the platform.
  - plainOldText 9y ago
    I think I fail to see your point. You can fetch any environment variable – e.g. `System.get_env("SHELL")` – and do whatever with it. But distillery has its own way of defining configurations. You can still mix and match.
  - anko 9y ago
    Yep, I think there should be some included Config behaviour module which handles the serialising / deserialising of a config file, and genservers can register themselves against it, and when the config changes the registered genservers get terminated and restarted gracefully. I don't think it's very much work - but everytime i try to build something like this I go down this path of trying to work out exactly what the architecture should be like. Maybe it's a config application you include in extra_applications. How would you tell it the format of the file? How could you do validations? And then maybe do an escript or similar for configuring it on the fly - something like "elixir_set <variable> <value>"
crymer11 9y ago
We use https://github.com/rentpath/ex_json_logger https://github.com/rentpath/ex_json_logger to make logs nicer for consumption by ELK and the like.
- anko 9y ago
  pro tip, thanks
agd 9y ago
> The error messages. Holy hell. What actually caused a pattern match to not work? what is it expecting? Elixir 1.5 has introduced improved messages for pattern matching errors in function clauses: https://github.com/elixir-lang/elixir/blob/v1.5/CHANGELOG.md#exceptionblame https://github.com/elixir-lang/elixir/blob/v1.5/CHANGELOG.md...
- anko 9y ago
  This is really great. Maybe it's been working for me recently and I didn't realise.
qohen 9y ago
How do you handle operating system signals? like when your autoscale group decides to scala down and you get a SIGTERM with some time to handle it, what do you do? This might help: from the release notes to Erlang R20 (i.e. latest): A new event manager to handle a subset of OS signals in Erlang It's documented in the docs for kernel here: http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/doc/html/kernel_app.html#erl_signal_server http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/d... You can write your own handler, which you'll install using gen_event (or :gen_event in Elixir), but even if you don't, there's a default handler which handles 3 signals, SIGUSER1, SIGQUIT and SIGTERM. Per the link above, SIGTERM is handled like so: The default handler will terminate Erlang normally. This is equivalent to calling init:stop(). Also worth a look, to see how os:set_signal/2 works (this allows you to decide which signals should be ignored, handled by the default handler or handled by your own handler): http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/doc/html/os.html#set_signal-2 http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/d... (And a shout out to legoscia on Stack Overflow for letting us know this might be available: https://stackoverflow.com/a/41872533 https://stackoverflow.com/a/41872533 )
- anko 9y ago
  This is brilliant, I can't believe I missed it! Thank you
  - qohen 9y ago
    If you hadn't mentioned it, I wouldn't have gone researching it and then finding legocia's 2017 comment to a 2010 StackOverflow thread where he says that perhaps a pull request involving an "event manager for signals" might make it into R20. So, you're welcome...and thank you for getting me to look :-)
abledon 9y ago
Do you play an instrument?
skrebbel 9y ago
I think it's good that this is mentioned for a change. Elixir is great and HN has rightly been raving about it lately, but 95% of the Elixir resources out there are "getting started" level tutorials and blog posts. The language is too young for a lot of patterns and approaches for slightly more complex problems to have settled in people's collective minds. Plus, unfortunately there is a strong culture of reinventing the wheel and not looking a lot at things coming out of much more mature Erlang community (with the notable exception of the OTP, but that's a builtin). For example, Webmachine is amazing and totally ignored in Elixir land - unrightfully so in my opinion. No, I believe that none of these issues are going to last very long - the community is strong, the average skill level of prolific Elixir coders and writers is pretty high, and there's tremendous traction. But currently it's the case and I'd wager that anyone raving on HN about how Elixir is the best thing since sliced bread probably didn't get super much further than a Phoenix CRUD app.
davidw 9y ago
> I mean I've got lots of books on OTP. But if i'm building a daemon which reads off a queue and talks to some databases - where are some patterns I can follow? This is a fair point, even in Erlang land. There are a zillion things encouraging you to "let it crash" and far fewer going beyond that. One thing that doesn't get mentioned often enough is a circuit-breaker like fuse: https://github.com/jlouis/fuse https://github.com/jlouis/fuse This also has some more advanced topics: https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/
josevalim 9y ago
This is very good feedback and show flaws in areas that could be improved. Thank you! I will add some comments below but they are not meant to dispute your claims but to find places where we can improve. > The log messages suck. If I want to use them in production and actually ship them to ELK or similar, the format really sucks. Do you mean the Logger messages? Their format is customizable. Can we make this clearer? > Getting information on what happens at compile time vs run time is hard. Which kind of information do you need? Is it something that would fit the meta-programming guide on the website? Or maybe we are missing a more detailed compiler guide? > How do you handle operating system signals? like when your autoscale group decides to scala down and you get a SIGTERM with some time to handle it, what do you do? wrapper scripts suck. It was not possible cleanly until OTP 20, so it is fairly new. At least, from 19.x onwards, the VM will shutdown cleanly on SIGTERM - which means shutting down all applications and supervision trees. A lot of guides (including the Elixir guides) cover supervisor only through the fault tolerance perspective, while they also encapsulate the start-up and shutdown logic. I have opened an issue for this here: https://github.com/elixir-lang/elixir/issues/6479 https://github.com/elixir-lang/elixir/issues/6479 > What command do I run? when do I run an app vs compiling it? What If I want command line args? Where's a good example of writing an app, which is a daemon, and using an escript to reconfigure it at runtime? Running your code is a matter of starting your application. We don't really have a `main` procedure as an entry point. Your system is designed as a series of applications and you start them. If you are using mix, it should be a matter of `mix run --no-halt`. If you are using releases, it automatically starts the system for you. Where would you expect to find this information? Did you see `mix run` at all? Maybe in the escripts page we should point to `mix run` as it seems an `escript` is really not required in your case? > The error messages. Holy hell. What actually caused a pattern match to not work? what is it expecting? especially a few function calls deep - like interaction between tuples and arrays and keyword lists. Those particular error message were fixed in Elixir v1.5 (with OTP 20+). Pull request is here: https://github.com/elixir-lang/elixir/pull/6127 https://github.com/elixir-lang/elixir/pull/6127 > Flow / Genstage and getting them into a supervision tree? what's the best practices. Please open up an issue. If this is not clear form the documentation, it is a "bug" in the docs. It is also worth mentioning that Ben Marx, Bruce Tate and I are working on book called "Adopting Elixir" (https://pragprog.com/book/tvmelixir/adopting-elixir https://pragprog.com/book/tvmelixir/adopting-elixir). The last three chapters focus exclusively on production aspects and it answers many of the questions here (log configuration, system termination, how to run it, design, etc).

by anko 9y ago

plainOldText 9y ago

I know it's probably not the answer you're looking for, but you can learn a lot by just reading the code of popular Elixir libraries, such as Ecto, Phoenix, etc. Error messages have gotten better in Elixir 1.5 The whole ecosystem is actually getting better with each release. For shipping to production I use 'distillery'. You create a 'release' and you can even instruct it to read some arguments from the env vars. But yeah, more resources would definitely be welcome. Hopefully as more people start using the language, more resources will be created.

anko 9y ago
I appreciate the response, at any rate. I will read more libraries; hopefully I can find some which use flow. I have used distillery but I find myself going down a pretty big rabbit hole just to release something which in most languages is pretty simple. I will look at it again.
- andruby 9y ago
  If it's any help: I use docker multistage to build a barebones alpine docker image. It uses distillery. I based my dockerfile on https://gist.github.com/bsedat/16cb74ebc8ab0ed61ac598a129b0a7ea https://gist.github.com/bsedat/16cb74ebc8ab0ed61ac598a129b0a... Blogpost: https://zorbash.com/post/docker-multi-stage-elixir-distillery-releases/ https://zorbash.com/post/docker-multi-stage-elixir-distiller... (I'm not the author)
  - thejosh 9y ago
    I just got the same setup this week. Docker multistage builds are amazing and make builds super small and awesome. You can ship releases without having to install Erlang or elixir on the imag, and just use what distillery has. Distillery really is amazing.
- tmat 9y ago
  Releases and deployments with Elixir are still the area that needs most work, but it's nothing that would ever prevent me from using it. I have several in production and once we figured it out, it was easy peasy from there on. We have some on DO and some on Heroku for clients. Both work well enough.
dudul 9y ago
> You create a 'release' and you can even instruct it to read some arguments from the env vars. That's my main complain about Elixir. Releasing and configuring at runtime is a huge pain. I honestly don't see any value in this whole "compile time configuration". If it's at compile time, I can put it in the code. What I call configuration is things I can change at runtime. The fact that something that is a given in any language (config via env vars) requires an extra dependency in Elixir is kind of sad. Otherwise I love the language, the paradigm and the platform.
- plainOldText 9y ago
  I think I fail to see your point. You can fetch any environment variable – e.g. `System.get_env("SHELL")` – and do whatever with it. But distillery has its own way of defining configurations. You can still mix and match.
- anko 9y ago
  Yep, I think there should be some included Config behaviour module which handles the serialising / deserialising of a config file, and genservers can register themselves against it, and when the config changes the registered genservers get terminated and restarted gracefully. I don't think it's very much work - but everytime i try to build something like this I go down this path of trying to work out exactly what the architecture should be like. Maybe it's a config application you include in extra_applications. How would you tell it the format of the file? How could you do validations? And then maybe do an escript or similar for configuring it on the fly - something like "elixir_set <variable> <value>"

crymer11 9y ago

We use https://github.com/rentpath/ex_json_logger https://github.com/rentpath/ex_json_logger to make logs nicer for consumption by ELK and the like.

anko 9y ago
pro tip, thanks

agd 9y ago

> The error messages. Holy hell. What actually caused a pattern match to not work? what is it expecting? Elixir 1.5 has introduced improved messages for pattern matching errors in function clauses: https://github.com/elixir-lang/elixir/blob/v1.5/CHANGELOG.md#exceptionblame https://github.com/elixir-lang/elixir/blob/v1.5/CHANGELOG.md...

anko 9y ago
This is really great. Maybe it's been working for me recently and I didn't realise.

qohen 9y ago

How do you handle operating system signals? like when your autoscale group decides to scala down and you get a SIGTERM with some time to handle it, what do you do? This might help: from the release notes to Erlang R20 (i.e. latest): A new event manager to handle a subset of OS signals in Erlang It's documented in the docs for kernel here: http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/doc/html/kernel_app.html#erl_signal_server http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/d... You can write your own handler, which you'll install using gen_event (or :gen_event in Elixir), but even if you don't, there's a default handler which handles 3 signals, SIGUSER1, SIGQUIT and SIGTERM. Per the link above, SIGTERM is handled like so: The default handler will terminate Erlang normally. This is equivalent to calling init:stop(). Also worth a look, to see how os:set_signal/2 works (this allows you to decide which signals should be ignored, handled by the default handler or handled by your own handler): http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/doc/html/os.html#set_signal-2 http://erlang.org/documentation/doc-9.0-rc2/lib/kernel-5.3/d... (And a shout out to legoscia on Stack Overflow for letting us know this might be available: https://stackoverflow.com/a/41872533 https://stackoverflow.com/a/41872533 )

anko 9y ago
This is brilliant, I can't believe I missed it! Thank you
- qohen 9y ago
  If you hadn't mentioned it, I wouldn't have gone researching it and then finding legocia's 2017 comment to a 2010 StackOverflow thread where he says that perhaps a pull request involving an "event manager for signals" might make it into R20. So, you're welcome...and thank you for getting me to look :-)

abledon 9y ago

Do you play an instrument?

skrebbel 9y ago

I think it's good that this is mentioned for a change. Elixir is great and HN has rightly been raving about it lately, but 95% of the Elixir resources out there are "getting started" level tutorials and blog posts. The language is too young for a lot of patterns and approaches for slightly more complex problems to have settled in people's collective minds. Plus, unfortunately there is a strong culture of reinventing the wheel and not looking a lot at things coming out of much more mature Erlang community (with the notable exception of the OTP, but that's a builtin). For example, Webmachine is amazing and totally ignored in Elixir land - unrightfully so in my opinion. No, I believe that none of these issues are going to last very long - the community is strong, the average skill level of prolific Elixir coders and writers is pretty high, and there's tremendous traction. But currently it's the case and I'd wager that anyone raving on HN about how Elixir is the best thing since sliced bread probably didn't get super much further than a Phoenix CRUD app.

davidw 9y ago

> I mean I've got lots of books on OTP. But if i'm building a daemon which reads off a queue and talks to some databases - where are some patterns I can follow? This is a fair point, even in Erlang land. There are a zillion things encouraging you to "let it crash" and far fewer going beyond that. One thing that doesn't get mentioned often enough is a circuit-breaker like fuse: https://github.com/jlouis/fuse https://github.com/jlouis/fuse This also has some more advanced topics: https://www.erlang-in-anger.com/ https://www.erlang-in-anger.com/

josevalim 9y ago

This is very good feedback and show flaws in areas that could be improved. Thank you! I will add some comments below but they are not meant to dispute your claims but to find places where we can improve. > The log messages suck. If I want to use them in production and actually ship them to ELK or similar, the format really sucks. Do you mean the Logger messages? Their format is customizable. Can we make this clearer? > Getting information on what happens at compile time vs run time is hard. Which kind of information do you need? Is it something that would fit the meta-programming guide on the website? Or maybe we are missing a more detailed compiler guide? > How do you handle operating system signals? like when your autoscale group decides to scala down and you get a SIGTERM with some time to handle it, what do you do? wrapper scripts suck. It was not possible cleanly until OTP 20, so it is fairly new. At least, from 19.x onwards, the VM will shutdown cleanly on SIGTERM - which means shutting down all applications and supervision trees. A lot of guides (including the Elixir guides) cover supervisor only through the fault tolerance perspective, while they also encapsulate the start-up and shutdown logic. I have opened an issue for this here: https://github.com/elixir-lang/elixir/issues/6479 https://github.com/elixir-lang/elixir/issues/6479 > What command do I run? when do I run an app vs compiling it? What If I want command line args? Where's a good example of writing an app, which is a daemon, and using an escript to reconfigure it at runtime? Running your code is a matter of starting your application. We don't really have a `main` procedure as an entry point. Your system is designed as a series of applications and you start them. If you are using mix, it should be a matter of `mix run --no-halt`. If you are using releases, it automatically starts the system for you. Where would you expect to find this information? Did you see `mix run` at all? Maybe in the escripts page we should point to `mix run` as it seems an `escript` is really not required in your case? > The error messages. Holy hell. What actually caused a pattern match to not work? what is it expecting? especially a few function calls deep - like interaction between tuples and arrays and keyword lists. Those particular error message were fixed in Elixir v1.5 (with OTP 20+). Pull request is here: https://github.com/elixir-lang/elixir/pull/6127 https://github.com/elixir-lang/elixir/pull/6127 > Flow / Genstage and getting them into a supervision tree? what's the best practices. Please open up an issue. If this is not clear form the documentation, it is a "bug" in the docs. It is also worth mentioning that Ben Marx, Bruce Tate and I are working on book called "Adopting Elixir" (https://pragprog.com/book/tvmelixir/adopting-elixir https://pragprog.com/book/tvmelixir/adopting-elixir). The last three chapters focus exclusively on production aspects and it answers many of the questions here (log configuration, system termination, how to run it, design, etc).