5 ms·

Under the Hood of Macros in Elixir

152 points by unripe_syntax 5y ago

yurishimo 5y ago
Really cool article! I've been learning Elixir for a couple of weeks now and metaprogramming is one thing that trips me up. Especially just browsing the source of Phoenix for the first time, it was/is really hard to follow what's happening. I come from a PHP background, using Laravel, which is regarded as a "magical" framework, but I think a stark difference is the lack of robust metaprogramming in PHP. Almost everything can be traced back to a parent object or a trait. The few magic methods you may see floating around are usually implemented in `__get()` and off the top of my head, I think they usually are only magic getters/setters on models. Laravel has a concept of macros that you can use to extend objects injected from the DI container, but each must be explicitly defined, so grepping for them is easy. In Phoenix, I could end up with a generated function at compile time that is nonexistent in my git repo. Not being able to click through to the definitions of these has tripped me up. A function called `{my-domain}_path()` feels like it should have an explicit definition. Still enjoying Elixir and hopefully I can work with it professionally at some point! I just wish it was easer to trace back where some of these things functions come from when you're using them far away from the definition.
- nesarkvechnep 5y ago
  It’s too early for you to try metaprogramming. It has very simple rules but proficiency with the language is advised when diving into macros. You might want to read Chris McCord’s “Metaprogramming Elixir”. The author is also one of Phoenix’s authors.
  - dugmartin 5y ago
    I agree. I've been using Elixir for almost five years and I wrote my first macro this morning. I needed _slightly_ customized code in each instance of set of GenServers and it turned out a macro was the cleanest way to do it. That said, understanding macros will help you sprinkle in your own "magic" in Phoenix though. I normally import a custom view helpers module inside the lib/<project>_web.ex file in this macro like this: defp view_helpers do quote do ... import <project>Web.ViewHelpers end end Then you can add methods in <project>Web.ViewHelpers and they are automagically imported into all your templates, eg. > lib/<project>_web/view_helpers.ex: defmodule <project>Web.ViewHelpers do use Phoenix.HTML # this allows you to use link(), ~E, etc def tagline, do: "Running light without overbyte" end > lib/<project>_web/templates/foo.html.heex ... <div class="tagline">{tagline()}</div>
    - dqv 5y ago
      And actually the <project>Web macros are a great way to show people when macros are useful. A fun exercise for beginners might be to have them manually import those modules. And it illuminates some aspects of Phoenix in general e.g. "oh I'm importing Phoenix.Controller in all my views." In other words, the fact that you're rendering a view from a controller is not the reason you have controller functions available in your views, it's because your importing Phoenix.Controller when you "use ProjectWeb, :view"
      - dugmartin 5y ago
        Yeah, the "magic" dispatcher code in <project>_web.ex is pretty powerful but I'm not sure it would be easy for beginners to grok it: @doc """ When used, dispatch to the appropriate controller/view/etc. """ defmacro __using__(which) when is_atom(which) do apply(__MODULE__, which, []) end That said I much prefer how Phoenix puts this code explicitly in each project and doesn't hide it in a gem like Rails.
        dqv 5y ago
        For example’s sake, it would be fine to just remove that and do it manually: defmacro __using__(:view), do: view() But I think explaining what each element does in that dispatcher would be helpful. Especially in cases with typos.
- weatherlight 5y ago
  Macros are nice when you are designing a DSL for a framework or a library. otherwise, they should probably be avoided.
  - dnautics 5y ago
    Yeah. It's super tempting to use macros for inheritance (and there are some anointed frameworks that even encourage that -- I'm looking at you ExUnit), but I think that is the wrong way to think of macros. The codebase I work on at work is littered with this antipattern, but I grin and bear it because elixir is such a joy to work with otherwise.
  - waynesonfire 5y ago
    everything is a DSL / framework / library. I'm starting to dislike this idea. The claim is a general programmer should avoid macros, but, some blessed developers are granted this privilege and thus have the tools to create useful -DSL / framework / library- code because they used this advanced capability. And useful code tends to become popular. It's a difficult pill to swallow when a an engineer comes in guns blazing trying to enhance your baby project with macros and thereby introducing a ton of new complexity. This is the issue. Macros are like a developer-ux enhancement. You better have your core library figured out pretty well before investing in this enhancement and most projects never reach that level of maturity. Especially, internal ones. It's not that macros should be avoided, they just need to be introduced at the right time.
    - klibertp 5y ago
      Macros in Elixir are a code-generation facility that happens to be Turing-complete, nothing more. You should use it whenever code generation makes sense. Code generation comes with a set of well-known problems, for example the error messages and stacktraces need special attention. If you feel that the advantages of code generation outweight the disadvantages in your specific situation, you should use it, no matter if you're designing a framework or writing a one-off script. One of the advantages of code generation is hiding the details of implementation, so that they can be changed more easily (because the description of the whole is shorter, and kept in one place). While you get the same benefits in Object Oriented languages via other means (interfaces and abstract classes, Template Class pattern), in Elixir you don't have much choice: you either implement a callback module (ie. behaviour; but then there are cross-module calls at runtime) or generate the code via macros (but then you lose the 1-to-1 correspondence between what's in the file and what's actually defined in the module). Of course, both can be (and frequently are, like in GenServer, Application, Supervisor, etc.) used at the same time. Not using macros until some point in time in the development (after it "reaches that level of maturity") risks getting that point in time wrong. It almost always will be wrong, actually, and more often than not you'll find that it's too late: introducing macros after first spending time on optimizing the functional interface makes you just do the same work for the second time, at which point the improvement in "developer ux" may not be worth it or even noticeable. Erlang doesn't have (well, it does, but not as easily accessed) macros, and the libraries still get written in it - frameworks too - and there's little point in adding macro-based wrappers (beyond some simple default implementation injections) to them, because their functional interfaces are already optimized from the dev UX angle. So, to make the most out of macros, you probably shouldn't wait until "after you're done" with something, but you should introduce them as soon as you see a good use for them, while considering all the potential downsides and pitfals that come with their use. Also of note, while you can compare macros to eg. Python (or Ruby, or Smalltalk) Metaclasses, there is a difference: Python offers a lot of mechanisms that you should prefer using over metaclasses if they would suffice. On the other hand, there's not really much you can use in Elixir, as it's a lot smaller and simpler language. So, while I don't think many Python programmers should be writing metaclasses (though all should know what they are!), I think almost all Elixir programmers should be able to use macros. The macro system in Elixir, BTW, is actually on a simpler side of Turing-complete macro systems, both defmacro from Lisp, syntax-case from Scheme, and imperative macros in Nim (and many others) are all much more complex and harder to use. So I think the Elixir designers wanted people to use macros and made an effort to make it safer and simpler. That's just my impression though, don't quote me on this :)
- sidkshatriya 5y ago
  You are correct about a lot of meta programming in Phoenix. As there is no inheritance in Elixir, macros are used a lot to inject code from another module. So as an example if Phoenix has an Endpoint module and your module is an endpoint the macros (especially “use”) injects endpoint related code. It’s impressive but ultimately all this magic gets too much. It gets very confusing when all these macros run because you have complications like bind_quoted, accumulating in module @ tags etc. And because there are no true static types you need to be very careful. Phoenix is slick and I think it’s doing its best given the underlying platform.
- dnautics 5y ago
  > A function called `{my-domain}_path()` 100% agree. I think it's generally good practice for macros which generate non-hidden functions to surface them as parameters for the user to include in their code so you can easily chase where it's coming from (e.g. EEx.function_from*). On the other hand, at least it's easy to find the source code for any given macro. If you want a gentler, less magickey introduction to elixir for web, I recommend starting with Plug. The good thing is that Phoenix builds on top of plug so everything you will learn is still applicable to Phoenix. The big danger is that you might be dragged kicking and screaming into Phoenix, as I was (but trust that most of the macros is phoenix are there for a reason and they do save time in most use cases at the expense of having to memorize a whole lot of shortcuts)
  - klibertp 5y ago
    Plug is actually very hard to learn, because its design is very simple, but knowing how to use this design to produce the effect you need requires a whole lot of mental gymnastics. A less generic design would account for, for example, "initialize this state on first time this plug is called" (I think something changed here, but don't remember if for the better) and similar things, but then it would be less generic and possibly not flexible enough to support all the things Phoenix and co. need from it. That being said, Plug is still better than Phoenix, so if you're not in rush to build something and can experiment, then learning Plug first will give you better understanding of Phoenix later on.
    - nelsonic 5y ago
      If anyone else is stuck learning Plug, the Elixir School tutorial is a good starting point: https://github.com/dwyl/elixir-plug-tutorial https://github.com/dwyl/elixir-plug-tutorial
eric4smith 5y ago
I still don't get it. Then again I'm not too much of a bright programmer. However Elixir has been paying the bills for 5 years now, so... meh.
- atonse 5y ago
  I literally run a company that builds products on elixir and my elixir code has touched millions during the pandemic, and I'm about 80% sure I'd fail an interview for an elixir developer job if it asked anything about macros, protocols, etc. Because I simply don't understand them (and haven't had the time to play around with them at a "Learn this language" level, as opposed to "build this feature we needed yesterday.") But I've also preferred not to do magical things and instead be explicit. But then again, I'm not building something like Ecto or Phoenix.
  - davidw 5y ago
    I've used Erlang for years, and this is more the mentality with that language. Just do the explicit thing even if it's a bit more code.

152 points by unripe_syntax 5y ago

Really cool article! I've been learning Elixir for a couple of weeks now and metaprogramming is one thing that trips me up. Especially just browsing the source of Phoenix for the first time, it was/is really hard to follow what's happening. I come from a PHP background, using Laravel, which is regarded as a "magical" framework, but I think a stark difference is the lack of robust metaprogramming in PHP. Almost everything can be traced back to a parent object or a trait. The few magic methods you may see floating around are usually implemented in `__get()` and off the top of my head, I think they usually are only magic getters/setters on models. Laravel has a concept of macros that you can use to extend objects injected from the DI container, but each must be explicitly defined, so grepping for them is easy. In Phoenix, I could end up with a generated function at compile time that is nonexistent in my git repo. Not being able to click through to the definitions of these has tripped me up. A function called `{my-domain}_path()` feels like it should have an explicit definition. Still enjoying Elixir and hopefully I can work with it professionally at some point! I just wish it was easer to trace back where some of these things functions come from when you're using them far away from the definition.

nesarkvechnep 5y ago
It’s too early for you to try metaprogramming. It has very simple rules but proficiency with the language is advised when diving into macros. You might want to read Chris McCord’s “Metaprogramming Elixir”. The author is also one of Phoenix’s authors.
- dugmartin 5y ago
  I agree. I've been using Elixir for almost five years and I wrote my first macro this morning. I needed _slightly_ customized code in each instance of set of GenServers and it turned out a macro was the cleanest way to do it. That said, understanding macros will help you sprinkle in your own "magic" in Phoenix though. I normally import a custom view helpers module inside the lib/<project>_web.ex file in this macro like this: defp view_helpers do quote do ... import <project>Web.ViewHelpers end end Then you can add methods in <project>Web.ViewHelpers and they are automagically imported into all your templates, eg. > lib/<project>_web/view_helpers.ex: defmodule <project>Web.ViewHelpers do use Phoenix.HTML # this allows you to use link(), ~E, etc def tagline, do: "Running light without overbyte" end > lib/<project>_web/templates/foo.html.heex ... <div class="tagline">{tagline()}</div>
  - dqv 5y ago
    And actually the <project>Web macros are a great way to show people when macros are useful. A fun exercise for beginners might be to have them manually import those modules. And it illuminates some aspects of Phoenix in general e.g. "oh I'm importing Phoenix.Controller in all my views." In other words, the fact that you're rendering a view from a controller is not the reason you have controller functions available in your views, it's because your importing Phoenix.Controller when you "use ProjectWeb, :view"
    - dugmartin 5y ago
      Yeah, the "magic" dispatcher code in <project>_web.ex is pretty powerful but I'm not sure it would be easy for beginners to grok it: @doc """ When used, dispatch to the appropriate controller/view/etc. """ defmacro __using__(which) when is_atom(which) do apply(__MODULE__, which, []) end That said I much prefer how Phoenix puts this code explicitly in each project and doesn't hide it in a gem like Rails.
      - dqv 5y ago
        For example’s sake, it would be fine to just remove that and do it manually: defmacro __using__(:view), do: view() But I think explaining what each element does in that dispatcher would be helpful. Especially in cases with typos.
weatherlight 5y ago
Macros are nice when you are designing a DSL for a framework or a library. otherwise, they should probably be avoided.
- dnautics 5y ago
  Yeah. It's super tempting to use macros for inheritance (and there are some anointed frameworks that even encourage that -- I'm looking at you ExUnit), but I think that is the wrong way to think of macros. The codebase I work on at work is littered with this antipattern, but I grin and bear it because elixir is such a joy to work with otherwise.
- waynesonfire 5y ago
  everything is a DSL / framework / library. I'm starting to dislike this idea. The claim is a general programmer should avoid macros, but, some blessed developers are granted this privilege and thus have the tools to create useful -DSL / framework / library- code because they used this advanced capability. And useful code tends to become popular. It's a difficult pill to swallow when a an engineer comes in guns blazing trying to enhance your baby project with macros and thereby introducing a ton of new complexity. This is the issue. Macros are like a developer-ux enhancement. You better have your core library figured out pretty well before investing in this enhancement and most projects never reach that level of maturity. Especially, internal ones. It's not that macros should be avoided, they just need to be introduced at the right time.
  - klibertp 5y ago
    Macros in Elixir are a code-generation facility that happens to be Turing-complete, nothing more. You should use it whenever code generation makes sense. Code generation comes with a set of well-known problems, for example the error messages and stacktraces need special attention. If you feel that the advantages of code generation outweight the disadvantages in your specific situation, you should use it, no matter if you're designing a framework or writing a one-off script. One of the advantages of code generation is hiding the details of implementation, so that they can be changed more easily (because the description of the whole is shorter, and kept in one place). While you get the same benefits in Object Oriented languages via other means (interfaces and abstract classes, Template Class pattern), in Elixir you don't have much choice: you either implement a callback module (ie. behaviour; but then there are cross-module calls at runtime) or generate the code via macros (but then you lose the 1-to-1 correspondence between what's in the file and what's actually defined in the module). Of course, both can be (and frequently are, like in GenServer, Application, Supervisor, etc.) used at the same time. Not using macros until some point in time in the development (after it "reaches that level of maturity") risks getting that point in time wrong. It almost always will be wrong, actually, and more often than not you'll find that it's too late: introducing macros after first spending time on optimizing the functional interface makes you just do the same work for the second time, at which point the improvement in "developer ux" may not be worth it or even noticeable. Erlang doesn't have (well, it does, but not as easily accessed) macros, and the libraries still get written in it - frameworks too - and there's little point in adding macro-based wrappers (beyond some simple default implementation injections) to them, because their functional interfaces are already optimized from the dev UX angle. So, to make the most out of macros, you probably shouldn't wait until "after you're done" with something, but you should introduce them as soon as you see a good use for them, while considering all the potential downsides and pitfals that come with their use. Also of note, while you can compare macros to eg. Python (or Ruby, or Smalltalk) Metaclasses, there is a difference: Python offers a lot of mechanisms that you should prefer using over metaclasses if they would suffice. On the other hand, there's not really much you can use in Elixir, as it's a lot smaller and simpler language. So, while I don't think many Python programmers should be writing metaclasses (though all should know what they are!), I think almost all Elixir programmers should be able to use macros. The macro system in Elixir, BTW, is actually on a simpler side of Turing-complete macro systems, both defmacro from Lisp, syntax-case from Scheme, and imperative macros in Nim (and many others) are all much more complex and harder to use. So I think the Elixir designers wanted people to use macros and made an effort to make it safer and simpler. That's just my impression though, don't quote me on this :)
sidkshatriya 5y ago
You are correct about a lot of meta programming in Phoenix. As there is no inheritance in Elixir, macros are used a lot to inject code from another module. So as an example if Phoenix has an Endpoint module and your module is an endpoint the macros (especially “use”) injects endpoint related code. It’s impressive but ultimately all this magic gets too much. It gets very confusing when all these macros run because you have complications like bind_quoted, accumulating in module @ tags etc. And because there are no true static types you need to be very careful. Phoenix is slick and I think it’s doing its best given the underlying platform.
dnautics 5y ago
> A function called `{my-domain}_path()` 100% agree. I think it's generally good practice for macros which generate non-hidden functions to surface them as parameters for the user to include in their code so you can easily chase where it's coming from (e.g. EEx.function_from*). On the other hand, at least it's easy to find the source code for any given macro. If you want a gentler, less magickey introduction to elixir for web, I recommend starting with Plug. The good thing is that Phoenix builds on top of plug so everything you will learn is still applicable to Phoenix. The big danger is that you might be dragged kicking and screaming into Phoenix, as I was (but trust that most of the macros is phoenix are there for a reason and they do save time in most use cases at the expense of having to memorize a whole lot of shortcuts)
- klibertp 5y ago
  Plug is actually very hard to learn, because its design is very simple, but knowing how to use this design to produce the effect you need requires a whole lot of mental gymnastics. A less generic design would account for, for example, "initialize this state on first time this plug is called" (I think something changed here, but don't remember if for the better) and similar things, but then it would be less generic and possibly not flexible enough to support all the things Phoenix and co. need from it. That being said, Plug is still better than Phoenix, so if you're not in rush to build something and can experiment, then learning Plug first will give you better understanding of Phoenix later on.
  - nelsonic 5y ago
    If anyone else is stuck learning Plug, the Elixir School tutorial is a good starting point: https://github.com/dwyl/elixir-plug-tutorial https://github.com/dwyl/elixir-plug-tutorial

eric4smith 5y ago

I still don't get it. Then again I'm not too much of a bright programmer. However Elixir has been paying the bills for 5 years now, so... meh.

atonse 5y ago
I literally run a company that builds products on elixir and my elixir code has touched millions during the pandemic, and I'm about 80% sure I'd fail an interview for an elixir developer job if it asked anything about macros, protocols, etc. Because I simply don't understand them (and haven't had the time to play around with them at a "Learn this language" level, as opposed to "build this feature we needed yesterday.") But I've also preferred not to do magical things and instead be explicit. But then again, I'm not building something like Ecto or Phoenix.
- davidw 5y ago
  I've used Erlang for years, and this is more the mentality with that language. Just do the explicit thing even if it's a bit more code.