4 ms·

I think that the metaprogramming situation is a little problematic in Elixir. Elixir makes heavy use of macros in basic libraries like the Phoenix or Ecto frame

I think that the metaprogramming situation is a little problematic in Elixir. Elixir makes heavy use of macros in basic libraries like the Phoenix or Ecto frameworks that are used by more or less everybody. I understand that macros are useful to build a DSL for a specific project but using macros in basic libraries like this leads to "way too much magic": Trying to understand (or somehow extend) Ecto is not really possible for all practical purposes and following the control flow in Phoenix is like a maze because of all the macro substitutions. This also leads to cryptic errors where you get an error in non existant lines of code. This heavy of macros is my main complain about Elixir and I think it's one of the main problems that new people face when trying the language resulting in either not using it at all (or if they start it they will abandon it). Finally, I understand that using macros in these libs allows for much more compact (and maybe beautiful) code, however I will always pick a more verbose (but less magic) version of this if I had the choice. I think this is the case for most programmers that want to use a language as a tool to do their job.

msie 5y ago
I agree. I was trying to learn Phoenix but had a hard time with the DSL because it was so non-functional in appearance.
- dqv 5y ago
  Are you talking about routes or Phoenix.HTML? Those are the only two places I can think of having a DSL. Phoenix.HTML isn't strictly required and the routes look like data to me.
cuddlecake 5y ago
Not sure if I would consider Phoenix basic. As far as I know, it's a wrapper around (and configurator for) Cowboy and Plug. I would consider Cowboy basic and Phoenix advanced.
iudqnolq 5y ago
It depends on what you mean by extending Ecto. I've read bits of the source code to better understand what commands are doing, and you're right you can't use an IDE to jump from the public api to the internal implementation. However, the project has put a lot of work into an extremely clear internal naming convention, so you can just go to the implementation directory and grep for the public api name. If you mean writing your own command that composes like the builtin ones, that's extremely easy. Often you can do it with just a plain function, and rarely you need a macro. Suppose you have something like this (haven't written elixir in a while, syntax may be slightly wrong) def where_admin_over(q, age) do: where(q, [u], u.age > ^age and u.is_admin) end def foo do from(User) |> where_admin_over(18) |> where([u], u.country == "de") |> Repo.all() end It's also quite easy to make that custom command run raw sql.
astrowilson 5y ago
Not sure I agree on Phoenix. There are macros, but only around plugs and routing, not much different to what you’d find in Django and definitely less magic than Rails. Plugs are straightforward function pipelines and a declarative DSL really helps structure your routes. The controllers and channels, which are responsible for coordinating with your domain layer, don’t really add anything in the macro department. Plus there is no inheritance, which in my opinion is the biggest source of confusion on most OO frameworks, often requiring you to chase a behavior up and down the class chain.
dqv 5y ago
I see this criticism a lot but I don't think it has anything to do with macros specifically and more so to do with lack of familiarity with Elixir. I've felt the same way about Django being magic because I had trouble following the class hierarchy. It makes a lot more sense now because I'm more familiar with Python and Django. But even today I'll be looking deeper at something and ask WTF it's doing. In that respect, Elixir codebases are easier to me. The module depth seems "shallower" and I don't have to disambiguate between what behavior is caused by class inheritance or an imported function. When I first tried to use Elixir several years ago Ecto.Schema [0] seemed complex and magical, but then I came to realize it's just converting module attributes to runtime code. There is not really that much complex macro logic going on. >This also leads to cryptic errors where you get an error in non existant lines of code. When was the last time you used Elixir? This isn't a problem I can recall having in the last 4 years or so of using Elixir. >following the control flow in Phoenix is like a maze because of all the macro substitutions. Can you clarify what you mean by this? A specific case as to where this happened for you would help. Phoenix's use of macros is actually pretty light [1] except for some very low level stuff. You can even see how frequently a developer will use macros in Phoenix by searching `__using__` in the codebase [2]. It's not used as much as people think. The majority is for views and controllers and only to provide a very thin layer of support on top of your regular use of code. As an example, the "macro magic" in Phoenix.Controller is just handling some basics for giving a layout and view to Plug and handling fallback actions for exceptions. You could do the plug calls manually and I think it would be safe to not use any macros in your controller code. Another familiarity issue with the language (and any language really) is understanding what is meaningful in a stack trace and what isn't. And the likely cause of the error in the first place. Is it syntax? Is it mistyping a variable? Is a function just used improperly? (wtf is init_p_do_apply and why does it show up in every stacktrace?) You're juggling all these different issues - learning a new paradigm, a new syntax, not knowing how to extend things. It's obviously going to be a little overwhelming and, if not strictly required, we might just pick a different language that we're more familiar with. [0]: https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/schema.ex https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/sch... [1]: https://github.com/thechangelog/changelog.com/blob/master/lib/changelog_web.ex https://github.com/thechangelog/changelog.com/blob/master/li... (not mine, I just go here to show the most frequent use of macros in Phoenix) [2]: https://github.com/phoenixframework/phoenix/search?q=__using__ https://github.com/phoenixframework/phoenix/search?q=__using...
natrys 5y ago
Macros can be de-mystified with proper editor support. In Emacs and Lisp you can (recursively) use "macroexpand" to see what code it produces in-place. I am not an Elixir user (though thinking about hopping on), apparently there is alchemist.el[1] that lets you do that (with caveat, and the project seems dead tbh). [1] https://github.com/tonini/alchemist.el https://github.com/tonini/alchemist.el
- lukaszsamson 5y ago
  Macro expansion has recently been added to elixir-ls[1] and is supported via custom command in vscode extension. [1] https://github.com/elixir-lsp/elixir-ls https://github.com/elixir-lsp/elixir-ls
  - elcritch 5y ago
    That’d be really handy! It’d make Phoenix that much more approachable. With good editor support, I much prefer macros to runtime OOP magic/design patterns.
  - natrys 5y ago
    That's awesome! Now, what needs to happen for this to be supported in generic LSP clients like Emacs lsp-mode/eglot or nvim-lsp?
dnautics 5y ago
Completely disagree with ecto[0] and plug. (I have some issues with ecto, but it has nothing to do with macros). The use of macros in those two libraries is parsimonious and exactly what you want in a DSL. For that matter ExUnit, too, has exactly the 'right' amount of macro usage. Phoenix, on the other hand, drives me up the wall. I would not call Phoenix a "basic library". It's definitely a heavy framework. You can of course use "just Plug" (though very few people do). Finally, I think the bigger problem is when you get devs that "take inspiration" from some of the way that Phoenix uses macros. It's a serious problem when a dev has built a macro that cleverly autogenerates a function call (so you can't search the codebase for that function to chase the call path) is not fun. This is the strategy that the Phoenix Routes (is that the right name? not to be confused with Router) uses. I hate it so much. [0] as an ecto consumer. if you ever try to develop an adapter for ecto, you'll run into these problems. I think for example this is why it took forever for someone to build an ecto3 SQLite adapter.
- tokamak-teapot 5y ago
  Yes I never got into Rails for perhaps the same reason. There were tiny source files with some very succinct and clever code that obviously caused a huge amount to go on behind the scenes but it was all magic and impossible to trace or diagnose anything - for me. Also discoverability simply wasn’t there in Rails and I feel the same with Phoenix. I have no idea what pieces might fit in anywhere. Having to know that you have to create a file with a particular name in a particular place if you want something to happen is a bit of a pain if you can’t find a solid reference on that, but also knowing that this is even a possibility - that a feature exists - means reading and re-reading huge documentation on every release just to have some idea of what might actually exist. It’s very likely that I really just haven’t ‘clicked’ with the right way to work with either, but I fear that the only real way to do so is to become an expert - and that would be a huge time investment. Still, I do enjoy looking at the code of projects using Phoenix, even if I can’t fathom how they knew to create certain files and what to put in them.
  - dnautics 5y ago
    I'm about three years into using elixir professionally and I still kind of hate phoenix. There's a lot more to love in elixir than phoenix, and to be honest, especially if you're getting paid to work on a phoenix codebase, you can kind of grin and bear it, ask for help on slack/elixirforums, and it's mostly fine. There are truly only a handful of things where "you kind of have to know the magic incantations", and it's very easy to find someone who knows them. Often it's a generator. Typically you only need to be shown once per project at most (and everything else is a normal copy/paste away). And the debug lines making no sense is relatively rare. In the best sense of "being a framework", once you're past these problems, you're into pure elixir and all is good again. Jose has stated that fixing internalized pain is a real priority for the elixir ecosystem moving forward, so it's entirely possible that these things will be fixed over time (or it's possible it won't!).

msie 5y ago

I agree. I was trying to learn Phoenix but had a hard time with the DSL because it was so non-functional in appearance.

dqv 5y ago
Are you talking about routes or Phoenix.HTML? Those are the only two places I can think of having a DSL. Phoenix.HTML isn't strictly required and the routes look like data to me.

cuddlecake 5y ago

Not sure if I would consider Phoenix basic. As far as I know, it's a wrapper around (and configurator for) Cowboy and Plug. I would consider Cowboy basic and Phoenix advanced.

iudqnolq 5y ago

It depends on what you mean by extending Ecto. I've read bits of the source code to better understand what commands are doing, and you're right you can't use an IDE to jump from the public api to the internal implementation. However, the project has put a lot of work into an extremely clear internal naming convention, so you can just go to the implementation directory and grep for the public api name. If you mean writing your own command that composes like the builtin ones, that's extremely easy. Often you can do it with just a plain function, and rarely you need a macro. Suppose you have something like this (haven't written elixir in a while, syntax may be slightly wrong) def where_admin_over(q, age) do: where(q, [u], u.age > ^age and u.is_admin) end def foo do from(User) |> where_admin_over(18) |> where([u], u.country == "de") |> Repo.all() end It's also quite easy to make that custom command run raw sql.

astrowilson 5y ago

Not sure I agree on Phoenix. There are macros, but only around plugs and routing, not much different to what you’d find in Django and definitely less magic than Rails. Plugs are straightforward function pipelines and a declarative DSL really helps structure your routes. The controllers and channels, which are responsible for coordinating with your domain layer, don’t really add anything in the macro department. Plus there is no inheritance, which in my opinion is the biggest source of confusion on most OO frameworks, often requiring you to chase a behavior up and down the class chain.

dqv 5y ago

I see this criticism a lot but I don't think it has anything to do with macros specifically and more so to do with lack of familiarity with Elixir. I've felt the same way about Django being magic because I had trouble following the class hierarchy. It makes a lot more sense now because I'm more familiar with Python and Django. But even today I'll be looking deeper at something and ask WTF it's doing. In that respect, Elixir codebases are easier to me. The module depth seems "shallower" and I don't have to disambiguate between what behavior is caused by class inheritance or an imported function. When I first tried to use Elixir several years ago Ecto.Schema [0] seemed complex and magical, but then I came to realize it's just converting module attributes to runtime code. There is not really that much complex macro logic going on. >This also leads to cryptic errors where you get an error in non existant lines of code. When was the last time you used Elixir? This isn't a problem I can recall having in the last 4 years or so of using Elixir. >following the control flow in Phoenix is like a maze because of all the macro substitutions. Can you clarify what you mean by this? A specific case as to where this happened for you would help. Phoenix's use of macros is actually pretty light [1] except for some very low level stuff. You can even see how frequently a developer will use macros in Phoenix by searching `__using__` in the codebase [2]. It's not used as much as people think. The majority is for views and controllers and only to provide a very thin layer of support on top of your regular use of code. As an example, the "macro magic" in Phoenix.Controller is just handling some basics for giving a layout and view to Plug and handling fallback actions for exceptions. You could do the plug calls manually and I think it would be safe to not use any macros in your controller code. Another familiarity issue with the language (and any language really) is understanding what is meaningful in a stack trace and what isn't. And the likely cause of the error in the first place. Is it syntax? Is it mistyping a variable? Is a function just used improperly? (wtf is init_p_do_apply and why does it show up in every stacktrace?) You're juggling all these different issues - learning a new paradigm, a new syntax, not knowing how to extend things. It's obviously going to be a little overwhelming and, if not strictly required, we might just pick a different language that we're more familiar with. [0]: https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/schema.ex https://github.com/elixir-ecto/ecto/blob/master/lib/ecto/sch... [1]: https://github.com/thechangelog/changelog.com/blob/master/lib/changelog_web.ex https://github.com/thechangelog/changelog.com/blob/master/li... (not mine, I just go here to show the most frequent use of macros in Phoenix) [2]: https://github.com/phoenixframework/phoenix/search?q=__using__ https://github.com/phoenixframework/phoenix/search?q=__using...

natrys 5y ago

Macros can be de-mystified with proper editor support. In Emacs and Lisp you can (recursively) use "macroexpand" to see what code it produces in-place. I am not an Elixir user (though thinking about hopping on), apparently there is alchemist.el[1] that lets you do that (with caveat, and the project seems dead tbh). [1] https://github.com/tonini/alchemist.el https://github.com/tonini/alchemist.el

lukaszsamson 5y ago
Macro expansion has recently been added to elixir-ls[1] and is supported via custom command in vscode extension. [1] https://github.com/elixir-lsp/elixir-ls https://github.com/elixir-lsp/elixir-ls
- elcritch 5y ago
  That’d be really handy! It’d make Phoenix that much more approachable. With good editor support, I much prefer macros to runtime OOP magic/design patterns.
- natrys 5y ago
  That's awesome! Now, what needs to happen for this to be supported in generic LSP clients like Emacs lsp-mode/eglot or nvim-lsp?

dnautics 5y ago

Completely disagree with ecto[0] and plug. (I have some issues with ecto, but it has nothing to do with macros). The use of macros in those two libraries is parsimonious and exactly what you want in a DSL. For that matter ExUnit, too, has exactly the 'right' amount of macro usage. Phoenix, on the other hand, drives me up the wall. I would not call Phoenix a "basic library". It's definitely a heavy framework. You can of course use "just Plug" (though very few people do). Finally, I think the bigger problem is when you get devs that "take inspiration" from some of the way that Phoenix uses macros. It's a serious problem when a dev has built a macro that cleverly autogenerates a function call (so you can't search the codebase for that function to chase the call path) is not fun. This is the strategy that the Phoenix Routes (is that the right name? not to be confused with Router) uses. I hate it so much. [0] as an ecto consumer. if you ever try to develop an adapter for ecto, you'll run into these problems. I think for example this is why it took forever for someone to build an ecto3 SQLite adapter.

tokamak-teapot 5y ago
Yes I never got into Rails for perhaps the same reason. There were tiny source files with some very succinct and clever code that obviously caused a huge amount to go on behind the scenes but it was all magic and impossible to trace or diagnose anything - for me. Also discoverability simply wasn’t there in Rails and I feel the same with Phoenix. I have no idea what pieces might fit in anywhere. Having to know that you have to create a file with a particular name in a particular place if you want something to happen is a bit of a pain if you can’t find a solid reference on that, but also knowing that this is even a possibility - that a feature exists - means reading and re-reading huge documentation on every release just to have some idea of what might actually exist. It’s very likely that I really just haven’t ‘clicked’ with the right way to work with either, but I fear that the only real way to do so is to become an expert - and that would be a huge time investment. Still, I do enjoy looking at the code of projects using Phoenix, even if I can’t fathom how they knew to create certain files and what to put in them.
- dnautics 5y ago
  I'm about three years into using elixir professionally and I still kind of hate phoenix. There's a lot more to love in elixir than phoenix, and to be honest, especially if you're getting paid to work on a phoenix codebase, you can kind of grin and bear it, ask for help on slack/elixirforums, and it's mostly fine. There are truly only a handful of things where "you kind of have to know the magic incantations", and it's very easy to find someone who knows them. Often it's a generator. Typically you only need to be shown once per project at most (and everything else is a normal copy/paste away). And the debug lines making no sense is relatively rare. In the best sense of "being a framework", once you're past these problems, you're into pure elixir and all is good again. Jose has stated that fixing internalized pain is a real priority for the elixir ecosystem moving forward, so it's entirely possible that these things will be fixed over time (or it's possible it won't!).