6 ms·

M4 (computer language)

70 points by laktak 8y ago

fusiongyro 8y ago
M4 is an interesting macro language, but developed (IMO) a somewhat undeserved reputation for being awful because it is how you build autoconf inputs as well as sendmail configuration files. Guilt by association. M4 has some "interesting" quirks but I have used it to build a static site. I briefly was maintaining my blog using make and M4. I no longer remember what the precise reason was that I stopped doing that, probably just that I wanted a through-the-web editor for content. I could dig up the "code" if there were interest in that.
- chris_wot 8y ago
  You should definitely put that in GitHub or the like :-)
  - totalperspectiv 8y ago
    I would also love to see that!
- gizmo686 8y ago
  My experience with M4 is using it to write SELinux policy. The reference policy (which serves as a baseline to build build custom policy on top of) is essentially a DSL implemented with M4. Its actually a good language, and looks a lot like how we would have designed it if we were actually going to write it properly [0]. However, debugging it when something goes wrong (eg, what should be a syntax error) is atrocious. I'm sure it is great if you stick to using it for macros. However, at this point, I view it in the same light as Microsoft Excel: good at what it does, but too powerful for its own good. If your not careful, it will grow until you wish you had been using a proper programming language from the begging. [0] Except for the backtick as opening quote. Objectively, I think this is a good design decision for any language, but it takes a lot of getting used to.
  - lolikoisuru 8y ago
    >[0] Except for the backtick as opening quote. Objectively, I think this is a good design decision for any language, but it takes a lot of getting used to. The `quoting' in m4 is pretty nice since it makes the quotes easy to nest. Sure braces would also be an option but I don't see any fundamental difference between the two, just visual.
- lolikoisuru 8y ago
  > I briefly was maintaining my blog using make and M4. I also wrote a static blog generator with M4 and GNU Make recently. It was the second time I've used M4 and it was pretty great. The first was a templating (probably missusing the term) system for my dotfiles to make them work across many distros and machines I have.
- wott 8y ago
  > M4 is an interesting macro language, but developed (IMO) a somewhat undeserved reputation for being awful because it is how you build autoconf inputs as well as sendmail configuration files. Guilt by association. Hmm...well, I remember my first Linux experience (1995): I rather messed with straight sendmail.cf than struggled with the M4 scripts which generated it. I never got those damn M4 scripts to do what I wanted. I guess M4 and I are incompatible.
- keithpeter 8y ago
  https://datagrok.github.io/makebakery/ https://datagrok.github.io/makebakery/ I'm quite interested in using a directory tree of markdown files to generate a web site so that the directory names become index files showing lists of titles contained within the markdown files &c Looks like make (possibly with m4 macros) could provide that
- tinus_hn 8y ago
  M4 used to be the easy way to manage Sendmail configuration.
totalperspectiv 8y ago
m4 is a fantastic utility. I only recently discovered it but was using it + make to create alternate builds of docker images.
kazinator 8y ago
Here is a GNU Makefile snippet I produced many years ago. It lets you have files preprocessed with the shell, C preprocessor (GNU cpp) or M4: %: %.ct cpp -P $(shell env | sed -e "s/.*/-D_ENV_'&'/") $< > $@ %: %.st ( export __FILE__=$< ; echo "cat <<!" ; cat $< ; echo "!" ) | bash > $@ %: %.mt rm -f $@.err m4 -D__FILE__=$< $(shell env | sed -e "s/$[^=]*$=$.*$/-D'_ENV_\\1=\\2'/") $< > $@ if [ -s $@.err ] ; then cat $@.err ; exit 1 ; fi rm -f $@.err .ct files are "C preprocessor Templates". Environment variables are available as preprocessor symbols prefixed with _ENV. .st files are "Shell here doc Templates". The content of the file is treated as the body of a here doc. The name of the file is available as __FILE__; other environoment variables are present naturally under their ordinary names. .mt files are "M4 Template". The __FILE__ macro is available with the name of the file. The environment is available as _ENV_* namespaced macros, like in the .ct case. I used this in a from-scratch embedded distro for preprocessing files in /etc and such.
lukeh 8y ago
Around 1995 I built some configuration management software based on M4, because I thought it was cool! Actually, it didn't do much in the way of management, but it did allow the ISP I was working for at the time to provision a customer gateway in a single step. Saved me a lot of error-prone and rather boring hand-editing of configuration files... (and probably saved the company some money, who knows)
1wd 8y ago
M4 is of historical interest, but nowadays it seem just using a templating system would be much better in almost all cases, no? Modern systems like Jinja2 or T4 are IMO actually more powerful, easier to to use, maintain and debug, and just plain nicer. Am I missing something?
- enriquto 8y ago
  What is the difference between a macro language and a templating system? I do some m4 from time to time and it is actually very easy to use and really nice.
  - 1wd 8y ago
    I'm not sure. I get the impression with such old-school macro languages (like M4) you have to use a lot of quoting tricks, evaluation order hacks, and use low level primitives like divert and dnl to build up even the simplest of useful things. Many scripts seem to start by inventing a looping construct. While modern templating systems (like Jinja2 or T4) use a proper programming language (like Python or C#) so using even high-level constructs and complex data models is trivial.
    - yayana 8y ago
      I think this last paragraph is a bit misleading: > use a proper programming language m4 is a small proper language that can be used directly. jinja2 is a DSL written in python that doesn't actually have access to much of python. m4 is harder to work with in the domain (quoting, escaping, and data modeling) since it is a general purpose language, but it is easier to do general purpose processing tasks with m4 if they violate the stereotypes of what should be done in the domain.
  - alxlaz 8y ago
    A macro processing language is pretty much a language purpose-built for describing how to copy text from here to there while performing certain changes in the process. A templating system is a set of language-specific constructs (a library, a module, whatever they happen to call in that language) that essentially deals only with the latter (i.e. describing how to perform the changes). It's generally up to you to deal with the "copy text from here to there" part, although most templating engines give you a specific interface that you have to adhere to. I suppose the difference is better illustrated by PyExpander ( http://pyexpander.sourceforge.net/ http://pyexpander.sourceforge.net/ ) which is a macro processing language based on python vs. Jinja (http://jinja.pocoo.org/ http://jinja.pocoo.org/) which is a templating engine for Python. I don't think any general statement about which on is "better" can be meaningful. I suppose that, if you have a full project already written in one language, with all work performed by a single program, it's easier to get what you need via templating engine. If your project is already a collection of tools, whose outputs you need to tie together, it's often less effort to bring in a macro language than write your processing logic from scratch in a non-macro language just to leverage a templating engine. Assuming, of course, that you have someone who knows the macro language in your team ;-). If all your team knows is Jinja2, you're gonna get Jinja2. FWIW, I also do a little M4 from time to time (and a long time ago I also worked with GPP) and find both of them fairly easy to use.
softbuilder 8y ago
About 5 years ago we had a nice talk on M4 from Bart Massey of Portland State. He gives a nice survey of the language and a demonstration: https://www.youtube.com/watch?v=ULZxHSPWn98 https://www.youtube.com/watch?v=ULZxHSPWn98
ubercow 8y ago
What's the best introduction out there for someone who's never touched M4 before?
- jolmg 8y ago
  The GNU manual is pretty good. You can run `info m4` or read it from the links here: https://www.gnu.org/software/m4/manual/index.html https://www.gnu.org/software/m4/manual/index.html
  - lolikoisuru 8y ago
    The default info pager is pretty horrible. I would recommend installing pinfo (usually available in official repos) which is an info pager with user interace similiar to lynx.
- kevin_thibedeau 8y ago
  Michael Breen has a nice introduction to m4. http://mbreen.com/m4.html http://mbreen.com/m4.html
hzhou321 8y ago
M4 is a general-purpose preprocessor. Its preprocessing ability focuses on the word level and it blends macros in appearance to the look of the target language on purpose. I especially having trouble with the latter. In editing, I would like to have a "macro" mode and "language" mode, and being confused about which mode I am in -- during writing or reading -- is not good. Then I also find the word level macros are too bottom-up. The most needed preprocessing is top-down management. `include` is an obvious example, `if-else` too. But a scoped block level facility is even more useful. As an alternative, I have developed and used MyDef: http://hz2.org/blog/mydef_general.html http://hz2.org/blog/mydef_general.html
ameixaseca 8y ago
I suggest to anyone considering M4 seriously to have a look at its substitution behaviour and syntax rules. Not only the default newline handling causes rules to become cluttered the more complex your macros get but the resubstitution of already parsed tokens recursively and the quoting that needs to be considered to avoid this can cause surprises - specially as the number of macros grows bigger. Maybe for config files or configuring source files m4 is great, but for anything more complex that need to be easier to read and understood, I'd avoid m4.

70 points by laktak 8y ago

M4 is an interesting macro language, but developed (IMO) a somewhat undeserved reputation for being awful because it is how you build autoconf inputs as well as sendmail configuration files. Guilt by association. M4 has some "interesting" quirks but I have used it to build a static site. I briefly was maintaining my blog using make and M4. I no longer remember what the precise reason was that I stopped doing that, probably just that I wanted a through-the-web editor for content. I could dig up the "code" if there were interest in that.

chris_wot 8y ago
You should definitely put that in GitHub or the like :-)
- totalperspectiv 8y ago
  I would also love to see that!
gizmo686 8y ago
My experience with M4 is using it to write SELinux policy. The reference policy (which serves as a baseline to build build custom policy on top of) is essentially a DSL implemented with M4. Its actually a good language, and looks a lot like how we would have designed it if we were actually going to write it properly [0]. However, debugging it when something goes wrong (eg, what should be a syntax error) is atrocious. I'm sure it is great if you stick to using it for macros. However, at this point, I view it in the same light as Microsoft Excel: good at what it does, but too powerful for its own good. If your not careful, it will grow until you wish you had been using a proper programming language from the begging. [0] Except for the backtick as opening quote. Objectively, I think this is a good design decision for any language, but it takes a lot of getting used to.
- lolikoisuru 8y ago
  >[0] Except for the backtick as opening quote. Objectively, I think this is a good design decision for any language, but it takes a lot of getting used to. The `quoting' in m4 is pretty nice since it makes the quotes easy to nest. Sure braces would also be an option but I don't see any fundamental difference between the two, just visual.
lolikoisuru 8y ago
> I briefly was maintaining my blog using make and M4. I also wrote a static blog generator with M4 and GNU Make recently. It was the second time I've used M4 and it was pretty great. The first was a templating (probably missusing the term) system for my dotfiles to make them work across many distros and machines I have.
wott 8y ago
> M4 is an interesting macro language, but developed (IMO) a somewhat undeserved reputation for being awful because it is how you build autoconf inputs as well as sendmail configuration files. Guilt by association. Hmm...well, I remember my first Linux experience (1995): I rather messed with straight sendmail.cf than struggled with the M4 scripts which generated it. I never got those damn M4 scripts to do what I wanted. I guess M4 and I are incompatible.
keithpeter 8y ago
https://datagrok.github.io/makebakery/ https://datagrok.github.io/makebakery/ I'm quite interested in using a directory tree of markdown files to generate a web site so that the directory names become index files showing lists of titles contained within the markdown files &c Looks like make (possibly with m4 macros) could provide that
tinus_hn 8y ago
M4 used to be the easy way to manage Sendmail configuration.

totalperspectiv 8y ago

m4 is a fantastic utility. I only recently discovered it but was using it + make to create alternate builds of docker images.

kazinator 8y ago

Here is a GNU Makefile snippet I produced many years ago. It lets you have files preprocessed with the shell, C preprocessor (GNU cpp) or M4: %: %.ct cpp -P $(shell env | sed -e "s/.*/-D_ENV_'&'/") $< > $@ %: %.st ( export __FILE__=$< ; echo "cat <<!" ; cat $< ; echo "!" ) | bash > $@ %: %.mt rm -f $@.err m4 -D__FILE__=$< $(shell env | sed -e "s/$[^=]*$=$.*$/-D'_ENV_\\1=\\2'/") $< > $@ if [ -s $@.err ] ; then cat $@.err ; exit 1 ; fi rm -f $@.err .ct files are "C preprocessor Templates". Environment variables are available as preprocessor symbols prefixed with _ENV. .st files are "Shell here doc Templates". The content of the file is treated as the body of a here doc. The name of the file is available as __FILE__; other environoment variables are present naturally under their ordinary names. .mt files are "M4 Template". The __FILE__ macro is available with the name of the file. The environment is available as _ENV_* namespaced macros, like in the .ct case. I used this in a from-scratch embedded distro for preprocessing files in /etc and such.

lukeh 8y ago

Around 1995 I built some configuration management software based on M4, because I thought it was cool! Actually, it didn't do much in the way of management, but it did allow the ISP I was working for at the time to provision a customer gateway in a single step. Saved me a lot of error-prone and rather boring hand-editing of configuration files... (and probably saved the company some money, who knows)

1wd 8y ago

M4 is of historical interest, but nowadays it seem just using a templating system would be much better in almost all cases, no? Modern systems like Jinja2 or T4 are IMO actually more powerful, easier to to use, maintain and debug, and just plain nicer. Am I missing something?

enriquto 8y ago
What is the difference between a macro language and a templating system? I do some m4 from time to time and it is actually very easy to use and really nice.
- 1wd 8y ago
  I'm not sure. I get the impression with such old-school macro languages (like M4) you have to use a lot of quoting tricks, evaluation order hacks, and use low level primitives like divert and dnl to build up even the simplest of useful things. Many scripts seem to start by inventing a looping construct. While modern templating systems (like Jinja2 or T4) use a proper programming language (like Python or C#) so using even high-level constructs and complex data models is trivial.
  - yayana 8y ago
    I think this last paragraph is a bit misleading: > use a proper programming language m4 is a small proper language that can be used directly. jinja2 is a DSL written in python that doesn't actually have access to much of python. m4 is harder to work with in the domain (quoting, escaping, and data modeling) since it is a general purpose language, but it is easier to do general purpose processing tasks with m4 if they violate the stereotypes of what should be done in the domain.
- alxlaz 8y ago
  A macro processing language is pretty much a language purpose-built for describing how to copy text from here to there while performing certain changes in the process. A templating system is a set of language-specific constructs (a library, a module, whatever they happen to call in that language) that essentially deals only with the latter (i.e. describing how to perform the changes). It's generally up to you to deal with the "copy text from here to there" part, although most templating engines give you a specific interface that you have to adhere to. I suppose the difference is better illustrated by PyExpander ( http://pyexpander.sourceforge.net/ http://pyexpander.sourceforge.net/ ) which is a macro processing language based on python vs. Jinja (http://jinja.pocoo.org/ http://jinja.pocoo.org/) which is a templating engine for Python. I don't think any general statement about which on is "better" can be meaningful. I suppose that, if you have a full project already written in one language, with all work performed by a single program, it's easier to get what you need via templating engine. If your project is already a collection of tools, whose outputs you need to tie together, it's often less effort to bring in a macro language than write your processing logic from scratch in a non-macro language just to leverage a templating engine. Assuming, of course, that you have someone who knows the macro language in your team ;-). If all your team knows is Jinja2, you're gonna get Jinja2. FWIW, I also do a little M4 from time to time (and a long time ago I also worked with GPP) and find both of them fairly easy to use.

softbuilder 8y ago

About 5 years ago we had a nice talk on M4 from Bart Massey of Portland State. He gives a nice survey of the language and a demonstration: https://www.youtube.com/watch?v=ULZxHSPWn98 https://www.youtube.com/watch?v=ULZxHSPWn98

ubercow 8y ago

What's the best introduction out there for someone who's never touched M4 before?

jolmg 8y ago
The GNU manual is pretty good. You can run `info m4` or read it from the links here: https://www.gnu.org/software/m4/manual/index.html https://www.gnu.org/software/m4/manual/index.html
- lolikoisuru 8y ago
  The default info pager is pretty horrible. I would recommend installing pinfo (usually available in official repos) which is an info pager with user interace similiar to lynx.
kevin_thibedeau 8y ago
Michael Breen has a nice introduction to m4. http://mbreen.com/m4.html http://mbreen.com/m4.html

hzhou321 8y ago

M4 is a general-purpose preprocessor. Its preprocessing ability focuses on the word level and it blends macros in appearance to the look of the target language on purpose. I especially having trouble with the latter. In editing, I would like to have a "macro" mode and "language" mode, and being confused about which mode I am in -- during writing or reading -- is not good. Then I also find the word level macros are too bottom-up. The most needed preprocessing is top-down management. `include` is an obvious example, `if-else` too. But a scoped block level facility is even more useful. As an alternative, I have developed and used MyDef: http://hz2.org/blog/mydef_general.html http://hz2.org/blog/mydef_general.html

ameixaseca 8y ago

I suggest to anyone considering M4 seriously to have a look at its substitution behaviour and syntax rules. Not only the default newline handling causes rules to become cluttered the more complex your macros get but the resubstitution of already parsed tokens recursively and the quoting that needs to be considered to avoid this can cause surprises - specially as the number of macros grows bigger. Maybe for config files or configuring source files m4 is great, but for anything more complex that need to be easier to read and understood, I'd avoid m4.