13 ms·

Why do web servers always seem to invent their own config file format? Whilst nginx seems to do it slightly more sanely than Apache, it still doesn't use someth

Why do web servers always seem to invent their own config file format? Whilst nginx seems to do it slightly more sanely than Apache, it still doesn't use something like YAML or JSON; is there a good/obvious reason for this I'm missing?

rdtsc 11y ago
JSON doesn't allow comments. So it is an ok-ish config format. Yaml would have been better. But they probably wrote everything from scratch to be optimized and decided if they are writing a parser for a config file might as well invent a config file format as well :-)
- FooBarWidget 11y ago
  Nginx was invented before the days of JSON and YAML...
  - nkozyra 11y ago
    JSON, sure, but I think YAML pre-dated the launch of nginx (if not the initial development). https://groups.yahoo.com/neo/groups/sml-dev/conversations/topics/4710 https://groups.yahoo.com/neo/groups/sml-dev/conversations/to...
    - FooBarWidget 11y ago
      YAML initial release: 2001 Nginx initial release: 2004 Well ok. But I'd still argue that Nginx was invented before the popularization of YAML. I think I didn't see YAML until I got in touch with Rails in 2007, which used the format extensively. And even then, almost everyone I met didn't take it seriously, saying that it is a joke until Microsoft, IBM, etc support it.
- CrLf 11y ago
  Not allowing comments doesn't make JSON an "ok-ish" config format, it makes it completely _unsuitable_ for it.
  - rdtsc 11y ago
    But it is used though pretty oftne. That's why it is an -ish format ;-) You can do things like { "comment": ["blah blah comment message"], "k1" : "v1", "k2" : "v2", ... } Or some silliness like that.
- samspot 11y ago
  { "//This": "works though!" }
Zikes 11y ago
> Since its public launch in 2004 YAML's initial release was 2001[0], likely not yet popular or stable enough for consideration. JSON was likewise "released" in 2002[1] with a similar story. In all likelihood, we're probably just lucky Igor didn't go with XML. [0] https://en.wikipedia.org/wiki/YAML https://en.wikipedia.org/wiki/YAML [1] https://en.wikipedia.org/wiki/JSON#History https://en.wikipedia.org/wiki/JSON#History
- euske 11y ago
  (s-expression (cough (cough)))
- pen2l 11y ago
  > In all likelihood, we're probably just lucky Igor didn't go with XML. Okay I'll bite, what's wrong with writing nginx to go with XML? XML is easily translatable to JSON, so... there's that.
  - herge 11y ago
    > XML is easily translatable to JSON Not really. XML is a document markup language, where order is important. JSON is a serialization format, and is a bit more bare-boned. For example, how would you transfer this XML to json, and back to XML? <foo bar="baz"><bork>foo1</bork><bork>foo2</bork></foo>
    - deleted 11y ago
      [deleted]
    - skissane 11y ago
      There are many ways to do it, you just need to pick a convention. For example, one convention is that you use "#" as key for element name, "." as key for element children, and attribute name as keys for the attributes. Following those conventions, you can produce: { "#" = "foo", "bar" = "baz", "." = [ { "#" = "bork", "." = "foo1" }, { "#" = "bork", "." = "foo2" } ] } Or, for something more verbose (but maybe more intelligible), you could use "$element" as key for element names, and "$children" as key for child elements / text. (The point of choosing $ as a prefix, is it is not a valid character in attribute names, so cannot conflict with them.)
blakeyrat 11y ago
I know it's probably not "open source-y" enough for you, but IIS just uses plain ol' XML files for configuration.
- chrshawkes 11y ago
  IIS sucks, not because it's not open source, but because it sucks to work with, when compared to NGINX or Apache.
- CrLf 11y ago
  Is that a positive thing? Of course, like in many other cases, IIS configuration is supposed to be generated from tools, but it always needs some tweaking and XML makes it a total pain to edit.
- GlennS 11y ago
  So I'd say Nginx is actually the least annoying web-server to config of those I've used: 1. Nginx 2. IIS 6 (strange metabase thing) 3. Apache (XML, mostly) 4. IIS 7, 7.5 (XML, but with some of the files scattered through your Windows directory, and also some values aren't valid in some of the files). 5. Tomcat (XML plus madness). I'd say the thing that makes all web-servers a pain is debugging which rules are passing/failing, and where are they sending their results to. I think the thing which makes Nginx easier than the others is probably that it doesn't try to support the 'shared hosting' scenario, which adds a lot of mess.
venning 11y ago
nginx configs can be difficult enough on their own without constraining them to formats that wouldn't allow directives that take arguments preceding a block, not to mention the extra escaping that would come with quoting already-quoted strings. This is a common formulation: location ~* \.(jpe?g|png|gif|ico)$ { ... } That would be pretty messy inside of JSON or YAML. You couldn't have the location line be a key for a map/hash, because you can have multiple blocks with the same "key". Besides, as bad as nginx configs are, just trying to understand which block "traps" a request is where you can spend most of your time; see ifIsEvil [1]. [1] https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ https://www.nginx.com/resources/wiki/start/topics/depth/ifis...
- nisa 11y ago
  Thanks for that comment. Running a complex nginx config with lot's of special treatment for subfolders on nginx - why and where a requests ends up in such a configuration is really difficult to tell. However Apache just can't do some things that help you shooting in your foot. Maybe the C++ vs. Java comparison is not too far fetched.
  - venning 11y ago
    nginx is one of those things that requires its own mentality to really get. Like SQL's set logic or CSS's forward-looking selectors, it's not quite like traditional imperative, control flow-based programming. But once you start thinking the way it wants you to think, there are benefits. I'm in the middle of a project built entirely in nginx and it's astoundingly performant. The restrictions on what I can do in (mainline) nginx force me to think through how I structure the blocks and directives with better logic representative of a web server and not an application server, which is what I'm used to. DigitalOcean has a decent tutorial on now nginx decides on server and location block [1]. And, related to ifIsEvil, this blog post [2] goes a little into explaining how nginx "traps" a request. If someone has better resources, I would appreciate them. [1] https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms https://www.digitalocean.com/community/tutorials/understandi... [2] http://agentzh.blogspot.com/2011/03/how-nginx-location-if-works.html http://agentzh.blogspot.com/2011/03/how-nginx-location-if-wo...
pnommensen 11y ago
Have you seen nginScript (1)? It's a syntax for embedding snippets of JavaScript in your NGINX configuration. This allows for scripting application logic into standard NGINX config files via a very common and simple language. There's some more details and simple examples at this blog post. (1) https://www.nginx.com/blog/launching-nginscript-and-looking-ahead/ https://www.nginx.com/blog/launching-nginscript-and-looking-... (disclaimer: I work at NGINX)
- nikolay 11y ago
  Okay, but you have to escape this in the Nginx config! Same issue with Lua! And this is terrible!
  - bungle 11y ago
    there is these days _by_lua_block.
    - nikolay 11y ago
      Thanks! This is really great: content_by_lua_block { ngx.say("hello, world") }
mholt 11y ago
I invented one and wrote an article about why: https://caddyserver.com/blog/deliberate-caddyfile-syntax https://caddyserver.com/blog/deliberate-caddyfile-syntax (You're definitely not the only one to ask that question.)
- commentzorro 11y ago
  This looks really interesting. Can't believe I haven't run across it before. I've been toying with getting a couple home servers going (replaced home service w/ business service, just installed two router based DMZ, looking at lightweight hardware -- probably will be Fit-PC products). I was going to run a separate reverse proxy and Lighttpd or similar but a quick glance makes it seem like Caddy could be used for both and more easily. Thanks for the link. (Edit. BTW, you're not here: https://en.wikipedia.org/wiki/Comparison_of_web_server_software)* https://en.wikipedia.org/wiki/Comparison_of_web_server_softw...
oblio 11y ago
The real question is: why didn't they use something like Lua for the config file format? Web servers can have notoriously complex configs, up to the point where designing a mini-language might a worse idea than stripping down an embeddable language, such as Lua. Lua, especially when sandboxed, seems like a fitting configuration language: http://stackoverflow.com/questions/1224708/how-can-i-create-a-secure-lua-sandbox http://stackoverflow.com/questions/1224708/how-can-i-create-...
- deathanatos 11y ago
  It's not just that Lua makes a good config file format, it's that serving HTTP requests is a complicated enough business with enough edge cases and corner cases that it merits a turning complete language. I actually have some special corner case API endpoints that nginx just simply can not handle in the manner I would prefer. Further, the regex based location syntax, and even the prefix based ones, are not really what you want; you want a "Path" object that's aware of what / in a URL means, and does the right thing if you do/don't add it to the URL. (And it's not as easy as "/foo/bar/baz/?")
  - SomeCallMeTim 11y ago
    In Nginx, you can use Lua for complex routing and responses, with the right extensions. [1] [1] http://openresty.org/ http://openresty.org/
    - deathanatos 11y ago
      And we do. We don't use openresty (at least at the moment), we use a custom-built nginx that includes — among other things — the lua module. It's a great boon, but something things are still hard or impossible[1], and we don't use it for every request, opting for the location syntax, and all its flaws, for most. [1]. Conditionally streaming an upload to a backend (i.e., if auth fails, don't stream) is impossible; nginx will buffer the entire request body, either in memory or on disk, and there is no way to change this behavior.
nikolay 11y ago
I am highly disappointed by Nginx in three major areas: 1. They created a paid version that has some additional basic features such as cache purging, dynamic upstream name resolution, and a few others. Charge for support, charge for some fancy management interface, monitoring, but for basic features (most of them available in Tengine [0]) - thanks, but no thanks. You lost me as an evangelist! In fact, in may aspects, they now are catching up with Tengine! 2. Instead of making LuaJIT integration standard and avoid the need to escape Lua in the configuration files, they invented some subpar JavaScript. People already use Lua widely, it's fast, it's great - don't you have anything better to do than invent yet another language!? I really can't believe pragmatic people would have done this, honestly! Speaks so badly about their thought process! I know can expect anything stupid from them! 3. The configuration language is not very intuitive. If they embedded Lua, the whole configuration could be a Lua script that initializes some internal state. This would have been a dream come true! [0] http://tengine.taobao.org http://tengine.taobao.org
dec0dedab0de 11y ago
It's not just web servers. I would say that it is because they want it to be human readable/editable, but JSON would probably be just as good in that regard. I would prefer it if it were more like openssh or supervisor. Though I suspect those styles of configs are would make some of the more advanced configurations a pain.
librvf 11y ago
JSON is horrible for a configuration language. Needing to put quotation marks around every single literal is insanely irritating if you ever have to write a lot of configuration. There's also a lot of application-specific syntax sugar that you cannot do if you stick with strict JSON. Note that it's not really fair to call Apache's configuration language "XML". Apache relies on XML for some structured data in its configuration file, but all the individual directives are parsed separately from the XML.
- nikolay 11y ago
  I agree. When people started to move towards the "better" JSON standard from XML, I was puzzled. The inability to have comments, the need to "escape" every key name, the lack (in the past) of a schema and query language standard, etc.
  - librvf 11y ago
    To be fair, while I agree that JSON is bad for config, I would argue that JSON's objects and arrays eliminate the need for DTDs and standard query languages altogether in many cases. Of course, with sufficiently lqrge and complex data stores, you'll want a documented structure and method for accessing that data no matter what format you use, but unlike XML, JSON provides structure to get started on a small scale very easily. Consider this python: import json import sys sys.stdout.write("%s\n" % json.dumps(json.load(sys.stdin).get(sys.argv[1], None))) This will accept JSON on standard input and will return the value of an object with the key name specified by the first command-line argument. $ echo '{ "value1" : { "sub-value": 5 }, "value2": 99 }' | python json-test.py value1 {"sub-value": 5} $ echo '{ "value1" : { "sub-value": 5 }, "value2": 99 }' | python json-test.py value2 99 Such a program will look similar in any language with a json library that maps objects and arrays to native data structures. Granted, my simple tool will fail if the JSON isn't an object, but it's a very simple matter to extend it to handle lists and literals. For many applications this is a huge advantage over XML, especially if the point of the JSON isn't configuration but rather inter-application communication (aka data serialization).

rdtsc 11y ago

JSON doesn't allow comments. So it is an ok-ish config format. Yaml would have been better. But they probably wrote everything from scratch to be optimized and decided if they are writing a parser for a config file might as well invent a config file format as well :-)

FooBarWidget 11y ago
Nginx was invented before the days of JSON and YAML...
- nkozyra 11y ago
  JSON, sure, but I think YAML pre-dated the launch of nginx (if not the initial development). https://groups.yahoo.com/neo/groups/sml-dev/conversations/topics/4710 https://groups.yahoo.com/neo/groups/sml-dev/conversations/to...
  - FooBarWidget 11y ago
    YAML initial release: 2001 Nginx initial release: 2004 Well ok. But I'd still argue that Nginx was invented before the popularization of YAML. I think I didn't see YAML until I got in touch with Rails in 2007, which used the format extensively. And even then, almost everyone I met didn't take it seriously, saying that it is a joke until Microsoft, IBM, etc support it.
CrLf 11y ago
Not allowing comments doesn't make JSON an "ok-ish" config format, it makes it completely _unsuitable_ for it.
- rdtsc 11y ago
  But it is used though pretty oftne. That's why it is an -ish format ;-) You can do things like { "comment": ["blah blah comment message"], "k1" : "v1", "k2" : "v2", ... } Or some silliness like that.
samspot 11y ago
{ "//This": "works though!" }

Zikes 11y ago

> Since its public launch in 2004 YAML's initial release was 2001[0], likely not yet popular or stable enough for consideration. JSON was likewise "released" in 2002[1] with a similar story. In all likelihood, we're probably just lucky Igor didn't go with XML. [0] https://en.wikipedia.org/wiki/YAML https://en.wikipedia.org/wiki/YAML [1] https://en.wikipedia.org/wiki/JSON#History https://en.wikipedia.org/wiki/JSON#History

euske 11y ago
(s-expression (cough (cough)))
pen2l 11y ago
> In all likelihood, we're probably just lucky Igor didn't go with XML. Okay I'll bite, what's wrong with writing nginx to go with XML? XML is easily translatable to JSON, so... there's that.
- herge 11y ago
  > XML is easily translatable to JSON Not really. XML is a document markup language, where order is important. JSON is a serialization format, and is a bit more bare-boned. For example, how would you transfer this XML to json, and back to XML? <foo bar="baz"><bork>foo1</bork><bork>foo2</bork></foo>
  - deleted 11y ago
    [deleted]
  - skissane 11y ago
    There are many ways to do it, you just need to pick a convention. For example, one convention is that you use "#" as key for element name, "." as key for element children, and attribute name as keys for the attributes. Following those conventions, you can produce: { "#" = "foo", "bar" = "baz", "." = [ { "#" = "bork", "." = "foo1" }, { "#" = "bork", "." = "foo2" } ] } Or, for something more verbose (but maybe more intelligible), you could use "$element" as key for element names, and "$children" as key for child elements / text. (The point of choosing $ as a prefix, is it is not a valid character in attribute names, so cannot conflict with them.)

blakeyrat 11y ago

I know it's probably not "open source-y" enough for you, but IIS just uses plain ol' XML files for configuration.

chrshawkes 11y ago
IIS sucks, not because it's not open source, but because it sucks to work with, when compared to NGINX or Apache.
CrLf 11y ago
Is that a positive thing? Of course, like in many other cases, IIS configuration is supposed to be generated from tools, but it always needs some tweaking and XML makes it a total pain to edit.
GlennS 11y ago
So I'd say Nginx is actually the least annoying web-server to config of those I've used: 1. Nginx 2. IIS 6 (strange metabase thing) 3. Apache (XML, mostly) 4. IIS 7, 7.5 (XML, but with some of the files scattered through your Windows directory, and also some values aren't valid in some of the files). 5. Tomcat (XML plus madness). I'd say the thing that makes all web-servers a pain is debugging which rules are passing/failing, and where are they sending their results to. I think the thing which makes Nginx easier than the others is probably that it doesn't try to support the 'shared hosting' scenario, which adds a lot of mess.

venning 11y ago

nginx configs can be difficult enough on their own without constraining them to formats that wouldn't allow directives that take arguments preceding a block, not to mention the extra escaping that would come with quoting already-quoted strings. This is a common formulation: location ~* \.(jpe?g|png|gif|ico)$ { ... } That would be pretty messy inside of JSON or YAML. You couldn't have the location line be a key for a map/hash, because you can have multiple blocks with the same "key". Besides, as bad as nginx configs are, just trying to understand which block "traps" a request is where you can spend most of your time; see ifIsEvil [1]. [1] https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/ https://www.nginx.com/resources/wiki/start/topics/depth/ifis...

nisa 11y ago
Thanks for that comment. Running a complex nginx config with lot's of special treatment for subfolders on nginx - why and where a requests ends up in such a configuration is really difficult to tell. However Apache just can't do some things that help you shooting in your foot. Maybe the C++ vs. Java comparison is not too far fetched.
- venning 11y ago
  nginx is one of those things that requires its own mentality to really get. Like SQL's set logic or CSS's forward-looking selectors, it's not quite like traditional imperative, control flow-based programming. But once you start thinking the way it wants you to think, there are benefits. I'm in the middle of a project built entirely in nginx and it's astoundingly performant. The restrictions on what I can do in (mainline) nginx force me to think through how I structure the blocks and directives with better logic representative of a web server and not an application server, which is what I'm used to. DigitalOcean has a decent tutorial on now nginx decides on server and location block [1]. And, related to ifIsEvil, this blog post [2] goes a little into explaining how nginx "traps" a request. If someone has better resources, I would appreciate them. [1] https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms https://www.digitalocean.com/community/tutorials/understandi... [2] http://agentzh.blogspot.com/2011/03/how-nginx-location-if-works.html http://agentzh.blogspot.com/2011/03/how-nginx-location-if-wo...

pnommensen 11y ago

Have you seen nginScript (1)? It's a syntax for embedding snippets of JavaScript in your NGINX configuration. This allows for scripting application logic into standard NGINX config files via a very common and simple language. There's some more details and simple examples at this blog post. (1) https://www.nginx.com/blog/launching-nginscript-and-looking-ahead/ https://www.nginx.com/blog/launching-nginscript-and-looking-... (disclaimer: I work at NGINX)

nikolay 11y ago
Okay, but you have to escape this in the Nginx config! Same issue with Lua! And this is terrible!
- bungle 11y ago
  there is these days _by_lua_block.
  - nikolay 11y ago
    Thanks! This is really great: content_by_lua_block { ngx.say("hello, world") }

mholt 11y ago

I invented one and wrote an article about why: https://caddyserver.com/blog/deliberate-caddyfile-syntax https://caddyserver.com/blog/deliberate-caddyfile-syntax (You're definitely not the only one to ask that question.)

commentzorro 11y ago
This looks really interesting. Can't believe I haven't run across it before. I've been toying with getting a couple home servers going (replaced home service w/ business service, just installed two router based DMZ, looking at lightweight hardware -- probably will be Fit-PC products). I was going to run a separate reverse proxy and Lighttpd or similar but a quick glance makes it seem like Caddy could be used for both and more easily. Thanks for the link. (Edit. BTW, you're not here: https://en.wikipedia.org/wiki/Comparison_of_web_server_software)* https://en.wikipedia.org/wiki/Comparison_of_web_server_softw...

oblio 11y ago

The real question is: why didn't they use something like Lua for the config file format? Web servers can have notoriously complex configs, up to the point where designing a mini-language might a worse idea than stripping down an embeddable language, such as Lua. Lua, especially when sandboxed, seems like a fitting configuration language: http://stackoverflow.com/questions/1224708/how-can-i-create-a-secure-lua-sandbox http://stackoverflow.com/questions/1224708/how-can-i-create-...

deathanatos 11y ago
It's not just that Lua makes a good config file format, it's that serving HTTP requests is a complicated enough business with enough edge cases and corner cases that it merits a turning complete language. I actually have some special corner case API endpoints that nginx just simply can not handle in the manner I would prefer. Further, the regex based location syntax, and even the prefix based ones, are not really what you want; you want a "Path" object that's aware of what / in a URL means, and does the right thing if you do/don't add it to the URL. (And it's not as easy as "/foo/bar/baz/?")
- SomeCallMeTim 11y ago
  In Nginx, you can use Lua for complex routing and responses, with the right extensions. [1] [1] http://openresty.org/ http://openresty.org/
  - deathanatos 11y ago
    And we do. We don't use openresty (at least at the moment), we use a custom-built nginx that includes — among other things — the lua module. It's a great boon, but something things are still hard or impossible[1], and we don't use it for every request, opting for the location syntax, and all its flaws, for most. [1]. Conditionally streaming an upload to a backend (i.e., if auth fails, don't stream) is impossible; nginx will buffer the entire request body, either in memory or on disk, and there is no way to change this behavior.

nikolay 11y ago

I am highly disappointed by Nginx in three major areas: 1. They created a paid version that has some additional basic features such as cache purging, dynamic upstream name resolution, and a few others. Charge for support, charge for some fancy management interface, monitoring, but for basic features (most of them available in Tengine [0]) - thanks, but no thanks. You lost me as an evangelist! In fact, in may aspects, they now are catching up with Tengine! 2. Instead of making LuaJIT integration standard and avoid the need to escape Lua in the configuration files, they invented some subpar JavaScript. People already use Lua widely, it's fast, it's great - don't you have anything better to do than invent yet another language!? I really can't believe pragmatic people would have done this, honestly! Speaks so badly about their thought process! I know can expect anything stupid from them! 3. The configuration language is not very intuitive. If they embedded Lua, the whole configuration could be a Lua script that initializes some internal state. This would have been a dream come true! [0] http://tengine.taobao.org http://tengine.taobao.org

dec0dedab0de 11y ago

It's not just web servers. I would say that it is because they want it to be human readable/editable, but JSON would probably be just as good in that regard. I would prefer it if it were more like openssh or supervisor. Though I suspect those styles of configs are would make some of the more advanced configurations a pain.

librvf 11y ago

JSON is horrible for a configuration language. Needing to put quotation marks around every single literal is insanely irritating if you ever have to write a lot of configuration. There's also a lot of application-specific syntax sugar that you cannot do if you stick with strict JSON. Note that it's not really fair to call Apache's configuration language "XML". Apache relies on XML for some structured data in its configuration file, but all the individual directives are parsed separately from the XML.

nikolay 11y ago
I agree. When people started to move towards the "better" JSON standard from XML, I was puzzled. The inability to have comments, the need to "escape" every key name, the lack (in the past) of a schema and query language standard, etc.
- librvf 11y ago
  To be fair, while I agree that JSON is bad for config, I would argue that JSON's objects and arrays eliminate the need for DTDs and standard query languages altogether in many cases. Of course, with sufficiently lqrge and complex data stores, you'll want a documented structure and method for accessing that data no matter what format you use, but unlike XML, JSON provides structure to get started on a small scale very easily. Consider this python: import json import sys sys.stdout.write("%s\n" % json.dumps(json.load(sys.stdin).get(sys.argv[1], None))) This will accept JSON on standard input and will return the value of an object with the key name specified by the first command-line argument. $ echo '{ "value1" : { "sub-value": 5 }, "value2": 99 }' | python json-test.py value1 {"sub-value": 5} $ echo '{ "value1" : { "sub-value": 5 }, "value2": 99 }' | python json-test.py value2 99 Such a program will look similar in any language with a json library that maps objects and arrays to native data structures. Granted, my simple tool will fail if the JSON isn't an object, but it's a very simple matter to extend it to handle lists and literals. For many applications this is a huge advantage over XML, especially if the point of the JSON isn't configuration but rather inter-application communication (aka data serialization).