8 ms·

Mongrel2 Sqlite3 Config System Working

25 points by twampss 16y ago

koenigdavidmj 16y ago
Mr. Shaw, do you ever sleep?
- mbreese 16y ago
  I heard a rumor once that he slept for 45 minutes. When he woke up, he sat down at his computer and typed out the most elegant program ever written. He had composed it all in his dreams. It was in pure x86 assembly. Then, looking upon the goodness of it all he said "fuck it, they don't deserve this" and then deleted it. At least, that's what I heard...
  - pjscott 16y ago
    Reality produces things that are even stranger: Ken Thompson and Dennis Ritchie once accidentally wrote the same program. It was 20 lines of assembly code, and what they had written was character-for-character identical.
    - stcredzero 16y ago
      Never play those two in Pictionary! You'll be sorry if you do.
  - zedshaw 16y ago
    Dude, why are you violating that restraining order? Do I have to call the cops. :-)
amix 16y ago
I like how nginx configuration works. It's simple, it's a DSL, it's a text file that's viewable by anyone, you can version it easily, you can comment in it, you can quickly take backups or revert back to old configurations, you can send a kill -HUP to reload the configuration without taking your site down. The bottom line: The nginx, and the usual text based approach, is simple and it works, I can't really see what he is solving with this database approach other than adding a lot of additional complexity.
- zedshaw 16y ago
  So, everything you said applies to SQL. SQL is a DSL (and a standardized one). SQL is text, so it can be versioned, you can back it up, and you can do it also the .sqlite file. You can also do -HUP to reload configs, nothing about SQL or SQLite prevents that, and in fact, nginx has to completely restart its world to make this happen. Mongrel2 will probably be able to reload without kicking over the process. The problem I'm solving is one of operations automation and design. Go take a look at the nginx code for loading configurations and then go write an nginx module. Figuring out how to get configuration settings out of the thing is horrible. Then, come talk to me when you can change the nginx config on 100 hosts based on the hostname and three sections of the configuration file using sed and awk. Hell, let me know if you can load a random nginx config and tell me what version of nginx it works for. That'd be a trick.
  - stcredzero 16y ago
    SQL is a DSL SQL is a DSL for dealing with Relations (n-tuples) and collections of Relations. In other words, its domain is Data in the most generic sense. It's not really Domain Specific to web server configuration per se. Maybe that doesn't matter, though. I know that Sinatra's config DSL is really doing nothing more than manipulating path data and associating that data with code. Ops people love manipulating stuff like that with SQL, so maybe it's a good fit.
  - amix 16y ago
    nginx does restart, but without taking the site down which is the most important aspect and the most important feature. Brining us to another point, I have deployed a lot of servers that use textual configurations and I can't really see what problem you are solving and I can't see how storing the configuration in a relational database will solve any problems.
    - zedshaw 16y ago
      I'm solving the problem of migrations, automation, tooling, and simplicity. Yes, you can do the many of the same things in both, and if you're an old school "I-ssh-into-100-boxes-all-day" sysadmin then they're the same thing. If you're a modern system administrator who automates, automates, automates, then this is better. For example, my challenge still stands, load an nginx config and tell me what version of nginx it is for. If you can't even do that then how can you automate it better than a sqlite3 file (remember, sqlite3 file, NOT RDBMS).
ErrantX 16y ago
INSERT INTO host (server_id, name, matching) VALUES ( last_insert_rowid(), 'mongrel2.org', '(.).mongrel2.org' ); I love this. (I can't think of any other way of putting that :))
- cageface 16y ago
  I'm having a very hard time seeing this as an improvement over apache/nginx/lighttpd text-file configuration. Web server configs also have a tendency to be heirarchical, which sounds extra-painful in SQL. Let's take a real-world nginx conf for example (comments and blank lines elided to save space). I can't imagine how trying to shoehorn something like this into sql is a win. Even Java-style XML situps seem preferable: http { include /usr/local/nginx/conf/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx_access.log main; error_log /var/log/nginx_error.log debug; sendfile on; tcp_nopush on; tcp_nodelay off; gzip on; gzip_http_version 1.0; gzip_comp_level 2; gzip_proxied any; gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript; upstream mongrel { server 127.0.0.1:5000; # server 127.0.0.1:5001; # server 127.0.0.1:5002; } server { listen 80; client_max_body_size 50M; root /var/www/apps/mysite.com/current/public; access_log /var/www/apps/mysite.com/shared/log/nginx.vhost.access.log main; if (-f $document_root/system/maintenance.html) { rewrite ^(.*)$ /system/maintenance.html last; break; } location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect false; proxy_max_temp_file_size 0; if (-f $request_filename) { break; } if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; } if (-f $request_filename.html) { rewrite (.*) $1.html break; } if (!-f $request_filename) { proxy_pass http://mongrel; break; } } error_page 500 502 503 504 /500.html; location = /500.html { root /var/www/apps/mysite.com/current/public; } }
  - 16y ago
stcredzero 16y ago
With Go's fast compile features, why couldn't a server start consist of using the configuration data to generate code which then gets compiled into a hella-fast optimized server? Paths could be highly-optimized hard-coded pattern-matching finite automata. Someone would have to up and write a Go web framework to use with that. They wouldn't have to, but it would be a shame not to.
- wmf 16y ago
  Varnish does this with VCL; I think it's translated to C, compiled to a .so, and dynamically loaded. (Using Clang you wouldn't even have to write out the .so.)
- zedshaw 16y ago
  That's possible, and probably not too hard, and man oh man would the hackers love a compiler right in their web server. :-)
  - stcredzero 16y ago
    They already pretty much have that with Perl, PHP, Python, Ruby, Smalltalk,... Not necessarily in the front end web server, though. Anyhow, that's easily solved. It wouldn't have to be in a process doing any web serving. The compiler would be available in a process that just does the starting up. The worker processes would then know which dynamically loadable module they need to load. So, no there wouldn't have to be a compiler in the web server.
cageface 16y ago
This claim seems completely specious: With a config file you have to hunt around for the right stanza, type in the right exact change, restart the server, and pray it works. With SQL you can actually query the database, and run updates (then pray that works) which are much easier to get right and do. With apache, I make an edit, check the syntax with apachectl configtest, then restart if the syntax checks out. The kinds of errors that get by the syntax check (bad regex in rewrite rule, typo in a host name etc) are just as likely in SQL. Text files allow me to use the same version control tools, text editors etc I use for the rest of development. Deployment is just a matter of copying a simple text file around etc. Putting this all in SQL is a solution in search of a problem.
- zedshaw 16y ago
  Nope, it's a different solution to the same problem. Like you said, if they're equivalent solutions, yet the sqlite3 one is simpler (code is smaller by far) and has other advantages (migrations, tooling, schema) then sqlite3 wins. Also, I still stand by that getting a SQL update correct is much easier than editing an apache config file. Only The Spaghetti Monster knows how Apache configs actually work (and he works for Microsoft).
  - cageface 16y ago
    They may be equivalent in terms of functionality, but I'm having a very hard time guessing how they might be equivalent in ergonomics. For instance, how would you represent something like this in SQL? http://topfunky.net/svn/shovel/nginx/conf/nginx.conf http://topfunky.net/svn/shovel/nginx/conf/nginx.conf
    - zedshaw 16y ago
      Ohhhhhh, I see you want huge swathes of commented out sections, bizarre syntax for routing, and lots of nesting that makes no sense and has lots of repetition. I'll get right on that. Seriously though, SQL's been proven capable of doing all of this, and for the half-assed turing tarpit if-statements in these files you'll get something better. I'm aiming for Lua or similar to let you really control it. But if your worry is that there will be some monster data structure you have to understand and work with manually in order to do anything then have no fear, you will get good tools.

25 points by twampss 16y ago

koenigdavidmj 16y ago

Mr. Shaw, do you ever sleep?

mbreese 16y ago
I heard a rumor once that he slept for 45 minutes. When he woke up, he sat down at his computer and typed out the most elegant program ever written. He had composed it all in his dreams. It was in pure x86 assembly. Then, looking upon the goodness of it all he said "fuck it, they don't deserve this" and then deleted it. At least, that's what I heard...
- pjscott 16y ago
  Reality produces things that are even stranger: Ken Thompson and Dennis Ritchie once accidentally wrote the same program. It was 20 lines of assembly code, and what they had written was character-for-character identical.
  - stcredzero 16y ago
    Never play those two in Pictionary! You'll be sorry if you do.
- zedshaw 16y ago
  Dude, why are you violating that restraining order? Do I have to call the cops. :-)

amix 16y ago

I like how nginx configuration works. It's simple, it's a DSL, it's a text file that's viewable by anyone, you can version it easily, you can comment in it, you can quickly take backups or revert back to old configurations, you can send a kill -HUP to reload the configuration without taking your site down. The bottom line: The nginx, and the usual text based approach, is simple and it works, I can't really see what he is solving with this database approach other than adding a lot of additional complexity.

zedshaw 16y ago
So, everything you said applies to SQL. SQL is a DSL (and a standardized one). SQL is text, so it can be versioned, you can back it up, and you can do it also the .sqlite file. You can also do -HUP to reload configs, nothing about SQL or SQLite prevents that, and in fact, nginx has to completely restart its world to make this happen. Mongrel2 will probably be able to reload without kicking over the process. The problem I'm solving is one of operations automation and design. Go take a look at the nginx code for loading configurations and then go write an nginx module. Figuring out how to get configuration settings out of the thing is horrible. Then, come talk to me when you can change the nginx config on 100 hosts based on the hostname and three sections of the configuration file using sed and awk. Hell, let me know if you can load a random nginx config and tell me what version of nginx it works for. That'd be a trick.
- stcredzero 16y ago
  SQL is a DSL SQL is a DSL for dealing with Relations (n-tuples) and collections of Relations. In other words, its domain is Data in the most generic sense. It's not really Domain Specific to web server configuration per se. Maybe that doesn't matter, though. I know that Sinatra's config DSL is really doing nothing more than manipulating path data and associating that data with code. Ops people love manipulating stuff like that with SQL, so maybe it's a good fit.
- amix 16y ago
  nginx does restart, but without taking the site down which is the most important aspect and the most important feature. Brining us to another point, I have deployed a lot of servers that use textual configurations and I can't really see what problem you are solving and I can't see how storing the configuration in a relational database will solve any problems.
  - zedshaw 16y ago
    I'm solving the problem of migrations, automation, tooling, and simplicity. Yes, you can do the many of the same things in both, and if you're an old school "I-ssh-into-100-boxes-all-day" sysadmin then they're the same thing. If you're a modern system administrator who automates, automates, automates, then this is better. For example, my challenge still stands, load an nginx config and tell me what version of nginx it is for. If you can't even do that then how can you automate it better than a sqlite3 file (remember, sqlite3 file, NOT RDBMS).

ErrantX 16y ago

INSERT INTO host (server_id, name, matching) VALUES ( last_insert_rowid(), 'mongrel2.org', '(.).mongrel2.org' ); I love this. (I can't think of any other way of putting that :))

cageface 16y ago
I'm having a very hard time seeing this as an improvement over apache/nginx/lighttpd text-file configuration. Web server configs also have a tendency to be heirarchical, which sounds extra-painful in SQL. Let's take a real-world nginx conf for example (comments and blank lines elided to save space). I can't imagine how trying to shoehorn something like this into sql is a win. Even Java-style XML situps seem preferable: http { include /usr/local/nginx/conf/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx_access.log main; error_log /var/log/nginx_error.log debug; sendfile on; tcp_nopush on; tcp_nodelay off; gzip on; gzip_http_version 1.0; gzip_comp_level 2; gzip_proxied any; gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript; upstream mongrel { server 127.0.0.1:5000; # server 127.0.0.1:5001; # server 127.0.0.1:5002; } server { listen 80; client_max_body_size 50M; root /var/www/apps/mysite.com/current/public; access_log /var/www/apps/mysite.com/shared/log/nginx.vhost.access.log main; if (-f $document_root/system/maintenance.html) { rewrite ^(.*)$ /system/maintenance.html last; break; } location / { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect false; proxy_max_temp_file_size 0; if (-f $request_filename) { break; } if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; } if (-f $request_filename.html) { rewrite (.*) $1.html break; } if (!-f $request_filename) { proxy_pass http://mongrel; break; } } error_page 500 502 503 504 /500.html; location = /500.html { root /var/www/apps/mysite.com/current/public; } }
- 16y ago

stcredzero 16y ago

With Go's fast compile features, why couldn't a server start consist of using the configuration data to generate code which then gets compiled into a hella-fast optimized server? Paths could be highly-optimized hard-coded pattern-matching finite automata. Someone would have to up and write a Go web framework to use with that. They wouldn't have to, but it would be a shame not to.

wmf 16y ago
Varnish does this with VCL; I think it's translated to C, compiled to a .so, and dynamically loaded. (Using Clang you wouldn't even have to write out the .so.)
zedshaw 16y ago
That's possible, and probably not too hard, and man oh man would the hackers love a compiler right in their web server. :-)
- stcredzero 16y ago
  They already pretty much have that with Perl, PHP, Python, Ruby, Smalltalk,... Not necessarily in the front end web server, though. Anyhow, that's easily solved. It wouldn't have to be in a process doing any web serving. The compiler would be available in a process that just does the starting up. The worker processes would then know which dynamically loadable module they need to load. So, no there wouldn't have to be a compiler in the web server.

cageface 16y ago

This claim seems completely specious: With a config file you have to hunt around for the right stanza, type in the right exact change, restart the server, and pray it works. With SQL you can actually query the database, and run updates (then pray that works) which are much easier to get right and do. With apache, I make an edit, check the syntax with apachectl configtest, then restart if the syntax checks out. The kinds of errors that get by the syntax check (bad regex in rewrite rule, typo in a host name etc) are just as likely in SQL. Text files allow me to use the same version control tools, text editors etc I use for the rest of development. Deployment is just a matter of copying a simple text file around etc. Putting this all in SQL is a solution in search of a problem.

zedshaw 16y ago
Nope, it's a different solution to the same problem. Like you said, if they're equivalent solutions, yet the sqlite3 one is simpler (code is smaller by far) and has other advantages (migrations, tooling, schema) then sqlite3 wins. Also, I still stand by that getting a SQL update correct is much easier than editing an apache config file. Only The Spaghetti Monster knows how Apache configs actually work (and he works for Microsoft).
- cageface 16y ago
  They may be equivalent in terms of functionality, but I'm having a very hard time guessing how they might be equivalent in ergonomics. For instance, how would you represent something like this in SQL? http://topfunky.net/svn/shovel/nginx/conf/nginx.conf http://topfunky.net/svn/shovel/nginx/conf/nginx.conf
  - zedshaw 16y ago
    Ohhhhhh, I see you want huge swathes of commented out sections, bizarre syntax for routing, and lots of nesting that makes no sense and has lots of repetition. I'll get right on that. Seriously though, SQL's been proven capable of doing all of this, and for the half-assed turing tarpit if-statements in these files you'll get something better. I'm aiming for Lua or similar to let you really control it. But if your worry is that there will be some monster data structure you have to understand and work with manually in order to do anything then have no fear, you will get good tools.