26 ms·
Althttpd: Simple webserver in a single C file
- ktpsns 5y agoHere is the actual single C-code file: https://sqlite.org/althttpd/file?name=althttpd.c https://sqlite.org/althttpd/file?name=althttpd.c Something I absolutely love about text based protocols such as HTTP/1 is how easy you can implement it in any virtually programming language. Sure, the implementation is not top-of-the-notch, but it just damned works, it is portable, it is understandable by humans. That's something what's got lost with HTTP/2 and HTTP/3, respectively.
- kijin 5y agoHTTP is serious about backward compatibility, so a client that speaks HTTP/1.1 (or HTTP/1.0 + Host header) can still talk to most servers out there. Similarly, if you write a server that only speaks HTTP/1.1 (or HTTP/1.0 + Host header), you can put it behind a reverse proxy or load balancer that handles higher versions, does connection management, and terminates TLS. It will work perfectly fine, only without some of the latest performance optimizations that you might or might not even need.
- bawolff 5y agoThat seems like a high risk of http desync attacks, if you're only implementing a subset of http/1.1.
- unixhero 5y agoHow would one attack a static page?
- ric2b 5y agoWhy would you even need a reverse proxy for a static page? This isn't about that.
- unixhero 5y agoI see. Thanks!
- bawolff 5y agoWho said anything about static? This webserver supports cgi, which means it supports php, perl, etc. However even if it didn't, js based client-side apps are probably still attackable in the right set of circumstances.
- chippiewill 5y ago> you can put it behind a reverse proxy or load balancer that handles higher versions, does connection management, and terminates TLS This is even standard practice in many production deployments. Typically you want the proxy or load balancer anyway and there's often little benefit (if any) to using HTTP/2 or HTTP/3 over a very low-latency, high reliability local network.
- saba2008 5y agoHTTP/1 simplicity comes not from text nature, but from it's clear concept and limited functionality. At core, it's just request-reply + key-value metadata. Whenever it's text, or binary, it does not matter much. But writing HTTP/2 frame types in letters would not make them any easier to understand.
- ahartmetz 5y agoThere are also keep-alive, caching (a big topic), chunked transfer encoding, header parsing peculiarities, and authentication in HTTP. The combination of these creates some nice opportunities for implementation bugs. Source: have worked on a client-side implementation. Now, HTTP/2 isn't even conceptually simple, I agree about that... it seems ugly.
- tokamak-teapot 5y agoOne good thing is that you don’t have to support everything if you’re writing a server. It just depends on what kind of use cases you want to support.
- jbverschoor 5y agoThat came with http/1.1 ;-)
- merb 5y ago> implementation bugs. Source: have worked on a client-side implementation well because the hard part is the client-side. caching is a client-side only thing with http, keep-alive is a thing that a server pushes to a client, the same with chunked transfer, which is not as easy to implement for a client like it was with content length. basically a server does only need to implement certain headers, but a client needs to know all. also most clients even accept bad servers, like content for head requests, etc.. most stateless protocols put a lot of burden into clients. h2 on the other hand is stateful and keeps the same hard semantics onto the client side and also makes servers more complex, because it's a state machine.
- secondcoming 5y ago
- TazeTSchnitzel 5y agoOnce you have to support HTTP/1.1 it's not really a text-based protocol, because support for chunked encoding is mandatory for clients. (Yes the chunk headers are technically text, but it's interspersed with arbitrary binary data. It's not something you can easily read in a text editor.)
- mbreese 5y agoHTTP/1 requests for binary files also aren’t text based under that criterion. The protocol itself is text based, but the payload(s) can be binary. That’s doesn’t change between HTTP/1 and /1.1.
- textmode 5y agoI make most HTTP requests using netcat or similar tcp clients so I write filters that read from stdin. Reading text files with the chunk sizes in hex interspersed is generally easy. Sometimes I do not even bother to remove the chunk sizes. Where it becomes an issue is when it breaks URLs. Here is a simple chunked transfer decoder that reads from stdin and removes the chunk sizes. flex -8iCrfa <<eof int fileno (FILE *); xa "\15"|"\12" xb "\15\12" %option noyywrap nounput noinput %% ^[A-Fa-f0-9]+{xa} {xa}+[A-Fa-f0-9]+{xa} {xb}[A-Fa-f0-9]+{xb} %% int main(){ yylex();exit(0);} eof cc -std=c89 -Wall -pipe lex.yy.c -static -o yy045 Example Yahoo! serves chunked pages printf 'GET / HTTP/1.1\r\nHost: us.yahoo.com\r\nConnection: close\r\n\r\n'|openssl s_client -connect us.yahoo.com:443 -ign_eof|./yy045
- charsi 5y agoI tried this but ended up with gibberish in my terminal. Also couldn't find an explanation for -a on flex's man page. I've never used the thing before.
- textmode 5y agoThe extra "a" is a typo but would have no effect. The "i" is also superfluous but harmless. Without more details on the "gibberish" it is difficult to guess what happened. The space before "int fileno (FILE *);" is required. All the other lines must be left-justified, no leading spaces, except the line with "int main()" which can be indented if desired.
- lmilcin 5y agoBinary protocols aren't (or at least don't have to be) any more difficult and frequently are even easier to implement. Text protocols have difficult problems like escaping or detecting the end of particular field that are frequent source of mistakes. The issue is that many (especially scripting) languages treat binary data as second class. The only real issue is that inspecting the binary message visually is little bit more difficult. You can usually easily tell if your data structure is broken if it is in text form. What I do is I usually have some simple converter that converts the binary message to text form and this helps me inspect the message easily.
- chrismorgan 5y agoAnd capitalisation issues, and encodings… give me a binary protocol to parse any time. They’re normally comparatively well-defined, whereas text protocols are seldom properly defined and so have undefined behaviour left, right and centre, which inevitably leads to security bugs—or even if they are well-defined, they’re probably done in a way that makes your language’s string type unsuitable, and makes parsing more complicated.
- touisteur 5y agoJust use recordflux for binary parser proved absent of runtime errors... ;-)
- einpoklum 5y agoCan you expand on that a little?
- michaelmior 5y agoRecordFlux[0] is a DSL written in Ada for specifying messages in a binary protocol. Code for parsing these messages is then generated automatically with a number of useful properties automatically proven including that no runtime errors will occur. [0] https://github.com/Componolit/RecordFlux https://github.com/Componolit/RecordFlux
- secondcoming 5y agoBeing text based is a huge flaw with HTTP in my opinion (and also elsewhere, like Redis). It leads to parsing bugs and overly verbose communications. Humans are good at reading text, CPUs prefer binary.
- mbreese 5y agoThe use of text for popular protocols is for a reason — computers don’t write programs, people do. And while CPUs prefer binary, it’s easier for programmers to read/write/reason about text. This makes it easier to work with a new protocol. From a practical perspective, with a binary protocol, it can be difficult to use across different languages or add support for a new language. If you use the simplest possible encoding, you’d send raw struct data. But this doesn’t always work across different OS/arch/versions/etc. if the server is in C, but the client is in Python, reading the binary protocol would require a far more complicated parser. Obviously a more formal encoding (protobuf, etc) would be preferred, but if you already need to use an encoding mechanism, why not wrap it in a text format? It’s easier to write clients that can read/write text protocols in any language. The reason why text protocols are so popular aren’t because they are necessarily “better” but easier to adopt. This is why the most popular protocols are text based.
- michaelmior 5y agoExcept it quickly gets messy when you start dealing with real data and making sure encoding and escaping is done correctly. > with a binary protocol, it can be difficult to use across different languages This is also true of text protocols that aren't well-designed. I don't think it's necessarily the case that binary protocols are more difficult to deal with. You just have a different set of concerns to address. > If you use the simplest possible encoding, you’d send raw struct data. This is the "simplest" in the sense that it's definitely easy to just copy this data on the wire, but I think this is a straw man. I don't think it's really any more difficult to write a simple protocol that uses binary data compared to text.
- mbreese 5y ago
- andai 5y agoFor people who like text-based protocols (and dislike surveillance and web bloat), I suggest taking a look at Gemini, which was designed so you could write a client for it over the weekend. https://gemini.circumlunar.space/ https://gemini.circumlunar.space/
- peterhil 5y agoGemini is quite interesting in that brings nostalgic feeling from when the web was a new thing, but is also modern. The Lagrange browser seems quite polished.
- strangeattractr 5y agoI prefer binary protocols but I think you make a good point about HTTP/1 especially from a learning perspective. I remember how enlightening it was when I followed a tutorial to create a http server and found that all I needed to do (ignoring TCP/IP) was write 'HTTP/1.1 200 OK' plus a few headers and some raw html in a string and it would actually show up in my browser.
- api 5y agoRecording state info in static variables makes it only suitable for embedded. I really hate when people do that. Static should only ever be const, init once, or something intrinsically singleton. There are very few exceptions.
- jmaygarden 5y agoAre you thinking of C++ issues with constructors? I don’t see anything wrong with the use of static variables in this single file C program. This isn’t a library; it’s a standalone web server.
- api 5y agoI am thinking of multithreading. Mutable static variables pretty much destroy any possibility of multithreading without a major refactor. Test harnessing is an issue too. But if this only ever wants to be an app binary, I guess it's sort of okay.
- jschwartzi 5y agoThat class of errors is not an issue here. This program isn't meant to be multithreaded, it's meant to be multi-/process/. The difference is that each instance of the application has its own independent address space. So that's one address space per instance. And because of that there are no threading issues. Also as a result any communication between different processes has to be explicitly defined. Making this program multi-threaded would be a big mistake because then you couldn't use any of the HTTP proxies out there to monitor for connections and hand them off to this program. This is arguably a much better architecture from a safety/security standpoint then trying to spin multiple threads each sharing an address space. It forces you to use OS-provided mechanisms for shared state rather than simply manipulating the memory to share information.
- TZubiri 5y agoReally simple, I implemented a couple. Remember that the letters after the status code are aesthetic, so make sure to put your style in there: 200 FINE 200 KTHX 404 MISS 403 NO 500 FUCK Another similar server in one file is busybox httpd command if you are interested https://git.busybox.net/busybox/tree/networking/httpd.c https://git.busybox.net/busybox/tree/networking/httpd.c
- Sohcahtoa82 5y agoI always wanted an HTTP response code for when the server detects a malicious request. Like, 430 DONT BE AN ASSHOLE
- zbentley 5y agoRelatedly, in every C project I've ever worked on, the custom (>255) error code for failures due to malicious looking input that could not possibly have been supplied by mistake is in the const EGETFUCT.
- slver 5y agoIt's not a property of text-based protocols, rather it's a property of simple protocols. HTTP/1 is not merely text based, it's ASCII based (technically ISO-8859-1, which includes ASCII). One char, one byte, one encoding. HTTP itself is mostly very simple, text "name: value" pairs separated by newlines, followed by arbitrary content as the body. I think the solution is to start with a simple protocol and upgrade to more complex protocols after that. While technically you don't need to support HTTP/1 to support 2 and 3, the upgrade over TCP happens mostly in that way.
- chrismorgan 5y ago> While technically you don't need to support HTTP/1 to support 2 and 3, the upgrade over TCP happens mostly in that way. This is not actually true of HTTP/2 (the upgrade doesn’t use HTTP/1), and slightly misleading of HTTP/3 (there’s nothing to upgrade because it sits beside TCP HTTP; but advertising HTTP/3 support is currently done over TCP HTTP). Now for the details: HTTP/2 upgrade is done at the TLS level, via ALPN. Essentially the client TLS handshake says “hello there! BTW do you do HTTP/2?” and the server either responds “hi!” and starts talking HTTP/1, or “hi! Let’s talk h2!” and starts talking HTTP/2. So it’s perfectly possible (though not a good idea) to have a fully-functioning HTTP/2 server that doesn’t speak a lick of HTTP/1. (HTTP/2 over cleartext, h2c, does use the old HTTP/1 Upgrade header mechanism, but h2c is more or less just not used by anyone.) HTTP/3 upgrade, well, “upgrade” is the wrong word. It’s operating over UDP rather than TCP, so you’re not upgrading an existing HTTP thing to HTTP/3, you’re starting a new connection because you’ve learned that the server supports HTTP/3. This bootstraping is currently done by advertising h3 support via the Alt-Svc header (HTTP/1+) or by an ALTSVC frame (HTTP/2+), which the client can then remember so it uses the best protocol next time. For best results, they’re working on allowing you to advertise HTTP/3 support on DNS, so that after a while it should be genuinely possible (though a very bad idea) to have a fully-functioning HTTP/3 server that doesn’t speak any TCP at all, yet works with sufficiently recent browsers in network environments that don’t break HTTP/3. https://blog.cloudflare.com/speeding-up-https-and-http-3-negotiation-with-dns/ https://blog.cloudflare.com/speeding-up-https-and-http-3-neg... is good info on this part.
- deathanatos 5y ago> it's ASCII based (technically ISO-8859-1, which includes ASCII) No, it is (some) ASCII plus "opaque octets". ("A recipient SHOULD treat other octets in field content (obs-text) as opaque data.") If you want to say that historically it was such, that's not correct either; it was that plus "supporting other charsets only through use of [RFC2047] encoding" which is a nightmare of an encoding scheme. > HTTP itself is mostly very simple, text "name: value" pairs separated by newlines Except when it isn't: headers can be folded across multiple lines. (Which is also obsolete, and discouraged.) Add to that 1xx responses, transfer encodings, chunks, chunk extensions, trailers. HTTP is far from "simple"…
- anthk 5y agoGopher is easier than that.
- geoffdunbar 5y agoI love C, but it's pretty scary sometime. 5 minutes ago, "I wonder if I can find a potential memory overwrite in 5 minutes?" Sure enough, the function StrAppend potentially overflows a size_t size (without checking), and then writes into memory could be past the end of the allocated buffer. Given 5 minutes, I didn't look thoroughly if this is actually exploitable, but it's definitely a red-flag for the code. Be careful out there! Hopefully I am missing something, or this is just a simple oversight, but I would carefully audit this code before using it. Submitted a ticket through the Althttpd website. static char StrAppend(char zPrior, const char zSep, const char zSrc){ char zDest; size_t size; size_t n0, n1, n2; if( zSrc==0 ) return 0; if( zPrior==0 ) return StrDup(zSrc); n0 = strlen(zPrior); n1 = strlen(zSep); n2 = strlen(zSrc); size = n0+n1+n2+1; zDest = (char*)SafeMalloc( size ); memcpy(zDest, zPrior, n0); free(zPrior); memcpy(&zDest[n0],zSep,n1); memcpy(&zDest[n0+n1],zSrc,n2+1); return zDest; }
- spacechild1 5y ago> Sure enough, the function StrAppend potentially overflows a size_t size How should this happen in practice? The three strings would have to be larger than the available address space...
- ectospheno 5y agoYeah. The function in question is called in only one place. It would seem you’d need to send the web server more than a size_t of data for this to be an issue.
- acqq 5y agoExactly. In a single file C nobody can expect to get universal library functions that work in any possible imaginable context. The only relevant context is the code the function is in. And in that context, the function is doing enough.
- geoffdunbar 5y agoYes, absolutely. If the webserver is compiled 32-bit, that is only 4GB of data, which might be feasible? I don't know enough to say. Assuming a hacker kindly won't overflow your buffer is never a good idea. However, the presence of one piece of code that is not integer-overflow safe definitely makes me nervous. This is just the one I found in 5 minutes, what else is in there?
- ecmascript 5y agoI'm all for SQLite and I am a fan of the author of the project but for a webserver I have turned my back away from Nginx for https://caddyserver.com/ https://caddyserver.com/ because of the simplicity. Caddy is just really awesome as a reverse proxy (2 line config!!) and I am in the processes of moving all my projects to it. It is fast enough as well since other things will be the bottle neck way before that. I am not affiliated with Caddy in any way, just blown away by the quality of it.
- eatonphil 5y agoA big missing feature in Caddy for me is an embedded language like Lua for nginx so you can write tiny hooks. The Caddy authors have indicated on HN a while ago that Caddy 2 may have an embedded scripting language but I can't find anything about it in their docs.
- coder543 5y agoSeems like it was postponed[0]. For very tiny hooks, you might be able to get away with using request matchers[1] and respond[2]. [0]: https://caddy.community/t/missing-starlark-documentation/9580/5 https://caddy.community/t/missing-starlark-documentation/958... [1]: https://caddyserver.com/docs/caddyfile/matchers https://caddyserver.com/docs/caddyfile/matchers [2]: https://caddyserver.com/docs/caddyfile/directives/respond https://caddyserver.com/docs/caddyfile/directives/respond
- francislavoie 5y agoWriting plugins for Caddy is so easy that it's generally not necessary to have scripting built-in. You can just build yourself a new binary with a Go plugin just by using the "xcaddy" build tool. https://caddyserver.com/docs/extending-caddy https://caddyserver.com/docs/extending-caddy But yeah, it's still something at the back of our minds, and we were considering Starlark for this, but that hasn't really materialized because it's usually easier to just go with the plugin route.
- terminalserver 5y agoThe thing I love most about caddy is it automatically does all the ssl certificate garbage which is so painful in every other web server ever. Yes certbot makes it less painful but it’s still a big PITA, unlike caddy where SSL is just like magic.
- terminalserver 5y agoInteresting but why? There’s a bazillion web servers out there, surely one of them can do the job? What have I missed?
- mro_name 5y agodependency awareness. External stuff bears surprises. Not everybody feels comfortable with that.
- terminalserver 5y agoThere’s plenty of lightweight minimal web servers with minimal dependencies. But hang on is dependency anxiety really the reason or did you just make that up?
- fraktl 5y agoYou missed the date in the comment of the C file: 2001-09-15 Back then, there weren't bazillion web servers out there. A patchy server was still.. patchy. And engine that solves problem X (c10k) was not created yet :) (for whoever reads my comment, I am referring to Apache and nginx)
- OJFord 5y ago> [Althttpd ...] has run the https://sqlite.org/ https://sqlite.org/ website since 2004 I don't know what the landscape was like in 2004 really, but probably at least an order of magnitude less than today's bazillion (whatever that would be!).
- mro_name 5y ago> separate process is started for each incoming connection wow. Thanks also for elaborating the xinetd & stunnel4 configs.
- skywal_l 5y agoI would like to see something like: althttpd -exec some_executable {}.method {}.body So you could quickly call executable from a browser and redirect the output in the response.
- pjc50 5y agoIsn't this just CGI again?
- pragma63 5y agoAlways has been.
- xrstf 5y agoYou mean CGI[1]? [1] https://www.wikiwand.com/en/Common_Gateway_Interface https://www.wikiwand.com/en/Common_Gateway_Interface
- 0xdeadbeefbabe 5y agoThat's a good idea. Don't know why you are being modded down.
- notRobot 5y agoAlso see: darkhttpd: https://github.com/emikulic/darkhttpd https://github.com/emikulic/darkhttpd
- chasil 5y agoSimilar is thttpd. I was not familiar with darkhttpd. Both of these are similar to the sqlite server in security design (chroot capability), but unlike in that a single process serves all requests and does not fork. I have used stunnel in front of thttpd, and chrome has no complaints. https://acme.com/software/thttpd/ https://acme.com/software/thttpd/
- mro_name 5y agoalthttpd has CGI, sometimes interesting.
- chasil 5y agoThe thttpd server also has CGI. I wrote about combining it with stunnel. The rfc-1867 tool is overly dated, and I've replaced it for my internal use: https://www.linuxjournal.com/content/secure-file-transfer https://www.linuxjournal.com/content/secure-file-transfer
- foobar33333 5y agoI'd be putting this in 1000 layers of sandboxing since its a C program with network access.
- Cthulhu_ 5y agoYour point being? It's been serving sqlite.org just fine for all this time. You seem to be making some pretty big assumptions about security here without actually explaining what your specific concerns are.
- foxes 5y agoI think the original comment is partially a joke. Maybe there isn't specifically anything wrong with this, but there is the fact that it is written in C. Historically there is a good precedent of this being an issue. C does not guarantee correctness to the same level as more modern languages. If it was written in Haskell or Rust for example you could be more sure about correctness. I believe for something like this, correctness is fairly important. Not to mention you probably wont even lose speed. As for if the code is understandable, it is a 2600 lines of terse-ish C [0]. Do you really think about the entire blob at the same time? [0] https://sqlite.org/althttpd/file?name=althttpd.c https://sqlite.org/althttpd/file?name=althttpd.c
- kahlonel 5y agoPeople need to stop assuming that memory safety == functional safety. They are two very different things. Rust ensures memory safety, but it won't stop you from making logical mistakes. You can't be "sure about correctness" unless proven mathematically.
- foxes 5y agoI agree, but it is an extra level above c. Same with using a more advanced type system, more easily encoding problem constraints in that type system.
- znpy 5y ago
- lnxg33k1 5y agoOne file of 2600 lines, guess if you give up common sense and follow this approach you can even create a kernel in one file
- mro_name 5y agohave you ever looked at the source of sqlite?
- littlecranky67 5y agoAFAIK FreeRTOS is a kernel and distributed as a single file.
- nneonneo 5y agoMaybe that was true at some point, but it’s no longer true. FreeRTOS consists of a core of 6-7 .c files (some of which may be optional) plus 2-3 board support source files. See their GitHub mirror: https://github.com/FreeRTOS/FreeRTOS-Kernel https://github.com/FreeRTOS/FreeRTOS-Kernel
- alex_smart 5y agoI am not sure exactly where you were trying to go with this, but 2600 lines is actually pretty short for a C program.
- Cthulhu_ 5y agoWhose common sense though? You seem to have one particular opinion on how to organize a project, but that's not the only one. Sometimes, having just one file is easier.
- mhd 5y agoFor parsing/protocol handling the implementations where this was distributed amongst several files and/or classes have usually been the worst, in my experience. But now I want to see how badly you could Uncle Bob this thing. My screen should be wide enough for the resulting function names.
- brobdingnagians 5y ago
- pjc50 5y ago/* ** Test procedure for ParseRfc822Date */ void TestParseRfc822Date(void){ time_t t1, t2; for(t1=0; t1<0x7fffffff; t1 += 127){ t2 = ParseRfc822Date(Rfc822Date(t1)); assert( t1==t2 ); } } There's only two billion integers, guess we can test them all. Well, substantially fewer than two billion with that skip. I wonder if that completes in a few seconds.
- ithkuil 5y agoComputers are insanely fast
- tyingq 5y agoAnother comment shows 10 seconds on a relatively decent CPU from 2017. So it is a fairly heavyweight task, though I suppose could be rewritten to use more than one core.
- ithkuil 5y agoyeah, that's a lot of iterations :-) I was curious to see how my M1 compares to my intel 2019 macbook pro: M1: /tmp/tt 8.42s user 0.01s system 99% cpu 8.435 total 2,6 GHz 6-Core Intel Core i7 /tmp/tt 15.69s user 0.03s system 98% cpu 15.888 total /tmp/tt 15.69s user 0.03s system 98% cpu 15.888 total
- nneonneo 5y agoThis loop doesn’t test them all - it tests every 1/127th integer, so it only runs about 16 million times.
- Cthulhu_ 5y agoAnd a modern CPU runs billions of instructions per second, so it shouldn't take too long at all. Depends on the implementation of the function as well though, and how the compiler can optimize, unroll or optimize both the function under test and the test itself.
- deleted 5y ago[deleted]
- cube00 5y ago> A separate process is started for each incoming connection, and that process is wholly focused on serving that one connection. It makes you wonder just how "heavy" operating system processes actually are. We may not need to worry about the complexity of trying to multiple run async requests in a single process/thread in all cases.
- frou_dh 5y agoI like to remember that threading APIs were a later add-on after Unix had already existed for years. They're not fundamental like the Process is. Isn't it an appealing model to not even have to talk about threads because every process is 1 thread by definition?
- astrange 5y agoMultiple processes are a bit easier to deal with than threads per servers - mostly because POSIX signals interacts poorly with threads. It's also more secure because you should not mix different users' requests in the same process if you can avoid it. nginx runs on a one process per core model and more or less does everything correctly.
- secondcoming 5y agoThe downside is excessive context switching, and sharing data between processes becomes difficult (counters, etc).
- sydthrowaway 5y agoSo what was the point of the async/callback web programming revolution if processes were good enough?
- TeMPOraL 5y agoThe point was that you didn't have the ability to spawn new threads at all. async lets you pretend you have threads, at the cost of everything being run through a hidden event loop.
- vmsp 5y agoBlocks some referers by default: static const char *azDisallow[] = { "skidrowcrack.com", "hoshiyuugi.tistory.com", "skidrowgames.net", }; Anyone know why?
- Hypergraphe 5y agoIt is located in a #ifdef 0 block. So my guess is it might be for testing purpose.
- mikkelam 5y agoI remember seeing these crackers around back in the days. Skidrow at least cracted some grand theft auto games. Not sure why they're blocked
- banana_giraffe 5y agoThere's also this: }else if( strcasecmp(zFieldName,"Referer:")==0 ){ zReferer = StrDup(zVal); if( strstr(zVal, "devids.net/")!=0 ){ zReferer = "devids.net.smut"; Forbidden(230); /* LOG: Referrer is devids.net */ } Which I can appreciate why it's there, it's still odd.
- iveqy 5y agoI thought forking webservers was too slow?
- layoutIfNeeded 5y agoToo slow for what?
- dspillett 5y agoIt depends how complex a process you are forking and what OS you are running on. It has been some time since I wrote any real code for Linux or Windows that would be significantly affected by such things, but it used to be that forking could be almost as efficient under Linux as starting a new thread in an existing process. Under Windows this was very much not the case. Threads make communication between parts easier (no IPC needed) but that isn't usually an issue for the work a web server does. Comparing to purely event based web servers forking can still be better as no request should fully block another, or usually crash another (which is more likely with threads), and thread or fork based servers can make better use of concurrency which is significant for CPU heavy jobs. So swings & roundabouts. Each type (event, thread, process, some hybrid of the above) has strengths, and of course weaknesses.
- cesarb 5y ago> but it used to be that forking could be almost as efficient under Linux as starting a new thread in an existing process. That's because internally it's nearly the same thing. Both forking and starting a new thread on Linux is a variant of the clone() system call, the only difference being which things are shared between parent and child.
- asddubs 5y agoI misread the title as "in a single line of C-code". I thought "must be a pretty long line"
- codazoda 5y ago91,933 characters. I was curious about the number of lines and calculating characters was a simple select all from there. There are 2,592 lines.
- spyke112 5y agoSure it lives a single file, but considering the length, wouldn't it actually be better to split it into multiple files? I'm not that familiar with C, it seems to be a "thing" in C to just have giant files.
- dmux 5y agoI've been thinking about this a lot recently. For most of my career I've been a Java programmer and for the majority of projects, each class is put in its own file. The amount of jumping (between files / following method calls) can get really tedious when you're trying to grok a code base. I've been working on Typescript projects recently where the standard has been to have slightly larger files -- possibly containing a class definition, but more often it's an entire module -- and it's actually been kind of nice to just read the entire thing top to bottom in one go. I've looked for studies on "locality" of source code, but haven't really found anything.
- GordonS 5y agoYes, it can be really jarring to have to constantly move between several small files. I mostly use C#, and a while back I settled on a middle ground, where closely-related classes and interfaces are grouped together in a single file. When I'm working on web apps/APIs, I usually follow the "feature folder" concept too, where all the most central parts are together in the same file.
- millerm 5y agoSQLite is actually maintained in many files, but they are concatenated into one file for distribution. Here is the reasoning: https://sqlite.org/amalgamation.html https://sqlite.org/amalgamation.html
- 01127790781 5y agoجهات اتصال
- Tepix 5y agoAlthttpd is less complex than other HTTPDs because it doesn't support encryption by itself and instead recommends to use stunnel4.
- thesnide 5y agoIsn't that the same philosophy behind varnish?
- henvic 5y agoTake a look at Fossil too: https://www.fossil-scm.org/ https://www.fossil-scm.org/ It's the distributed version control (and more) used by SQLite. Most people have no idea about how cool the SQLite ecosystem is, and how it's used even on avionics!
- pdimitar 5y agoThe lack of ability to squash PRs into single commits is a deal breaker for many, myself included. No, having a commit "fix typo" in the main branch's history is not at all useful and won't ever be. It's noise. In a work setting it's much better to reduce noise.
- emn13 5y agoI never understood the point of squashing if you just want the short version, only read the merge commits; if you want the full details to figure out some bug or whatever, then yes, I want those fix typo commits most definitely, because as often as not, those are at fault. Squashing buys you next to nothing, and costs you the ability to dive into the history in greater detail. I suppose if your project is truly huge, it becomes worth it to reduce load on your VCS, but beyond that...
- pdimitar 5y agoHaving full history in the feature branch is mandatory -- nobody is disputing that, myself included. All due diligence is done there, not in the main branch. The main branch only needs to have one big commit saying "merging PR #2169". If you need more details you'll go that PR/branch and get your info. The "fix typo" commit being in the main branch buys you nothing. It's only useful in its separate branch.
- emn13 5y agoBut that's what a merge commit is - the diff along the merge commit is what the squashed commit would be; the only thing squashing does is "forget" that the second parent exists (terminology for non-git VCS's may be slightly different). Why not merge?
- niutech 5y agoThere is also redbean (https://justine.lol/redbean/ https://justine.lol/redbean/) - a single-file web server with embedded Lua interpreter as an Actually Portable Executable by Justine Tunney, the creator of Cosmopolitan.
- jedimastert 5y agoEverytime I come across Justine's work, I'm always amazed.
- john-tells-all 5y agoThe idea you can have an executable that is incredibly small, and runs on macOS/Linux/Windows, and is very fast, and has features, is mind-blowing. Justine Tunney is a treasure!
- pstuart 5y agoThat is amazing. And it looks like she may embed SQLite in as well.
- paulclinger 5y agoAlready happened; https://github.com/jart/cosmopolitan/pull/185 https://github.com/jart/cosmopolitan/pull/185 (Add sqlite3 support to Lua scripts in Redbean) has been merged.
- dmux 5y agoThere's a lot of cool technology in the Tcl and SQLite ecosystem. Wapp [0] is a tiny web application framework (a single file) also written by D. Richard Hipp [0] https://wapp.tcl.tk/home/doc/trunk/README.md https://wapp.tcl.tk/home/doc/trunk/README.md
- 0xbadcafebee 5y ago> As of 2018, the althttpd instance for sqlite.org answers about 500,000 HTTP requests per day (about 5 or 6 per second) delivering about 50GB of content per day (about 4.6 megabits/second) on a $40/month Linode. The load average on this machine normally stays around 0.1 or 0.2 Interesting. If the load avg is consistently low, it could mean they're over-paying for CPU. If this was a non-dedicated AWS instance you might want low load so you don't chew up CPU credits, but you'd also want to use an instance type that creates some load so you're utilizing what you're paying for. Linode VPCs don't use cpu credits so the calculation is a bit simpler. I'm also curious how much of that bandwidth couldn't be offset by a CDN or mirrors. If you were using a serverless platform, you'd ideally want to use something like static site hosting feature where you're mostly just paying for storage and egress. Or a serverless application platform to auto-scale traffic as needed. The main problem with doing this, of course, is the cost of egress: cloud providers with fancy serverless platforms often have redonkulous egress costs, so using a plain old VM on a VPC provider can be cheaper if you have more bandwidth demands than compute. (I am aware none of this is a concern if you'd rather just spend $40 and forget about it. I am a nerd.)
- GordonS 5y ago> I'm also curious how much of that bandwidth couldn't be offset by a CDN or mirrors As you say, at $40/m it's academic for a lot of people, but AFAIK, the whole site is static, so presumably if you put it behind Cloudflare’s free tier it would serve all but the file downloads from the edge. A pure guess, but I'd imagine that would mean serving 75% of requests from the edge.
- SQLite 5y agoLook again. The entire Althttpd website is 100% dynamic. Notice that the hyperlink at the very top of this HN article is to a Markdown file (althttpd.md). A CGI runs to convert this into HTML for your web-browser. The core SQLite website has a lot of static content, but there are dynamic elements, such as Search (https://www.sqlite.org/search?s=d&q=sqlite https://www.sqlite.org/search?s=d&q=sqlite) and the source code repository (https://www.sqlite.org/src/timeline?n=100&y=ci https://www.sqlite.org/src/timeline?n=100&y=ci). So far today, 23.48% of HTTP requests to the sqlite.org domain are for dynamic content, according to server logs.
- abriosi 5y agoIt takes years of complexity to achieve this level of simplicity
- geocrasher 5y agoI have yet to try running this for anything, but I do appreciate how it really sticks to the "do one thing well" ethos. Modern web servers can be extremely complicated with a lot of moving parts. This boils it down to just one thing and lets a person focus on the project instead of the infrastructure. Granted, it's very simplistic, but that's its strength.
- tyingq 5y agoI do respect the technical chops around sqlite. However, I think a "fork for every single http request" server isn't really useful in many situations. That the sqlite website is able to run this way is more a testament to Linux's work on a lightweight/fast fork() than anything else. This would perform terribly on a more traditional Unix.
- deleted 5y ago[deleted]
- vidarh 5y agoI ran a webmail service with 2m users that forked and exec'd a CGI for every request 20 years ago. 20 year old hardware was already fast enough that we were usually IO bound on the storage backend rather than constrained by the (much cheaper) frontends. Forking for every request is slow, sure. But if your code is written with it in mind it's faster than most people might expect, and most people never get to a scale where it matters. It's not the right choice for everything, but people have ironically gotten obsessed with things we introduced a long time ago as workarounds for slow hardware (and fork used to be slow on Linux too) decades after the original problems were largely solved. I do agree there are times this won't be useful, though.
- tyingq 5y ago"I ran a webmail service with 2m users that forked and exec'd a CGI for every request" Yes, but that met expectations of that time period, and expectations for a webmail service. I'm curious if you also forked for every static asset...that's what this setup appears to do. I just don't see the benefit of mysql choosing to use this today. It works, but there are other minimal http servers that would be just as simple, but would be faster and use fewer resources. I suppose they don't need to change it, but it's not really a great example of anything other than "fork is cheap on linux" to me.
- ankurpatel 5y agoPerformance is really bad. This is good for running a small HTTP server on an embedded device but if plan is to use it for HTTP server to serve production web traffic performance is really bad. Below is report of running the server and hitting a minimal index.html page and hitting it with artillery. All virtual users finished Summary report @ 09:39:57(-0400) 2021-06-08 Scenarios launched: 33645 Scenarios completed: 2573 Requests completed: 2573 Mean response/sec: 42.57 Response time (msec): min: 0 max: 9029 median: 2 p95: 6027.7 p99: 8778.8 Scenario counts: Get index.html: 33645 (100%) Codes: 200: 2573 Errors: ETIMEDOUT: 31008 EPIPE: 48 ECONNRESET: 16
- dmux 5y ago>but if plan is to use it for HTTP server to serve production web traffic performance is really bad. But it seems to be "good enough", no? As stated on the page, it serves 500k requests a day. Were you running your tests using xinetd or stunnel?
- SQLite 5y agoIndeed. And a Citation biz-jet is way faster, flies higher, goes further, and carries more passengers than a Carbon Cub. On the other hand, the Citation costs more, burns more gas, takes more maintenance, and is more complex to fly, and you should not try to land a Citation on a sandbar in a remote Alaskan river. Choose the right tool for the job. Changing the https://sqlite.org/ https://sqlite.org/ website to run off of Nginx or Apache instead of alhttpd would just increase the time I spend on administration and configuration auditing.
- dsalzman 5y agoLove the Carbon Cub reference. STOL!!
- jmercouris 5y agoIt is not clear whether you would spend more time on administration with another webserver. I don't have experience with your webserver, but mine are 'set it' and 'forget it' affairs.
- millerm 5y agoI really wish more developers took the time to comment code as well as the sqlite.org devs.
- fortran77 5y agoSee Jef’s original thttpd https://acme.com/software/thttpd/ https://acme.com/software/thttpd/
- callumprentice 5y agoI use SQLite in applications via their C/C++ interface and it’s spectacular. I’d love to see a version of this that I could embed too.
- andrewmcwatters 5y agoI have a great appreciation for D. Richard Hipp's work. ** May you do good and not evil. ** May you find forgiveness for yourself and forgive others. ** May you share freely, never taking more than you give.
- 0xdeadbeefbabe 5y agoDoes z in zTmpNam or zProtocol signify global variable? https://sqlite.org/althttpd/file/althttpd.c https://sqlite.org/althttpd/file/althttpd.c Also, I'd like to complain about the hn hug of death, because it isn't happening.
- SQLite 5y agoThe "z" prefix is intended to denote a "zero-terminated string", or more specifically a pointer to a zero-terminated string.
- zanethomas 5y agothe beauty and power of simplicity
- ulises314 5y agoThis pretty much sounds like my networks class final project.