8 ms·

A fast and static web server for serving web apps

77 points by timothy89 13y ago

tunnuz 13y ago
You're from Lund! I've been studying in Lund! By the way, can you show some statistics and performance metrics with respect to nginx, Apache and such? To put it another way: why did you choose to write your own webserver instead of using, say, nginx?
- davidcollantes 13y ago
  I am curious too, on the performance metrics. He is still using Nginx though.
  - timothy89 13y ago
    Nginx is just acting as a reverse proxy because I host other sites on the same server. Benchmarks will come!
- timothy89 13y ago
  Lund is awesome! ;) I will perform some benchmarks and compare it to nginx (and maybe apache) in a while. By the time (2010, 2011) I was working at a company that needed a tracking server for tracking clicks and views in email. So I started to mod lighttpd and thought that I could make it faster. So in true "challenge accepted" spirit I started to write my own HTTP server. And since then I've made it better and better. It's not meant to be an nginx killer. It will always be very basic, fast and lightweight. It's great for projects where you need a web server included. As I wrote in another comment I have used the code base myself in various projects such as: * Tracking clicks, view, etc. for AlphaMail * The send-API server for AlphaMail * A DMARC report server (https://github.com/amail/comfirm-dmarc-report-server https://github.com/amail/comfirm-dmarc-report-server) * A REST server with redis as storage I've never used it as a static web server in production yet. Mostly because of the lack of a reverse proxy and other great features that nginx has and shortfin not.
sciurus 13y ago
There's a sample configuration at https://raw.github.com/timothyej/Shortfin/master/config/shortfin.conf https://raw.github.com/timothyej/Shortfin/master/config/shor...
babuskov 13y ago
I don't get it. You create a server to solve C10K problem, and then host its own website with nginx? Why? Having HN crowd coming and testing your server would be a great test, no?
- clone1018 13y ago
  The actual site is down for me too, http://shortfin.io:88/ http://shortfin.io:88/
  - SirSkidmore 13y ago
    The link works fine for me.
    - clone1018 13y ago
      Work firewall, duh, sorry.
- kajecounterhack 13y ago
  I think in this case nginx is being used as a reverse proxy.
- untothebreach 13y ago
  From the footer: > It was created in 2011 by Timothy E. Johansson and is the base for the send-API and the tracking server for AlphaMail. > It also powers this website (though behind nginx). You can reach it directly on port 88: shortfin.io:88
  - babuskov 13y ago
    I read that. But why? What's the point in using this if you're going to have nginx in front?
    - nolok 13y ago
      I suppose he is hosting other stuff too, and nginx proxies.
dictum 13y ago
>wget http://shortfin.io/install.sh http://shortfin.io/install.sh && sh install.sh Isn't it recommended that any project doing one-line installs use HTTPS?
- chc 13y ago
  In the same way that it's recommended for any other program, yes.
- bru 13y ago
  Side-note for the author: the download can be made more elegant that way $ curl http://shortfin.io/install.sh | sh I definitely agree with the HTTPS. For example this is what meteor proposes: $ curl https://install.meteor.com | sh
  - pnathan 13y ago
    asdfsdfgkjlsdfg! Streaming your install script directly into computer execution without even giving it a once-over, md5 compare, etc is atrociously insecure.
    - nmcfarl 13y ago
      This seems to be a common complaint but I can't see any reason you can't give it a once over when it's hosted on the web…
    - chc 13y ago
      That's completely orthogonal to what we're talking about here. If you're already not doing it, you're not doing it any less by being more efficient. And it doesn't really matter if you're running it in a throwaway VM like I do.
    - jgreen10 13y ago
      curl https://install.meteor.com | less
- sehrope 13y ago
jfb 13y ago
Just run the following command as root to install the server. # wget http://shortfin.io/install.sh && sh install.sh What? Are they serious? Is this becoming a thing? Please don't tell me that this is a thing.
- chrissnell 13y ago
  This has been a thing since the mid-90s when you could execute 'telnet sci.dixie.edu 1 | sh' to install an IRC client. It preys on the clueless and one of these days, some project that uses this install method is going to get owned and a bunch of machines will get malware. Then again, most of us blindly run what we assume to be GNU autoconf scripts and never read Makefiles before we 'make install' as root. Nor do most of us read or audit source code, even for security-sensitive applications.
- ripter 13y ago
  a lot of times it's curl instead of wget.
- chc 13y ago
  I don't see any rational basis for this. If you weren't vetting the code you downloaded before, you don't lose anything by blindly running the commands on one line instead of three. And if you do want to vet the code, you know what to do. Practically no installation instructions include a "vet the code" step, so you aren't losing anything here. The lack of TLS is worrisome, but that's not so much "becoming a thing" as it is "a bad thing we're just finally starting to get rid of."
  - rsync 13y ago
    Although this isn't new, it should make you throw up in your mouth a little bit. Malicious compromise is not the only way that an install script can become broken, or corrupted, on a remote site. The ability to checksum what you're running is a very useful sanity check and you should be doing it with all downloaded scripts/packages. I suppose this is tilting at windmills, though.
    - 13y ago
ahknight 13y ago
okay ... why? what's special about this? features? benchmarks? anything?
- grey-area 13y ago
  Why not? The author claims it's 'high-performance and open-source' - sounds good to me. It's not as if we have a hundred web servers all competing as we do static blogging systems, the more web servers the better in my opinion, just as with web browsers. The zip file is 312KB as linked from here, so it's pretty compact, possibly suitable for embedding in apps, etc: https://github.com/timothyej/Shortfin https://github.com/timothyej/Shortfin
tuananh 13y ago
author should add a benchmark, comparison. or maybe it's just a fun, personal project and not ready for production.
reidrac 13y ago
I was going to say that there are *~ files in the repository, they should tidy a little bit because things like that don't look good. Then I thought may be I was being a little bit to picky and unfair, so I checked the code of some random files and... well, it looks like it was part of a learning experience but it definitely needs work before being considered a viable option.
ds9 13y ago
What's a "static web server"? I haven't seen this term. A webserver for only static pages, i.e. it won't talk to a runtime? But it's "for serving web apps", this means all the dynamic content is pulled from client side JS? Then you would still need programs on the server to answer AJAX requests. I did not find an explanation on the page or by Googling.
- sehrope 13y ago
  A server that only serves static files. It doesn't enable the use of dynamic content like CGI[1]. Compare this to something like Apache or nginx. Both can serve static files but also support a number of ways of either directly running scripts (ex: mod_perl or mod_php in Apache) or proxying to other servers that handle dynamic content. [1]: https://en.wikipedia.org/wiki/Common_Gateway_Interface https://en.wikipedia.org/wiki/Common_Gateway_Interface
  - deleted 13y ago
    [deleted]
  - ds9 13y ago
    OK, then, sorry if I'm being dense, but how can it be "for serving web apps"? App seems an odd term for a site consisting of only static files.
    - packetslave 13y ago
      Perhaps you've heard of this new language, JavaScript?
      - ds9 13y ago
        Still not much of an app unless the JS does AJAX, but then it's talking to a server that's running some dynamic process, maybe with DB access etc. - right?
        Isofarro 13y ago
ksec 13y ago
Waiting For Benchmarks.
nine_k 13y ago
I think the landing page would greatly benefit from two short paragraphs: Why Shortfin when we have Nginx/Lighttpd? Why Shortfin when we have thttpd?
- timothy89 13y ago
  1. Shortfin is not a replacement for nginx. I use it myself to create other projects, e.g. fast API servers with one purpose only and a REST server that uses redis as storage. 2. See #1. 3. I don't try to sell anything, it's just an open-source project that I think maybe someone could benefit from. I've learned a lot while coding it.
  - nine_k 13y ago
    OK, Shortfin is just trying to be an efficient HTTP server, without a radical differentiating feature or implementation approach. Well, let a hundred flowers blossom.
eksith 13y ago
While I appreciate the time and effort that went into this, I'm more concerned with security than just speed alone. It's still relatively easy to throw hardware at the load problem (up to a point, naturally), but safety shouldn't take a back seat to performance. The wget shell script installation does make me nervous and I'm glad the source is available separately. Blind installation of scripts was never "a thing" with me. Of course, I'm still glad people are writing proper web servers (as opposed to simple 2-10 line ones). That creates an opportunity to explore the field with fresh ideas.
ChuckMcM 13y ago
Close. Can add this to the list like thttpd. Something I keep thinking I'll build is a fast, high connection count, limited HTTP server, something that is essentially a wrapper around a program that works like 'regular' and emits HTML. It is a corner case in a custom corner, but the target it something which is essentially a 'transponder.' In the 'Internet of Things' I want to build a wrapper/environment such that my program can be main(int argc, char *argv[]) { uint16_t sense; uint16_t chan; sense = adc_read(atoi(argv[1])); printf("Sensor %d reads : %d\n", chan, sense); } And then link my 'wrap around web server on it' and then it can be accessed with wget 'http://ip:port/?1' http://ip:port/?1' have it do the right thing. The keys are low memory footprint, lots of connections, easily wrapped around a 'regular' program.
- tlrobinson 13y ago
  "easily wrapped around a 'regular' program" Sounds like you want CGI :)
  - ChuckMcM 13y ago
    Yes, "static CGI" if such a thing was a thing :-). I've got a thttpd tree that I do this sort of hack in (these are great for monitoring large numbers of servers for example).
    - skrebbel 13y ago
      I don't understand (and am interested). What's "static" about it? With Apache and mod_rewrite and CGI, I could make the functionality you want (which is, of course, completely not low footprint, but I don't understand how what you want is different from CGI).
      - ChuckMcM 13y ago
        Static in the sense that it always executes the one function (doesn't load it from disk, its part of the executable) and so there is never any risk that some other path might get you 'out' of the docs directory and into the cgi_bin directory. So in this case static is code for 'compiled in' versus 'dynamically loaded.'
BlackDeath3 13y ago
Very cool. I've been doing the same thing lately, writing my own little web server that serves static files. It's been a fun learning experience!
- timothy89 13y ago
  Nice, in what language? I've really learned a lot from creating shortin! Shortfin is written in C so it's a nice project to spend some time on when I get tired of writing javascript and html ;)
  - BlackDeath3 13y ago
    I'm writing mine in C as well. I've never been too into web programming, but the lower-level stuff is quite interesting to me. Keep it up!
joeblau 13y ago
I just ran two speed tests since there aren't any benchmarks provided 1. First Test is using http://www.webpagetest.org/ http://www.webpagetest.org/ Results shortfin.io:88 - http://www.webpagetest.org/result/130806_71_13HK/ http://www.webpagetest.org/result/130806_71_13HK/ First View: 1.945s Repeat View: 1.632s Results shortfin.io - http://www.webpagetest.org/result/130806_QV_13F7/ http://www.webpagetest.org/result/130806_QV_13F7/ First View 2.099s Repeat View 0.084s 2. Second Test is using Apache Bench Results (Best results of 3 runs): ab -n 100 -c 100 http://shortfin.io:88/ http://shortfin.io:88/ Time taken for tests: 12.125 seconds Requests per second: 8.25 [#/sec] (mean) Time per request: 12124.501 [ms] (mean) Time per request: 121.245 [ms] (mean, across all concurrent requests) Transfer rate: 38.46 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 210 1114 1517.2 360 4041 Processing: 5017 8407 2034.0 8715 11762 Waiting: 169 1084 3054.8 188 11408 Total: 9057 9521 1068.7 9058 12123 Results (Best results of 3 runs): ab -n 100 -c 100 http://shortfin.io/ http://shortfin.io/ Time taken for tests: 5.790 seconds Requests per second: 17.27 [#/sec] (mean) Time per request: 5789.949 [ms] (mean) Time per request: 57.899 [ms] (mean, across all concurrent requests) Transfer rate: 82.97 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 171 249 35.7 263 290 Processing: 172 1798 2345.5 212 5524 Waiting: 170 1055 1825.1 212 4949 Total: 344 2047 2368.2 469 5789
- rarestblog 13y ago
  You need some better hardware $ ab -n 100 -c 100 http://shortfin.io/ http://shortfin.io/ Requests per second: 1363.85 [#/sec] (mean)
  - rorrr2 13y ago
    Benchmarking a fast remote static server is pointless. You're basically benchmarking your internet connection.
riledhel 13y ago
I can't find anything related to the response headers the server can send to the client, and the OP site just serves responses with content-length, content-type and server. Anyone?
c-a 13y ago
The request parsing code seems to be of dubious quality https://github.com/timothyej/Shortfin/blob/master/src/request.c#L39 https://github.com/timothyej/Shortfin/blob/master/src/reques... From a cursory glance: 1. https://github.com/timothyej/Shortfin/blob/master/src/request.c#L69 https://github.com/timothyej/Shortfin/blob/master/src/reques... should be (data_len - i >= 4) since it accesses data[i+3] 2. https://github.com/timothyej/Shortfin/blob/master/src/request.c#L125 https://github.com/timothyej/Shortfin/blob/master/src/reques... shouldn't headers[header_count]->key also be null terminated? 3. https://github.com/timothyej/Shortfin/blob/master/src/request.c#L144 https://github.com/timothyej/Shortfin/blob/master/src/reques... header_count can become greater than 49 which causes a heap overflow at https://github.com/timothyej/Shortfin/blob/master/src/request.c#L148 https://github.com/timothyej/Shortfin/blob/master/src/reques... and https://github.com/timothyej/Shortfin/blob/master/src/request.c#L154 https://github.com/timothyej/Shortfin/blob/master/src/reques...
- timothy89 13y ago
  I'll look them up, thanks for noticing!
- andrewcooke 13y ago
  the code in general is not that great, imho. as well as the header count (which i came here to post), there's another suspicious hard-coded size limit in the number of servers (1000). although you could only crash the system in that case by configuring too many. there's very little error handling. good c code returns error codes all over the damn place. this hardly has any. i wouldn't use this.
el-mapache 13y ago
My favorite part is this, under the config section: # vim /etc/shortfin/shortfin.conf # /etc/init.d/shortfind restart No explanation of what the configs are, just that "its important to configure it right." Why not put this information on the site?
zzzcpan 13y ago
I find it very hard to trust any C code without tests. Is this a thing for C code? Why most of the new projects in C lack tests?
timothy89 13y ago
A lot of you wanted benchmarks so here's one comparing Shortfin with Nginx. The tests was performed with a 56.1 kB PNG image with keep-alive turned off on my laptop. The best result out of 3 tests is shown below. tl;dr: Shortfin: 18 914 req/sec Nginx: 15 603 req/sec SHORTFIN sudo ab -n 100 -c 100 http://127.0.0.1:40/timothy-johansson.png Server Software: shortfin/0.9.5 Server Hostname: 127.0.0.1 Server Port: 40 Document Path: /timothy-johansson.png Document Length: 56089 bytes Concurrency Level: 100 Time taken for tests: 0.005 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 5618000 bytes HTML transferred: 5608900 bytes Requests per second: 18914.32 [#/sec] (mean) Time per request: 5.287 [ms] (mean) Time per request: 0.053 [ms] (mean, across all concurrent requests) Transfer rate: 1037701.56 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 2 0.0 1 2 Processing: 2 2 0.1 2 2 Waiting: 1 1 0.2 1 2 Total: 4 4 0.1 4 4 ERROR: The median and mean for the initial connection time are more than twice the standard deviation apart. These results are NOT reliable. Percentage of the requests served within a certain time (ms) 50% 4 66% 4 75% 4 80% 4 90% 4 95% 4 98% 4 99% 4 100% 4 (longest request) NGINX sudo ab -n 100 -c 100 http://127.0.0.1:41/timothy-johansson.png Server Software: nginx/1.2.6 Server Hostname: 127.0.0.1 Server Port: 41 Document Path: /timothy-johansson.png Document Length: 56089 bytes Concurrency Level: 100 Time taken for tests: 0.006 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 5631000 bytes HTML transferred: 5608900 bytes Requests per second: 15603.06 [#/sec] (mean) Time per request: 6.409 [ms] (mean) Time per request: 0.064 [ms] (mean, across all concurrent requests) Transfer rate: 858015.83 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 2 0.7 2 3 Processing: 1 2 0.5 1 3 Waiting: 0 1 0.7 1 3 Total: 2 4 0.9 4 5 WARNING: The median and mean for the processing time are not within a normal deviation These results are probably not that reliable. Percentage of the requests served within a certain time (ms) 50% 4 66% 4 75% 4 80% 4 90% 5 95% 5 98% 5 99% 5 100% 5 (longest request)
nickzoic 13y ago
Lots of people have asked "Why not Nginx? Why not thttpd?" ... but if all you want to host is static files, why not AWS S3 or similar CDN?
simon_vetter 13y ago
I would recommend adding ipv6 support. Should be fairly easy to do in socket.{c,h} and in your configuration parser.
halayli 13y ago
This code is far off from being production ready. Skimming through the code quickly, there's barely any error handling. HTTP parsing is not compliant either. https://github.com/timothyej/Shortfin/blob/master/src/response.c#L5 https://github.com/timothyej/Shortfin/blob/master/src/respon... I am not sure what's the point of using it instead of nginx.

77 points by timothy89 13y ago

You're from Lund! I've been studying in Lund! By the way, can you show some statistics and performance metrics with respect to nginx, Apache and such? To put it another way: why did you choose to write your own webserver instead of using, say, nginx?

davidcollantes 13y ago
I am curious too, on the performance metrics. He is still using Nginx though.
- timothy89 13y ago
  Nginx is just acting as a reverse proxy because I host other sites on the same server. Benchmarks will come!
timothy89 13y ago
Lund is awesome! ;) I will perform some benchmarks and compare it to nginx (and maybe apache) in a while. By the time (2010, 2011) I was working at a company that needed a tracking server for tracking clicks and views in email. So I started to mod lighttpd and thought that I could make it faster. So in true "challenge accepted" spirit I started to write my own HTTP server. And since then I've made it better and better. It's not meant to be an nginx killer. It will always be very basic, fast and lightweight. It's great for projects where you need a web server included. As I wrote in another comment I have used the code base myself in various projects such as: * Tracking clicks, view, etc. for AlphaMail * The send-API server for AlphaMail * A DMARC report server (https://github.com/amail/comfirm-dmarc-report-server https://github.com/amail/comfirm-dmarc-report-server) * A REST server with redis as storage I've never used it as a static web server in production yet. Mostly because of the lack of a reverse proxy and other great features that nginx has and shortfin not.

sciurus 13y ago

There's a sample configuration at https://raw.github.com/timothyej/Shortfin/master/config/shortfin.conf https://raw.github.com/timothyej/Shortfin/master/config/shor...

babuskov 13y ago

I don't get it. You create a server to solve C10K problem, and then host its own website with nginx? Why? Having HN crowd coming and testing your server would be a great test, no?

clone1018 13y ago
The actual site is down for me too, http://shortfin.io:88/ http://shortfin.io:88/
- SirSkidmore 13y ago
  The link works fine for me.
  - clone1018 13y ago
    Work firewall, duh, sorry.
kajecounterhack 13y ago
I think in this case nginx is being used as a reverse proxy.
untothebreach 13y ago
From the footer: > It was created in 2011 by Timothy E. Johansson and is the base for the send-API and the tracking server for AlphaMail. > It also powers this website (though behind nginx). You can reach it directly on port 88: shortfin.io:88
- babuskov 13y ago
  I read that. But why? What's the point in using this if you're going to have nginx in front?
  - nolok 13y ago
    I suppose he is hosting other stuff too, and nginx proxies.

dictum 13y ago

>wget http://shortfin.io/install.sh http://shortfin.io/install.sh && sh install.sh Isn't it recommended that any project doing one-line installs use HTTPS?

chc 13y ago
In the same way that it's recommended for any other program, yes.
bru 13y ago
Side-note for the author: the download can be made more elegant that way $ curl http://shortfin.io/install.sh | sh I definitely agree with the HTTPS. For example this is what meteor proposes: $ curl https://install.meteor.com | sh
- pnathan 13y ago
  asdfsdfgkjlsdfg! Streaming your install script directly into computer execution without even giving it a once-over, md5 compare, etc is atrociously insecure.
  - nmcfarl 13y ago
    This seems to be a common complaint but I can't see any reason you can't give it a once over when it's hosted on the web…
  - chc 13y ago
    That's completely orthogonal to what we're talking about here. If you're already not doing it, you're not doing it any less by being more efficient. And it doesn't really matter if you're running it in a throwaway VM like I do.
  - jgreen10 13y ago
    curl https://install.meteor.com | less
sehrope 13y ago

jfb 13y ago

Just run the following command as root to install the server. # wget http://shortfin.io/install.sh && sh install.sh What? Are they serious? Is this becoming a thing? Please don't tell me that this is a thing.

chrissnell 13y ago
This has been a thing since the mid-90s when you could execute 'telnet sci.dixie.edu 1 | sh' to install an IRC client. It preys on the clueless and one of these days, some project that uses this install method is going to get owned and a bunch of machines will get malware. Then again, most of us blindly run what we assume to be GNU autoconf scripts and never read Makefiles before we 'make install' as root. Nor do most of us read or audit source code, even for security-sensitive applications.
ripter 13y ago
a lot of times it's curl instead of wget.
chc 13y ago
I don't see any rational basis for this. If you weren't vetting the code you downloaded before, you don't lose anything by blindly running the commands on one line instead of three. And if you do want to vet the code, you know what to do. Practically no installation instructions include a "vet the code" step, so you aren't losing anything here. The lack of TLS is worrisome, but that's not so much "becoming a thing" as it is "a bad thing we're just finally starting to get rid of."
- rsync 13y ago
  Although this isn't new, it should make you throw up in your mouth a little bit. Malicious compromise is not the only way that an install script can become broken, or corrupted, on a remote site. The ability to checksum what you're running is a very useful sanity check and you should be doing it with all downloaded scripts/packages. I suppose this is tilting at windmills, though.
  - 13y ago

ahknight 13y ago

okay ... why? what's special about this? features? benchmarks? anything?

grey-area 13y ago
Why not? The author claims it's 'high-performance and open-source' - sounds good to me. It's not as if we have a hundred web servers all competing as we do static blogging systems, the more web servers the better in my opinion, just as with web browsers. The zip file is 312KB as linked from here, so it's pretty compact, possibly suitable for embedding in apps, etc: https://github.com/timothyej/Shortfin https://github.com/timothyej/Shortfin

tuananh 13y ago

author should add a benchmark, comparison. or maybe it's just a fun, personal project and not ready for production.

reidrac 13y ago

I was going to say that there are *~ files in the repository, they should tidy a little bit because things like that don't look good. Then I thought may be I was being a little bit to picky and unfair, so I checked the code of some random files and... well, it looks like it was part of a learning experience but it definitely needs work before being considered a viable option.

ds9 13y ago

What's a "static web server"? I haven't seen this term. A webserver for only static pages, i.e. it won't talk to a runtime? But it's "for serving web apps", this means all the dynamic content is pulled from client side JS? Then you would still need programs on the server to answer AJAX requests. I did not find an explanation on the page or by Googling.

sehrope 13y ago
A server that only serves static files. It doesn't enable the use of dynamic content like CGI[1]. Compare this to something like Apache or nginx. Both can serve static files but also support a number of ways of either directly running scripts (ex: mod_perl or mod_php in Apache) or proxying to other servers that handle dynamic content. [1]: https://en.wikipedia.org/wiki/Common_Gateway_Interface https://en.wikipedia.org/wiki/Common_Gateway_Interface
- deleted 13y ago
  [deleted]
- ds9 13y ago
  OK, then, sorry if I'm being dense, but how can it be "for serving web apps"? App seems an odd term for a site consisting of only static files.
  - packetslave 13y ago
    Perhaps you've heard of this new language, JavaScript?
    - ds9 13y ago
      Still not much of an app unless the JS does AJAX, but then it's talking to a server that's running some dynamic process, maybe with DB access etc. - right?
      - Isofarro 13y ago

ksec 13y ago

Waiting For Benchmarks.

nine_k 13y ago

I think the landing page would greatly benefit from two short paragraphs: Why Shortfin when we have Nginx/Lighttpd? Why Shortfin when we have thttpd?

timothy89 13y ago
1. Shortfin is not a replacement for nginx. I use it myself to create other projects, e.g. fast API servers with one purpose only and a REST server that uses redis as storage. 2. See #1. 3. I don't try to sell anything, it's just an open-source project that I think maybe someone could benefit from. I've learned a lot while coding it.
- nine_k 13y ago
  OK, Shortfin is just trying to be an efficient HTTP server, without a radical differentiating feature or implementation approach. Well, let a hundred flowers blossom.

eksith 13y ago

While I appreciate the time and effort that went into this, I'm more concerned with security than just speed alone. It's still relatively easy to throw hardware at the load problem (up to a point, naturally), but safety shouldn't take a back seat to performance. The wget shell script installation does make me nervous and I'm glad the source is available separately. Blind installation of scripts was never "a thing" with me. Of course, I'm still glad people are writing proper web servers (as opposed to simple 2-10 line ones). That creates an opportunity to explore the field with fresh ideas.

ChuckMcM 13y ago

Close. Can add this to the list like thttpd. Something I keep thinking I'll build is a fast, high connection count, limited HTTP server, something that is essentially a wrapper around a program that works like 'regular' and emits HTML. It is a corner case in a custom corner, but the target it something which is essentially a 'transponder.' In the 'Internet of Things' I want to build a wrapper/environment such that my program can be main(int argc, char *argv[]) { uint16_t sense; uint16_t chan; sense = adc_read(atoi(argv[1])); printf("Sensor %d reads : %d\n", chan, sense); } And then link my 'wrap around web server on it' and then it can be accessed with wget 'http://ip:port/?1' http://ip:port/?1' have it do the right thing. The keys are low memory footprint, lots of connections, easily wrapped around a 'regular' program.

tlrobinson 13y ago
"easily wrapped around a 'regular' program" Sounds like you want CGI :)
- ChuckMcM 13y ago
  Yes, "static CGI" if such a thing was a thing :-). I've got a thttpd tree that I do this sort of hack in (these are great for monitoring large numbers of servers for example).
  - skrebbel 13y ago
    I don't understand (and am interested). What's "static" about it? With Apache and mod_rewrite and CGI, I could make the functionality you want (which is, of course, completely not low footprint, but I don't understand how what you want is different from CGI).
    - ChuckMcM 13y ago
      Static in the sense that it always executes the one function (doesn't load it from disk, its part of the executable) and so there is never any risk that some other path might get you 'out' of the docs directory and into the cgi_bin directory. So in this case static is code for 'compiled in' versus 'dynamically loaded.'

BlackDeath3 13y ago

Very cool. I've been doing the same thing lately, writing my own little web server that serves static files. It's been a fun learning experience!

timothy89 13y ago
Nice, in what language? I've really learned a lot from creating shortin! Shortfin is written in C so it's a nice project to spend some time on when I get tired of writing javascript and html ;)
- BlackDeath3 13y ago
  I'm writing mine in C as well. I've never been too into web programming, but the lower-level stuff is quite interesting to me. Keep it up!

joeblau 13y ago

I just ran two speed tests since there aren't any benchmarks provided 1. First Test is using http://www.webpagetest.org/ http://www.webpagetest.org/ Results shortfin.io:88 - http://www.webpagetest.org/result/130806_71_13HK/ http://www.webpagetest.org/result/130806_71_13HK/ First View: 1.945s Repeat View: 1.632s Results shortfin.io - http://www.webpagetest.org/result/130806_QV_13F7/ http://www.webpagetest.org/result/130806_QV_13F7/ First View 2.099s Repeat View 0.084s 2. Second Test is using Apache Bench Results (Best results of 3 runs): ab -n 100 -c 100 http://shortfin.io:88/ http://shortfin.io:88/ Time taken for tests: 12.125 seconds Requests per second: 8.25 [#/sec] (mean) Time per request: 12124.501 [ms] (mean) Time per request: 121.245 [ms] (mean, across all concurrent requests) Transfer rate: 38.46 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 210 1114 1517.2 360 4041 Processing: 5017 8407 2034.0 8715 11762 Waiting: 169 1084 3054.8 188 11408 Total: 9057 9521 1068.7 9058 12123 Results (Best results of 3 runs): ab -n 100 -c 100 http://shortfin.io/ http://shortfin.io/ Time taken for tests: 5.790 seconds Requests per second: 17.27 [#/sec] (mean) Time per request: 5789.949 [ms] (mean) Time per request: 57.899 [ms] (mean, across all concurrent requests) Transfer rate: 82.97 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 171 249 35.7 263 290 Processing: 172 1798 2345.5 212 5524 Waiting: 170 1055 1825.1 212 4949 Total: 344 2047 2368.2 469 5789

rarestblog 13y ago
You need some better hardware $ ab -n 100 -c 100 http://shortfin.io/ http://shortfin.io/ Requests per second: 1363.85 [#/sec] (mean)
- rorrr2 13y ago
  Benchmarking a fast remote static server is pointless. You're basically benchmarking your internet connection.

riledhel 13y ago

I can't find anything related to the response headers the server can send to the client, and the OP site just serves responses with content-length, content-type and server. Anyone?

c-a 13y ago

The request parsing code seems to be of dubious quality https://github.com/timothyej/Shortfin/blob/master/src/request.c#L39 https://github.com/timothyej/Shortfin/blob/master/src/reques... From a cursory glance: 1. https://github.com/timothyej/Shortfin/blob/master/src/request.c#L69 https://github.com/timothyej/Shortfin/blob/master/src/reques... should be (data_len - i >= 4) since it accesses data[i+3] 2. https://github.com/timothyej/Shortfin/blob/master/src/request.c#L125 https://github.com/timothyej/Shortfin/blob/master/src/reques... shouldn't headers[header_count]->key also be null terminated? 3. https://github.com/timothyej/Shortfin/blob/master/src/request.c#L144 https://github.com/timothyej/Shortfin/blob/master/src/reques... header_count can become greater than 49 which causes a heap overflow at https://github.com/timothyej/Shortfin/blob/master/src/request.c#L148 https://github.com/timothyej/Shortfin/blob/master/src/reques... and https://github.com/timothyej/Shortfin/blob/master/src/request.c#L154 https://github.com/timothyej/Shortfin/blob/master/src/reques...

timothy89 13y ago
I'll look them up, thanks for noticing!
andrewcooke 13y ago
the code in general is not that great, imho. as well as the header count (which i came here to post), there's another suspicious hard-coded size limit in the number of servers (1000). although you could only crash the system in that case by configuring too many. there's very little error handling. good c code returns error codes all over the damn place. this hardly has any. i wouldn't use this.

el-mapache 13y ago

My favorite part is this, under the config section: # vim /etc/shortfin/shortfin.conf # /etc/init.d/shortfind restart No explanation of what the configs are, just that "its important to configure it right." Why not put this information on the site?

zzzcpan 13y ago

I find it very hard to trust any C code without tests. Is this a thing for C code? Why most of the new projects in C lack tests?

timothy89 13y ago

A lot of you wanted benchmarks so here's one comparing Shortfin with Nginx. The tests was performed with a 56.1 kB PNG image with keep-alive turned off on my laptop. The best result out of 3 tests is shown below. tl;dr: Shortfin: 18 914 req/sec Nginx: 15 603 req/sec SHORTFIN sudo ab -n 100 -c 100 http://127.0.0.1:40/timothy-johansson.png Server Software: shortfin/0.9.5 Server Hostname: 127.0.0.1 Server Port: 40 Document Path: /timothy-johansson.png Document Length: 56089 bytes Concurrency Level: 100 Time taken for tests: 0.005 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 5618000 bytes HTML transferred: 5608900 bytes Requests per second: 18914.32 [#/sec] (mean) Time per request: 5.287 [ms] (mean) Time per request: 0.053 [ms] (mean, across all concurrent requests) Transfer rate: 1037701.56 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 2 0.0 1 2 Processing: 2 2 0.1 2 2 Waiting: 1 1 0.2 1 2 Total: 4 4 0.1 4 4 ERROR: The median and mean for the initial connection time are more than twice the standard deviation apart. These results are NOT reliable. Percentage of the requests served within a certain time (ms) 50% 4 66% 4 75% 4 80% 4 90% 4 95% 4 98% 4 99% 4 100% 4 (longest request) NGINX sudo ab -n 100 -c 100 http://127.0.0.1:41/timothy-johansson.png Server Software: nginx/1.2.6 Server Hostname: 127.0.0.1 Server Port: 41 Document Path: /timothy-johansson.png Document Length: 56089 bytes Concurrency Level: 100 Time taken for tests: 0.006 seconds Complete requests: 100 Failed requests: 0 Write errors: 0 Total transferred: 5631000 bytes HTML transferred: 5608900 bytes Requests per second: 15603.06 [#/sec] (mean) Time per request: 6.409 [ms] (mean) Time per request: 0.064 [ms] (mean, across all concurrent requests) Transfer rate: 858015.83 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 2 0.7 2 3 Processing: 1 2 0.5 1 3 Waiting: 0 1 0.7 1 3 Total: 2 4 0.9 4 5 WARNING: The median and mean for the processing time are not within a normal deviation These results are probably not that reliable. Percentage of the requests served within a certain time (ms) 50% 4 66% 4 75% 4 80% 4 90% 5 95% 5 98% 5 99% 5 100% 5 (longest request)

nickzoic 13y ago

Lots of people have asked "Why not Nginx? Why not thttpd?" ... but if all you want to host is static files, why not AWS S3 or similar CDN?

simon_vetter 13y ago

I would recommend adding ipv6 support. Should be fairly easy to do in socket.{c,h} and in your configuration parser.

halayli 13y ago

This code is far off from being production ready. Skimming through the code quickly, there's barely any error handling. HTTP parsing is not compliant either. https://github.com/timothyej/Shortfin/blob/master/src/response.c#L5 https://github.com/timothyej/Shortfin/blob/master/src/respon... I am not sure what's the point of using it instead of nginx.