16 ms·

Kore: a fast web server for writing web apps in C

271 points by api 11y ago

ejcx 11y ago
This looks pretty neat actually. A ton of effort clearly went into it and it looks like the code is really well written with pretty well thought out interfaces "Its main goals are security.." Is it actually? I also don't really see an advantage of using something like this over something like the Go net/http package. Web-type API stuff is usually high enough level that something like C doesn't make sense. Go has nice enough standard packages for system things that even if I was doing a lot of system-y stuff I would be alright. I don't really see the type of work I would be doing where I want to use this.
- mambodog 11y ago
  It does seem like if your goal was security than perhaps Rust would be a better choice.
  - ZoFreX 11y ago
    If your goal is security literally any high level language is a better choice.
    - homarp 11y ago
      High level language like php ? Security and secure code is a mindset. You code without thinking that bad things will happen, you will get bitten. Sure high level language can help with memory management... but plenty of CVE are because of sloppy coding, not because of low level language.
      - girvo 11y ago
        While you are correct in that picking a higher level language doesn't shield you from writing insecure code, insecure C/C++ failure modes are usually quite a bit worse than other languages used for that purpose. I don't trust myself to write code that handles memory 100% correctly and is also network-facing. If much better developers than I manage to screw that up, what chance do I have?
        CHY872 11y ago
        Mmm, it can be difficult to take advantage from C exploits (in the context of a webserver). On a well written C system, you might expect most bugs to lead to crashes. PHP bugs tend to be more exploitable, because you're doing something supported by the language.
        girvo 11y ago
        But what's stopping someone from writing an app-level vulnerability in C vs any other language? Most of them are because of horrible handling of strings, which is something C is also not that great at. I'm not seeing the security benefit here.
        CHY872 11y ago
        My gut feeling is that if you really want security in C, you have fewer constructs to misuse and so you get less unexpected behaviour. At the same time, you get more protection mechanisms; guard pages etc. I.e. harder to be secure, but if you really want to harden, you can get harder than in higher level languages. I'm not putting any kind of weight behind that, though; I just feel that it's a bit odd for people (not specifically meaning you, just the whole thread) to criticise this purely on language choice and not put any substance behind their criticisms that actually relate to the software in question.
      - hvm 11y ago
        I think the idea still stands. PHP is a fake high level language in my opinion. A high level language should reduce boilerplate and 'force' you to write concise and predictable code. PHP does none of that especially in the context of error/exception handling.
        krapp 11y ago
        None of your standards for what a high-level language should be have any bearing on what a high-level language is, according to the definition that people actually use, and those standards might exclude Python, Ruby, C++ and Java among others.
      - ZoFreX 11y ago
        Actually yes. For security, even PHP is a better choice than C (for certain versions of the idea of "security"). There's entire classes of security problems that are literally impossible in PHP.
  - vidarh 11y ago
    Rust won't help you on the dozens of platforms it isn't available.
- jzelinskie 11y ago
  I can see this being useful for embedded systems where Go is not suitable. I'm not super familiar with that whole world, though.
  - ejcx 11y ago
    Ahh that is a use case I hadn't considered. I guess it would be pretty useful if some type of basic webserver was required.
- deleted 11y ago
  [deleted]
  - nkozyra 11y ago
    This kind of statement warrants a lot more support.
  - pekk 11y ago
    Though Rust is very popular around here, and is an exciting project, this is a very low-effort comment which doesn't cast any light on the subject.
  - jerf 11y ago
    http://arewewebyet.com/ http://arewewebyet.com/ If you need something running on HTTP next week or next month, it isn't. If you can wait a year or two, it'll probably be a closer thing.
    - SwellJoe 11y ago
      Given that this project is implementing a web server from the ground up, that site doesn't really have advice. The milestones for "Are we web yet?" include a web server, someone writing a new web server in Rust would contribute toward the goal, rather than relying on the dependencies being discussed. "Are we web yet?" is about whether you can effectively build web apps in Rust, not whether you can effectively build core networking infrastructure in Rust. That's not to say I'm arguing this should have been written in Rust (if it were me, I might have done so, but it's not, so I don't get a say). Also, you can't possibly argue that C satisfies all, or even most, of the milestones given for "Are we web yet?"
      - jerf 11y ago
        "Also, you can't possibly argue that C satisfies all, or even most, of the milestones given for "Are we web yet?"" C has all the libraries. Like, all of them, ever. I don't like it necessarily, but it's true. It has all the HTTP servers, all the database drivers, all the email and all of the "misc". (Yes, not literally all. But it's a closer thing than we'd like to admit!) Except for the fact that it's basically completely unsuitable for use in a network environment due to its design flaws, and one "security-focused" framework can't change that because you've still got all the rest of C's problems and large set of libraries that also really shouldn't be put on the network (and frankly I still trust the "security-focused" framework about as far as I can throw its immaterial self, because it's still written in C), C would be the perfect web programming language, and passes, yes, darned near everything. Also, I don't know if you got here after the message was deleted. I think my reply makes more sense in the original context it appeared in.
- jvink 11y ago
  It is. For example being strict on the network input path and doing proper validation of incoming data is a strong part of the design. Or was the question more related to, it is C therefor security cannot be part of the process?
- vidarh 11y ago
  > I also don't really see an advantage of using something like this over something like the Go net/http package. C runs "everywhere". Your old C64 from the 80's? Has C compilers. Go does not, even with gccgo. I'd bet there are also still likely at least two orders of magnitude more programmers that know C well, and still more programmers with in depth experience of embedded development in C than have even tried Go. I've still yet to meet anyone outside of the startup devops bubble that have written any Go, and often what language the developers have experience with matters more.
  - pjmlp 11y ago
    > Your old C64 from the 80's? Has C compilers Yeah, but the idea to use them back then was like trying to use Ruby for real time applications in modern days, given how shitty they were. Home computers might have had implementations of C and Pascal dialects, but we all used Assembly when stepping out of the built-in Basic and Forth enviroments.
danieltillett 11y ago
Has anyone done any performance testing on kore? Yes I know I can do this myself, but why invent the wheel.
- delinka 11y ago
  You'd invent the wheel to get you places. You wouldn't reinvent the wheel when someone's already done a perfectly good job.
  - danieltillett 11y ago
    So much for me trying to be clever and avoid a cliche :)
  - rch 11y ago
    as in... http://okws.org http://okws.org
    - danieltillett 11y ago
      Interesting. Any performance data on this beyond the vague it is faster?
      - rch 11y ago
        Anything I've seen is 10-ish years old. It was relatively fast, but also complex and covered by the GPL (for better or worse).
        danieltillett 11y ago
        I found this which seems interesting https://github.com/koanlogic/klone https://github.com/koanlogic/klone
        rch 11y ago
        The portability aspect is interesting. I use uwsgi extensively (not just with python), and I think it sets the bar these days.
yuvipanda 11y ago
Awesome! I've used https://www.gnu.org/software/libmicrohttpd/ https://www.gnu.org/software/libmicrohttpd/ in the past (for fun stuff I was building to learn C), will try this out instead. But.. > Only HTTPS connections allowed Seems a bit painful. HTTPS on the webserver itself is a fair bit more painful to setup and administer than HTTPS in your reverse proxy / loadbalancer (I prefer nginx). Web servers should support plain HTTP.
- bubblesorting 11y ago
  I'm not trying to antagonize here, but what do you find so bad? I would call it a little bit more painful; three lines to add to an nginx config, along with generating the cert. Maybe 10 minutes of work? Thirty if you're getting a CA to sign your cert for you. I could see pain if you need to wait for finance to approve, or if you're trying to get domains validated on behalf of a customer. And I suppose it adds another setup step to wireshark (if you need to debug neat bugs), but that's a set-it-up-once-and-forget-about-it thing.
  - Canada 11y ago
    It's a scalability problem. If you have many web servers behind a reverse proxy that takes care of TLS it's often pointless to waste processing power on encryption in between.
    - anthony_franco 11y ago
      Yep. And on top of that a reverse proxy is unable to inspect the requests to route them smartly (based on path, cookies, etc.).
      - Canada 11y ago
        Well, the reverse proxy would have made the backend request itself, so it's free to inspect or modify anything.
- jvink 11y ago
rdtsc 11y ago
The site and documentations looks well done, great job! Architecture looks pretty interesting too. Wonder why was there a need for an accept lock? Ordinary accept() socket call already allows for simultaneous threads/process wait on a single socket.
- luikore 11y ago
  I think the authors want to avoid thundering herd. You can find this basic pattern in the book UNIX Network Programming.
  - rdtsc 11y ago
    Hmm, thought that was fixed 10 years ago or so. I can't believe the standard accept() call will wake up all threads on a modern Linux kernel. Maybe if it is using epoll/select/etc. it would exhibit the thundering herd issue.
  - jvink 11y ago
    Correct. The accepting socket is shared between multiple workers which each have its own fd for epoll or kqueue. Because of this a form of serialising the accepts between said workers is needed to avoid unnecessary wakeups.
    - rdtsc 11y ago
      Actually that is being changed: http://lwn.net/Articles/633422/ http://lwn.net/Articles/633422/ See part about EPOLLEXCLUSIVE
      - jvink 11y ago
        That is great, thanks for sharing.
        rdtsc 11y ago
        If you are the author, thanks for sharing the project. You did a great job and made the right choice of having per CPU worker processes each with their own epoll loop.
deleted 11y ago
[deleted]
slowmovintarget 11y ago
Writing a web application in C sounds like a good trigger for an utterance from the Jargon File: "You could do that, but that'd be like kicking dead whales down the beach." We've advanced the state of the art quite a bit with dramatically more expressive languages than C that are sufficiently efficient in terms of memory and CPU. This is especially true when communications are occurring over HTTP and not direct socket-to-socket comms. Why use C instead of D, Rust, Go, C#, Java, Perl, Python, Ruby, Scala, Clojure, Erlang, Elixir, Haskell, Swift, OCaml, Objective-C...? I didn't miss C++, it just seems a worse alternative than C.
- nulltype 11y ago
  Thank god it's written in C and not C++
  - CountSessine 11y ago
    Yeah. Thank God. I love broken string implementationsssssdd##%^$*$#@@#$%$$$###ludvik^0knox^0shamen^0password^0pass^012345^0qwerty^0dreamdevil^0 C is just awesome.
    - nulltype 11y ago
      We got that solved. https://github.com/antirez/sds https://github.com/antirez/sds
      - pjmlp 11y ago
        Except it isn't neither part of ANSI C, nor is used by any relevant third party C library, but it is solved....
      - CountSessine 11y ago
        This seems to be an adaptation of the earlier SafeStr lib. SafeStr is nice except that the C community is still too busy arguing about their awful lstr* and str*_s functions to take any notice and incorporate something genuinely secure like SafeStr into the standard lib. And the NVD just keeps growing... So... you guys added something random and silly like <complex.h> to the standard, but still couldn't get around to a working string implementation? OK. Well, good luck with all that. Not solved yet.
dang 11y ago
Previously: https://news.ycombinator.com/item?id=5995298 https://news.ycombinator.com/item?id=5995298.
windlep 11y ago
Was quite excited to try out a little websocket server with Kore till I saw it fork's per connection. I don't really want 20k processes for handling 20k connections, I was really hoping for an event loop.
- altano 11y ago
  Evented io is great for extremely high concurrency, but that isn't always the right thing to optimize for. A forking web server might be faster for users depending on the application. Lastly, you can't just have an event loop without also creating an entirely async platform. For an event loop to work well, all operations from file reading to network requests need to be completely async.
  - bazookajoes 11y ago
    Out of curiosity in what scenarios do you see a forking web server being faster than a evented server that balances requests across cores and can direct a request to the core with the best cache for the request? I completely agree with need to async. The hard part is that many operations are async without an async interface. For example memory allocation, or even memory usage if the memory was not truly allocated by malloc.
    - nomel 11y ago
      Any non asynchronous application.
      - dhaivatpandya 11y ago
        Why would you have a web server that doesn't process requests async?
        otterley 11y ago
        There are a few reasons I can think of. 1) Client libraries you might need to use in your web service might not be available in asynchronous versions. 2) Writing blocking code is much easier to write than asynchronous code. 3) Your server code is CPU bound, so there's no benefit to an asynchronous model. 4) If your web app runs in an asynchronous server and your app crashes, it'll crash the whole server. On the other hand, in a forking model, only the client that the child is serving will be impacted; the other workers will be unaffected. 5) Memory leaks are easier to contain in a forking model, assuming the child can exit or be killed after N requests.
rezacks 11y ago
I've used https://github.com/cesanta/mongoose https://github.com/cesanta/mongoose in the past maybe I could try this one.
deleted 11y ago
[deleted]
randomfool 11y ago
I'm curious- why C? Strings, scoped objects and C++11 move operators seems much safer and clearer from an API perspective. The complaints about C++ seem to mostly be around the ability to abuse the language, not specific issues that C solves. Something like https://github.com/facebook/proxygen https://github.com/facebook/proxygen seems like a better API. And I don't quite buy portability- if it's not a modern compiler with decent security checks then I'm note sure it should be building web-facing code.
- luikore 11y ago
  Why not? I'd love to build web applications on a router with pure C.
  - tomcam 11y ago
    Man, me too. I feel more comfortable knowing where all my memory is, and that my types are being checked at compile time.
    - pjmlp 11y ago
      Do you really know where all you memory is being used without Valgrid?
      - nitrogen 11y ago
        Valgrind should definitely be a part of every C developer's workflow, if possible integrated into the unit and integration tests.
        pjmlp 11y ago
        It doesn't help that is isn't available in all platforms, e.g. embedded, real-time OS, commercial UNIXes, mainframes.
        askfakasdsfsfk 11y ago
poindontcare 11y ago
hmm maybe put it in the kernel and use it for debugging? cant use userspace threading libraries though.
golergka 11y ago
So, on one hand, it's one process per connection and ease of development of C, but on the other hand, it has execution speed of C. Can anyone who works in web (I don't, but I'm curious) explain what kind of services is this good and bad for?
- tomjen3 11y ago
  Control interfaces to embedded devices?
  - golergka 11y ago
    Thanks, that actually seems to be the perfect use-case (from my almost non-existent experience with web development).
- jvink 11y ago
  It is not a forking web server. It is evented I/O with multiple worker processes. It is literally in the documentation and easily spottable in the code.
vbernat 11y ago
When an HTTP API is just an additional feature of a larger project, it may make sense to keep using C: a toolchain available everywhere and well known (including cross-compilation and full bootstrap), a small memory footprint, easy use of any library needed for the project. I am doing a lot of that and will keep a look at Kore. Unfortunately, HTTPS only and non-evented core is a no-go for me. I am currently relying on the web server embedded in libevent, as well as wslay for websockets and some additional code for SSE. To easily start a project, I am using a cookiecutter template: https://github.com/vincentbernat/bootstrap.c-web https://github.com/vincentbernat/bootstrap.c-web
- jvink 11y ago
  Perhaps you can check out the websocket and SSE examples. https://github.com/jorisvink/kore/tree/master/examples/sse https://github.com/jorisvink/kore/tree/master/examples/sse https://github.com/jorisvink/kore/tree/master/examples/websocket https://github.com/jorisvink/kore/tree/master/examples/webso...
unixprogrammer 11y ago
It speaks poorly of this community that you get second-guessed more for building an application in C than for building one in JavaScript + Node.js.
- j-pb 11y ago
  If you want to use C today you need a really good excuse. The tradeoff between security and performance is simply not there anymore. Unless you write on an embedded system, a game, or a high performance number crunching application, C is premature optimisation. And even in the above we see drastic changes today, embedded systems have become so powerful that they can run scripting languages (http://www.eluaproject.net http://www.eluaproject.net), game engines are written in C and scripted with other things (http://docs.unity3d.com/ScriptReference/ http://docs.unity3d.com/ScriptReference/), and inmemory-bigdata systems like spark offer significant advantages over classical HPC frameworks like MPI (http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/). While JS is horrid, it at least doesn't have manual memory management.
  - fla 11y ago
    C is alot about portability, not necessarly performance.
    - j-pb 11y ago
      If you only care for portability you can always use one of the billion safe languages whose interpreters are written in C. Lua for example.
      - vidarh 11y ago
        You may not only care about portability. And "safety" sometimes also means being able to meet realtime requirements, and also often means having the ability to carefully account for resource use in ways that many interpreters does poorly (and no, kernel enforced system limits are not always an option). Many interpreters also make assumptions (e.g. expecting a POSIX'y system) about the host system that many embedded platforms doesn't necessarily meet. I've more than once looked at interpreters for embedding and found most of the alternatives sorely lacking. Very few interpreters are well suited for embedding on constrained platforms at all (Lua, admittedly is probably one of the more solid exceptions). And once you start having to write lots of support code in C to port or sandbox your interpreter of choice, the reason for considering an interpreter quickly becomes less compelling.
stefantalpalaru 11y ago
You can also do that with uWSGI: http://uwsgi.readthedocs.org/en/latest/Symcall.html http://uwsgi.readthedocs.org/en/latest/Symcall.html
zippie 11y ago
Process per connection does not scale no matter how light weight, even with COW. kore's connection handling model is the reason apache2 mpm_prefork fell out of favor many iterations ago. The only valid argument to avoid a single event based I/O is some sort of hard blocking I/O such as disk or non-queuing chardev. However I'm still not biting, this is solved...and as usual the answer is somewhere in between. For example, in RIBS2 there are two models for connection handling, event loops for connections and "ribbons" for the non-queuing bits [1]. RIBS2 is also written in C for C. [1] https://github.com/Adaptv/ribs2/blob/master/README https://github.com/Adaptv/ribs2/blob/master/README Edit - mention RIBS2 is also written in C
- jvink 11y ago
  Kore does not fork per connection. It uses per cpu worker processes which multiplex I/O over either epoll or kqueue.
  - zippie 11y ago
    A clone is a clone. The main inplementation difference between a thread and process in Linux is COW, for all intensive purposes a worker process and a fork are the same in this use case. Neither have led to scalable web servers.
    - jvink 11y ago
      Except you are basing yourself on the fact it creates a single worker process per connection. It does not. Workers are spawned when the server is started. Each of them deals with tens of thousands of connections on its own via the listening socket they share. This is a common technique and scales incredible well.
- otterley 11y ago
  > apache2 mpm_prefork fell out of favor many iterations ago. By whom, exactly? There are still plenty of reasons to use a forking web server (see my other comment in this discussion). Saying it "does not scale" is misleading; even with a event-driven model there are only so many CPU resources that can be used to serve responses to clients. Event-driven webservers are fantastic compared to forking ones, for keeping open many thousands of relatively idle connections (if that is your definition of "scale"). But many web services simply don't do that. Preforking webservers, like event-driven ones, still have a rightful place in this world. As with all things technology, you have to pick the right tool for the job.
faragon 11y ago
Writing high level C applications can be easy, if you use a library that frees you from using dynamic memory on typical data structures (e.g. strings, vectors, sorted binary trees, maps). I'm developing a C library for high-level C code, with hard real-time in mind, is already functional for static linking: https://github.com/faragon/libsrt https://github.com/faragon/libsrt
maciej 11y ago
If you think Kore is interesting, then also check out Tntnet <http://www.tntnet.org> http://www.tntnet.org>. I've checked it out a few years ago and it felt good - stable, complete, easy to use etc.
bitL 11y ago
Good job & good luck! ;-)
sownkunz 11y ago
this is AWESOME!
talnet 11y ago
Anyway I love this.
z3t4 11y ago
Kore mapped to JavaScript (NodeJS): var kore = require("kore"); kore.on("request", http_request); function http_request(req, resp) { var statusCode = 200; resp.write("Hello world", statusCode); }
bubbba 11y ago
Probably it would be better to have also a lib or dll, not just a program that render c/c++ servlets. It seems that all (that the kore executes from the feeded code) is running in servlet threads or at least started from servlet thread. Or it would be cool to add some "application" framework, not only "C-servlet" framework
unwind 11y ago
Great! This looks really nice, and interesting. Some fantastically quick points from a very cursory glance at the code. Feel free to ignore this. - The code uses the convention to put the argument of return inside parentheses, making it look like a function call. This is very strange, to me. - It treats sizeof as a function too (i.e. always parentheses the argument). - It is not C99, which always seems so fantastically defensive these days. - It's not (in my opinion) sufficiently const-happy. - I saw at least one instance (in cli.c) of a long string not being written as a auto-concatenated literal but instead leading to multiple fprintf() calls. Very obviously not in a performance-critical place, so perhaps it's not indicative of anything. It just made me take notice.
- jvink 11y ago
  Author here. I see you picked out the few things that I consistently hear on the coding style I adopted which is based on my time hacking on openbsd. I have no real points to argue against those as it is based on preference in my opinion. I am curious why you arrived on it not being sufficiently constified however. I'll gladly make sensible changes. As for the multiple fprintf() calls ... to me it just reads better and the place it occurs in is as you stated pretty obvious non performance critical.
  - unwind 11y ago
    Right. I could have guessed these were based on some coding style guide from somewhere. I still don't see the point, or why any sane guide would prefer to treat return as a function. It just never seems helpful to me, and always wasteful/more complicated. I realize it's just two tokens, so it's probably not "important" in any real sense of the word, but it irks me. I like to point it out since it can help others cargo-culting this. It's not sufficiently const if there are places where a variable could be const but still isn't. :) To be super-specific, the variable 'r' here: https://github.com/jorisvink/kore/blob/master/src/cli.c#L542 https://github.com/jorisvink/kore/blob/master/src/cli.c#L542 is one such case. It should be declared inside the loop, i.e. as "const ssize_t r = write(...);" since once assigned the return value from write(), it's read-only. Of course, many ancient-smelling style guides seem to outlaw declaring variables as close to their point of usage, too. Note that declaring variables inside scopes other than the "root" one in a function isn't even C99, but many people seem to think you can't do that.

271 points by api 11y ago

ejcx 11y ago

This looks pretty neat actually. A ton of effort clearly went into it and it looks like the code is really well written with pretty well thought out interfaces "Its main goals are security.." Is it actually? I also don't really see an advantage of using something like this over something like the Go net/http package. Web-type API stuff is usually high enough level that something like C doesn't make sense. Go has nice enough standard packages for system things that even if I was doing a lot of system-y stuff I would be alright. I don't really see the type of work I would be doing where I want to use this.

mambodog 11y ago
It does seem like if your goal was security than perhaps Rust would be a better choice.
- ZoFreX 11y ago
  If your goal is security literally any high level language is a better choice.
  - homarp 11y ago
    High level language like php ? Security and secure code is a mindset. You code without thinking that bad things will happen, you will get bitten. Sure high level language can help with memory management... but plenty of CVE are because of sloppy coding, not because of low level language.
    - girvo 11y ago
      While you are correct in that picking a higher level language doesn't shield you from writing insecure code, insecure C/C++ failure modes are usually quite a bit worse than other languages used for that purpose. I don't trust myself to write code that handles memory 100% correctly and is also network-facing. If much better developers than I manage to screw that up, what chance do I have?
      - CHY872 11y ago
        Mmm, it can be difficult to take advantage from C exploits (in the context of a webserver). On a well written C system, you might expect most bugs to lead to crashes. PHP bugs tend to be more exploitable, because you're doing something supported by the language.
        girvo 11y ago
        But what's stopping someone from writing an app-level vulnerability in C vs any other language? Most of them are because of horrible handling of strings, which is something C is also not that great at. I'm not seeing the security benefit here.
        CHY872 11y ago
        My gut feeling is that if you really want security in C, you have fewer constructs to misuse and so you get less unexpected behaviour. At the same time, you get more protection mechanisms; guard pages etc. I.e. harder to be secure, but if you really want to harden, you can get harder than in higher level languages. I'm not putting any kind of weight behind that, though; I just feel that it's a bit odd for people (not specifically meaning you, just the whole thread) to criticise this purely on language choice and not put any substance behind their criticisms that actually relate to the software in question.
    - hvm 11y ago
      I think the idea still stands. PHP is a fake high level language in my opinion. A high level language should reduce boilerplate and 'force' you to write concise and predictable code. PHP does none of that especially in the context of error/exception handling.
      - krapp 11y ago
        None of your standards for what a high-level language should be have any bearing on what a high-level language is, according to the definition that people actually use, and those standards might exclude Python, Ruby, C++ and Java among others.
    - ZoFreX 11y ago
      Actually yes. For security, even PHP is a better choice than C (for certain versions of the idea of "security"). There's entire classes of security problems that are literally impossible in PHP.
- vidarh 11y ago
  Rust won't help you on the dozens of platforms it isn't available.
jzelinskie 11y ago
I can see this being useful for embedded systems where Go is not suitable. I'm not super familiar with that whole world, though.
- ejcx 11y ago
  Ahh that is a use case I hadn't considered. I guess it would be pretty useful if some type of basic webserver was required.
deleted 11y ago
[deleted]
- nkozyra 11y ago
  This kind of statement warrants a lot more support.
- pekk 11y ago
  Though Rust is very popular around here, and is an exciting project, this is a very low-effort comment which doesn't cast any light on the subject.
- jerf 11y ago
  http://arewewebyet.com/ http://arewewebyet.com/ If you need something running on HTTP next week or next month, it isn't. If you can wait a year or two, it'll probably be a closer thing.
  - SwellJoe 11y ago
    Given that this project is implementing a web server from the ground up, that site doesn't really have advice. The milestones for "Are we web yet?" include a web server, someone writing a new web server in Rust would contribute toward the goal, rather than relying on the dependencies being discussed. "Are we web yet?" is about whether you can effectively build web apps in Rust, not whether you can effectively build core networking infrastructure in Rust. That's not to say I'm arguing this should have been written in Rust (if it were me, I might have done so, but it's not, so I don't get a say). Also, you can't possibly argue that C satisfies all, or even most, of the milestones given for "Are we web yet?"
    - jerf 11y ago
      "Also, you can't possibly argue that C satisfies all, or even most, of the milestones given for "Are we web yet?"" C has all the libraries. Like, all of them, ever. I don't like it necessarily, but it's true. It has all the HTTP servers, all the database drivers, all the email and all of the "misc". (Yes, not literally all. But it's a closer thing than we'd like to admit!) Except for the fact that it's basically completely unsuitable for use in a network environment due to its design flaws, and one "security-focused" framework can't change that because you've still got all the rest of C's problems and large set of libraries that also really shouldn't be put on the network (and frankly I still trust the "security-focused" framework about as far as I can throw its immaterial self, because it's still written in C), C would be the perfect web programming language, and passes, yes, darned near everything. Also, I don't know if you got here after the message was deleted. I think my reply makes more sense in the original context it appeared in.
jvink 11y ago
It is. For example being strict on the network input path and doing proper validation of incoming data is a strong part of the design. Or was the question more related to, it is C therefor security cannot be part of the process?
vidarh 11y ago
> I also don't really see an advantage of using something like this over something like the Go net/http package. C runs "everywhere". Your old C64 from the 80's? Has C compilers. Go does not, even with gccgo. I'd bet there are also still likely at least two orders of magnitude more programmers that know C well, and still more programmers with in depth experience of embedded development in C than have even tried Go. I've still yet to meet anyone outside of the startup devops bubble that have written any Go, and often what language the developers have experience with matters more.
- pjmlp 11y ago
  > Your old C64 from the 80's? Has C compilers Yeah, but the idea to use them back then was like trying to use Ruby for real time applications in modern days, given how shitty they were. Home computers might have had implementations of C and Pascal dialects, but we all used Assembly when stepping out of the built-in Basic and Forth enviroments.

danieltillett 11y ago

Has anyone done any performance testing on kore? Yes I know I can do this myself, but why invent the wheel.

delinka 11y ago
You'd invent the wheel to get you places. You wouldn't reinvent the wheel when someone's already done a perfectly good job.
- danieltillett 11y ago
  So much for me trying to be clever and avoid a cliche :)
- rch 11y ago
  as in... http://okws.org http://okws.org
  - danieltillett 11y ago
    Interesting. Any performance data on this beyond the vague it is faster?
    - rch 11y ago
      Anything I've seen is 10-ish years old. It was relatively fast, but also complex and covered by the GPL (for better or worse).
      - danieltillett 11y ago
        I found this which seems interesting https://github.com/koanlogic/klone https://github.com/koanlogic/klone
        rch 11y ago
        The portability aspect is interesting. I use uwsgi extensively (not just with python), and I think it sets the bar these days.

yuvipanda 11y ago

Awesome! I've used https://www.gnu.org/software/libmicrohttpd/ https://www.gnu.org/software/libmicrohttpd/ in the past (for fun stuff I was building to learn C), will try this out instead. But.. > Only HTTPS connections allowed Seems a bit painful. HTTPS on the webserver itself is a fair bit more painful to setup and administer than HTTPS in your reverse proxy / loadbalancer (I prefer nginx). Web servers should support plain HTTP.

bubblesorting 11y ago
I'm not trying to antagonize here, but what do you find so bad? I would call it a little bit more painful; three lines to add to an nginx config, along with generating the cert. Maybe 10 minutes of work? Thirty if you're getting a CA to sign your cert for you. I could see pain if you need to wait for finance to approve, or if you're trying to get domains validated on behalf of a customer. And I suppose it adds another setup step to wireshark (if you need to debug neat bugs), but that's a set-it-up-once-and-forget-about-it thing.
- Canada 11y ago
  It's a scalability problem. If you have many web servers behind a reverse proxy that takes care of TLS it's often pointless to waste processing power on encryption in between.
  - anthony_franco 11y ago
    Yep. And on top of that a reverse proxy is unable to inspect the requests to route them smartly (based on path, cookies, etc.).
    - Canada 11y ago
      Well, the reverse proxy would have made the backend request itself, so it's free to inspect or modify anything.
jvink 11y ago

rdtsc 11y ago

The site and documentations looks well done, great job! Architecture looks pretty interesting too. Wonder why was there a need for an accept lock? Ordinary accept() socket call already allows for simultaneous threads/process wait on a single socket.

luikore 11y ago
I think the authors want to avoid thundering herd. You can find this basic pattern in the book UNIX Network Programming.
- rdtsc 11y ago
  Hmm, thought that was fixed 10 years ago or so. I can't believe the standard accept() call will wake up all threads on a modern Linux kernel. Maybe if it is using epoll/select/etc. it would exhibit the thundering herd issue.
- jvink 11y ago
  Correct. The accepting socket is shared between multiple workers which each have its own fd for epoll or kqueue. Because of this a form of serialising the accepts between said workers is needed to avoid unnecessary wakeups.
  - rdtsc 11y ago
    Actually that is being changed: http://lwn.net/Articles/633422/ http://lwn.net/Articles/633422/ See part about EPOLLEXCLUSIVE
    - jvink 11y ago
      That is great, thanks for sharing.
      - rdtsc 11y ago
        If you are the author, thanks for sharing the project. You did a great job and made the right choice of having per CPU worker processes each with their own epoll loop.

deleted 11y ago

[deleted]

slowmovintarget 11y ago

Writing a web application in C sounds like a good trigger for an utterance from the Jargon File: "You could do that, but that'd be like kicking dead whales down the beach." We've advanced the state of the art quite a bit with dramatically more expressive languages than C that are sufficiently efficient in terms of memory and CPU. This is especially true when communications are occurring over HTTP and not direct socket-to-socket comms. Why use C instead of D, Rust, Go, C#, Java, Perl, Python, Ruby, Scala, Clojure, Erlang, Elixir, Haskell, Swift, OCaml, Objective-C...? I didn't miss C++, it just seems a worse alternative than C.

nulltype 11y ago
Thank god it's written in C and not C++
- CountSessine 11y ago
  Yeah. Thank God. I love broken string implementationsssssdd##%^$*$#@@#$%$$$###ludvik^0knox^0shamen^0password^0pass^012345^0qwerty^0dreamdevil^0 C is just awesome.
  - nulltype 11y ago
    We got that solved. https://github.com/antirez/sds https://github.com/antirez/sds
    - pjmlp 11y ago
      Except it isn't neither part of ANSI C, nor is used by any relevant third party C library, but it is solved....
    - CountSessine 11y ago
      This seems to be an adaptation of the earlier SafeStr lib. SafeStr is nice except that the C community is still too busy arguing about their awful lstr* and str*_s functions to take any notice and incorporate something genuinely secure like SafeStr into the standard lib. And the NVD just keeps growing... So... you guys added something random and silly like <complex.h> to the standard, but still couldn't get around to a working string implementation? OK. Well, good luck with all that. Not solved yet.

dang 11y ago

Previously: https://news.ycombinator.com/item?id=5995298 https://news.ycombinator.com/item?id=5995298.

windlep 11y ago

Was quite excited to try out a little websocket server with Kore till I saw it fork's per connection. I don't really want 20k processes for handling 20k connections, I was really hoping for an event loop.

altano 11y ago
Evented io is great for extremely high concurrency, but that isn't always the right thing to optimize for. A forking web server might be faster for users depending on the application. Lastly, you can't just have an event loop without also creating an entirely async platform. For an event loop to work well, all operations from file reading to network requests need to be completely async.
- bazookajoes 11y ago
  Out of curiosity in what scenarios do you see a forking web server being faster than a evented server that balances requests across cores and can direct a request to the core with the best cache for the request? I completely agree with need to async. The hard part is that many operations are async without an async interface. For example memory allocation, or even memory usage if the memory was not truly allocated by malloc.
  - nomel 11y ago
    Any non asynchronous application.
    - dhaivatpandya 11y ago
      Why would you have a web server that doesn't process requests async?
      - otterley 11y ago
        There are a few reasons I can think of. 1) Client libraries you might need to use in your web service might not be available in asynchronous versions. 2) Writing blocking code is much easier to write than asynchronous code. 3) Your server code is CPU bound, so there's no benefit to an asynchronous model. 4) If your web app runs in an asynchronous server and your app crashes, it'll crash the whole server. On the other hand, in a forking model, only the client that the child is serving will be impacted; the other workers will be unaffected. 5) Memory leaks are easier to contain in a forking model, assuming the child can exit or be killed after N requests.

rezacks 11y ago

I've used https://github.com/cesanta/mongoose https://github.com/cesanta/mongoose in the past maybe I could try this one.

deleted 11y ago

[deleted]

randomfool 11y ago

I'm curious- why C? Strings, scoped objects and C++11 move operators seems much safer and clearer from an API perspective. The complaints about C++ seem to mostly be around the ability to abuse the language, not specific issues that C solves. Something like https://github.com/facebook/proxygen https://github.com/facebook/proxygen seems like a better API. And I don't quite buy portability- if it's not a modern compiler with decent security checks then I'm note sure it should be building web-facing code.

luikore 11y ago
Why not? I'd love to build web applications on a router with pure C.
- tomcam 11y ago
  Man, me too. I feel more comfortable knowing where all my memory is, and that my types are being checked at compile time.
  - pjmlp 11y ago
    Do you really know where all you memory is being used without Valgrid?
    - nitrogen 11y ago
      Valgrind should definitely be a part of every C developer's workflow, if possible integrated into the unit and integration tests.
      - pjmlp 11y ago
        It doesn't help that is isn't available in all platforms, e.g. embedded, real-time OS, commercial UNIXes, mainframes.
        askfakasdsfsfk 11y ago

poindontcare 11y ago

hmm maybe put it in the kernel and use it for debugging? cant use userspace threading libraries though.

golergka 11y ago

So, on one hand, it's one process per connection and ease of development of C, but on the other hand, it has execution speed of C. Can anyone who works in web (I don't, but I'm curious) explain what kind of services is this good and bad for?

tomjen3 11y ago
Control interfaces to embedded devices?
- golergka 11y ago
  Thanks, that actually seems to be the perfect use-case (from my almost non-existent experience with web development).
jvink 11y ago
It is not a forking web server. It is evented I/O with multiple worker processes. It is literally in the documentation and easily spottable in the code.

vbernat 11y ago

When an HTTP API is just an additional feature of a larger project, it may make sense to keep using C: a toolchain available everywhere and well known (including cross-compilation and full bootstrap), a small memory footprint, easy use of any library needed for the project. I am doing a lot of that and will keep a look at Kore. Unfortunately, HTTPS only and non-evented core is a no-go for me. I am currently relying on the web server embedded in libevent, as well as wslay for websockets and some additional code for SSE. To easily start a project, I am using a cookiecutter template: https://github.com/vincentbernat/bootstrap.c-web https://github.com/vincentbernat/bootstrap.c-web

jvink 11y ago
Perhaps you can check out the websocket and SSE examples. https://github.com/jorisvink/kore/tree/master/examples/sse https://github.com/jorisvink/kore/tree/master/examples/sse https://github.com/jorisvink/kore/tree/master/examples/websocket https://github.com/jorisvink/kore/tree/master/examples/webso...

unixprogrammer 11y ago

It speaks poorly of this community that you get second-guessed more for building an application in C than for building one in JavaScript + Node.js.

j-pb 11y ago
If you want to use C today you need a really good excuse. The tradeoff between security and performance is simply not there anymore. Unless you write on an embedded system, a game, or a high performance number crunching application, C is premature optimisation. And even in the above we see drastic changes today, embedded systems have become so powerful that they can run scripting languages (http://www.eluaproject.net http://www.eluaproject.net), game engines are written in C and scripted with other things (http://docs.unity3d.com/ScriptReference/ http://docs.unity3d.com/ScriptReference/), and inmemory-bigdata systems like spark offer significant advantages over classical HPC frameworks like MPI (http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/ http://www.dursi.ca/hpc-is-dying-and-mpi-is-killing-it/). While JS is horrid, it at least doesn't have manual memory management.
- fla 11y ago
  C is alot about portability, not necessarly performance.
  - j-pb 11y ago
    If you only care for portability you can always use one of the billion safe languages whose interpreters are written in C. Lua for example.
    - vidarh 11y ago
      You may not only care about portability. And "safety" sometimes also means being able to meet realtime requirements, and also often means having the ability to carefully account for resource use in ways that many interpreters does poorly (and no, kernel enforced system limits are not always an option). Many interpreters also make assumptions (e.g. expecting a POSIX'y system) about the host system that many embedded platforms doesn't necessarily meet. I've more than once looked at interpreters for embedding and found most of the alternatives sorely lacking. Very few interpreters are well suited for embedding on constrained platforms at all (Lua, admittedly is probably one of the more solid exceptions). And once you start having to write lots of support code in C to port or sandbox your interpreter of choice, the reason for considering an interpreter quickly becomes less compelling.

stefantalpalaru 11y ago

You can also do that with uWSGI: http://uwsgi.readthedocs.org/en/latest/Symcall.html http://uwsgi.readthedocs.org/en/latest/Symcall.html

zippie 11y ago

Process per connection does not scale no matter how light weight, even with COW. kore's connection handling model is the reason apache2 mpm_prefork fell out of favor many iterations ago. The only valid argument to avoid a single event based I/O is some sort of hard blocking I/O such as disk or non-queuing chardev. However I'm still not biting, this is solved...and as usual the answer is somewhere in between. For example, in RIBS2 there are two models for connection handling, event loops for connections and "ribbons" for the non-queuing bits [1]. RIBS2 is also written in C for C. [1] https://github.com/Adaptv/ribs2/blob/master/README https://github.com/Adaptv/ribs2/blob/master/README Edit - mention RIBS2 is also written in C

jvink 11y ago
Kore does not fork per connection. It uses per cpu worker processes which multiplex I/O over either epoll or kqueue.
- zippie 11y ago
  A clone is a clone. The main inplementation difference between a thread and process in Linux is COW, for all intensive purposes a worker process and a fork are the same in this use case. Neither have led to scalable web servers.
  - jvink 11y ago
    Except you are basing yourself on the fact it creates a single worker process per connection. It does not. Workers are spawned when the server is started. Each of them deals with tens of thousands of connections on its own via the listening socket they share. This is a common technique and scales incredible well.
otterley 11y ago
> apache2 mpm_prefork fell out of favor many iterations ago. By whom, exactly? There are still plenty of reasons to use a forking web server (see my other comment in this discussion). Saying it "does not scale" is misleading; even with a event-driven model there are only so many CPU resources that can be used to serve responses to clients. Event-driven webservers are fantastic compared to forking ones, for keeping open many thousands of relatively idle connections (if that is your definition of "scale"). But many web services simply don't do that. Preforking webservers, like event-driven ones, still have a rightful place in this world. As with all things technology, you have to pick the right tool for the job.

faragon 11y ago

Writing high level C applications can be easy, if you use a library that frees you from using dynamic memory on typical data structures (e.g. strings, vectors, sorted binary trees, maps). I'm developing a C library for high-level C code, with hard real-time in mind, is already functional for static linking: https://github.com/faragon/libsrt https://github.com/faragon/libsrt

maciej 11y ago

If you think Kore is interesting, then also check out Tntnet <http://www.tntnet.org> http://www.tntnet.org>. I've checked it out a few years ago and it felt good - stable, complete, easy to use etc.

bitL 11y ago

Good job & good luck! ;-)

sownkunz 11y ago

this is AWESOME!

talnet 11y ago

Anyway I love this.

z3t4 11y ago

Kore mapped to JavaScript (NodeJS): var kore = require("kore"); kore.on("request", http_request); function http_request(req, resp) { var statusCode = 200; resp.write("Hello world", statusCode); }

bubbba 11y ago

Probably it would be better to have also a lib or dll, not just a program that render c/c++ servlets. It seems that all (that the kore executes from the feeded code) is running in servlet threads or at least started from servlet thread. Or it would be cool to add some "application" framework, not only "C-servlet" framework

unwind 11y ago

Great! This looks really nice, and interesting. Some fantastically quick points from a very cursory glance at the code. Feel free to ignore this. - The code uses the convention to put the argument of return inside parentheses, making it look like a function call. This is very strange, to me. - It treats sizeof as a function too (i.e. always parentheses the argument). - It is not C99, which always seems so fantastically defensive these days. - It's not (in my opinion) sufficiently const-happy. - I saw at least one instance (in cli.c) of a long string not being written as a auto-concatenated literal but instead leading to multiple fprintf() calls. Very obviously not in a performance-critical place, so perhaps it's not indicative of anything. It just made me take notice.

jvink 11y ago
Author here. I see you picked out the few things that I consistently hear on the coding style I adopted which is based on my time hacking on openbsd. I have no real points to argue against those as it is based on preference in my opinion. I am curious why you arrived on it not being sufficiently constified however. I'll gladly make sensible changes. As for the multiple fprintf() calls ... to me it just reads better and the place it occurs in is as you stated pretty obvious non performance critical.
- unwind 11y ago
  Right. I could have guessed these were based on some coding style guide from somewhere. I still don't see the point, or why any sane guide would prefer to treat return as a function. It just never seems helpful to me, and always wasteful/more complicated. I realize it's just two tokens, so it's probably not "important" in any real sense of the word, but it irks me. I like to point it out since it can help others cargo-culting this. It's not sufficiently const if there are places where a variable could be const but still isn't. :) To be super-specific, the variable 'r' here: https://github.com/jorisvink/kore/blob/master/src/cli.c#L542 https://github.com/jorisvink/kore/blob/master/src/cli.c#L542 is one such case. It should be declared inside the loop, i.e. as "const ssize_t r = write(...);" since once assigned the return value from write(), it's read-only. Of course, many ancient-smelling style guides seem to outlaw declaring variables as close to their point of usage, too. Note that declaring variables inside scopes other than the "root" one in a function isn't even C99, but many people seem to think you can't do that.