3 ms·

I've always wondered, what is the overhead of an OS (say Linux), compared to a stand-alone binary? Maybe some here could answer the question: What kind of diffe

by mruts 8y ago

I've always wondered, what is the overhead of an OS (say Linux), compared to a stand-alone binary? Maybe some here could answer the question: What kind of differences would you see for nginx on HermiTux vs Linux? Less memory usage, more requests? Also, what about a standalone web server operating system that just implemented a TCP stack? How would the performance ande memory usage of those three things compare?

sanxiyn 8y ago
OSv benchmarked memcached: "An unmodified memcached running on OSv was able to handle about 20% more requests per second than the same memcached version on Linux. A modified memcached, designed to use OSv-specific network APIs, had nearly four times the throughput." http://osv.io/benchmarks http://osv.io/benchmarks
convolvatron 8y ago
not a huge difference i think, at a high level the work involved is exactly the same. but there are a few differences. raw system call overhead: this isn't alot, but in an i/o heavy application it shows up as a reasonable percentage of work. general purpose network machinery: things like kernel thread context switches for interrupts and networking likely wouldn't be there in the more bare bones systems. also none of the firewall/packet filter/netgraph checks will be there. its also possible that a single application network service might revert to a pure polling model and avoid interrupt context switches entirely. unix interface: your standalone web server interface might be very well written to not use the copying tcp read(), and use direct callbacks or thread handoffs instead of select/poll/kqueue other system gunk: none of the usual daemons and their kernel state needs to be there, and they wont be waking up and stealing cycles from your service. you should expect a much tighter service latency another factor at play here is that alot of the general OS paths have had substantial work on them (interrupt coalescing, thread scheduling, memory management etc). that kind of effort may not be replicated in the simple standalone version. so by throwing that away you're losing some of the good along with the bad (where the bad here is otherwise useful generality, not something really objectively bad) as mentioned in the HT writeup, startup time can be vanishingly small if thats of importance to you. overall there should be some modest performance gains, but really not enough to drive adoption solely on that basis. for me the real win is radically simplified management. given that everyone is building/using management layers over OS instances in a cloud context - is it really useful to carry around that large amount of additional machinery to manage the per-OS configuration? if I can trivially build and deploy new instances and wire them into my service, do I need chef anymore? do I need to fuss with user permissions in a world where no one is really logging in? why manage a firewall, which assumes that all sorts of random junk might be running and we need to implement a policy for safety. with tools like this I can construct a system which by construction only responds to requests for the service of interest. do I need to maintain a shell environment for monitoring and debugability? maybe - but effective distributed deployments already use distributed monitoring and tracing tools that are arguably more effective - even for general purpose OS deployments.
- mruts 8y ago
  I mean, ideally you could get rid of the performance hit from syscalls as you could run the app and the barebones kernel in the same address space. You could also get rid of pages, virtual memory, and have a very fast malloc implementation. read()/write() could also be super fast. In addition, you can get rid of the scheduler (assuming no threads) and a ton of other complexity. Also the cpu cache lines would be hella consistent. Of course, you’re throwing away a lot. But for certain applications (like HFT), the potential benefits seem very attractive.

by mruts 8y ago

sanxiyn 8y ago

OSv benchmarked memcached: "An unmodified memcached running on OSv was able to handle about 20% more requests per second than the same memcached version on Linux. A modified memcached, designed to use OSv-specific network APIs, had nearly four times the throughput." http://osv.io/benchmarks http://osv.io/benchmarks

convolvatron 8y ago

not a huge difference i think, at a high level the work involved is exactly the same. but there are a few differences. raw system call overhead: this isn't alot, but in an i/o heavy application it shows up as a reasonable percentage of work. general purpose network machinery: things like kernel thread context switches for interrupts and networking likely wouldn't be there in the more bare bones systems. also none of the firewall/packet filter/netgraph checks will be there. its also possible that a single application network service might revert to a pure polling model and avoid interrupt context switches entirely. unix interface: your standalone web server interface might be very well written to not use the copying tcp read(), and use direct callbacks or thread handoffs instead of select/poll/kqueue other system gunk: none of the usual daemons and their kernel state needs to be there, and they wont be waking up and stealing cycles from your service. you should expect a much tighter service latency another factor at play here is that alot of the general OS paths have had substantial work on them (interrupt coalescing, thread scheduling, memory management etc). that kind of effort may not be replicated in the simple standalone version. so by throwing that away you're losing some of the good along with the bad (where the bad here is otherwise useful generality, not something really objectively bad) as mentioned in the HT writeup, startup time can be vanishingly small if thats of importance to you. overall there should be some modest performance gains, but really not enough to drive adoption solely on that basis. for me the real win is radically simplified management. given that everyone is building/using management layers over OS instances in a cloud context - is it really useful to carry around that large amount of additional machinery to manage the per-OS configuration? if I can trivially build and deploy new instances and wire them into my service, do I need chef anymore? do I need to fuss with user permissions in a world where no one is really logging in? why manage a firewall, which assumes that all sorts of random junk might be running and we need to implement a policy for safety. with tools like this I can construct a system which by construction only responds to requests for the service of interest. do I need to maintain a shell environment for monitoring and debugability? maybe - but effective distributed deployments already use distributed monitoring and tracing tools that are arguably more effective - even for general purpose OS deployments.

mruts 8y ago
I mean, ideally you could get rid of the performance hit from syscalls as you could run the app and the barebones kernel in the same address space. You could also get rid of pages, virtual memory, and have a very fast malloc implementation. read()/write() could also be super fast. In addition, you can get rid of the scheduler (assuming no threads) and a ton of other complexity. Also the cpu cache lines would be hella consistent. Of course, you’re throwing away a lot. But for certain applications (like HFT), the potential benefits seem very attractive.