24 ms·
This matters in so few applications, and they already have alternatives (mostly message queue systems). Bulk data transfer is rarely operationally time-sensitiv
by contingencies 13y ago
This matters in so few applications, and they already have alternatives (mostly message queue systems). Bulk data transfer is rarely operationally time-sensitive. To justify bothering investing time to investigate this, maybe we could see some benchmarks versus popular MQ solutions and traditional sockets for different messaging scenarios. Also factor in the various TCP optimizations, etc. (Actually, something like this should exist anyway... it would be a great resource.)
- RyanZAG 13y agoAs far as I can tell, the main usage for this MegaPipe system would be creating a better API for webservers like apache and nginx to plug into to bypass some of the overhead inside BSD sockets. Most popular MQ solutions also use TCP/IP and hence BSD sockets, so they're not alternatives - they would actually replace their BSD socket API calls with MegaPipe API calls and receive performance benefits (in very large message/recipient situations). I think either you have misunderstood the article, or I have.
- lcampbell 13y agoFor static file serving, Nginx can bypass BSD socket overhead with the sendfile (or sendfile-like) syscall, though it is disabled by default. On FreeBSD, the sendfile syscall accepts not only the fd to pipe, but can additionally be passed a header/footer such that you don't need to call write(2) at all, though this is not true on either Linux or Windows (AFAIK). The whitepaper's macrobenchmarks make no mention of sendfile; I presume their tests with Nginx run without sendfile enabled (since it would defeat the point of the benchmark by nature of it skipping all of the write(2) calls). The paper does not appear to detail the configuration used in their benchmarks. I believe that the performance benefit of Nginx+MegaPipe, when compared to an Nginx server configured to use sendfile, is much, much less than +75% throughput. That said, sendfile has very narrow uses (it's effectively only for sending static files) -- I wish the paper benchmarked Nginx against Nginx+MegaPipe as a reverse proxy rather than a static file server.
- justincormack 13y agoLinux has splice now which can support more generic uses (and tee). You can feed one network stream into another and add headers etc. They are not yet widely used but some decent performance figures I believe. But on the other hand user space networking is also performing.
- dinkumthinkum 13y agoSo, I take it you don't know much about research apparently ...