5 ms·
Well, the idea that simply 'queueing the rest in userspace' is OK illustrates a pretty disconcerting lack of understanding in writing networking code. This is
by sshumaker 16y ago
Well, the idea that simply 'queueing the rest in userspace' is OK illustrates a pretty disconcerting lack of understanding in writing networking code. This is not an acceptable solution in the real world.
I want to correct that assumption right now - it's one of those places that higher level networking libraries often fall down - because simply 'fire and forget' is a leaky abstraction.
Let's say you're streaming a large file over TCP to a client. Generally, disk reads are far faster than you can send data over the internet. If you naively keep sending data, you'll quickly fill up the OS buffers - and begin to consume memory as it's queued on the client side. In an extreme case, if you're sending a multi-GB file, you'll probably end up with most of the file sitting in RAM (assuming you don't run of ram and start thrashing). And for what? You really only need to load additional data in from the file as fast as you can send it in the socket.
In a real-life case, streaming multiple files to multiple clients, you'll quickly run out of RAM with even moderate size files.
Some high level libraries provide an event to let you know that the OS buffers have been emptied out. For example, Node.js has a drain event. But this has to be handled explicitly, and isn't really highlighted in the docs. But if you write your code naively, you will definitely run into problems.
While it's great that many of these high level networking libraries have tried to abstract away many of the complexities of network programming, they're also a trap for the unwary. Just remember that they aren't a silver bullet that let you handle sending data over the internet as a simple function call.
- ekidd 16y agoHave you checked out the ZeroMQ documentation on flow control and data rate limiting? I can't speak for anyone else's use case, but there's a nice selection of blocking primitives and data rate limits that have proven sufficient for the smaller programs I've written. It's hard for me to tell if your criticisms are based on real-world experience with the library, a review of the available documentation, or just my description of a single feature intended to reduce the cost traversing the userspace/kernel barrier. I'm definitely interested in real-world experience with ZeroMQ and any problems you've encountered. We may begin some larger-scale testing soon, and would love to know about any problems.
- ehsanul 16y agoInstead of rate-limiting, there are two other options that solve the GP's concerns: the high water mark which controls the maximum queue length and the swap size, which saves messages to disk when the queue is full. http://api.zeromq.org/zmq_setsockopt.html http://api.zeromq.org/zmq_setsockopt.html
- nkurz 16y agoIf you are streaming a large file over TCP, ZeroMQ, with its emphasis on small messages sent and received apparently atomically, would be a stunningly poor choice. It's also wholly unsuitable as a floor wax or a dessert topping. :) Does your criticism also apply to its intended use case? At a glance, it doesn't strike me as an unreasonable design choice, but I haven't thought about it deeply.
- sandGorgon 16y agowhat would be the best way of sending large files server-to-server ? this is a question that has been puzzling me. Is there like a library or a protocol that is tuned towards transferring multi-gb files ?
- nkurz 16y agoAre you intending to have many of these transfers happening in parallel, or are you sending a single file and hoping to saturate your link? Do you have full control of both ends, or only the server? Are you streaming or is the whole file always transferred? Do you need encryption? For simultaneous large files with no special needs, my instinct would be to just choose any lightweight HTTP server that supports sendfile(). Basically, pass it off to the kernel, and let it handle it. But maybe there are more subtle keys to good large file performance: http://www.psc.edu/networking/projects/tcptune/ http://www.psc.edu/networking/projects/tcptune/ For cases where you are controlling both ends, you might find useful suggestions here: http://serverfault.com/questions/111813/what-is-the-best-way-to-transfer-a-single-large-file-over-a-high-speed-high-late http://serverfault.com/questions/111813/what-is-the-best-way... In particular, Tsunami and uftp look interesting.
- tedunangst 16y agoWhat OS are you using that has infinite sized TCP send buffers?
- sustrik 16y ago0MQ is built on good old principles the Internet stack (TCP, IP) is based upon. Thus is has tx/rx buffers. It doesn't even try to do fire-and-forget, because -- as you correctly say -- it's an leaky abstraction. Instead it aims for end-to-end reliability. And yes, moving it into kernel would make it somehow more reliable. Watch for my talk at Linux Kongress (eeptember 23rd) which deals with moving the functionality into kernel, standardisation and integration with Internet stack.
- phintjens 16y ago0MQ has what is called a high-water mark on (outgoing) queues. You can then decide whether to block if the queue gets full, or poll the socket to know when it's ready for output. This is also how TCP works. 0MQ abstracts the complexities of the network that you don't want to see, but it exposes those parts you still need to. It would make a very nice transport for file transfer, and I'll probably add such an example to the 0MQ Guide. Try it, you'll be surprised, I guarantee it.