3 ms·

Hi, I'm the guy who came up with the splice idea. It's based on what I learned doing this: www.connectathon.org/talks96/bds.pdf which was for the EIS (Earth

Hi, I'm the guy who came up with the splice idea. It's based on what I learned doing this: www.connectathon.org/talks96/bds.pdf which was for the EIS (Earth Imaging System) project, a government effort to image the earth about 15 years ago. That project eventually had 200Mhz MIPS SMP boxes moving data through NFS at close to 1Gbyte/sec 24x7. So far as I know, nobody else has ever come close to that even with 10x faster CPUs. Most of the people in this thread pretty clearly don't understand the issues involved, Rob included (sorry, Rob, go talk to Greg). Moving lots and lots of data very quickly precludes looking at each byte by the CPU. The only thing that should look at each byte is a DMA engine. Sendfile(2) is a hack, that's true. It is a subset of what I imagined splice(2) could be (actually splice(3), the syscalls are pull(2) and push(2)). But it's a necessary hack. Jens' splice() implementation was a start but wasn't really what I imagined for splice(), to really go there you need to rethink how the OS thinks about moving data. Unless the buyin is pervasive splice() is sort of a wart.

deleted 16y ago
[deleted]
fragmede 16y ago
> So far as I know, nobody else has ever come close to that even with 10x faster CPUs. With 10-gigabit becoming commonplace in high-end server rooms, dual 10-gigabit cards hitting the market, and 100-gigabit on its way, I've got to ask, what you mean by no one has come close? (Certainly, 1 GByte/sec is achievable; splice is a large part of that.)
- luckydude 16y ago
  The big print giveth, the fine print taketh away :) Just because you have a 10 gigabit pipe (which is 1.2Gbyte/sec) that doesn't mean you can fill it. And filling it with a benchmark is different than filling it with NFS traffic. So far as I know, nobody has come close to what SGI could do years ago, i.e, stream 100's of MB/sec of data off the disk and out the network and keep doing it. I've tried with Linux to build a disk array that would do gigabit, not 10-gigabit, and couldn't come anywhere close. I think about 60MB/sec was where things started to fail. I'd love to hear about a pair of boxes that could fill multiple gigabit pipes with the following benchmark: $ tar cf - big_data | rsh server "tar xf -" You can set any block size you want in tar, I don't care, the files can be reasonably large, etc. If you manage to fill one pipe, then try it with 10-gigabit and let's see what that does. I'd love to be wrong but so far as I can tell, we're nowhere near what SGI could do.
  - alexgartrell 16y ago
    Serious question, why tar it? I bet you could get pretty sweet throughput with an array of SSDs and 10gig pipes.
    - luckydude 16y ago
      In my case, it's because it's a pretty good emulator of what we do. We're the oh-so-loved BitKeeper guys and we're working on tech for very large repositories. A bk clone is essentially the described benchmark. Or should be, if we're slower than that we're doing something wrong.
  - fragmede 16y ago
    :) Benchmarks, especially those from card manufactures are suspect. Just because you've got a network ping-pong test running at essentially line speed, doesn't mean I'll see any of that. I fully agree. Filling any pipe with your 'rsh' benchmark is harder - on all the Linuxes I have on hand, rsh is aliased to ssh, and I've never seen a system deal with the overhead of ssh adequately. (There's high perf ssh if you really want; I've never tried it. http://www.psc.edu/networking/projects/hpn-ssh/ http://www.psc.edu/networking/projects/hpn-ssh/) (Would you accept numbers over netcat?) If I'm not getting 90MBytes/sec over Gigabit, then something, somewhere, is slowing it down. We sell servers that sustain 300+ MBytes/s per client, via Samba for two clients. Those clients are very fussy about latency (video editing/DI). That's what we sell, so it's empirically that fast; ie not a benchmark. The server itself will actually do 900+ MBytes a second (to three clients, the same as page 14 on your linked pdf), but my small print is the market doesn't want that configuration. These numbers are what I know, which doesn't include NFS. NFS-RDMA and pNFS are supposedly good for performance, but I don't have any first hand knowledge. None of what I just stated contradicts you. But if I'm at 900 MBytes/sec with my level of hardware, I find it hard to believe someone with more budget isn't capable of a full Gigabyte/sec over Infiniband, arguably HIPPI's successor. I, too, lament the demise of SGI.
    - luckydude 16y ago
      Who is "we"? I might be another customer :) I meant rsh, not ssh, wasn't trying to stick extra processing in there. Debian, bless 'em, still support rsh, I think it's rsh-clients and rsh-server. For your 300Mbytes/sec, what's the network? And is the data in memory or on disk? If on disk, what sort of drives and what size files? Thanks!
aliguori 16y ago
I'm assuming that the real objection to an interface like sendfile() is that you shouldn't need to copy_from_user() just to construct an SKB if the user address being referred to is pointing at a file backed VMA. If we could do a zero-copy send() from userspace, there would be no need for sendfile() in the first place. IOW, { data = mmap(fd, ...); ...; send(fd, data + offset, length); } could be just as fast as sendfile() if we were smarter. We're starting to build infrastructure to do zero-copy networking in userspace specifically for KVM. So far, it's not at the socket level but instead at the macvtap level but that's arguably another problem with the networking stack--macvtap/tap should just be an address family :-)
- caf 16y ago
  I'm not so sure. The problem is that when send() returns, the application is free to modify the buffer and expect that the modifications won't be visible to the other side of the link - the data has already been "sent" as far as the app is concerned. So this implies that if you wished to send by DMAing directly from the page cache to the NIC, send() would have to block - not just until the data (and all previous buffered written data on the connection) has been passed to the NIC, but until the TCP ACK from the other side has been recieved (or RST). (Non-blocking sockets could obviously never use this method - they would have to perform the copy). sendfile() obviously has this limitation already, but if you were to do it for send() you'd at least have to hide it behind a setsockopt(). Otherwise the weird behaviour is likely to upset existing applications (eg. select() says that write() will not block - but it does anyway, because the buffer you sent from was mmap()ed?)
  - aliguori 16y ago
    No, TCP ACK has nothing to do with this. All that matters is when the NIC has completed DMA'ing the packet. At that point, userspace can be unblocked. Non-blocking sockets only make sense when there's an intermediate buffer.
    - mad 16y ago
      What happens if you need to retransmit a packet?

deleted 16y ago

[deleted]

fragmede 16y ago

> So far as I know, nobody else has ever come close to that even with 10x faster CPUs. With 10-gigabit becoming commonplace in high-end server rooms, dual 10-gigabit cards hitting the market, and 100-gigabit on its way, I've got to ask, what you mean by no one has come close? (Certainly, 1 GByte/sec is achievable; splice is a large part of that.)

luckydude 16y ago
The big print giveth, the fine print taketh away :) Just because you have a 10 gigabit pipe (which is 1.2Gbyte/sec) that doesn't mean you can fill it. And filling it with a benchmark is different than filling it with NFS traffic. So far as I know, nobody has come close to what SGI could do years ago, i.e, stream 100's of MB/sec of data off the disk and out the network and keep doing it. I've tried with Linux to build a disk array that would do gigabit, not 10-gigabit, and couldn't come anywhere close. I think about 60MB/sec was where things started to fail. I'd love to hear about a pair of boxes that could fill multiple gigabit pipes with the following benchmark: $ tar cf - big_data | rsh server "tar xf -" You can set any block size you want in tar, I don't care, the files can be reasonably large, etc. If you manage to fill one pipe, then try it with 10-gigabit and let's see what that does. I'd love to be wrong but so far as I can tell, we're nowhere near what SGI could do.
- alexgartrell 16y ago
  Serious question, why tar it? I bet you could get pretty sweet throughput with an array of SSDs and 10gig pipes.
  - luckydude 16y ago
    In my case, it's because it's a pretty good emulator of what we do. We're the oh-so-loved BitKeeper guys and we're working on tech for very large repositories. A bk clone is essentially the described benchmark. Or should be, if we're slower than that we're doing something wrong.
- fragmede 16y ago
  :) Benchmarks, especially those from card manufactures are suspect. Just because you've got a network ping-pong test running at essentially line speed, doesn't mean I'll see any of that. I fully agree. Filling any pipe with your 'rsh' benchmark is harder - on all the Linuxes I have on hand, rsh is aliased to ssh, and I've never seen a system deal with the overhead of ssh adequately. (There's high perf ssh if you really want; I've never tried it. http://www.psc.edu/networking/projects/hpn-ssh/ http://www.psc.edu/networking/projects/hpn-ssh/) (Would you accept numbers over netcat?) If I'm not getting 90MBytes/sec over Gigabit, then something, somewhere, is slowing it down. We sell servers that sustain 300+ MBytes/s per client, via Samba for two clients. Those clients are very fussy about latency (video editing/DI). That's what we sell, so it's empirically that fast; ie not a benchmark. The server itself will actually do 900+ MBytes a second (to three clients, the same as page 14 on your linked pdf), but my small print is the market doesn't want that configuration. These numbers are what I know, which doesn't include NFS. NFS-RDMA and pNFS are supposedly good for performance, but I don't have any first hand knowledge. None of what I just stated contradicts you. But if I'm at 900 MBytes/sec with my level of hardware, I find it hard to believe someone with more budget isn't capable of a full Gigabyte/sec over Infiniband, arguably HIPPI's successor. I, too, lament the demise of SGI.
  - luckydude 16y ago
    Who is "we"? I might be another customer :) I meant rsh, not ssh, wasn't trying to stick extra processing in there. Debian, bless 'em, still support rsh, I think it's rsh-clients and rsh-server. For your 300Mbytes/sec, what's the network? And is the data in memory or on disk? If on disk, what sort of drives and what size files? Thanks!

aliguori 16y ago

I'm assuming that the real objection to an interface like sendfile() is that you shouldn't need to copy_from_user() just to construct an SKB if the user address being referred to is pointing at a file backed VMA. If we could do a zero-copy send() from userspace, there would be no need for sendfile() in the first place. IOW, { data = mmap(fd, ...); ...; send(fd, data + offset, length); } could be just as fast as sendfile() if we were smarter. We're starting to build infrastructure to do zero-copy networking in userspace specifically for KVM. So far, it's not at the socket level but instead at the macvtap level but that's arguably another problem with the networking stack--macvtap/tap should just be an address family :-)

caf 16y ago
I'm not so sure. The problem is that when send() returns, the application is free to modify the buffer and expect that the modifications won't be visible to the other side of the link - the data has already been "sent" as far as the app is concerned. So this implies that if you wished to send by DMAing directly from the page cache to the NIC, send() would have to block - not just until the data (and all previous buffered written data on the connection) has been passed to the NIC, but until the TCP ACK from the other side has been recieved (or RST). (Non-blocking sockets could obviously never use this method - they would have to perform the copy). sendfile() obviously has this limitation already, but if you were to do it for send() you'd at least have to hide it behind a setsockopt(). Otherwise the weird behaviour is likely to upset existing applications (eg. select() says that write() will not block - but it does anyway, because the buffer you sent from was mmap()ed?)
- aliguori 16y ago
  No, TCP ACK has nothing to do with this. All that matters is when the NIC has completed DMA'ing the packet. At that point, userspace can be unblocked. Non-blocking sockets only make sense when there's an intermediate buffer.
  - mad 16y ago
    What happens if you need to retransmit a packet?