6 ms·

That was my first reaction as well. But to be fair, the kdbus people always said the real speedup will be seen for large packets. However, the real issue is ho

by deng 12y ago

That was my first reaction as well. But to be fair, the kdbus people always said the real speedup will be seen for large packets. However, the real issue is how slow overall DBus seems to be according to those numbers. As Linus says earlier in the thread: "No way should it take 4+ seconds to send a 1000b message to back and forth 20k times." This is indeed rather shocking.

lmilcin 12y ago
I would expect any half decent HTTP server to be able to handle more messages per second than that. If that is going to be the future fabric underlying all application ecosystem on Linux I would be as wary of it as Linus is. I will put it another way: if it can't be more performant than a simple HTTP client/server it doesn't deserve to be included into mainline kernel especially when the costs are so high.
- azernik 12y ago
  Well, he's not claiming that kdbus is that slow. He's claiming that kdbus is faster not because it's in the kernel, but because it's not shitty code like userspace dbus (completely different implementation).
fulafel 12y ago
So what's the use case for sending large packets of data over dbus where the performance gains warrant integrating the dbus server to the kernel? Are they planning to migrate the X server to dbus?
- deng 12y ago
  IIRC the use case is to stream media over DBus, which currently no one is doing because it is too slow.
  - semi-extrinsic 12y ago
    So: if DBus was suddenly fast enough to stream media over, what applications would be using this feature, for what sets of streaming endpoints?
    - deng 12y ago
      I don't know. I'm not advocating kdbus for multimedia, I'm just repeating what I've heard. I think one of the driving forces behind kdbus is Samsung who are interested in kdbus for Tizen.
    - scrollaway 12y ago
      KDBus was designed with sandboxed applications in mind. So use-case wise think android-style "use the camera app to take a picture/video", where the camera app transfers the media to another app over kdbus, thus without leaking information about the system.
      - zurn 12y ago
        You don't fling uncompressed video between apps like that, it goes straight to the hardware-accelerated h.264 (or whatever) encoder and then a low bandwidth stream comes out. Local AF_UNIX sockets (aka userspace dbus) provide ample bandwidth for that use case.
        detaro 12y ago
        I think the idea is "why use dbus to coordinate creation, rights assignment, ... of local sockets when you can push the data over dbus"?
        zurn 12y ago
        Yes, that's what I was talking about. Current dbus is is using local AF_UNIX sockets for transport -> the current transport is plenty fast enough to push encoded video over.
      - jschwartzi 12y ago
  - audidude 12y ago
    I think wayland was one possibility.
  - kryptiskt 12y ago
    I don't get the use case, why can't the programs use DBus to negotiate the streaming over some low-overhead transport? Why are people so insistent on ramming a square peg into a round hole?
    - mburns 12y ago
      because they want the square hole for a reason, the square hole being a unified in-kernel IPC system. Greg KH has given a few talks/blogs about the design goals of kdbus. https://lwn.net/Articles/551969/ https://lwn.net/Articles/551969/
      - mcguire 12y ago
        "because they want the square hole for a reason, the square hole being a unified in-kernel IPC system." Another one?
        mburns 12y ago
        No, a first one. The others aren't unified, they are niche solutions. Those specific shortcomings are covered in those KDBus talks. If the existing solutions were sufficient, why would the 'new' way ever get merged into mainline?
    - justincormack 12y ago
      Well actually thats what kdbus would do anyway, all the large transfers will happen over memfd buffers...
  - hobarrera 12y ago
    I'd much prefer a an IPC that lets me pass a socket and use that socket on both ends, instead of this. I really still don't see a good-enough scenario for this.
  - comex 12y ago
    I'm not terribly convinced by this point, since it's relatively easily solved by the "transparent direct connection" idea that's been suggested in LKML. That is, rather than making the application set up its own Unix socket connection for the media, the DBus library would have a simple call to switch to a direct socket connection as a transport for a given DBus connection. This wouldn't work for multicast, but who needs to multicast raw media data to whatever local process feels like receiving it?
- ploxiln 12y ago
  Large packets, I'm not sure. For a long time, you've been able to attach a file descriptor (e.g. one end of a pipe) to a message, and then send bulk data over the file descriptor. My understanding of this whole thing was that most of the car companies thought they should try using linux for their console / entertainment / etc systems. Electrical and mechanical engineers seem to all only use windows and not know anything about linux, so they asked themselves "what's the bus on linux?" and found D-bus, the desktop bus used by desktop environments and utilities, for a handful of messages a minute. They used it to send frequent sensor data to all sorts of services that might be interested, like digital gauges, sound distribution, etc. They didn't realize that there are lots of good low-level IPC mechanisms available on linux, and they should have probably used 0mq, or a scheme based on unix sockets with filesystem permissions based access control, etc. Anyway, they realized that D-bus had way too much overhead, and instead of porting to some other IPC, got to work accelerating D-bus with additional kernel functionality, such as the AF_BUS which was rejected by the linux networking maintainers. (Though that was probably a better idea than kdbus.) How exactly GregKH got into this project, I'm not entirely sure. He is relatively friendly with the systemd developers. He's paid by the Linux Foundation, and many of those car companies are members of the Linux Consortium, or something like that, I think. I dunno. Apparently a lot of people used D-bus accidentally, and now the thing to do is make the kernel make it faster.
  - digi_owl 12y ago
    The same car companies that put the entertainment system and the brake servos on the same bus...
- bitwize 12y ago
  X is going to be migrated into the recycling bin. Lennart wants to migrate PA to dbus, and the possibility exists for doing the same for some future version of Wayland.
  - digi_owl 12y ago
    Ugh, and here i thought Wayland was doing good...
  - mcguire 12y ago
    Ah, Wayland. Yet another example of the successor to a system being less capable than the system it replaces. "Wayland does not currently provide network transparency, but it may in the future. It was attempted as a Google Summer of Code project in 2011, but was not successful. Adam Jackson has envisioned providing remote access to a Wayland application by either 'pixel-scraping' (like VNC) or getting it to send a "rendering command stream" across the network (as in RDP, SPICE or X11). As of early 2013, Høgsberg is experimenting with network transparency using a proxy Wayland server which sends compressed images to the real compositor." [1] https://en.wikipedia.org/wiki/Wayland_%28display_server_protocol%29 https://en.wikipedia.org/wiki/Wayland_%28display_server_prot...
    - bitwize 12y ago
      It turns out that in this day and age, desktop users much prefer pixel-perfect tear-free rendering to network transparency. Wayland is X11 without the legacy cruft -- and it turns out that the only thing left in X11 that's not legacy cruft -- like seriously, bleeding-edge tech for 1986 -- is a centralized way to hand out direct-rendering frame buffers to user processes and composite them together for the final image. Which is exactly the extent to which modern apps use X11. Remote access can be achieved by running a special Wayland server on the remote side that connects to a Wayland client on the local side and exchanges framebuffer and event data with that client. In this setup, apps can talk Wayland and not even be aware that their display is being networked; they are also completely decoupled from the remoting protocol which can be anything. RDP and streaming h.264 would be good choices, certainly better than legacy X protocol. Wayland is objectively a huge win.
anon4 12y ago
In that case, they should implement just the functionality they need in kernel, not a full DBUS implementation.

by deng 12y ago

lmilcin 12y ago

I would expect any half decent HTTP server to be able to handle more messages per second than that. If that is going to be the future fabric underlying all application ecosystem on Linux I would be as wary of it as Linus is. I will put it another way: if it can't be more performant than a simple HTTP client/server it doesn't deserve to be included into mainline kernel especially when the costs are so high.

azernik 12y ago
Well, he's not claiming that kdbus is that slow. He's claiming that kdbus is faster not because it's in the kernel, but because it's not shitty code like userspace dbus (completely different implementation).

fulafel 12y ago

So what's the use case for sending large packets of data over dbus where the performance gains warrant integrating the dbus server to the kernel? Are they planning to migrate the X server to dbus?

deng 12y ago
IIRC the use case is to stream media over DBus, which currently no one is doing because it is too slow.
- semi-extrinsic 12y ago
  So: if DBus was suddenly fast enough to stream media over, what applications would be using this feature, for what sets of streaming endpoints?
  - deng 12y ago
    I don't know. I'm not advocating kdbus for multimedia, I'm just repeating what I've heard. I think one of the driving forces behind kdbus is Samsung who are interested in kdbus for Tizen.
  - scrollaway 12y ago
    KDBus was designed with sandboxed applications in mind. So use-case wise think android-style "use the camera app to take a picture/video", where the camera app transfers the media to another app over kdbus, thus without leaking information about the system.
    - zurn 12y ago
      You don't fling uncompressed video between apps like that, it goes straight to the hardware-accelerated h.264 (or whatever) encoder and then a low bandwidth stream comes out. Local AF_UNIX sockets (aka userspace dbus) provide ample bandwidth for that use case.
      - detaro 12y ago
        I think the idea is "why use dbus to coordinate creation, rights assignment, ... of local sockets when you can push the data over dbus"?
        zurn 12y ago
        Yes, that's what I was talking about. Current dbus is is using local AF_UNIX sockets for transport -> the current transport is plenty fast enough to push encoded video over.
    - jschwartzi 12y ago
- audidude 12y ago
  I think wayland was one possibility.
- kryptiskt 12y ago
  I don't get the use case, why can't the programs use DBus to negotiate the streaming over some low-overhead transport? Why are people so insistent on ramming a square peg into a round hole?
  - mburns 12y ago
    because they want the square hole for a reason, the square hole being a unified in-kernel IPC system. Greg KH has given a few talks/blogs about the design goals of kdbus. https://lwn.net/Articles/551969/ https://lwn.net/Articles/551969/
    - mcguire 12y ago
      "because they want the square hole for a reason, the square hole being a unified in-kernel IPC system." Another one?
      - mburns 12y ago
        No, a first one. The others aren't unified, they are niche solutions. Those specific shortcomings are covered in those KDBus talks. If the existing solutions were sufficient, why would the 'new' way ever get merged into mainline?
  - justincormack 12y ago
    Well actually thats what kdbus would do anyway, all the large transfers will happen over memfd buffers...
- hobarrera 12y ago
  I'd much prefer a an IPC that lets me pass a socket and use that socket on both ends, instead of this. I really still don't see a good-enough scenario for this.
- comex 12y ago
  I'm not terribly convinced by this point, since it's relatively easily solved by the "transparent direct connection" idea that's been suggested in LKML. That is, rather than making the application set up its own Unix socket connection for the media, the DBus library would have a simple call to switch to a direct socket connection as a transport for a given DBus connection. This wouldn't work for multicast, but who needs to multicast raw media data to whatever local process feels like receiving it?
ploxiln 12y ago
Large packets, I'm not sure. For a long time, you've been able to attach a file descriptor (e.g. one end of a pipe) to a message, and then send bulk data over the file descriptor. My understanding of this whole thing was that most of the car companies thought they should try using linux for their console / entertainment / etc systems. Electrical and mechanical engineers seem to all only use windows and not know anything about linux, so they asked themselves "what's the bus on linux?" and found D-bus, the desktop bus used by desktop environments and utilities, for a handful of messages a minute. They used it to send frequent sensor data to all sorts of services that might be interested, like digital gauges, sound distribution, etc. They didn't realize that there are lots of good low-level IPC mechanisms available on linux, and they should have probably used 0mq, or a scheme based on unix sockets with filesystem permissions based access control, etc. Anyway, they realized that D-bus had way too much overhead, and instead of porting to some other IPC, got to work accelerating D-bus with additional kernel functionality, such as the AF_BUS which was rejected by the linux networking maintainers. (Though that was probably a better idea than kdbus.) How exactly GregKH got into this project, I'm not entirely sure. He is relatively friendly with the systemd developers. He's paid by the Linux Foundation, and many of those car companies are members of the Linux Consortium, or something like that, I think. I dunno. Apparently a lot of people used D-bus accidentally, and now the thing to do is make the kernel make it faster.
- digi_owl 12y ago
  The same car companies that put the entertainment system and the brake servos on the same bus...
bitwize 12y ago
X is going to be migrated into the recycling bin. Lennart wants to migrate PA to dbus, and the possibility exists for doing the same for some future version of Wayland.
- digi_owl 12y ago
  Ugh, and here i thought Wayland was doing good...
- mcguire 12y ago
  Ah, Wayland. Yet another example of the successor to a system being less capable than the system it replaces. "Wayland does not currently provide network transparency, but it may in the future. It was attempted as a Google Summer of Code project in 2011, but was not successful. Adam Jackson has envisioned providing remote access to a Wayland application by either 'pixel-scraping' (like VNC) or getting it to send a "rendering command stream" across the network (as in RDP, SPICE or X11). As of early 2013, Høgsberg is experimenting with network transparency using a proxy Wayland server which sends compressed images to the real compositor." [1] https://en.wikipedia.org/wiki/Wayland_%28display_server_protocol%29 https://en.wikipedia.org/wiki/Wayland_%28display_server_prot...
  - bitwize 12y ago
    It turns out that in this day and age, desktop users much prefer pixel-perfect tear-free rendering to network transparency. Wayland is X11 without the legacy cruft -- and it turns out that the only thing left in X11 that's not legacy cruft -- like seriously, bleeding-edge tech for 1986 -- is a centralized way to hand out direct-rendering frame buffers to user processes and composite them together for the final image. Which is exactly the extent to which modern apps use X11. Remote access can be achieved by running a special Wayland server on the remote side that connects to a Wayland client on the local side and exchanges framebuffer and event data with that client. In this setup, apps can talk Wayland and not even be aware that their display is being networked; they are also completely decoupled from the remoting protocol which can be anything. RDP and streaming h.264 would be good choices, certainly better than legacy X protocol. Wayland is objectively a huge win.

anon4 12y ago

In that case, they should implement just the functionality they need in kernel, not a full DBUS implementation.