5 ms·

The paper reads like it's suggesting moving the burden of complexity in dealing with varying hardware interfaces from the kernel to userland so that userland ca

The paper reads like it's suggesting moving the burden of complexity in dealing with varying hardware interfaces from the kernel to userland so that userland can take direct advantage of higher performance hardware when it's available. I could see that for some very small niches, but in general I think it would be a terrible development for the industry. Hardware vendors don't like to share. They don't share code, they don't share common interfaces, they don't even share documentation. As it is now, these are all problems which most userland developers don't have to care about -- those problems get dealt with in the kernel, by developers who specialize in building support for uncooperative hardware. The average application developer doesn't want to have to figure out how many queues are supported by a NIC just to open a connection on the network. Further: the average application developer isn't experienced enough to do this correctly. Given the niche where these tradeoffs make sense, I'm not sure why the paper bothers to emphasize security at all.

johnm1019 7y ago
Disclaimer: IANA kernel developer. Is there any benefit to having those specialized developers create frameworks or libraries in Userland which other developers can leverage? This way they remain the interface to uncooperative hardware, but the code is in Userland so the bold folks can try their own approach.
- loeg 7y ago
  That's basically DPDK / SDPK.
- dorlaor 7y ago
  Seastar.io is another one, it's an async engine that utilizes all of the cores in a modern system and is the heart of Scylla's DB. However, it's complex to use it correctly
Q6T46nT668w6i3m 7y ago
I enjoyed the paper. My impression is that you’d shift the burden to the runtimes that, for many applications, currently sit between POSIX and applications (e.g. see the Q&A about POSIX).
- asn1parse 7y ago
  Considering the poor quality of userland software, this seems like a terrible idea.
  - ncmncm 7y ago
    Anyone willing to spend the money can have top quality userland software. Getting a better OS is not always accessible, even with deep pockets.
josephg 7y ago
I don’t think anyone is suggesting that every application explicitly code for each network device. All of that tricky logic could be put into a userland library instead of the kernel. If we wanted, we could even replicate a kernel device driver style API in userland that network device drivers program against, done as a set of userland library files loaded dynamically based on detected hardware. The tricky part wouldn’t be sharing code between applications. We know how to do that. The hard part would be figuring out a clean way to share the hardware between all running applications, given that any app could be terminated at any time, apps might be mutually untrustworthy and apps would have to play nice to share resources. I can imagine a hybrid approach where the kernel allocates network queues to applications and suggests userland device drivers. While running, the apps would have direct access to the hardware. And when the app is terminated the kernel would reclaim the assigned hardware for reuse by other applications.
- sp33der 7y ago
  Microkernels tend to do a lot of what you're talking about. They accomplish this via interrupts and virtual mapping of the device memory to userspace for the userland app to access. "If we wanted, we could even replicate a kernel device driver style API in userland that network device drivers program against, done as a set of userland library files loaded dynamically based on detected hardware." This is how many microkernel based embedded RTOS's work. Common API's written for a hardware device, say a NOR flash chip, and the driver for the hardware onboard implements the interface specific to that hardware. You can then dynamically link the driver specific to your hardware. That's what I was doing as an embedded C developer some 10 years ago, and that solution had already been implemented for at least a decade (or 2) before that.
deleted 7y ago
[deleted]
robbyt 7y ago
As I was reading this, I remembered the days of my youth setting the IRQ and DMA address for my soundblaster (compatible) soundcard.
- asn1parse 7y ago
  Haha, me too.
- organsnyder 7y ago
  Be sure to avoid IRQ7. That conflict with LPT1 and/or LTP2 can be brutal.
baruch 7y ago
At least for NVMe SSDs that is already solved, there is a standard that all NVMe SSDs implement and you only need one driver for all of them. If you want you can use SPDK or one of the few other drivers and you get the full speed block access. What you don't get however is sharing the disk between multiple processes.
saltcured 7y ago
There is a constant dance at the fringe of high performance systems. It leads to a recurring pattern of "revolutionizing" with some kind of bypass or coprocessor architecture, then eventually reverting to traditional structures as the new performance realities reach commodity levels. Part of it is the economics at the fringe can pursue speed at any cost. And part is the heady appeal of doing things differently for researchers and advanced practitioners. But, in the long view, I think you are right that it is a bad idea. If you care about maintenance and sustainability, you usually find that these bypass solutions get abandoned as soon as the more conventional approaches can approximate their speed on newer commodity hardware. So there is huge churn in these specialist devices with specialist APIs and tooling. There is a recurring theme in high performance networking where crazy things are tried and all sorts of fancy protocol offloading written, then eventually deprecated because it is seen as a support burden and a source of bugs. Because each of these specialized stacks has a smaller user base, they are have less economy of scale to invest in maintenance and stabilization.
- baybal2 7y ago
  > There is a constant dance at the fringe of high performance systems. It leads to a recurring pattern of "revolutionizing" with some kind of bypass or coprocessor architecture, then eventually reverting to traditional structures as the new performance realities reach commodity levels. Truly this is what DC hardware been about for 20+ years. Look at HPC space, and than look for "uncle Joe hosting." The moment Joe starts looking for HPC and invest heavily for some super duper RDMA powered disk, a mainstream hardware maker comes and spoils everything by releasing a mass market product doing just that if not better, thus ruining the grand scheme of gaining competitive advantage through some "secret sauce hardware" Look at video transcoding or that new "AI" neural network thing. After few years of entertaining themselves with idea of purpose made hardware, big boys simply went the route of just using a lot of mainstream hardware.
baybal2 7y ago
> Hardware vendors don't like to share. They don't share code, they don't share common interfaces, they don't even share documentation. As it is now, these are all problems which most userland developers don't have to care about -- those problems get dealt with in the kernel, by developers who specialize in building support for uncooperative hardware. Thus more money for us :) I think this fact almost begs to be taken advantage of. See, how things work in corporate storage products, with EMC being the prime example
TheSoftwareGuy 7y ago
An OS is more than just a kernel. OS’s would simply ship with user land drivers instead of kernel space drivers
atq2119 7y ago
You should take a look at GPUs. All of the complexity of 3D rendering is implemented in userspace, and that has been the case universally in production for more than 5 years now -- closer to 10, really. If you replace "all" by "almost all", you can go back much further than that, really all the way to the beginning of GPUs' existence. And yet normal application developers don't have to care, because the driver does it for them. The point is, drivers don't have to live in kernel space, they can live in user space as well. Networking folks may start being more serious about this nowadays, but it has been the reality in GPUs for a long time.

johnm1019 7y ago

Disclaimer: IANA kernel developer. Is there any benefit to having those specialized developers create frameworks or libraries in Userland which other developers can leverage? This way they remain the interface to uncooperative hardware, but the code is in Userland so the bold folks can try their own approach.

loeg 7y ago
That's basically DPDK / SDPK.
dorlaor 7y ago
Seastar.io is another one, it's an async engine that utilizes all of the cores in a modern system and is the heart of Scylla's DB. However, it's complex to use it correctly

Q6T46nT668w6i3m 7y ago

I enjoyed the paper. My impression is that you’d shift the burden to the runtimes that, for many applications, currently sit between POSIX and applications (e.g. see the Q&A about POSIX).

asn1parse 7y ago
Considering the poor quality of userland software, this seems like a terrible idea.
- ncmncm 7y ago
  Anyone willing to spend the money can have top quality userland software. Getting a better OS is not always accessible, even with deep pockets.

josephg 7y ago

I don’t think anyone is suggesting that every application explicitly code for each network device. All of that tricky logic could be put into a userland library instead of the kernel. If we wanted, we could even replicate a kernel device driver style API in userland that network device drivers program against, done as a set of userland library files loaded dynamically based on detected hardware. The tricky part wouldn’t be sharing code between applications. We know how to do that. The hard part would be figuring out a clean way to share the hardware between all running applications, given that any app could be terminated at any time, apps might be mutually untrustworthy and apps would have to play nice to share resources. I can imagine a hybrid approach where the kernel allocates network queues to applications and suggests userland device drivers. While running, the apps would have direct access to the hardware. And when the app is terminated the kernel would reclaim the assigned hardware for reuse by other applications.

sp33der 7y ago
Microkernels tend to do a lot of what you're talking about. They accomplish this via interrupts and virtual mapping of the device memory to userspace for the userland app to access. "If we wanted, we could even replicate a kernel device driver style API in userland that network device drivers program against, done as a set of userland library files loaded dynamically based on detected hardware." This is how many microkernel based embedded RTOS's work. Common API's written for a hardware device, say a NOR flash chip, and the driver for the hardware onboard implements the interface specific to that hardware. You can then dynamically link the driver specific to your hardware. That's what I was doing as an embedded C developer some 10 years ago, and that solution had already been implemented for at least a decade (or 2) before that.

deleted 7y ago

[deleted]

robbyt 7y ago

As I was reading this, I remembered the days of my youth setting the IRQ and DMA address for my soundblaster (compatible) soundcard.

asn1parse 7y ago
Haha, me too.
organsnyder 7y ago
Be sure to avoid IRQ7. That conflict with LPT1 and/or LTP2 can be brutal.

baruch 7y ago

At least for NVMe SSDs that is already solved, there is a standard that all NVMe SSDs implement and you only need one driver for all of them. If you want you can use SPDK or one of the few other drivers and you get the full speed block access. What you don't get however is sharing the disk between multiple processes.

saltcured 7y ago

There is a constant dance at the fringe of high performance systems. It leads to a recurring pattern of "revolutionizing" with some kind of bypass or coprocessor architecture, then eventually reverting to traditional structures as the new performance realities reach commodity levels. Part of it is the economics at the fringe can pursue speed at any cost. And part is the heady appeal of doing things differently for researchers and advanced practitioners. But, in the long view, I think you are right that it is a bad idea. If you care about maintenance and sustainability, you usually find that these bypass solutions get abandoned as soon as the more conventional approaches can approximate their speed on newer commodity hardware. So there is huge churn in these specialist devices with specialist APIs and tooling. There is a recurring theme in high performance networking where crazy things are tried and all sorts of fancy protocol offloading written, then eventually deprecated because it is seen as a support burden and a source of bugs. Because each of these specialized stacks has a smaller user base, they are have less economy of scale to invest in maintenance and stabilization.

baybal2 7y ago
> There is a constant dance at the fringe of high performance systems. It leads to a recurring pattern of "revolutionizing" with some kind of bypass or coprocessor architecture, then eventually reverting to traditional structures as the new performance realities reach commodity levels. Truly this is what DC hardware been about for 20+ years. Look at HPC space, and than look for "uncle Joe hosting." The moment Joe starts looking for HPC and invest heavily for some super duper RDMA powered disk, a mainstream hardware maker comes and spoils everything by releasing a mass market product doing just that if not better, thus ruining the grand scheme of gaining competitive advantage through some "secret sauce hardware" Look at video transcoding or that new "AI" neural network thing. After few years of entertaining themselves with idea of purpose made hardware, big boys simply went the route of just using a lot of mainstream hardware.

baybal2 7y ago

> Hardware vendors don't like to share. They don't share code, they don't share common interfaces, they don't even share documentation. As it is now, these are all problems which most userland developers don't have to care about -- those problems get dealt with in the kernel, by developers who specialize in building support for uncooperative hardware. Thus more money for us :) I think this fact almost begs to be taken advantage of. See, how things work in corporate storage products, with EMC being the prime example

TheSoftwareGuy 7y ago

An OS is more than just a kernel. OS’s would simply ship with user land drivers instead of kernel space drivers

atq2119 7y ago

You should take a look at GPUs. All of the complexity of 3D rendering is implemented in userspace, and that has been the case universally in production for more than 5 years now -- closer to 10, really. If you replace "all" by "almost all", you can go back much further than that, really all the way to the beginning of GPUs' existence. And yet normal application developers don't have to care, because the driver does it for them. The point is, drivers don't have to live in kernel space, they can live in user space as well. Networking folks may start being more serious about this nowadays, but it has been the reality in GPUs for a long time.