6 ms·

It's a nice exercise, but I don't see t going anywhere. 1st. I don't see that many bugs in the TCP/IP, filesystems, etc. specially those exploitable by black h

It's a nice exercise, but I don't see t going anywhere. 1st. I don't see that many bugs in the TCP/IP, filesystems, etc. specially those exploitable by black hat hackers, specially remote exploits. We usually see wide-spread vulnerabilities in the middleware like OpenSSH, which is in the userspace, putting everything in the app is not going to solve that, quite the contrary, we need to wait for the library maintainer to fix the bug, for the applications to integrate with the newer version of the library, and then, update all the applications. 2nd. Side-channel attacks. I'm not really confident in letting applications have that low level of hardware access. I mean, it's really easy for one application to steal the resources from another. We need authority if we are sharing peripherals. Well, we have one peripheral already that is shared by processes, that's memory and look at the mess that is a modern MMU and all the attacks on those that we have seen throughout the decades. 3rd. If we are going to implement that as a shared library, the result is exactly the same as today's micro and hybrid kernels, if we are not going to save memory and implement that as a static library, it's exactly the save as virtualization. Correct me if I'm wrong.

justicezyx 8y ago
Unikernel is about install controls to application writer. Once developers get into the workflow of writing applixations that can have fuller control of its environment, rarely one would return back to the world of opaque infrastructure and framework.
MaxBarraclough 8y ago
I'm not an expert in unikernels by any means, but it's not the case that it's the equivalent of a lightweight Linux distro plus some neat debug tools. I don't think the linked article does the idea justice. Here [0] is a good discussion of unikernels (TheRaven knows what he's talking about). ctrl-f for "You absolutely can not do the same thing with a conventional *NIX OS" https://soylentnews.org/comments.pl?sid=11888&cid=296337 https://soylentnews.org/comments.pl?sid=11888&cid=296337
- weberc2 8y ago
  I agree. One of the appeals of unikernels in my opinion is that (I think) they will eventually give us the things that Docker currently gives us, but with one less layer of abstraction (unikernels running on a VM orchestrator instead of containers running on a container orchestrator running on a VM orchestrator). I imagine that this is inevitable as a matter of efficiency, but it's probably such a small practical gain that we won't see much traction in the next decade.
  - rhinoceraptor 8y ago
    Or you could just run containers without the abhorrent mess that is hardware virtualization...
    - mato 8y ago
      The post discusses running unikernels without "the abhorrent mess that is hardware virtualization".
  - MaxBarraclough 8y ago
    > I imagine that this is inevitable as a matter of efficiency, but it's probably such a small practical gain that we won't see much traction in the next decade. I agree that if the advantages boil down to saving a hundred MB of RAM of two, and enabling additional compiler optimisations, it won't be compelling to that many folks. Taken to the extreme though, we can start spinning up instances 'just in time' to deal with incoming requests. Doing this, you could run (and hopefully pay for) your instances for only a tiny fraction of the time your service is in a working/available state. This has apparently already been demonstrated [0], but their demo has been broken for as long as I can remember. I presume it worked at one point. I don't know if any commercial Xen-based offerings can spin up an instance as quickly as their configuration did, but the idea is there. [0] http://erlangonxen.org/zerg http://erlangonxen.org/zerg
    - weberc2 8y ago
      I think container orchestration offerings are pretty much at the pricing model you describe. You can spin them up just in time and only pay for what you use. The advantage to unikernels is that cloud operators don't have to operate two layers of orchestrators, so they could pass those savings onto unikernel users; however, I don't think that will be the lowest hanging fruit for a long time.
mato 8y ago
Co-author of the paper referenced in the blog post here. 1. Regarding bugs: There is a lot of code in your monolithic OS kernel. There have been, and will be, a lot of vulnerabilities in that code. Various sandboxing mechanisms notwithstanding, the vast majority of processes running on your system can potentially use all of that code as an attack vector. Unikernels let you switch off access to most of that "host" code, and, through the use of a library OS, contain only the minimal set of libraries needed to run your application in the "guest". Regarding updates: Valid point, but no different from any modern application of substantial complexity, except that now you also have to update (e.g.) the library providing its TCP stack. 2. Unikernels don't make that problem any worse. In fact, if you run on something like Muen (https://muen.sk/ https://muen.sk/), it'll mitigate a bunch of these attacks by giving you a less precise RDTSC at the subject ("VM") level. 3. I don't follow. Implement what?
- nostrademons 8y ago
  If the unikernel is running as a normal process, all that potentially vulnerable code is still on the box, and still potentially executable in the event of a vulnerability. The security selling point of a standard unikernel is that the all the potentially-vulnerable code doesn't even exist, because only the libraries that are actually used get compiled and linked into the app, and it has no code relevant to functionality that it's not actually using. The paper suggests using system-call filtering via seccomp or similar to block off this attack surface, but if you trust seccomp to have zero bugs, you might as well do traditional process sandboxing a la Google Chrome. The problem unikernels solve is that traditionally, a vulnerability in C code means "game over" and the box is completely pwned with all memory and storage directly accessible, while with a unikernel, a vulnerability means only that an attacker can access other functionality within the app, and can't rely on eg. a shell or a debugger being available to arbitrarily move bytes around.
  - mato 8y ago
    > If the unikernel is running as a normal process, all that potentially vulnerable code is still on the box [...] Correct. Conceptually the same applies to all Type 2 hypervisors. Type 1 less so, but you could still potentially exploit Xen and you have all of the dom0 to play around in. > The paper suggests using system-call filtering via seccomp or similar to block off this attack surface, but if you trust seccomp to have zero bugs, you might as well do traditional process sandboxing a la Google Chrome. If you do that, you will have to do a line-by-line analysis of the code you want to sandbox in order to determine exactly which syscalls it's using. With the approach presented in our paper the developer does not need to care about this. For example, she can develop her MirageOS unikernel as a normal UNIX process and switch to a guaranteed-to-work-minimal-seccomp sandbox with a simple change of target (build-time configuration option). But yes, you are now trusting seccomp instead of KVM. I believe in giving people the ability to easily make that choice. > The problem unikernels solve is that traditionally [...] can't rely on eg. a shell or a debugger being available to arbitrarily move bytes around. That part does not change with the sandboxing mechanism changing. The stuff that's inside is still only your unikernel.
candiodari 8y ago
> We usually see wide-spread vulnerabilities in the middleware like OpenSSH There has been exactly 1 remotely exploitable bug and exactly 1 information leak bug in openssh in more than a decade (I believe the exploit is actually more than a decade old, need to check though), it is by far the most secure daemon ever. It is hard to overstate just how much of an accomplishment that is. NOTHING matches it in security. This is even more impressive given how widespread it's usage it. > We need authority if we are sharing peripherals. I think you'll find that large websites ... just don't do this. The only real purpose of multiple processes on such webservers is administration. > f we are not going to save memory and implement that as a static library ... The reason you want to do this is that the slowest thing in modern processors is context-switching. Going from user-space to kernel space. If you have, say, a networking stack in user space you can avoid such context switching entirely. The speed gains are enormous, especially for webservers. Ridiculous performance in networking also enables many more applications that just won't work without. What do you think about cluster-wide disks, for instance ? SQLite like databases that run against Petabyte-sized files, concurrently, safely. You're just not going to do that efficiently by having it done in the OS.
- eyberg 8y ago
  Just to touch on this point - you hit the nail on the head. Microsoft and others have measured this cost to be north of 30%.
corysama 8y ago
1st. It’s not going to solve that. But, I don’t think solving that is a goal of this approach. 2nd. In a pure-unikernel approach, applications can’t steal from each other because there is only one application. Isolation is left to the VM. The goal is to get rid of the redundant isolation of running a multiplexing OS over a multiplexing VM. In the as-processes model, the Linux kernel still provides sandboxing. The goal there is to be a bridge between traditional and unikernel development. 3rd. The goal is to boil down to just virtualization and only the libraries specifically needed by a single specific application.
toast0 8y ago
I would (in theory), love a unikernel for the work I do. Our bare metal boxes run about 20 Unix processes -- cron, syslog, sshd, etc plus one server daemon. It's nice to have a familiar environment, but we could run on fewer boxes without a general purpose kernel and general purpose memory protection that we don't strictly need, especially for nodes that could easily be tweaked to forgo local disk access (because filesystems are tricky, I'd rather leave nonephemeral storage with something proven) That said, we're not moving in the direction of unikernels, but it has a clear application in my mind. It's a fair bit of work to migrate, and I've yet to see a practical comparison, so it's unclear what the actual benefits over conceptual benefits would be.
- dman 8y ago
  What is the problem domain?
  - toast0 8y ago
    Chat service, in Erlang. I'd love to just boot to Erlang, and have as much of the system as possible be hotloadable, including the tcp stack. But I may be suffering from too much exposure to hot loading. :)
    - dman 8y ago
      Sounds like a great problem to work on! When you have such a well defined problem it can be a great ticket to do good work since in some ways you can control the stack top to bottom. Have fun!
      - toast0 8y ago
        I'm not actually working on it, unfortunately. There's too much risk, not enough time, and we're in the middle of migrating to a different environment where it would be an even weirder thing. But I think it's a good example of a place it could work, and it would be a lot of fun.
eyberg 8y ago
A few thoughts: 1) (Putting on my black hat) Attackers don't care about bugs or exploits. They care about running their code on your system. Whether that is as simple as mysqldump or wget'ng a monero cryptominer to run on there it all is based on the premise that the monolithic operating system (whose design is ~50 years old and linux is > 27 years old) is explicitly designed to run multiple programs by multiple users. Keep in mind this design pre-dates commercialized virtualization (eg: vmware) and pre-date "cloud" (eg: aws). If we assume that you are already utilizing VMs (and you are if you are on any public cloud and you are in most private on-prem deployments) the VM then becomes your isolation model. Can you still attack the underlying infrastructure? Sure - but if you can root GCE or AWS I'd say we all have some serious thinking to do on the current state of cloud infrastructure. Contrast and compare that to all the ridiculous headlines you see every single day and the fact that every single RCE that is worth doing entails forking/execve a new process. It's one thing to have the instruction pointer - it's quite another to launch a shell that doesn't exist, link your program to libraries that don't exist, as a user that doesn't exist, download new code when you can't.... etc. 2) Not to belabor this point but side-channel attacks affect everyone and Intel has been taking the hard (in terms of market) approach of simply disabling hyper-threading on some of their hardware. Security is the number one selling point for unikernels imo.
rayiner 8y ago
We are already seeing exokernel concepts. On a Linix machine, most 3D graphics is handled in the userspace GL library. The kernel driver handles modesetting, manages memory, and arranges for the userspace library to be able to directly submit command buffers to the GPU via memory mapped areas. DPDK works the same way. The NIC ring buffer is mapped directly into the userspace process, which can then perform packet I/O without calling into the kernel.
- pcwalton 8y ago
  Well, that's roughly how it's supposed to work in theory for graphics, but in practice the kernel ends up having to do fairly deep parsing of GPU command buffers for security. See, for example: https://www.kernel.org/doc/html/v4.11/gpu/i915.html#batchbuffer-parsing https://www.kernel.org/doc/html/v4.11/gpu/i915.html#batchbuf...
  - atq2119 8y ago
    Intel's GPUs aren't exactly known for stellar performance, but I'm pretty sure that's old info even for Intel. Certainly this kind of parsing hasn't been required on AMD and Nvidia GPUs' 3D engines for almost ten years now.
    - pcwalton 8y ago
      There's still a good bit of it necessary even for new GPUs like Radeon R600. See r600_packet0_check, r600_packet3_check: https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/radeon/r600_cs.c https://github.com/torvalds/linux/blob/master/drivers/gpu/dr...
      - rayiner 8y ago
        Thanks for the very interesting references and discussion. GPUs are hard, huh? AFAIK, this sort of validation isn't necessary for NICs. (Question: why is parsing necessary to ensure security, instead of say relying on the IO-MMU?)
madmax96 8y ago
Re: 1 - that might be true, but you still probably wouldn't want to co-tenant a sandbox'd process with a total stranger. The overhead of a total Linux kernel is still larger than a unikernel-native application, so if you're a cloud provider, encouraging and using unikernel technology makes a lot of sense. Re: 2 - this is managed by the hypervisor. SEL4 can be run as a hypervisor and has been formally verified to be correct [1]. Trusting a proven system like SEL4 is a far-cry from trusting Linux's isolation primitives because we can make hard-guarantees about the behavior. Unikernels have the advantage that they are basically backwards-compatible (you can run any VM on the infrastructure you develop, even if that infrastructure is tuned for unikernel-native applications). With Unikernels, you can achieve VMs that are lighter than containers [2] thereby increasing your customer-per-server ratio. [1] https://sel4.systems/Info/FAQ/proof.pml https://sel4.systems/Info/FAQ/proof.pml [2] http://cnp.neclab.eu/projects/lightvm/lightvm.pdf http://cnp.neclab.eu/projects/lightvm/lightvm.pdf
blattimwind 8y ago
> We usually see wide-spread vulnerabilities in the middleware like OpenSSH > Correct me if I'm wrong. OpenSSH is probably the single most secure piece of C network software. It isn't exactly a good example for "wide-spread vulnerabilities in middleware".

justicezyx 8y ago

Unikernel is about install controls to application writer. Once developers get into the workflow of writing applixations that can have fuller control of its environment, rarely one would return back to the world of opaque infrastructure and framework.

MaxBarraclough 8y ago

I'm not an expert in unikernels by any means, but it's not the case that it's the equivalent of a lightweight Linux distro plus some neat debug tools. I don't think the linked article does the idea justice. Here [0] is a good discussion of unikernels (TheRaven knows what he's talking about). ctrl-f for "You absolutely can not do the same thing with a conventional *NIX OS" https://soylentnews.org/comments.pl?sid=11888&cid=296337 https://soylentnews.org/comments.pl?sid=11888&cid=296337

weberc2 8y ago
I agree. One of the appeals of unikernels in my opinion is that (I think) they will eventually give us the things that Docker currently gives us, but with one less layer of abstraction (unikernels running on a VM orchestrator instead of containers running on a container orchestrator running on a VM orchestrator). I imagine that this is inevitable as a matter of efficiency, but it's probably such a small practical gain that we won't see much traction in the next decade.
- rhinoceraptor 8y ago
  Or you could just run containers without the abhorrent mess that is hardware virtualization...
  - mato 8y ago
    The post discusses running unikernels without "the abhorrent mess that is hardware virtualization".
- MaxBarraclough 8y ago
  > I imagine that this is inevitable as a matter of efficiency, but it's probably such a small practical gain that we won't see much traction in the next decade. I agree that if the advantages boil down to saving a hundred MB of RAM of two, and enabling additional compiler optimisations, it won't be compelling to that many folks. Taken to the extreme though, we can start spinning up instances 'just in time' to deal with incoming requests. Doing this, you could run (and hopefully pay for) your instances for only a tiny fraction of the time your service is in a working/available state. This has apparently already been demonstrated [0], but their demo has been broken for as long as I can remember. I presume it worked at one point. I don't know if any commercial Xen-based offerings can spin up an instance as quickly as their configuration did, but the idea is there. [0] http://erlangonxen.org/zerg http://erlangonxen.org/zerg
  - weberc2 8y ago
    I think container orchestration offerings are pretty much at the pricing model you describe. You can spin them up just in time and only pay for what you use. The advantage to unikernels is that cloud operators don't have to operate two layers of orchestrators, so they could pass those savings onto unikernel users; however, I don't think that will be the lowest hanging fruit for a long time.

mato 8y ago

Co-author of the paper referenced in the blog post here. 1. Regarding bugs: There is a lot of code in your monolithic OS kernel. There have been, and will be, a lot of vulnerabilities in that code. Various sandboxing mechanisms notwithstanding, the vast majority of processes running on your system can potentially use all of that code as an attack vector. Unikernels let you switch off access to most of that "host" code, and, through the use of a library OS, contain only the minimal set of libraries needed to run your application in the "guest". Regarding updates: Valid point, but no different from any modern application of substantial complexity, except that now you also have to update (e.g.) the library providing its TCP stack. 2. Unikernels don't make that problem any worse. In fact, if you run on something like Muen (https://muen.sk/ https://muen.sk/), it'll mitigate a bunch of these attacks by giving you a less precise RDTSC at the subject ("VM") level. 3. I don't follow. Implement what?

nostrademons 8y ago
If the unikernel is running as a normal process, all that potentially vulnerable code is still on the box, and still potentially executable in the event of a vulnerability. The security selling point of a standard unikernel is that the all the potentially-vulnerable code doesn't even exist, because only the libraries that are actually used get compiled and linked into the app, and it has no code relevant to functionality that it's not actually using. The paper suggests using system-call filtering via seccomp or similar to block off this attack surface, but if you trust seccomp to have zero bugs, you might as well do traditional process sandboxing a la Google Chrome. The problem unikernels solve is that traditionally, a vulnerability in C code means "game over" and the box is completely pwned with all memory and storage directly accessible, while with a unikernel, a vulnerability means only that an attacker can access other functionality within the app, and can't rely on eg. a shell or a debugger being available to arbitrarily move bytes around.
- mato 8y ago
  > If the unikernel is running as a normal process, all that potentially vulnerable code is still on the box [...] Correct. Conceptually the same applies to all Type 2 hypervisors. Type 1 less so, but you could still potentially exploit Xen and you have all of the dom0 to play around in. > The paper suggests using system-call filtering via seccomp or similar to block off this attack surface, but if you trust seccomp to have zero bugs, you might as well do traditional process sandboxing a la Google Chrome. If you do that, you will have to do a line-by-line analysis of the code you want to sandbox in order to determine exactly which syscalls it's using. With the approach presented in our paper the developer does not need to care about this. For example, she can develop her MirageOS unikernel as a normal UNIX process and switch to a guaranteed-to-work-minimal-seccomp sandbox with a simple change of target (build-time configuration option). But yes, you are now trusting seccomp instead of KVM. I believe in giving people the ability to easily make that choice. > The problem unikernels solve is that traditionally [...] can't rely on eg. a shell or a debugger being available to arbitrarily move bytes around. That part does not change with the sandboxing mechanism changing. The stuff that's inside is still only your unikernel.

candiodari 8y ago

> We usually see wide-spread vulnerabilities in the middleware like OpenSSH There has been exactly 1 remotely exploitable bug and exactly 1 information leak bug in openssh in more than a decade (I believe the exploit is actually more than a decade old, need to check though), it is by far the most secure daemon ever. It is hard to overstate just how much of an accomplishment that is. NOTHING matches it in security. This is even more impressive given how widespread it's usage it. > We need authority if we are sharing peripherals. I think you'll find that large websites ... just don't do this. The only real purpose of multiple processes on such webservers is administration. > f we are not going to save memory and implement that as a static library ... The reason you want to do this is that the slowest thing in modern processors is context-switching. Going from user-space to kernel space. If you have, say, a networking stack in user space you can avoid such context switching entirely. The speed gains are enormous, especially for webservers. Ridiculous performance in networking also enables many more applications that just won't work without. What do you think about cluster-wide disks, for instance ? SQLite like databases that run against Petabyte-sized files, concurrently, safely. You're just not going to do that efficiently by having it done in the OS.

eyberg 8y ago
Just to touch on this point - you hit the nail on the head. Microsoft and others have measured this cost to be north of 30%.

corysama 8y ago

1st. It’s not going to solve that. But, I don’t think solving that is a goal of this approach. 2nd. In a pure-unikernel approach, applications can’t steal from each other because there is only one application. Isolation is left to the VM. The goal is to get rid of the redundant isolation of running a multiplexing OS over a multiplexing VM. In the as-processes model, the Linux kernel still provides sandboxing. The goal there is to be a bridge between traditional and unikernel development. 3rd. The goal is to boil down to just virtualization and only the libraries specifically needed by a single specific application.

toast0 8y ago

I would (in theory), love a unikernel for the work I do. Our bare metal boxes run about 20 Unix processes -- cron, syslog, sshd, etc plus one server daemon. It's nice to have a familiar environment, but we could run on fewer boxes without a general purpose kernel and general purpose memory protection that we don't strictly need, especially for nodes that could easily be tweaked to forgo local disk access (because filesystems are tricky, I'd rather leave nonephemeral storage with something proven) That said, we're not moving in the direction of unikernels, but it has a clear application in my mind. It's a fair bit of work to migrate, and I've yet to see a practical comparison, so it's unclear what the actual benefits over conceptual benefits would be.

dman 8y ago
What is the problem domain?
- toast0 8y ago
  Chat service, in Erlang. I'd love to just boot to Erlang, and have as much of the system as possible be hotloadable, including the tcp stack. But I may be suffering from too much exposure to hot loading. :)
  - dman 8y ago
    Sounds like a great problem to work on! When you have such a well defined problem it can be a great ticket to do good work since in some ways you can control the stack top to bottom. Have fun!
    - toast0 8y ago
      I'm not actually working on it, unfortunately. There's too much risk, not enough time, and we're in the middle of migrating to a different environment where it would be an even weirder thing. But I think it's a good example of a place it could work, and it would be a lot of fun.

eyberg 8y ago

A few thoughts: 1) (Putting on my black hat) Attackers don't care about bugs or exploits. They care about running their code on your system. Whether that is as simple as mysqldump or wget'ng a monero cryptominer to run on there it all is based on the premise that the monolithic operating system (whose design is ~50 years old and linux is > 27 years old) is explicitly designed to run multiple programs by multiple users. Keep in mind this design pre-dates commercialized virtualization (eg: vmware) and pre-date "cloud" (eg: aws). If we assume that you are already utilizing VMs (and you are if you are on any public cloud and you are in most private on-prem deployments) the VM then becomes your isolation model. Can you still attack the underlying infrastructure? Sure - but if you can root GCE or AWS I'd say we all have some serious thinking to do on the current state of cloud infrastructure. Contrast and compare that to all the ridiculous headlines you see every single day and the fact that every single RCE that is worth doing entails forking/execve a new process. It's one thing to have the instruction pointer - it's quite another to launch a shell that doesn't exist, link your program to libraries that don't exist, as a user that doesn't exist, download new code when you can't.... etc. 2) Not to belabor this point but side-channel attacks affect everyone and Intel has been taking the hard (in terms of market) approach of simply disabling hyper-threading on some of their hardware. Security is the number one selling point for unikernels imo.

rayiner 8y ago

We are already seeing exokernel concepts. On a Linix machine, most 3D graphics is handled in the userspace GL library. The kernel driver handles modesetting, manages memory, and arranges for the userspace library to be able to directly submit command buffers to the GPU via memory mapped areas. DPDK works the same way. The NIC ring buffer is mapped directly into the userspace process, which can then perform packet I/O without calling into the kernel.

pcwalton 8y ago
Well, that's roughly how it's supposed to work in theory for graphics, but in practice the kernel ends up having to do fairly deep parsing of GPU command buffers for security. See, for example: https://www.kernel.org/doc/html/v4.11/gpu/i915.html#batchbuffer-parsing https://www.kernel.org/doc/html/v4.11/gpu/i915.html#batchbuf...
- atq2119 8y ago
  Intel's GPUs aren't exactly known for stellar performance, but I'm pretty sure that's old info even for Intel. Certainly this kind of parsing hasn't been required on AMD and Nvidia GPUs' 3D engines for almost ten years now.
  - pcwalton 8y ago
    There's still a good bit of it necessary even for new GPUs like Radeon R600. See r600_packet0_check, r600_packet3_check: https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/radeon/r600_cs.c https://github.com/torvalds/linux/blob/master/drivers/gpu/dr...
    - rayiner 8y ago
      Thanks for the very interesting references and discussion. GPUs are hard, huh? AFAIK, this sort of validation isn't necessary for NICs. (Question: why is parsing necessary to ensure security, instead of say relying on the IO-MMU?)

madmax96 8y ago

Re: 1 - that might be true, but you still probably wouldn't want to co-tenant a sandbox'd process with a total stranger. The overhead of a total Linux kernel is still larger than a unikernel-native application, so if you're a cloud provider, encouraging and using unikernel technology makes a lot of sense. Re: 2 - this is managed by the hypervisor. SEL4 can be run as a hypervisor and has been formally verified to be correct [1]. Trusting a proven system like SEL4 is a far-cry from trusting Linux's isolation primitives because we can make hard-guarantees about the behavior. Unikernels have the advantage that they are basically backwards-compatible (you can run any VM on the infrastructure you develop, even if that infrastructure is tuned for unikernel-native applications). With Unikernels, you can achieve VMs that are lighter than containers [2] thereby increasing your customer-per-server ratio. [1] https://sel4.systems/Info/FAQ/proof.pml https://sel4.systems/Info/FAQ/proof.pml [2] http://cnp.neclab.eu/projects/lightvm/lightvm.pdf http://cnp.neclab.eu/projects/lightvm/lightvm.pdf

blattimwind 8y ago

> We usually see wide-spread vulnerabilities in the middleware like OpenSSH > Correct me if I'm wrong. OpenSSH is probably the single most secure piece of C network software. It isn't exactly a good example for "wide-spread vulnerabilities in middleware".