14 ms·

Running Postgres as a Unikernel

90 points by eyberg 3y ago

arghwhat 3y ago
Unikernels are cool, but half this official post is pointlessly throwing rocks at postgres, the other half having to hack around to get this extremely common piece of software working with your own product. No thanks, I will get my unikernel fix from someone else. Plus, if you often have to LD_PRELOAD getuid, just add a bogus impl to your libc or as a syscall to reduce onboarding friction.
- anonymous_sorry 3y ago
  Is it throwing rocks? "I'm not knocking postgres here. You have to keep in mind [a paragraph of contextualising postgres's design decisions]"
  - aljarry 3y ago
    The rant in the middle is very dismissive over voices against in the discussion, and that makes me agree with OP.
  - giraffe_lady 3y ago
    Saying you're not doing something is not actually the same as not doing it.
    - richbell 3y ago
      I'm not disagreeing with you, but I think you're completely wrong about that.
- eyberg 3y ago
  If you think I was throwing rocks that most definitely was not the intention. The whole point of writing the post was to point out all the abstractions that don't quite gel well in unikernel land. The LD_PRELOAD wasn't used for this particular port but it does get used sometimes . We don't actually have real uid calls but some applications will require specific ones that we stub from time to time. Very very few apps actually require this.
dvdplm 3y ago
So what would be the reasons to run pg as a unikernel? To get max performance on bare metal? Smaller attack surface? Or?
- FridgeSeal 3y ago
  The less between your application and the hardware the better you can utilise it. There’d be basically no contention for resources, IO could go basically straight to the drive with less fsync-shenanigans etc.
  - tpetry 3y ago
    But is the impact really noticeable? I believe it would then we would have seen more ready and stable solutions.
    - rwmj 3y ago
      With our Linux-based unikernel you can make small adjustments to the application -- usually just a handful of lines of code -- and get big performance improvements, like 25% or so. We had a paper about this at Eurosys'23: https://dl.acm.org/doi/abs/10.1145/3552326.3587458 https://dl.acm.org/doi/abs/10.1145/3552326.3587458
      - tpetry 3y ago
        That‘s the problem of unikernels. There are papers how great they are but almost nobody is using them. If applications could easily get a 25% boost without much work they would do it. But nobody is using it. So I am not trusting the „benefits“.
        rwmj 3y ago
        There are a ton of well known techniques in computer science that will make software faster, more reliable or better in other ways, that people don't use, yet really work.
        3y ago
imiric 3y ago
I wonder why unikernels haven't caught on. Years ago when they started to be hyped, I imagined the technology would surpass the adoption of containers. In principle, they have the security and isolation of VMs which containers lack, but also have even less overhead than containers, and can run on bare metal. It sounds like the best of both worlds. The main issues years ago were developer tooling, packaging, and difficulty with debugging, so the UX was far from what Docker did for containers. But I was hoping that these issues would be addressed (are they?), and that a defacto standard would emerge that would propel them into the mainstream. Instead, we have these disparate niche tools and services that seem to be competing, but there's no widespread industry adoption. I'm sure that the evolution of the container ecosystem made unikernels less desirable, and the increased popularity of container orchestration engines like Kubernetes cemented this trend. But I wish that the unikernel ecosystem were more mature so that it's at least a topic of conversation in tech circles, and k8s wasn't such an obvious choice. Though I might be in the wrong circles and have outdated information, so I'm curious to know what HN thinks.
- vbezhenar 3y ago
  I feel like you’re mixing layers. Alternative to VM is Linux namespaces. They provide isolation. Docker, Kubernetes, LXC, systemd-spawn and whatnot are just thin wrappers around Linux API. There’s nothing that would prevent you from running workloads in VMs and managing them with Kubernetes. Just write proper adapters. Actually there are projects in the wild which do exactly that. As to why VM is supposed to be lighter than container, I don’t really understand. I think it’s the other way around. Industry does not really care about security and losing performance for unclear benefits won’t fly for masses.
  - imiric 3y ago
    > Alternative to VM is Linux namespaces. They're really not. Containers (and by that I mean their underlying technology) are not alternatives to VMs, in the same way that unikernels are not alternatives to containers (yet, anyway). They're technologies that can be used for similar purposes, but in practice, they all have their tradeoffs that should be considered before use. For one, containers share the same kernel of the host machine. The isolation provided by VMs is on a deeper level, which also makes them more secure. Unikernels are more like VMs in this sense. My point is that the only drawback of unikernels in theory is the immature tooling and support around it. So I'm wondering if there is another reason that they haven't been as widely adopted as containers. > As to why VM is supposed to be lighter than container, I don’t really understand. You misunderstood. I said that unikernels, not VMs, have less overhead than containers. Containers require a minimal, but fully functioning, OS to work, while unikernels only require a cut-down version of a kernel. Whether unikernels are more lightweight is debatable, since containers are merely another process running on the same host OS with improved isolation, while unikernels depend on a hypervisor to work, but if we define "lightweight" to include boot time and performance, unikernels have certainly surpassed containers in this sense as well[1]. So the main benefit containers had over VMs is also an attribute of unikernels. [1]: https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2.pdf https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2...
cmrdporcupine 3y ago
To me it's more sensible to consider writing a database from scratch for the unikernel. Or at least the storage engine portion of it. Plenty of recent DB research about running up against the wall of what the Linux VM subsystem can provide for them in terms of memory management. LeanStore[1], Umbra [2], and research [3] since then show that to crank the most out of buffer pool mgmt, we're getting closer and closer to the TLB itself. Fiddling with mmap & vm overcommit, pointer swizzling (or not), userfaultfd & custom page fault handling, even custom kernel drivers, etc. To really crank performance on in-memory and hybrid in-memory/disk systems -- why even bother with Linux then? Let's run direct on hypervisor! On the whole, DBs already manage their own persistent storage, so don't strictly need a filesystem (esp in the New Cloud World where pages often go up into S3 buckets etc); they manage their own memory; and often user accounts, etc too, and often managing their own concurrency. They're really an OS within an OS in many respects. Virtualization already handles abstracting things enough that drivers for network and disk etc aren't as big of benefit from the OS side. Security and monitoring can be handled at the per-VM level. We're no longer held back by the requirement to have to have a pile of drivers for different hardware configurations. At least in broad strokes. But I wouldn't start with Postgres as a base, that's for sure. If you're building enough of libc and a POSIX/Unix ABI that you can run stock programs, there's likely little benefit at all. I doubt you'd get much win for analytical workloads, but for very high throughput transactional workloads ... what fun! Let's go all the way baby! [1] https://db.in.tum.de/~leis/papers/leanstore.pdf https://db.in.tum.de/~leis/papers/leanstore.pdf [2] https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf [3] https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_uploads/vmcache.pdf https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...
- kjs3 3y ago
  Doesn't Oracle (and I think reaches waaaaay back in the memory banks DB2) do something similar in the sense that it is packaged up with its own VM/storage subsystems that bypasses the native OSes facilities and is hyper-optimized for the DB use case?
  - cmrdporcupine 3y ago
ComputerGuru 3y ago
I think Postgres could learn to write to a raw block devices (instead of over a fs) before jumping to this step.
jolux 3y ago
not sure how long it's going to take to get there but I think unikernels/library OS are the future. cloud computing is moving in the direction of lots of small tasks running in strong isolation and communicating over the network instead of via IPC and shared memory. there's a lot of surface area in traditional operating systems that is completely unused by most modern applications and just sits there consuming cycles and waiting to be attacked. it’s the tooling that’s lacking now. Whoever gets it right could make a lot of money.
- immibis 3y ago
  Unikernels with a hypervisor are isomorphic to processes with an operating system, and there are a lot more tools to deal with the latter.
  - jolux 3y ago
    “Isomorphic to” in what sense? Surely you’re not claiming process-level isolation is as strong as a virtual machine. There are similar roles in both but the abstractions are pretty different.
    - wazzaps 3y ago
      Why would processes under seccomp(2) be less isolated than VMs?

90 points by eyberg 3y ago

Unikernels are cool, but half this official post is pointlessly throwing rocks at postgres, the other half having to hack around to get this extremely common piece of software working with your own product. No thanks, I will get my unikernel fix from someone else. Plus, if you often have to LD_PRELOAD getuid, just add a bogus impl to your libc or as a syscall to reduce onboarding friction.

anonymous_sorry 3y ago
Is it throwing rocks? "I'm not knocking postgres here. You have to keep in mind [a paragraph of contextualising postgres's design decisions]"
- aljarry 3y ago
  The rant in the middle is very dismissive over voices against in the discussion, and that makes me agree with OP.
- giraffe_lady 3y ago
  Saying you're not doing something is not actually the same as not doing it.
  - richbell 3y ago
    I'm not disagreeing with you, but I think you're completely wrong about that.
eyberg 3y ago
If you think I was throwing rocks that most definitely was not the intention. The whole point of writing the post was to point out all the abstractions that don't quite gel well in unikernel land. The LD_PRELOAD wasn't used for this particular port but it does get used sometimes . We don't actually have real uid calls but some applications will require specific ones that we stub from time to time. Very very few apps actually require this.

dvdplm 3y ago

So what would be the reasons to run pg as a unikernel? To get max performance on bare metal? Smaller attack surface? Or?

FridgeSeal 3y ago
The less between your application and the hardware the better you can utilise it. There’d be basically no contention for resources, IO could go basically straight to the drive with less fsync-shenanigans etc.
- tpetry 3y ago
  But is the impact really noticeable? I believe it would then we would have seen more ready and stable solutions.
  - rwmj 3y ago
    With our Linux-based unikernel you can make small adjustments to the application -- usually just a handful of lines of code -- and get big performance improvements, like 25% or so. We had a paper about this at Eurosys'23: https://dl.acm.org/doi/abs/10.1145/3552326.3587458 https://dl.acm.org/doi/abs/10.1145/3552326.3587458
    - tpetry 3y ago
      That‘s the problem of unikernels. There are papers how great they are but almost nobody is using them. If applications could easily get a 25% boost without much work they would do it. But nobody is using it. So I am not trusting the „benefits“.
      - rwmj 3y ago
        There are a ton of well known techniques in computer science that will make software faster, more reliable or better in other ways, that people don't use, yet really work.
      - 3y ago

imiric 3y ago

I wonder why unikernels haven't caught on. Years ago when they started to be hyped, I imagined the technology would surpass the adoption of containers. In principle, they have the security and isolation of VMs which containers lack, but also have even less overhead than containers, and can run on bare metal. It sounds like the best of both worlds. The main issues years ago were developer tooling, packaging, and difficulty with debugging, so the UX was far from what Docker did for containers. But I was hoping that these issues would be addressed (are they?), and that a defacto standard would emerge that would propel them into the mainstream. Instead, we have these disparate niche tools and services that seem to be competing, but there's no widespread industry adoption. I'm sure that the evolution of the container ecosystem made unikernels less desirable, and the increased popularity of container orchestration engines like Kubernetes cemented this trend. But I wish that the unikernel ecosystem were more mature so that it's at least a topic of conversation in tech circles, and k8s wasn't such an obvious choice. Though I might be in the wrong circles and have outdated information, so I'm curious to know what HN thinks.

vbezhenar 3y ago
I feel like you’re mixing layers. Alternative to VM is Linux namespaces. They provide isolation. Docker, Kubernetes, LXC, systemd-spawn and whatnot are just thin wrappers around Linux API. There’s nothing that would prevent you from running workloads in VMs and managing them with Kubernetes. Just write proper adapters. Actually there are projects in the wild which do exactly that. As to why VM is supposed to be lighter than container, I don’t really understand. I think it’s the other way around. Industry does not really care about security and losing performance for unclear benefits won’t fly for masses.
- imiric 3y ago
  > Alternative to VM is Linux namespaces. They're really not. Containers (and by that I mean their underlying technology) are not alternatives to VMs, in the same way that unikernels are not alternatives to containers (yet, anyway). They're technologies that can be used for similar purposes, but in practice, they all have their tradeoffs that should be considered before use. For one, containers share the same kernel of the host machine. The isolation provided by VMs is on a deeper level, which also makes them more secure. Unikernels are more like VMs in this sense. My point is that the only drawback of unikernels in theory is the immature tooling and support around it. So I'm wondering if there is another reason that they haven't been as widely adopted as containers. > As to why VM is supposed to be lighter than container, I don’t really understand. You misunderstood. I said that unikernels, not VMs, have less overhead than containers. Containers require a minimal, but fully functioning, OS to work, while unikernels only require a cut-down version of a kernel. Whether unikernels are more lightweight is debatable, since containers are merely another process running on the same host OS with improved isolation, while unikernels depend on a hypervisor to work, but if we define "lightweight" to include boot time and performance, unikernels have certainly surpassed containers in this sense as well[1]. So the main benefit containers had over VMs is also an attribute of unikernels. [1]: https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2.pdf https://www.ietf.org/proceedings/95/slides/slides-95-nfvrg-2...

cmrdporcupine 3y ago

To me it's more sensible to consider writing a database from scratch for the unikernel. Or at least the storage engine portion of it. Plenty of recent DB research about running up against the wall of what the Linux VM subsystem can provide for them in terms of memory management. LeanStore[1], Umbra [2], and research [3] since then show that to crank the most out of buffer pool mgmt, we're getting closer and closer to the TLB itself. Fiddling with mmap & vm overcommit, pointer swizzling (or not), userfaultfd & custom page fault handling, even custom kernel drivers, etc. To really crank performance on in-memory and hybrid in-memory/disk systems -- why even bother with Linux then? Let's run direct on hypervisor! On the whole, DBs already manage their own persistent storage, so don't strictly need a filesystem (esp in the New Cloud World where pages often go up into S3 buckets etc); they manage their own memory; and often user accounts, etc too, and often managing their own concurrency. They're really an OS within an OS in many respects. Virtualization already handles abstracting things enough that drivers for network and disk etc aren't as big of benefit from the OS side. Security and monitoring can be handled at the per-VM level. We're no longer held back by the requirement to have to have a pile of drivers for different hardware configurations. At least in broad strokes. But I wouldn't start with Postgres as a base, that's for sure. If you're building enough of libc and a POSIX/Unix ABI that you can run stock programs, there's likely little benefit at all. I doubt you'd get much win for analytical workloads, but for very high throughput transactional workloads ... what fun! Let's go all the way baby! [1] https://db.in.tum.de/~leis/papers/leanstore.pdf https://db.in.tum.de/~leis/papers/leanstore.pdf [2] https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf [3] https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_uploads/vmcache.pdf https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...

kjs3 3y ago
Doesn't Oracle (and I think reaches waaaaay back in the memory banks DB2) do something similar in the sense that it is packaged up with its own VM/storage subsystems that bypasses the native OSes facilities and is hyper-optimized for the DB use case?
- cmrdporcupine 3y ago

ComputerGuru 3y ago

I think Postgres could learn to write to a raw block devices (instead of over a fs) before jumping to this step.

jolux 3y ago

not sure how long it's going to take to get there but I think unikernels/library OS are the future. cloud computing is moving in the direction of lots of small tasks running in strong isolation and communicating over the network instead of via IPC and shared memory. there's a lot of surface area in traditional operating systems that is completely unused by most modern applications and just sits there consuming cycles and waiting to be attacked. it’s the tooling that’s lacking now. Whoever gets it right could make a lot of money.

immibis 3y ago
Unikernels with a hypervisor are isomorphic to processes with an operating system, and there are a lot more tools to deal with the latter.
- jolux 3y ago
  “Isomorphic to” in what sense? Surely you’re not claiming process-level isolation is as strong as a virtual machine. There are similar roles in both but the abstractions are pretty different.
  - wazzaps 3y ago
    Why would processes under seccomp(2) be less isolated than VMs?