10 ms·

Great idea - let's rewrite the most security-critical piece of container tech in plain C! Red Hat literally took a memory safe Go program and rewrote it in C f

• coder543 6y ago
  Yeah. I just don’t think “runc” has ever been a bottleneck in any systems I’ve worked with. It’s been years since I’ve been able to get excited about rewriting systems from a memory safe language into a memory unsafe language for “performance reasons”. As an industry, we have too much evidence that even the best humans make mistakes. Quality C codebases still consistently run into CVEs that a memory safe language would have prevented. Rust exists, if performance is so critical here and if Go were somehow at fault... but it sounds like most of the performance difference is due to a different architecture, so rewriting runc in Go probably could have had the same outcome. I wish I could be excited about crun... I’m sure the author has put a lot of effort into it. Instead, I’m more excited about things like Firecracker or gVisor that enable people to further isolate containers, even though such a thing naturally comes with some performance impact.
  • lima 6y ago
    > I wish I could be excited about crun... I’m sure the author has put a lot of effort into it. Instead, I’m more excited about things like Firecracker that attempt to enable people to further isolate containers, even though such a thing naturally comes with some performance impact. gVisor is also very exciting due to how it handles memory allocation and scheduling. Syscalls and I/O are more expensive, though.
    • coder543 6y ago
      Heh, I actually edited gVisor in right before your reply. I agree completely!
  • SEJeff 6y ago
    “Any of the systems you’ve worked with” I suspect it has been a bottleneck in some of the systems redhat has worked with, or they would have no benefit in writing it. As the authors of openshift I’m sure they’ve seen some pretty crazy use cases. Just because it isn’t your use case doesn’t make it invalid.
    • deleted 6y ago
      [deleted]
• SEJeff 6y ago
  It is almost 50% faster for launching containers. That’s considerable, and at scale could be an actual cost savings. If nothing else, efficiency is good. Also, small c projects can be written properly with discipline. This is a very talented team on a very small and well scoped project. It can be written properly.
  • jnwatson 6y ago
    There's also the copious amount of static analysis and warnings built into gcc and clang these days, not to mention safer build options. Writing C today is quite a bit better than even 10 years ago.
    • SEJeff 6y ago
      Indeed. Pretending it isn’t is equally silly.
    • lima 6y ago
      Better, yes. Safe, no. Even with ASAN and modern C++, vulnerabilities happen all the time.
      • SEJeff 6y ago
        This is C, not C++. Modern C++ is kind of a mess with how much stuff they’ve added to the language. This is not relevant.
        • MaxBarraclough 6y ago
          The Linux kernel still has memory safety issues to this day. The C++ approach is to offer a feature-rich language so the programmer doesn't have to reinvent common abstractions. The C approach is to offer a minimal and stable language and let the programmer take it from there. It's not obvious a priori which approach should result in fewer memory safety issues. If I had to guess my money would be on C++ being the better choice, as it has smart pointers.
        • 6y ago
• zapita 6y ago
  It’s not for performance. It’s to settle an old grudge with Docker. All code coming from Docker must be erased. See also: podman; cri-o; buildah.
  • SEJeff 6y ago
    Grudge? You mean when Jessie Frazelle walked around dockercon with a badge saying, “I will not merge patches for systemd” just out of principal? In hindsight it is fairly obvious picking a fight at redhat summit was a very poor long term strategy from docker. It forced them to ultimately sell the orchestration parts of their business and massively limit the free tier of docker hub.
    • zapita 6y ago
      Thank you for illustrating my point perfectly. “Grudge?” (proceeds with details of grudge).
      • kbenson 6y ago
        What was described could be the basis of a grudge, or it could be that Red Hat were convinced that the current project would not accept changes Red Hat would benefit by even if it helped many other people. At that point, and at the size of Red Hat and how important this technology is becoming for them, it only makes sense for them to secure some of the technological foundations they rely on. It costs them very little in resources in relation to the business they are basing on that technology. I don't think it's a stretch to look at the events as described and see them as Red Hat doing what they need to work around someone else's grudge. It could be any number of things, but I don't think the information presented so far lends itself towards Red Hat having a grudge.
        • zapita 6y ago
          runc, containerd and the docker client are critical infrastructure to a whole lot of people, and they seem to be doing just fine running it and, when needed, contributing back. Only Red Hat seems to have a problem with using those projects without forking or re-writing them. Could it be because Red Hat considers it of strategic importance for their business, not just for those projects to be reliable (they are), but to be controlled by them to a sufficient degree that they can act as gatekeepers to upstream, thus preserving their business model?
• vishvananda 6y ago
  Go is actually a really poor choice for the container runtime because much of the container setup cannot be done from multithreaded code[0], so it has to be done in C before the go runtime initializes. I do think rust is a better choice for this layer than C because there are still security risks, but getting rid of Go for this layer is a win. I'm not sure why RH chose to rewrite it in C rather than using rust[1]. [0]: https://www.weave.works/blog/linux-namespaces-and-go-don-t-mix https://www.weave.works/blog/linux-namespaces-and-go-don-t-m... [1]: https://github.com/drahnr/railcar https://github.com/drahnr/railcar
  • SEJeff 6y ago
    This I entirely agree with you on but I’d expect it due to the resources available at the time were more fluent with C than rust. A shame really.
    • Brian_K_White 6y ago
      Or they want to be more platform-agnostic than rust and llvm.
      • coder543 6y ago
        Which platforms does Red Hat support Linux containers on that Rust doesn't support? Which platforms is anyone else frequently using Linux containers on that Rust doesn't support? I'm pretty sure the venn diagram here would be a single circle. Rust supports the major architectures. Portability seems like an unlikely reason here, but I would pick memory safety over portability every single day of the week. Rust is far from the only language that offers memory safety, but it is certainly one of the most compelling ones for projects that would otherwise be written in C. "runc" is battle tested, written almost entirely in Go, which offers memory safety, and the performance delta under discussion is only about 2x for a part of the system that seemingly only has a measurable performance impact if you're starting containers as quickly as possible. This just isn't a very compelling use case for C.
        • 6y ago
• 8fingerlouie 6y ago
  Contrary to popular belief, memory safe C is possible to write, and with modern linters it’s even easier. Yes it’s easier to shoot yourself in the foot, but with practice you can learn to miss every time.
  • tshtf 6y ago
    It probably isn't worth trying. djb wrote remotely exploitable C for qmail, applications written by the OpenBSD team have had exploitable memory issues, Microsoft still doesn't get it right, neither do Linux kernel devs.
    • CivBase 6y ago
      You say that as if the likes of Microsoft or Linux kernel devs consist exclusively of elite C devs. I'm very glad languages like Go and Rust exist, but saying you shouldn't write C because you might create memory leaks is kind of like saying you shouldn't write multi-threaded code because you might create race conditions. Yeah, it adds complexity to your code, but it's sometimes worth the overhead it saves. Whether that trade-off is worth it is always up for debate.
      • tshtf 6y ago
        Oh, I agree it is a tradeoff. But the parent said "you can learn to miss every time". Can anyone point me to a C language project where the developers consistently miss every time?
        • CivBase 6y ago
          Are you saying that every project written in C has suffered from a memory leak issue at some point? Most C/C++ code I've personally written doesn't even use the heap. I work on avionics and we use C/C++ now and then. We have a ton of rules regarding memory management (pretty much everything stays on the stack) and I can't recall anything I've ever been involved with suffering from a memory leak.
• deleted 6y ago
  [deleted]
• gobblegobble2 6y ago
  The most security-critical part of container tech lives in Linux kernel and has always been in plain C.
• hda111 6y ago
  Most security software is written in C. WireGuard, OpenSSH, OpenSSL, OpenVPN, all the Software from OpenBSD folks. Crun is being used without root privilege in most cases so it’s not a huge problem if the runtime is ever escaped.
• kragniz 6y ago
  Remember that runc contains quite a few lines of C: https://github.com/opencontainers/runc/blob/4d4d19ce528ac40cc357ef92cd3a6931dba19316/libcontainer/nsenter/nsexec.c https://github.com/opencontainers/runc/blob/4d4d19ce528ac40c... https://github.com/opencontainers/runc/blob/4d4d19ce528ac40cc357ef92cd3a6931dba19316/libcontainer/nsenter/cloned_binary.c https://github.com/opencontainers/runc/blob/4d4d19ce528ac40c...
  • BenTheElder 6y ago
    Is what appears to be less than 2,000 lines including detailed comments etc. "quite a lot"? This seems like a manageable amount to very carefully maintain and not at all like writing something entirely in C.
• deleted 6y ago
  [deleted]
• xerxes901 6y ago
  I did some hacking on crun this week to fix a bug, so maybe I can offer a perspective about why it’s written in C. A huge part of a container runtime is pretty much just issuing Linux syscalls - and often the new ones too, that don’t have any glibc bindings yet and need to be called by number. Plus, very careful control of threading is needed, and it’s not just a case of calling runtime.LockOSThread() in go - some of these namespace related syscalls can only be called if there is a single thread in the process. It’s _possible_ to do this in go, of course; runc exists after all. But it’s certainly very convenient to be able to flip back and forth between kernel source and the crun code base, to use exactly the actual structs that are defined in the kernel headers, and to generally be using the lingua franca of interfacing with the Linux kernel. It lowers cognitive overhead (eg what is the runtime going to do here?) and lets you focus on making exactly the calls to the kernel that you want.
• cpuguy83 6y ago
  To be fair, much of runc is written in C as well. And go is really terrible specifically for the case of runc.