5 ms·

I can’t go into too much detail but: IME the benefits are somewhat doubtful and it can run into weird issues that are hard to debug (because C) at scale. I woul

by pm90 3y ago

I can’t go into too much detail but: IME the benefits are somewhat doubtful and it can run into weird issues that are hard to debug (because C) at scale. I wouldn’t use it unless I had a maintainer or a C expert on my team. The other OCI runtimes are written in go and are (generally) easier to debug.

yjftsjthsd-h 3y ago
Why is Go easier to debug? It kinda sounds like you're more familiar with one language than another and are basing your assessment of the tool on that.
- Gigachad 3y ago
  Less undefined behavior I would assume.
- winrid 3y ago
  Easier to track allocations and tie them to structures.
  - dilyevsky 3y ago
    Technically if you use jemalloc, which most everyone should do anyway, it comes with built-in instrumentation but you need to enable it compile time and generally not many are aware of this.
    - winrid 3y ago
      I work on many C/C++ projects and none of them use jemalloc. I also use several databases written in C, or C++, and none of them use jemalloc either. I do however wish Mongo offered the option: https://jira.mongodb.org/browse/SERVER-39325 https://jira.mongodb.org/browse/SERVER-39325
      - dilyevsky 3y ago
        I think tcmalloc will output protos too now which google pprof tool understands. If you’re using standard glibc malloc you’re probably leaving a lot of performance on the table
        winrid 3y ago
        Performance wise it doesn't matter for most projects like games where allocations aren't done in hot paths anyway. Problem with tcmalloc is fragmentation. Luckily I can then on aggressive decommit in some scenarios and that helps.
      - zX41ZdbW 3y ago
        ClickHouse uses jemalloc as the only option: https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/ We have contributed patches and bugfixes to jemalloc as well.
        winrid 3y ago
        I'd expect nothing less for clickhouse :)
- dilyevsky 3y ago
  Unless unsafe package is used which is rare or there’s a bug in the compiler which is rarer still you will not get silent memory corruption bugs which are stupid common in C and hard to debug to boot
  - throwaway173738 3y ago
    So your whole argument isn’t that crun itself is unstable but that C can be hard for non-experts to write programs in. That could be said of tons of stuff including the OS kernel you’re using to run containers, so I’d be curious to hear what you’re using to replace Linux in your work. The principal issue with Go applications in my work continues to be the massive size of the executables.
    - dilyevsky 3y ago
      I can't speak for crun specifically since I haven't used it personally but almost every single C application I've had to deal with had these hard to debug memory corruption bugs which required an expert (me) days/weeks to fix. And that included the Linux kernel itself btw, the difference being it was almost always fixed in the upstream by the time I found it. > The principal issue with Go applications in my work continues to be the massive size of the executables. That is certainly true in general case, however out of curiosity I went to see the size of containerd runtimes on my machine and: -rwxr-xr-x 1 1001 121 46889792 Sep 22 2022 /usr/local/bin/containerd -rwxr-xr-x 1 1001 121 9699328 Sep 22 2022 /usr/local/bin/containerd-shim-runc-v2 Does not seem that bad... (note that only the shim is launched per container, containerd process is a single systemd service)
      - throwaway173738 3y ago
        Some systems I’ve worked in have entire root filesystems that are merely twice the size of that one binary. And in those cases sending an extra 40 megabytes over the network connection is a big imposition. So there are still places where executable size matters and that’s why we’d want crun and not runc. If someone wrote an alternative in Rust I’d be interested but golang is just too piggy.
- jrockway 3y ago
  Compare these two programs: char foo[123]; int x; foo[124] = 10; and: var foo [123]byte var x int foo[124] = 10; In C, this surprisingly changes the value of x. (Well, it's undefined, so it could do anything!) In Go, the program crashes with the error that index 124 is out of bounds. C is the absolute best programming languages for programmers who don't make mistakes. I'm not one of them, and I've never met one of them. If it works for you, that's impressive!
  - linkdd 3y ago
    You know sanitizers and static analysis tools exist, and have existed for decades, and have been the basis for the work done in Rust and other "safe" languages? Also, a disciplined C programmer will always keep the size of the buffer near the buffer itself. For example, using something like this is perfectly safe (though, we can see the performance hit of those if-statements if used intensively): struct buffer { size_t size; char *data; }; void buffer_alloc(struct buffer *buf, size_t size) { if (buf != NULL) { buf->data = calloc(size, sizeof(char)); if (buf->data != NULL) { buf->size = size; } } } void buffer_free(struct buffer *buf) { if (buf != NULL && buf->data != NULL) { free(buf->data); buf->size = 0; } } bool buffer_read(struct buffer buf, size_t offset, char *dest) { if (dest != NULL) { if (buf.data != NULL && offset >= 0 && offset < buf.size) { *dest = buf.data[offset]; return true; } } return false; } int main() { struct buffer buf = {0}; buffer_alloc(&buf, 123); char c; buffer_read(buf, 124, &c); buffer_free(&buf); return 0; } It is possible to write safe C code, but it takes understanding of the language, sometimes of the compiler as well. The whole "There is no programmer who don't make mistakes" argument is fallacious at best. C is like a chainsaw, there are certain rules on how to use it safely. Yes you can cut limbs with a chainsaw, that does not make the chainsaw useless.
    - gigatexal 3y ago
      This comes up from time-to-time. Surely, there's some caveat here either for performance or other reasons (I'm not a solid C programmer enough to know the if this hypothesis is viable or not). If it was so simple this approach would be ubiquitous and C would be safe. What am I missing?
osigurdson 3y ago
>> hard to debug (because C) at scale I think what the crun author is positing is the container runtime is closer to the kernal (cgroups) than it is the orchestrator (Kubernetes). I tend to agree. Of course kernel / cgroups / container runtime need to be rock solid. There should be no need to debug these in most use cases.
re-thc 3y ago
There's also 1 in Rust: https://github.com/containers/youki https://github.com/containers/youki
chippiewill 3y ago
> I wouldn’t use it unless I had a maintainer or a C expert on my team I suppose you keep a Linux kernel dev or C expert on your team when you run on Linux too?

by pm90 3y ago

yjftsjthsd-h 3y ago

Why is Go easier to debug? It kinda sounds like you're more familiar with one language than another and are basing your assessment of the tool on that.

Gigachad 3y ago
Less undefined behavior I would assume.
winrid 3y ago
Easier to track allocations and tie them to structures.
- dilyevsky 3y ago
  Technically if you use jemalloc, which most everyone should do anyway, it comes with built-in instrumentation but you need to enable it compile time and generally not many are aware of this.
  - winrid 3y ago
    I work on many C/C++ projects and none of them use jemalloc. I also use several databases written in C, or C++, and none of them use jemalloc either. I do however wish Mongo offered the option: https://jira.mongodb.org/browse/SERVER-39325 https://jira.mongodb.org/browse/SERVER-39325
    - dilyevsky 3y ago
      I think tcmalloc will output protos too now which google pprof tool understands. If you’re using standard glibc malloc you’re probably leaving a lot of performance on the table
      - winrid 3y ago
        Performance wise it doesn't matter for most projects like games where allocations aren't done in hot paths anyway. Problem with tcmalloc is fragmentation. Luckily I can then on aggressive decommit in some scenarios and that helps.
    - zX41ZdbW 3y ago
      ClickHouse uses jemalloc as the only option: https://github.com/ClickHouse/ClickHouse/ https://github.com/ClickHouse/ClickHouse/ We have contributed patches and bugfixes to jemalloc as well.
      - winrid 3y ago
        I'd expect nothing less for clickhouse :)
dilyevsky 3y ago
Unless unsafe package is used which is rare or there’s a bug in the compiler which is rarer still you will not get silent memory corruption bugs which are stupid common in C and hard to debug to boot
- throwaway173738 3y ago
  So your whole argument isn’t that crun itself is unstable but that C can be hard for non-experts to write programs in. That could be said of tons of stuff including the OS kernel you’re using to run containers, so I’d be curious to hear what you’re using to replace Linux in your work. The principal issue with Go applications in my work continues to be the massive size of the executables.
  - dilyevsky 3y ago
    I can't speak for crun specifically since I haven't used it personally but almost every single C application I've had to deal with had these hard to debug memory corruption bugs which required an expert (me) days/weeks to fix. And that included the Linux kernel itself btw, the difference being it was almost always fixed in the upstream by the time I found it. > The principal issue with Go applications in my work continues to be the massive size of the executables. That is certainly true in general case, however out of curiosity I went to see the size of containerd runtimes on my machine and: -rwxr-xr-x 1 1001 121 46889792 Sep 22 2022 /usr/local/bin/containerd -rwxr-xr-x 1 1001 121 9699328 Sep 22 2022 /usr/local/bin/containerd-shim-runc-v2 Does not seem that bad... (note that only the shim is launched per container, containerd process is a single systemd service)
    - throwaway173738 3y ago
      Some systems I’ve worked in have entire root filesystems that are merely twice the size of that one binary. And in those cases sending an extra 40 megabytes over the network connection is a big imposition. So there are still places where executable size matters and that’s why we’d want crun and not runc. If someone wrote an alternative in Rust I’d be interested but golang is just too piggy.
jrockway 3y ago
Compare these two programs: char foo[123]; int x; foo[124] = 10; and: var foo [123]byte var x int foo[124] = 10; In C, this surprisingly changes the value of x. (Well, it's undefined, so it could do anything!) In Go, the program crashes with the error that index 124 is out of bounds. C is the absolute best programming languages for programmers who don't make mistakes. I'm not one of them, and I've never met one of them. If it works for you, that's impressive!
- linkdd 3y ago
  You know sanitizers and static analysis tools exist, and have existed for decades, and have been the basis for the work done in Rust and other "safe" languages? Also, a disciplined C programmer will always keep the size of the buffer near the buffer itself. For example, using something like this is perfectly safe (though, we can see the performance hit of those if-statements if used intensively): struct buffer { size_t size; char *data; }; void buffer_alloc(struct buffer *buf, size_t size) { if (buf != NULL) { buf->data = calloc(size, sizeof(char)); if (buf->data != NULL) { buf->size = size; } } } void buffer_free(struct buffer *buf) { if (buf != NULL && buf->data != NULL) { free(buf->data); buf->size = 0; } } bool buffer_read(struct buffer buf, size_t offset, char *dest) { if (dest != NULL) { if (buf.data != NULL && offset >= 0 && offset < buf.size) { *dest = buf.data[offset]; return true; } } return false; } int main() { struct buffer buf = {0}; buffer_alloc(&buf, 123); char c; buffer_read(buf, 124, &c); buffer_free(&buf); return 0; } It is possible to write safe C code, but it takes understanding of the language, sometimes of the compiler as well. The whole "There is no programmer who don't make mistakes" argument is fallacious at best. C is like a chainsaw, there are certain rules on how to use it safely. Yes you can cut limbs with a chainsaw, that does not make the chainsaw useless.
  - gigatexal 3y ago
    This comes up from time-to-time. Surely, there's some caveat here either for performance or other reasons (I'm not a solid C programmer enough to know the if this hypothesis is viable or not). If it was so simple this approach would be ubiquitous and C would be safe. What am I missing?

osigurdson 3y ago

>> hard to debug (because C) at scale I think what the crun author is positing is the container runtime is closer to the kernal (cgroups) than it is the orchestrator (Kubernetes). I tend to agree. Of course kernel / cgroups / container runtime need to be rock solid. There should be no need to debug these in most use cases.

re-thc 3y ago

There's also 1 in Rust: https://github.com/containers/youki https://github.com/containers/youki

chippiewill 3y ago

> I wouldn’t use it unless I had a maintainer or a C expert on my team I suppose you keep a Linux kernel dev or C expert on your team when you run on Linux too?