5 ms·

Some context: In master branch of Zig right now (before https://github.com/ziglang/zig/pull/10330 https://github.com/ziglang/zig/pull/10330 is merged), status

Some context: In master branch of Zig right now (before https://github.com/ziglang/zig/pull/10330 https://github.com/ziglang/zig/pull/10330 is merged), status quo: $ cat ../lib/libc/glibc/*.txt | wc -c 205219 $ cat ../lib/libc/glibc/*.txt | xz | wc -c 22976 Zig supports targeting every version of glibc for any target architecture. The information required to do this takes up 200 KB installation size / 22 KB tarball size, and it has the following problems (which are causing several bugs): * fails to represent symbols that migrate from one library to another between glibc versions * fails to represent a distinction between functions and objects * fails to represent object sizes With this new glibc-abi-tool: $ cat abilists | wc -c 169421 $ cat abilists | xz | wc -c 24880 165 KB installation size / 24 KB tarball size, and all the above issues are solved. It also should be slightly faster to load/parse for the compiler since it is fewer KB to load from disk. The code that uses this file is here: https://github.com/ziglang/zig/blob/0d7331b6f6af85bd45829514668075c8a9910019/src/glibc.zig#L681-L1010 https://github.com/ziglang/zig/blob/0d7331b6f6af85bd45829514... It creates c.s, pthread.s, dl.s, etc., which are assembled into libc.so, libpthread.so, libdl.so, etc., which are placed on the linker line when cross compiling. Thanks to this improved dataset, they can now accurately model the glibc that will be on any CPU architecture, any glibc version. If I just naively shipped every version of the .abilist files: $ cat (find glibc/ -name "*.abilist") | wc -c 37041906 $ cat (find glibc/ -name "*.abilist") | xz | wc -c 205036 ...it would be 35 MiB installation size, 200 KB tarball size. So I have effectively achieved a compression ratio of 219:1 by implementing a bespoke encoding of this information.

formerly_proven 5y ago
Totally off-topic: I installed Zig on Saturday afternoon and by the evening I had my first program do some useful stuff, on Sunday I added some C code, then added some tests and by Sunday evening I had a pure Zig project again (and afterwards learned that translate-to-zig is a thing). I gotta say I'm very impressed with Zig and partial evaluation is just such a nice, mentally simple technique for generics and lots of other things. The syntax is pretty nice, too, though the prefix-[] are kinda weird (just way too used to C/C++'s postfix array notation) and ranged loops are somewhat awkward. The @ naming for builtins confused me for a bit because I thought the @ itself was syntax. And array literals are kinda weird sometimes though I've seen that apparently in 0.9 "&[_][]const u8{ "-std=c99", "-O3" }" can just be written as ".{ "-std=c99", "-O3" }". And I was kinda surprised about array types in function signatures being implicitly const, I think they're like that to make array types behave mostly like value types while only passing a pointer?
- dnautics 5y ago
  > the prefix-[] are kinda weird (just way too used to C/C++'s postfix array notation) I think over time the C/C++ way is going to go the way of the dodo; IIRC, rust and go both use the prefix-[]. It creates a clear, unambiguous way of defining the type (no "spiral typing" jokes for the modern languages).
  - nyanpasu64 5y ago
    I wish Rust used prefix &[]i32 for a slice, and [5][2]i32 for a 5x2 array. Instead it uses &[i32] for a slice, and [[i32; 2]; 5] for a 5x2 array. Reading types backwards is alive and well.
    - dnautics 5y ago
      eh, the type surrounded by the [] "operator" is equally unambiguous as prefix-[], so it's not the worst.
    - mmastrac 5y ago
      I prefer the unambiguous nesting of Rust's style. I can never remember the array shape when array bounds are listed in either prefix or postfix style. It's a tiny bit of extra typing that ensures that my understanding of the code is always 100% correct.
      - nyanpasu64 5y ago
        I find Rust's array style more confusing than prefix, since prefix matches the order you index into the arrays (arr: [5][2]i32 can be validly indexed as arr[<5][<2]), and Rust's style is written backwards from how you index the array. Can you clarify how you find Rust unambiguous?
        mmastrac 5y ago
        It's just obvious to me that it works the correct way by way of elimination of alternative parsing. The example below can be unwrapped mentally and unambiguously as "an array of 5 items of (an array of 2 items of i32)". Indexing into the array is not backwards: the outer array is deref'd first, giving you one of those five items which has length two. fn main() { let mut a: [[i32; 2]; 5] = [[0,0],[0,0],[0,0],[0,0],[0,0]]; println!("{:?}", a[0]); } Note that the Rust array declarations are also homologous to other types in the ecosystem, such as nested Option types or nested collections (eg: Option<Option<T>> or List<List<T>>). Contrast w/the C version, where it's obvious to a skilled C programmer but you need to remember to read the bounds from right to left to determine the memory layout (ie: an array of 5 arrays of two elements each): int main() { int i[5][2] = {{1,2},{3,4}, /*...*/}; printf("%ld\n", sizeof(i[0])/sizeof(int)); return 0; } In this case you are required to mentally combine the type on the left of the variable with the bounds on the right of the variable to parse a type declaration in your head that eventually looks like the Rust version: (int[2])[5] I've been doing a lot of professional C work lately so it's currently "swapped in", but I find that any time I pick C up after spending time away I find it ambiguous and I have to reason through the ordering of the declarations. Obviously this is all IMHO and YMMV, but this developer does find it significantly easier to unambiguously parse Rust types.
lhorie 5y ago
How much work is it to then maintain this bespoke encoding? Without having much context on it, it sounds similar to having to maintain unicode identifier character tables for unicode-aware parsers (e.g. Javascript's). Updating unicode tables after unicode version releases can be done w/ somewhat hack-ish scripting of parsing txt spec files but it's admittedly not super clean, I'm curious what's involved when dealing w/ abi lists over time.
- Spex_guy 5y ago
  The process of adding new versions of glibc to the data set is described in the readme. If something changes in a way the format doesn't handle, we can just change it and update the code in the compiler at the same time. The file ships as part of the compiler distribution so we don't have to worry about backwards compatibility.
gavinray 5y ago
Can I use this incredible feature with other toolchains, or existing artifacts that have a glibc dependency? Say if I am using a language/toolchain which pulls in a C/C++ compiler, are you able to substitute "zig cc" there and explicitly cross-compile to a different GLIBC version?
- AndyKelley 5y ago
  Yep! Example: https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...
  - gavinray 5y ago
    Incredible, thanks a ton for doing this work!

formerly_proven 5y ago

Totally off-topic: I installed Zig on Saturday afternoon and by the evening I had my first program do some useful stuff, on Sunday I added some C code, then added some tests and by Sunday evening I had a pure Zig project again (and afterwards learned that translate-to-zig is a thing). I gotta say I'm very impressed with Zig and partial evaluation is just such a nice, mentally simple technique for generics and lots of other things. The syntax is pretty nice, too, though the prefix-[] are kinda weird (just way too used to C/C++'s postfix array notation) and ranged loops are somewhat awkward. The @ naming for builtins confused me for a bit because I thought the @ itself was syntax. And array literals are kinda weird sometimes though I've seen that apparently in 0.9 "&[_][]const u8{ "-std=c99", "-O3" }" can just be written as ".{ "-std=c99", "-O3" }". And I was kinda surprised about array types in function signatures being implicitly const, I think they're like that to make array types behave mostly like value types while only passing a pointer?

dnautics 5y ago
> the prefix-[] are kinda weird (just way too used to C/C++'s postfix array notation) I think over time the C/C++ way is going to go the way of the dodo; IIRC, rust and go both use the prefix-[]. It creates a clear, unambiguous way of defining the type (no "spiral typing" jokes for the modern languages).
- nyanpasu64 5y ago
  I wish Rust used prefix &[]i32 for a slice, and [5][2]i32 for a 5x2 array. Instead it uses &[i32] for a slice, and [[i32; 2]; 5] for a 5x2 array. Reading types backwards is alive and well.
  - dnautics 5y ago
    eh, the type surrounded by the [] "operator" is equally unambiguous as prefix-[], so it's not the worst.
  - mmastrac 5y ago
    I prefer the unambiguous nesting of Rust's style. I can never remember the array shape when array bounds are listed in either prefix or postfix style. It's a tiny bit of extra typing that ensures that my understanding of the code is always 100% correct.
    - nyanpasu64 5y ago
      I find Rust's array style more confusing than prefix, since prefix matches the order you index into the arrays (arr: [5][2]i32 can be validly indexed as arr[<5][<2]), and Rust's style is written backwards from how you index the array. Can you clarify how you find Rust unambiguous?
      - mmastrac 5y ago
        It's just obvious to me that it works the correct way by way of elimination of alternative parsing. The example below can be unwrapped mentally and unambiguously as "an array of 5 items of (an array of 2 items of i32)". Indexing into the array is not backwards: the outer array is deref'd first, giving you one of those five items which has length two. fn main() { let mut a: [[i32; 2]; 5] = [[0,0],[0,0],[0,0],[0,0],[0,0]]; println!("{:?}", a[0]); } Note that the Rust array declarations are also homologous to other types in the ecosystem, such as nested Option types or nested collections (eg: Option<Option<T>> or List<List<T>>). Contrast w/the C version, where it's obvious to a skilled C programmer but you need to remember to read the bounds from right to left to determine the memory layout (ie: an array of 5 arrays of two elements each): int main() { int i[5][2] = {{1,2},{3,4}, /*...*/}; printf("%ld\n", sizeof(i[0])/sizeof(int)); return 0; } In this case you are required to mentally combine the type on the left of the variable with the bounds on the right of the variable to parse a type declaration in your head that eventually looks like the Rust version: (int[2])[5] I've been doing a lot of professional C work lately so it's currently "swapped in", but I find that any time I pick C up after spending time away I find it ambiguous and I have to reason through the ordering of the declarations. Obviously this is all IMHO and YMMV, but this developer does find it significantly easier to unambiguously parse Rust types.

lhorie 5y ago

How much work is it to then maintain this bespoke encoding? Without having much context on it, it sounds similar to having to maintain unicode identifier character tables for unicode-aware parsers (e.g. Javascript's). Updating unicode tables after unicode version releases can be done w/ somewhat hack-ish scripting of parsing txt spec files but it's admittedly not super clean, I'm curious what's involved when dealing w/ abi lists over time.

Spex_guy 5y ago
The process of adding new versions of glibc to the data set is described in the readme. If something changes in a way the format doesn't handle, we can just change it and update the code in the compiler at the same time. The file ships as part of the compiler distribution so we don't have to worry about backwards compatibility.

gavinray 5y ago

Can I use this incredible feature with other toolchains, or existing artifacts that have a glibc dependency? Say if I am using a language/toolchain which pulls in a C/C++ compiler, are you able to substitute "zig cc" there and explicitly cross-compile to a different GLIBC version?

AndyKelley 5y ago
Yep! Example: https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...
- gavinray 5y ago
  Incredible, thanks a ton for doing this work!