6 ms·

Using Zig as cross-platform C toolchain

196 points by insraq 5y ago

synergy20 5y ago
I'm almost sold, I checked zig briefly at its home page in the past and never used it. the cross platform build(similar to golang and rust) for c is very interesting. what about the mac-os(or linux,etc) native libraries on windows where you're building the demo code? The 60MB zig compiler and native clang on Windows will not have them, how is that done? Is it totally on clang side?
- _0w8t 5y ago
  Apple does not allow to use their SDK on non-Apple hardware. So if you need to use MacOS headers and libraries, the hardware where the compiler runs must be from Apple. Apple has no problem if you run Linux on that and cross-compile from it to Mac. At work we could not for that reason use the same compilation server setup to cross-compile for Linux, MacOS, Windows and ended up with a separated MacPro box for the compilation farm. EDIT: cross-compilation from Windows works after one gets Windows SDK and Visual Studio libraries (the later is not free, but free alternatives often are sufficient). The biggest headache is that Windows headers assume a case-insensitive file system and includes, for example, windows.h when the file is Windows.h. One can either use case-insensitive mount option for ext4 or symlink relevant files to the proper case.
  - smeenai 5y ago
    VS community edition is free (but not an option for commercial work). Another Clang-specific way to handle the case insensitivity is by making use of a virtual file system (VFS) overlay. https://github.com/llvm/llvm-project/blob/4976d1fe58f89169efaca565c02c5e424ce266cb/llvm/cmake/platforms/WinMsvc.cmake#L102-L131 https://github.com/llvm/llvm-project/blob/4976d1fe58f89169ef... demonstrates how that works. (There's no linker equivalent that I'm aware of, so you do have to use the symlinks or something like ciopfs for that; https://github.com/llvm/llvm-project/blob/4976d1fe58f89169efaca565c02c5e424ce266cb/llvm/cmake/platforms/WinMsvc.cmake#L133-L157 https://github.com/llvm/llvm-project/blob/4976d1fe58f89169ef... is an example of symlink generation.)
  - slimsag 5y ago
    Note however Zig actually provides libSystem.tbd stubs out of the box, so you can cross-compile from any OS to macOS too as long as you only depend on libSystem (the macOS libc.) If you need the rest of the Apple SDK, then you'd need a sysroot (and face the challenges you described.)
RodgerTheGreat 5y ago
TIL about clang's -fsanitize=undefined, and immediately discovered and corrected a handful of subtle bugs lurking in one of my projects. Thanks, Zig, and thanks, Clang.
- slimsag 5y ago
  Crazy how often this catches undefined behavior. Compiling GLFW with Zig we've found like ~4 separate UB issues, one I described here[0] Wish it could default to on with clang, but probably it'd break too many things. Really wonder how many severe issues we'd find just from turning this on by default everywhere, though. [0] https://devlog.hexops.com/2021/perfecting-glfw-for-zig-and-finding-undefined-behavior https://devlog.hexops.com/2021/perfecting-glfw-for-zig-and-f...
  - bregma 5y ago
    UBSAN adds a lot of runtime overhead. Turning it on my default would mean most software built with clang (or GCC, which has also had it for years) would suddenly be dog slow, like MSVCC in debug mode. Few developers have any idea how to use their toolchain correctly, so the defaults need to be chosen wisely.
  - a_e_k 5y ago
    I recently started testing a project with -fsanitize=integer as well. It'll report a few potential issues that -fsanitize=undefined won't, like left shifting a 1-bit off the end.
- brandmeyer 5y ago
  GCC supports many of the same sanitizers as well, since they were originally developed for GCC. We use GNU's ubsan and asan in our automated test suite.
extheat 5y ago
So basically a clang wrapper?
- faraaz98 5y ago
  At first glance, yeah probably. But the advantage with zig is that it has c interoperability and the concept around the programmable build system (build.zig)
- rbehrends 5y ago
  Its caching system is also smart enough to handle #include dependencies, so for simple C/C++ projects, it can also function as a make alternative (without having to write a Makefile). Plus, it makes cross-compilation really easy.
  - cpuguy83 5y ago
    Now that's interesting. This is something that breaks go's cache (for cgo).
- user-the-name 5y ago
  It's a clang wrapper that makes clang just work for cross-compilation. In addition to being its own programming language and build system as well.
haberman 5y ago
I recently learned that Clang supports this kind of cross-compiling out of the box. https://mcilloni.ovh/2021/02/09/cxx-cross-clang/ https://mcilloni.ovh/2021/02/09/cxx-cross-clang/ The main difference is that Clang does not ship with headers/libraries for different platforms, as Zig appears to do. You need to give Clang a "sysroot" -- a path that has the headers/libraries for the platform you want to compile for. If you create a bunch of sysroots for various architectures, you can do some pretty "easy" cross-compiling with just a single compiler binary. Docker can be a nice way of packaging up these sysroots (especially combined with Docker images like manylinux: https://github.com/pypa/manylinux https://github.com/pypa/manylinux). Gone are the days when you had to build a separate GCC cross-compiler for each platform you want to target.
- jcelerier 5y ago
  Yes, for instance for ossia score, the sequencer I'm working on which does some runtime compiling of c++ through clang & llvmjit, I ship a sdk for each platform with all the libc, libc++ headers. What a pain it was ! My scripts are the create-sdk-... Here if that can be useful to anyone: https://github.com/ossia/score/tree/master/ci https://github.com/ossia/score/tree/master/ci and the SDKs are available at https://github.com/ossia/sdk https://github.com/ossia/sdk
faraaz98 5y ago
I'm really liking the enthusiasm about zig these days. Kinda mirrors what first had when it was starting to become popular
emidoots 5y ago
"So, a clang wrapper?" is a common thought, so here's how Zig differs from clang out of the box: * Links MachO binaries for Apple Silicon via the custom zld linker it ships. LLVM cannot do this currently. * Provides (deduplicated) libc headers for pretty much every platform, including macOS and glibc/musl. https://github.com/ziglang/zig/tree/master/lib/libc/include https://github.com/ziglang/zig/tree/master/lib/libc/include * Provides a libc implementation (libSystem for macOS, musl and glibc, mingw for Windows, and WASI) * Deals with lots of the deep depths of hell, like enabling you to target any version of glibc out of the box by building symbol mappings: https://github.com/ziglang/glibc-abi-tool/ https://github.com/ziglang/glibc-abi-tool/ And that doesn't mention the most important part, IMO, which is that it lets you cross compile _out of the box_. No fiddling with sysroots, system packages, etc. to get a cross compiling toolchain working.
- cyber_kinetist 5y ago
  Does it also work with C++? If so, it seems like a very good alternative compared to the MSVC + clang-cl toolchain in Windows (which you need to install gigabytes of libraries before doing any development). Also, does it also work with build systems like CMake flawlessly? (I know that you can use Zig as a build system, but I would still want to leverage the ecosystem of C++ libraries compatible with CMake.)
  - afranchuk 5y ago
    Best to try it to figure it out. I tried compiling a trivial main function with `zig c++` (without cross compile) and the produced binary crashed when run. I think it's far more battle tested for C. But I look forward to C++ being improved as well.
  - nicoburns 5y ago
    I haven't actually used it myself, but I believe yes to both. It wraps clang, and while it's not quite as simple as "just a wrapper" it should be able to do anything that clang can do.
    - 5y ago
cube2222 5y ago
There's a great article about using Zig as a cross-platform C toolchain for compiling CGo, to bring easy cross-platform compilation to it, which is usually a pain: https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...
- shp0ngle 5y ago
  huh that’s… a great idea.
- deleted 5y ago
  [deleted]
nicoburns 5y ago
Zig's cross-compilation tooling and C interop is really impressive. I love to see projects that put so much emphasis on solving real-world practical problems like this and implement it so well. It makes me wonder whether cross-language tooling could be made to work more broadly. It would be awesome if you could have one compiler that would seamlessly compile zig/C/C++/Rust, and perhaps Fortran/ADA too.
githubholobeat 5y ago
"...an absolute joy to work with" I agree. I used zig to make a game in WASM and had a blast. I also had some spare Arduino Nanos laying around and so I build a simple firmware in zig to test it. It worked great, so you can use zig for your embedded projects as well.
jedisct1 5y ago
Zig can also target WebAssembly (standalone and WASI), so you don't need to download and install yet another LLVM toolchain, or fiddle with libclang_rt.builtins-wasm32.a.

196 points by insraq 5y ago

I'm almost sold, I checked zig briefly at its home page in the past and never used it. the cross platform build(similar to golang and rust) for c is very interesting. what about the mac-os(or linux,etc) native libraries on windows where you're building the demo code? The 60MB zig compiler and native clang on Windows will not have them, how is that done? Is it totally on clang side?

_0w8t 5y ago
Apple does not allow to use their SDK on non-Apple hardware. So if you need to use MacOS headers and libraries, the hardware where the compiler runs must be from Apple. Apple has no problem if you run Linux on that and cross-compile from it to Mac. At work we could not for that reason use the same compilation server setup to cross-compile for Linux, MacOS, Windows and ended up with a separated MacPro box for the compilation farm. EDIT: cross-compilation from Windows works after one gets Windows SDK and Visual Studio libraries (the later is not free, but free alternatives often are sufficient). The biggest headache is that Windows headers assume a case-insensitive file system and includes, for example, windows.h when the file is Windows.h. One can either use case-insensitive mount option for ext4 or symlink relevant files to the proper case.
- smeenai 5y ago
  VS community edition is free (but not an option for commercial work). Another Clang-specific way to handle the case insensitivity is by making use of a virtual file system (VFS) overlay. https://github.com/llvm/llvm-project/blob/4976d1fe58f89169efaca565c02c5e424ce266cb/llvm/cmake/platforms/WinMsvc.cmake#L102-L131 https://github.com/llvm/llvm-project/blob/4976d1fe58f89169ef... demonstrates how that works. (There's no linker equivalent that I'm aware of, so you do have to use the symlinks or something like ciopfs for that; https://github.com/llvm/llvm-project/blob/4976d1fe58f89169efaca565c02c5e424ce266cb/llvm/cmake/platforms/WinMsvc.cmake#L133-L157 https://github.com/llvm/llvm-project/blob/4976d1fe58f89169ef... is an example of symlink generation.)
- slimsag 5y ago
  Note however Zig actually provides libSystem.tbd stubs out of the box, so you can cross-compile from any OS to macOS too as long as you only depend on libSystem (the macOS libc.) If you need the rest of the Apple SDK, then you'd need a sysroot (and face the challenges you described.)

RodgerTheGreat 5y ago

TIL about clang's -fsanitize=undefined, and immediately discovered and corrected a handful of subtle bugs lurking in one of my projects. Thanks, Zig, and thanks, Clang.

slimsag 5y ago
Crazy how often this catches undefined behavior. Compiling GLFW with Zig we've found like ~4 separate UB issues, one I described here[0] Wish it could default to on with clang, but probably it'd break too many things. Really wonder how many severe issues we'd find just from turning this on by default everywhere, though. [0] https://devlog.hexops.com/2021/perfecting-glfw-for-zig-and-finding-undefined-behavior https://devlog.hexops.com/2021/perfecting-glfw-for-zig-and-f...
- bregma 5y ago
  UBSAN adds a lot of runtime overhead. Turning it on my default would mean most software built with clang (or GCC, which has also had it for years) would suddenly be dog slow, like MSVCC in debug mode. Few developers have any idea how to use their toolchain correctly, so the defaults need to be chosen wisely.
- a_e_k 5y ago
  I recently started testing a project with -fsanitize=integer as well. It'll report a few potential issues that -fsanitize=undefined won't, like left shifting a 1-bit off the end.
brandmeyer 5y ago
GCC supports many of the same sanitizers as well, since they were originally developed for GCC. We use GNU's ubsan and asan in our automated test suite.

extheat 5y ago

So basically a clang wrapper?

faraaz98 5y ago
At first glance, yeah probably. But the advantage with zig is that it has c interoperability and the concept around the programmable build system (build.zig)
rbehrends 5y ago
Its caching system is also smart enough to handle #include dependencies, so for simple C/C++ projects, it can also function as a make alternative (without having to write a Makefile). Plus, it makes cross-compilation really easy.
- cpuguy83 5y ago
  Now that's interesting. This is something that breaks go's cache (for cgo).
user-the-name 5y ago
It's a clang wrapper that makes clang just work for cross-compilation. In addition to being its own programming language and build system as well.

haberman 5y ago

I recently learned that Clang supports this kind of cross-compiling out of the box. https://mcilloni.ovh/2021/02/09/cxx-cross-clang/ https://mcilloni.ovh/2021/02/09/cxx-cross-clang/ The main difference is that Clang does not ship with headers/libraries for different platforms, as Zig appears to do. You need to give Clang a "sysroot" -- a path that has the headers/libraries for the platform you want to compile for. If you create a bunch of sysroots for various architectures, you can do some pretty "easy" cross-compiling with just a single compiler binary. Docker can be a nice way of packaging up these sysroots (especially combined with Docker images like manylinux: https://github.com/pypa/manylinux https://github.com/pypa/manylinux). Gone are the days when you had to build a separate GCC cross-compiler for each platform you want to target.

jcelerier 5y ago
Yes, for instance for ossia score, the sequencer I'm working on which does some runtime compiling of c++ through clang & llvmjit, I ship a sdk for each platform with all the libc, libc++ headers. What a pain it was ! My scripts are the create-sdk-... Here if that can be useful to anyone: https://github.com/ossia/score/tree/master/ci https://github.com/ossia/score/tree/master/ci and the SDKs are available at https://github.com/ossia/sdk https://github.com/ossia/sdk

faraaz98 5y ago

I'm really liking the enthusiasm about zig these days. Kinda mirrors what first had when it was starting to become popular

emidoots 5y ago

"So, a clang wrapper?" is a common thought, so here's how Zig differs from clang out of the box: * Links MachO binaries for Apple Silicon via the custom zld linker it ships. LLVM cannot do this currently. * Provides (deduplicated) libc headers for pretty much every platform, including macOS and glibc/musl. https://github.com/ziglang/zig/tree/master/lib/libc/include https://github.com/ziglang/zig/tree/master/lib/libc/include * Provides a libc implementation (libSystem for macOS, musl and glibc, mingw for Windows, and WASI) * Deals with lots of the deep depths of hell, like enabling you to target any version of glibc out of the box by building symbol mappings: https://github.com/ziglang/glibc-abi-tool/ https://github.com/ziglang/glibc-abi-tool/ And that doesn't mention the most important part, IMO, which is that it lets you cross compile _out of the box_. No fiddling with sysroots, system packages, etc. to get a cross compiling toolchain working.

cyber_kinetist 5y ago
Does it also work with C++? If so, it seems like a very good alternative compared to the MSVC + clang-cl toolchain in Windows (which you need to install gigabytes of libraries before doing any development). Also, does it also work with build systems like CMake flawlessly? (I know that you can use Zig as a build system, but I would still want to leverage the ecosystem of C++ libraries compatible with CMake.)
- afranchuk 5y ago
  Best to try it to figure it out. I tried compiling a trivial main function with `zig c++` (without cross compile) and the produced binary crashed when run. I think it's far more battle tested for C. But I look forward to C++ being improved as well.
- nicoburns 5y ago
  I haven't actually used it myself, but I believe yes to both. It wraps clang, and while it's not quite as simple as "just a wrapper" it should be able to do anything that clang can do.
  - 5y ago

cube2222 5y ago

There's a great article about using Zig as a cross-platform C toolchain for compiling CGo, to bring easy cross-platform compilation to it, which is usually a pain: https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...

shp0ngle 5y ago
huh that’s… a great idea.
deleted 5y ago
[deleted]

nicoburns 5y ago

Zig's cross-compilation tooling and C interop is really impressive. I love to see projects that put so much emphasis on solving real-world practical problems like this and implement it so well. It makes me wonder whether cross-language tooling could be made to work more broadly. It would be awesome if you could have one compiler that would seamlessly compile zig/C/C++/Rust, and perhaps Fortran/ADA too.

githubholobeat 5y ago

"...an absolute joy to work with" I agree. I used zig to make a game in WASM and had a blast. I also had some spare Arduino Nanos laying around and so I build a simple firmware in zig to test it. It worked great, so you can use zig for your embedded projects as well.

jedisct1 5y ago

Zig can also target WebAssembly (standalone and WASI), so you don't need to download and install yet another LLVM toolchain, or fiddle with libclang_rt.builtins-wasm32.a.