11 ms·
Zig is now self–hosted by default
- AndyKelley 4y agoTo clarify what this means, since things can get confusing with subtle differences in wordings: * stage1 is when zig is built from C++ code[1] (the "bootstrap" compiler) using system C/C++ compiler toolchain. * stage2 is when zig is built from Zig code[2] using stage1. * stage3 is when zig is rebuilt from the same Zig code[2] using stage2. Before today, zig would give you stage1 by default, and you could opt in to stage2 using `-fno-stage1`. After today, zig gives you stage3 by default, and you can opt in to stage1 using `-fstage1`. In all three cases, LLVM is being used. Although Zig has started to fully self-host by providing backends that have no dependency on LLVM, none of these fully self-hosted backends are complete. Why bother self-hosting? Because: * Zig produces smaller, faster binaries than C++ that use less memory[4]. Case in point: the new self-hosted compiler is 1.5x faster than the C++ implementation and uses 3x less peak RAM. * Development velocity in Zig is much faster than C++, and debugging is a breeze in comparison due to Zig being much safer than C++ and having better debug tooling. Similarly, comptime features let us add more assertions and debug checks that are not possible in C++. * Zig compiles much faster than C++. * Zig cross compiles better than C++ making it easier to create builds of the compiler for every target. * The self-hosted compiler does not need a softfloat library dependency to support f16 or f128 operations. * Zig's std lib data structures are incredibly useful, especially compared to the C++ STL. There are still many known bugs; not every project will be able to upgrade immediately. See the full upgrade guide [3] for help deciding when and how to upgrade. [1]: https://github.com/ziglang/zig/tree/master/src/stage1 https://github.com/ziglang/zig/tree/master/src/stage1 [2]: https://github.com/ziglang/zig/tree/master/src https://github.com/ziglang/zig/tree/master/src [3]: https://github.com/ziglang/zig/wiki/Self-Hosted-Compiler-Upgrade-Guide/ https://github.com/ziglang/zig/wiki/Self-Hosted-Compiler-Upg... [4]: https://media.handmade-seattle.com/practical-data-oriented-design/ https://media.handmade-seattle.com/practical-data-oriented-d...
- kuon 4y agoCongratulations. This is an important milestone.
- childintime 4y agoFor Andrew it must feel he's now on the home stretch towards 1.0, finally!
- ksec 4y agoIn a previous presentation I think he mentions they are still aiming at something like 2025.
- lewurm 4y agoThanks for the context! What's the difference between the stage2 and stage3 binary? Does stage1 produce different binaries for the same input compared to stage2/stage3?
- amaranth 4y agoIdeally there should be no difference and building stage 3 is basically a sanity check to ensure the compiler is working correctly.
- AndyKelley 4y agoNot quite - what you said is true for a hypothetical "stage4" however there is a distinct difference between stage2 and stage3. While they are built from the same source code, and therefore have the same logic, they are lowered by different backends, meaning they will have potentially drastically different performance characteristics depending on the differences between the stage1 and stage2 backend, respectively. Related: https://github.com/ziglang/zig/issues/12183 https://github.com/ziglang/zig/issues/12183
- llimllib 4y ago> having better debug tooling What debug tooling are you referring to? It seems like people just use lldb to debug zig programs? I'm honestly interested, and searching doesn't turn up good tools as far as I can tell.
- cturtle 4y agoMaybe the runtime safety in debug builds? Like for out of bounds, int overflows, etc. In my zig projects those checks have saved me from using a debugger or valgrind in many cases.
- llimllib 4y agoI guess I don't call runtime safety features "debug tooling"? But maybe that's the disconnect here
- klyrs 4y agoI'd call tools like ASAN and friends debug tooling. I'm not sure if zig has tooling that doesn't have an analogue in the c++ world, but having them built in to the compiler by default is a notable feature.
- tialaramex 4y agoRight, defaults matter. The fact sanitizers exist is much less of a game changer than the default behaviour being sanitized. Because C++ defaults are notoriously all wrong, in programming language design you could usually do worse than consider, "Is there an alternative to what C++ does here by default?" and if there is one, choose that alternative as your default because it's more likely to be correct. Sometimes C++ helps you out, it might have a keyword "do_it_right" and you can just default to that, if you feel people might want the C++ default behaviour, feel free to reserve "do_it_wrong" in case people wanted that, but in many cases they will never ask you to implement the C++ behaviour because it's just wrong. Examples: "const" in C++ doesn't mean "constant" it means "immutable", so, make immutable your default and offer "mutable" as the option. "explicit" in C++ makes constructors need an explicit cast to be used for conversion, once again of course you want that by default but feel free to reserve "implicit" in case some people are sure they need the C++ behaviour.
- conaclos 4y ago> Case in point: the new self-hosted compiler is 1.5x faster than the C++ implementation and uses 3x less peak RAM. I am not sure it is comparable: the self-hosted compiler uses a different architecture [1]. [1] https://vimeo.com/649009599/7f1fda43ff https://vimeo.com/649009599/7f1fda43ff
- Laremere 4y agoThis is true, but I don't think your point entirely fair either. Zig's design lends itself to writing this style of code. In contrast the usual way(s) C++ is used encourages code that makes these types of optimizations hard/impossible. In practical terms, if you wanted to write fast software, the ergonomics of the language allowing you to use the desired architecture matter a lot. (Also, you're replying to the primary author of Zig and presenter of that talk, if you weren't aware.)
- kaba0 4y agoWhile ergonomics matter, I don’t believe the question is settled at all - for very high performance applications C++ is the answer for a long time to come.
- chris2860 4y agoHi Andrew, sorry for the unrelated question but where can we find the financial reports of the Zig Software Foundation for 2022? The Finances spreadsheet hasn't been updated for the past half a year: https://docs.google.com/spreadsheets/d/14_ljFHGFXY5NhBhlfjgkO0RZHqeVv04fAmCxw3ZusYc/edit#gid=99104331 https://docs.google.com/spreadsheets/d/14_ljFHGFXY5NhBhlfjgk...
- killingtime74 4y agoIsn’t it quite common to only do financial reports only every quarter, 6 months, yearly?
- dilap 4y ago> * Zig's std lib data structures are incredibly useful, especially compared to the C++ STL. I think elaborating on this could make for a very interesting article. :-) (I have a guess already at one item, which is MultiArrayList, something that kind of blew my mind when I first learned about it, and is in my opinion a very impressive testament to both the power and ease of use of Zig's approach to compile-time computation.)
- sekao 4y agoI didn't even know about it until today, very impressive. Generic SoA in less than 500 lines of userland code... https://github.com/ziglang/zig/blob/master/lib/std/multi_array_list.zig https://github.com/ziglang/zig/blob/master/lib/std/multi_arr...
- jcelerier 4y agoSoA in C++ isn't that long, here's my version (which for sure does not have the utility functions defined here though, but is useful enough for me): https://github.com/celtera/ahsohtoa/blob/main/include/ahsohtoa/ahsohtoa.hpp https://github.com/celtera/ahsohtoa/blob/main/include/ahsoht...
- dilap 4y agoThat's pretty sweet, I didn't know C++ could do this now! Zig's still got it beat handily in simplicity though -- I think what's really cool about Zig's idea of comp-time evaluation is you get something like the power of C++ with something like the simplicity of C.
- labrador 4y ago> Zig produces smaller, faster binaries than C++ This seems fairly significant to me and makes me wonder how this is possible with all the effort that has gone into optimizing C++
- kristoff_it 4y agoright after that sentence there's a link to a talk that explains one way this is true
- labrador 4y agoWe can hardly be expected to watch a 46 minute video on "A Practical Guide to Applying Data-Oriented Design" to find our answer. It seems unpromising on the surface.
- JonChesterfield 4y agoData oriented design is why the thing is faster.
- labrador 4y agoI guess I'll wait for the benchmarks comparing C++ and Zig over a variety of problem solutions to see which one is faster
- rk06 4y agoBecause they can't do breaking changes and must live with legacy baggage. While zig can work use modern techniques without such issues
- labrador 4y agoThis is wrong because you can strip out C++ code you don't need. Green code doesn't need to worry about legacy baggage
- christophilus 4y ago> Memory usage is improved by a factor of about 3x. For Zig, building itself went from using 9.1 GiB to 2.7 GiB. Pretty impressive.
- mhd 4y agoHow many processes does this involve, i.e. `make -jX` for which X?
- dralley 4y agoAndrew did a nice talk on the kinds of modifications that were necessary to get those improvements https://vimeo.com/649009599 https://vimeo.com/649009599
- diffxx 4y ago
- dom96 4y agoNot bad. For comparison Nim builds itself using just 668MiB.
- deleted 4y ago[deleted]
- AndyKelley 4y agoNice work! I suspect the difference mainly comes down to the fact that Zig compiles everything into a single compilation unit. As you can see, this has tradeoffs. It produces better code (single-compilation-unit is what "LTO" approximates) but it requires more memory and more code needs to be rebuilt on changes.
- tomcam 4y agoThat is stunning
- endgame 4y agoVery exciting. Are there plans to maintain the bootstrap path over the longer term? It always makes me sad when I hear about compilers which must bootstrap from magic binary blobs.
- anonymoushn 4y agoI hope so, but it seems like even if they didn't maintain it you could build today's stage1 using LLVM 14 and then work from the resulting binary instead of a binary you downloaded from online.
- hiccuphippo 4y agoThey have a project for maintaining this: https://github.com/ziglang/zig-bootstrap https://github.com/ziglang/zig-bootstrap
- Schroedingersat 4y agoI believe the goal is to replace the C++ with a C bootstrap that is initially auto generated from the zig code by zig but manually cleaned up and maintained to match
- pabs3 4y agoStarting from auto-generated code isn't considered a proper bootstrap process by the Bootstrappable Builds project. You need to be able to start without any binaries or auto-generated code from the project itself. Usually that would mean starting with a basic implementation in another language, but starting with an older version of the same project that was written in another language and then going through several older versions of different milestones is often easier than reimplementing the language from scratch in another language. https://bootstrappable.org/ https://bootstrappable.org/
- Schroedingersat 4y agoPlease read comments before responding
- overflyer 4y agoOh boi, oh boi, oh boi, oh boi, this is a historic moment. I think at the current rate I will start focusing my energy on learning and using Zig when version 0.12.0 is out :) Awesome work guys. I am so looking forward to this language!
- anonymoushn 4y agoYou can use it today!
- badpun 4y agoIs there any IDE support? Any debugger?
- _trackno5 4y agoany debugger support DWARF should work just fine. Also imagine it's the same on Windows. Anything that can manage a PDB file should work
- hiccuphippo 4y agoThere's a Language Server: https://github.com/zigtools/zls https://github.com/zigtools/zls
- anonymoushn 4y agoThere's a language server. It seems to work pretty well.
- badpun 4y agoI found this IntelliJ IDEA plugin, the list of features looks pretty good already: https://plugins.jetbrains.com/plugin/10560-zig https://plugins.jetbrains.com/plugin/10560-zig
- rudedogg 4y agoThat one hasn't been updated in a while, but this one is under active development: https://plugins.jetbrains.com/plugin/18062-zig-support https://plugins.jetbrains.com/plugin/18062-zig-support
- dang 4y agoWe changed the URL from https://github.com/ziglang/zig/issues/89 https://github.com/ziglang/zig/issues/89.
- BratishkaErik 4y agoOriginally it was meant to be https://github.com/ziglang/zig/issues/89#issuecomment-1221188076 https://github.com/ziglang/zig/issues/89#issuecomment-122118..., but somehow last aprt was stripped. But that doesn't matter, thank you so much! New URL is definitely better.
- fezfight 4y agoNew programming languages are so much fun. I think it maybe scratches the same itch as home renovation. It also makes me nostalgic for the first time I compiled a program (on the c64).
- NeutralForest 4y agoCongrats to the team, that's a big milestone!
- rnhmjoj 4y agoI never understood why "self-hosting" is considered a good thing. Sure, the compiler developers can write in their favorite language and I guess it means the language is stable enough to be used in a complex project; however requiring an (older) compiler to build the compiler seems a significant complication for software distributions. You either have to set up an ever increasing chain of compilers, as the complexity of the language and the features required to build the compiler grows, or rely on pre-existing binaries. Either way, it seems like a nightmare compared to keeping it in plain old C and building with any compiler of your choice. Just Imagine if any project did this, like, building Firefox now requires an existing Firefox installation.
- damagednoob 4y agoFor me, compilers tend to be complex so it's a way to show that it can be done. Also, dogfooding[1]. [1]: https://en.m.wikipedia.org/wiki/Eating_your_own_dog_food https://en.m.wikipedia.org/wiki/Eating_your_own_dog_food
- nindalf 4y agoYou don’t know if the product you’re building is any good unless you’re using it yourself.
- hypertele-Xii 4y agoAlso known as dogfooding.
- nicoburns 4y ago> Either way, it seems like a nightmare compared to keeping it in plain old C and building with any compiler of your choice. Just Imagine if any project did this, like, building Firefox now requires an existing Firefox installation. But this is exactly the same for C! Building C compiler requires an existing C compiler. C compilers are admittedly more widely available at the moment, but that probably won't be the case forever. Languages like zig aim to supplant C. How are they supposed to do that when they're directly dependent on C?
- 4y ago
- selfhosted 4y agoCongrats to the zig team! This is a big milestone and they must have put an enormous amount of work into making it happen. At the same time, the reason this milestone seems so significant is because the language is already quite complex. It seems likely to get even more complex over time. Some metrics of complexity that jump out are the amount of compiler source code, compiler compilation time and required resources (cpu/memory). There are other languages that have simpler and more efficient bootstrap compilers. Ideally, a self hosting compiler contains only the minimum amount of information that is strictly necessary to compile itself. It is possible to write one for a simple, imperative, javascript style language in at most a few thousand lines of code in pretty much any reasonable turing complete language (something like lisp with barely any syntax can be done even more succinctly). Because the simple language is so small, the implementation can also be written in a high level language that targets that same high level language, i.e. the bootstrap compiler could be a transpiler to javascript written in javascript. There is little point in compiling to native assembly during the bootstrap process because the language is so small that a good javascript implementation should be capable of compiling the bootstrap source in under 200ms on a decent laptop. Once the compiler can compile itself, then one can add more features to it, such as a more robust type system, a native backend or automatic memory management without gc. A positive feedback loop emerges because these features are implemented as optimizations to the compiler itself. It gets faster as more features are added to it but it is always fast because it was fast from the beginning. With good benchmarking in place, it should never get slower at compiling itself. Such a compiler/language would be very minimal by design but a more batteries included language can be built on top of it, much like how an os kernel is extended by user space programs. A historical example of this kind of self-hosting compiler is Forth. It is easy to bootstrap (see e.g. JonesForth). Derived programs are written by extending the compiler with a new vocabulary specific to the problem at hand. Development is often done in a REPL environment for fast feedback. REPL sessions can be saved as source files once the program works as expected. Forth syntax is unfortunately inscrutable to most and the stack based design is not great for every problem so I wouldn't recommend actually using it, but it contains important ideas that can be adopted into more modern languages. What the zig team has done strikes me as very, very difficult, so I tip my cap for the effort it must have required. In the long run though, it feels almost inevitable that a simpler language with the more desirable high level properties described above will eat its lunch. As painful as it would be, I believe the best thing the zig team could do to ensure the language's long term survival would be to completely rewrite the language from scratch (zag?) using the knowledge that they've gained during their initial bootstrap process to distill zig to its essence.
- SLWW 4y agoCongratulations Zig! That's a pretty monumental moment. Can't wait to see what comes down the pipeline for such a fantastic language!