5 ms·

Writing your obscure language in a popular language has an overwhelming benefit: simplicity of bootstrapping. The user can easily obtain an executable C compil

Writing your obscure language in a popular language has an overwhelming benefit: simplicity of bootstrapping. The user can easily obtain an executable C compiler from their distro or whatever. With that, they build your language. If your language is written in itself entirely, then the user needs an executable of the compiler for your language. There is a second benefit: C is an external language. Nothing that you do in your program changes the C language. If your language is written in itself, and you make changes to it, you're changing the language in which your language is written. Mistakes in this can cause hair-pulling bugs. Bootstrapping with another language like C doesn't have to mean that anything remains that is written in C in the boostrapped product! A self-in-self thing can be turned into a C boostrapped thing by simply adding an independent, minimal implementation of that language written in C that the user can compile and run, which is capable of running the real self-in-self compiler to compile that compiler and the run-time. Once that is done, the real compiler can be used to again compiler the compiler into a faster compiler and the run-time into a faster run-time. After that, the C boostrapper is not used for anything, and there is no evidence of it in the product.

matheusmoreira 3y ago
Could be a good idea to maintain a very simple stable reference C implementation for bootstrapping and testing purposes while later developing a more complex one in the language itself.
TylerE 3y ago
While nothing you said is technically wrong, my experience has been that languages that force themselves to dogfood are generally much more pleasant to use than those that are not, both as languages and as tooling. I strongly feel this is not a coincidence. Today's "ease of bootstrapping" is 3 years from now "incoherent library hell".
- regularfry 3y ago
  It is fiendishly hard to disentangle cause from effect here. The sort of person to implement a dogfooded language is likely also the sort of person to care deeply about the quality of that language. It may be that just being in a position to use the language a lot (because you're implementing it) and to change that language (because you're the one building it) is enough to get the effect you've observed, but I wouldn't be at all surprised if you would only start down that path in the first place if you already had strong opinions about what a good language looks like.
- nickpsecurity 3y ago
  Another counterpoint is the mismatch between the intended use of languages. Languages are designed for many types of goals. The language you’re building should be tested against the goals it’s designed to achieve. Whereas, some languages are really good at making compilers. Then, some tools are specifically designed for program transformation and analysis. Using those languages can dramatically improve your compiler which might indirectly increase the number of goals the other language hits.
- wavemode 3y ago
  I'm curious about specific examples you are referring to, going in either direction. Off the top of my head, Java is not implemented in itself yet has perfectly fine tooling. Whereas Haskell is implemented in itself and its tooling leaves much to be desired (despite my love for the language overall).
  - comex 3y ago
    I'm curious too. Among the major language implementations I can think of, almost every single one is dogfooded as much as it can be. In other words, dogfooding is almost always derivative of other technical decisions, not an independent factor. Which makes it hard to distinguish correlation from causation. Fully dogfooded: C++, Rust, Go, D, Zig, [Java with Graal or Jikes], [Python with PyPy] Not dogfooded: JavaScript, Python, Ruby, Lisp, PHP, Lua, Swift Compiler is dogfooded but VM/runtime is not: Java, C#, Haskell, Kotlin Native The compiler (i.e. whatever parses source files and generates either native code or bytecode) is almost always written in the language itself, except for purely interpreted languages where that would be too slow. If there's a VM or runtime, it's written in C or C++ if the language is slow; it's written in the language itself if the language aims to be as fast as C; and for the many languages in the middle, it varies. The biggest exception I can think of is Swift, which is mostly not dogfooded, although it's becoming more so. And even that is explainable by other design decisions. Like C++ or Rust, Swift requires the compiler to do a lot of work to compile it properly. Like Java or Haskell, Swift is really slow at runtime. The combination of those would be deadly for a dogfooded compiler. Besides that, C and Objective-C are technically exceptions because the most popular compilers for them are all written in C++. But I didn't count them because C++ is in the same language family.
    - neonsunset 3y ago
      Compiler and runtime are partially dogfooded now in .NET: - CoreLib is entirely C# (let alone anything outside it) - Low-level facilities like thread-locals, instead of C++, are now C# + compiler idioms + C# to unmanaged exports mostly - Type system facilities have more of their code moved to C# as well - Memmove/memcpy and other performance-sensitive paths, that are part of CoreLib, are either pure C# using its SIMD API or special-cased in compiler (like unrolling of copies, comparisons (including text) of constant lengths and similar last-mile optimizations) In addition, ILC (IL AOT Compiler) is fully written in C#, although uses the same back-end as JIT. JIT could have been reasonably written in C# now that there is NativeAOT but would require more work (also considering that it's a GC-based language), apparently it was considered at some point but ultimately there isn't just capacity to do so - .NET compiler team is fairly small.
pests 3y ago
That does introduce the possibility of a Trusting Trust attack where the original C program could compile the compiler or runtime with a vulnerability that persists until the final compile. Or a similar vector to the xz attack where everything looks completely normal, except for this old script that was subtly changed and if you bootstrap from c you are vulnerable but if you bootstrapped from the language source itself you are good.
- Isamu 3y ago
  >but if you bootstrapped from the language source itself you are good No, that’s exactly what Ken Thompson was describing in Trusting Trust. You always have to bootstrap the compiler, but once you do, you use the compiler to compile itself and all future versions. So you use that circularity to hide the evidence of a back door in the compiler. It applies to all bootstrapped languages.
- kazinator 3y ago
  You cannot get around the Ken Thompson problem because all bootstrapping starts with some trusted binary that compiles your level 0 sources. Now, sure, if your chain does not involve C, then it is immune to a compromised C compilers; how could C compilers do anything to it, that are not installed or run? However, you're vulnerable to a doctored binary of your language. Bootstrapping your Language X from C helps you here also, for multiple reasons. Reason 1: there is no recursive cycle when you externally boostrap from another language. Even if a C compiler is doctored in order to recognize that it's boostrapping Language X, and do something malicious, that malicious thing will not propagate to new Language X installations. Reason being that the build output of Language X is not used for further bootstrapping new Language X instances. The other language is always used for boostrapping. All proliferation of the hack has to propagate through the ecosystem of that other language. Reason 2: Language X is less popular than the boostrapping language, which makes its ecosystem vulnerable. If a vulnerable language with a tiny ecosystem is boostrapped using ready-made binaries of itself, those binaries are likely only available from one site. If those are infected, it's game over. Reason 3: Language X development does not develop the host language (such as C). C is not of interest to the Language X project, and can be conservatively used, so that Language X will build fine with a 20 year old GCC. Even if a bad actor attacks Language X by getting malicious code upstreamed into GCC, that attack won't appear in old versions of GCC. (By the same token, Language X written in Language X could also use a conservative dialect of Language X for boostrapping, so that a many-years-old implementation of it with a widely known SHA-256 can be used.)
danybittel 3y ago
So, you are arguing that It's easier to download the source and compile it yourself than downloading the compiler?
- kazinator 3y ago
  The goal is to boostrap the compiler. So we need two things: the source code, and the compiler which understands it. If the compiler which understands it is in some widely used language, things are easier because we likely already have it. If we don't have it, it's likely nicely packaged. We also likely have it on a platform where that language we are trying to build has not been ported. That gives us a big head start in porting.
- nilamo 3y ago
  If the compiler exists only as a binary, and you cannot compile it yourself, how can you be sure what the compiler is actually doing?
  - pjmlp 3y ago
    As proven by Ken Thompson, unless you are a security guru, having source code won't help.
nickpsecurity 3y ago
I’ll add one more benefit to using languages like C or Java for output or building. That is their tooling ecosystem. Almost all research in static analysis, code generation, etc was for these languages. Using them means you can use all their tooling. Talent, too, for contributions and reviewers.
galdosdi 3y ago
> If your language is written in itself, and you make changes to it, you're changing the language in which your language is written. Mistakes in this can cause hair-pulling bugs. Fair, but, hold on, shouldn't you just make sure that if you're working on version 1.2 of your language, you compile it using your compiler from a stable 1.1 build? (other than maybe in tests where you need to test new features etc) This is not a problem specific to programming languages, but to any other system that uses part of itself

matheusmoreira 3y ago

Could be a good idea to maintain a very simple stable reference C implementation for bootstrapping and testing purposes while later developing a more complex one in the language itself.

TylerE 3y ago

While nothing you said is technically wrong, my experience has been that languages that force themselves to dogfood are generally much more pleasant to use than those that are not, both as languages and as tooling. I strongly feel this is not a coincidence. Today's "ease of bootstrapping" is 3 years from now "incoherent library hell".

regularfry 3y ago
It is fiendishly hard to disentangle cause from effect here. The sort of person to implement a dogfooded language is likely also the sort of person to care deeply about the quality of that language. It may be that just being in a position to use the language a lot (because you're implementing it) and to change that language (because you're the one building it) is enough to get the effect you've observed, but I wouldn't be at all surprised if you would only start down that path in the first place if you already had strong opinions about what a good language looks like.
nickpsecurity 3y ago
Another counterpoint is the mismatch between the intended use of languages. Languages are designed for many types of goals. The language you’re building should be tested against the goals it’s designed to achieve. Whereas, some languages are really good at making compilers. Then, some tools are specifically designed for program transformation and analysis. Using those languages can dramatically improve your compiler which might indirectly increase the number of goals the other language hits.
wavemode 3y ago
I'm curious about specific examples you are referring to, going in either direction. Off the top of my head, Java is not implemented in itself yet has perfectly fine tooling. Whereas Haskell is implemented in itself and its tooling leaves much to be desired (despite my love for the language overall).
- comex 3y ago
  I'm curious too. Among the major language implementations I can think of, almost every single one is dogfooded as much as it can be. In other words, dogfooding is almost always derivative of other technical decisions, not an independent factor. Which makes it hard to distinguish correlation from causation. Fully dogfooded: C++, Rust, Go, D, Zig, [Java with Graal or Jikes], [Python with PyPy] Not dogfooded: JavaScript, Python, Ruby, Lisp, PHP, Lua, Swift Compiler is dogfooded but VM/runtime is not: Java, C#, Haskell, Kotlin Native The compiler (i.e. whatever parses source files and generates either native code or bytecode) is almost always written in the language itself, except for purely interpreted languages where that would be too slow. If there's a VM or runtime, it's written in C or C++ if the language is slow; it's written in the language itself if the language aims to be as fast as C; and for the many languages in the middle, it varies. The biggest exception I can think of is Swift, which is mostly not dogfooded, although it's becoming more so. And even that is explainable by other design decisions. Like C++ or Rust, Swift requires the compiler to do a lot of work to compile it properly. Like Java or Haskell, Swift is really slow at runtime. The combination of those would be deadly for a dogfooded compiler. Besides that, C and Objective-C are technically exceptions because the most popular compilers for them are all written in C++. But I didn't count them because C++ is in the same language family.
  - neonsunset 3y ago
    Compiler and runtime are partially dogfooded now in .NET: - CoreLib is entirely C# (let alone anything outside it) - Low-level facilities like thread-locals, instead of C++, are now C# + compiler idioms + C# to unmanaged exports mostly - Type system facilities have more of their code moved to C# as well - Memmove/memcpy and other performance-sensitive paths, that are part of CoreLib, are either pure C# using its SIMD API or special-cased in compiler (like unrolling of copies, comparisons (including text) of constant lengths and similar last-mile optimizations) In addition, ILC (IL AOT Compiler) is fully written in C#, although uses the same back-end as JIT. JIT could have been reasonably written in C# now that there is NativeAOT but would require more work (also considering that it's a GC-based language), apparently it was considered at some point but ultimately there isn't just capacity to do so - .NET compiler team is fairly small.

pests 3y ago

That does introduce the possibility of a Trusting Trust attack where the original C program could compile the compiler or runtime with a vulnerability that persists until the final compile. Or a similar vector to the xz attack where everything looks completely normal, except for this old script that was subtly changed and if you bootstrap from c you are vulnerable but if you bootstrapped from the language source itself you are good.

Isamu 3y ago
>but if you bootstrapped from the language source itself you are good No, that’s exactly what Ken Thompson was describing in Trusting Trust. You always have to bootstrap the compiler, but once you do, you use the compiler to compile itself and all future versions. So you use that circularity to hide the evidence of a back door in the compiler. It applies to all bootstrapped languages.
kazinator 3y ago
You cannot get around the Ken Thompson problem because all bootstrapping starts with some trusted binary that compiles your level 0 sources. Now, sure, if your chain does not involve C, then it is immune to a compromised C compilers; how could C compilers do anything to it, that are not installed or run? However, you're vulnerable to a doctored binary of your language. Bootstrapping your Language X from C helps you here also, for multiple reasons. Reason 1: there is no recursive cycle when you externally boostrap from another language. Even if a C compiler is doctored in order to recognize that it's boostrapping Language X, and do something malicious, that malicious thing will not propagate to new Language X installations. Reason being that the build output of Language X is not used for further bootstrapping new Language X instances. The other language is always used for boostrapping. All proliferation of the hack has to propagate through the ecosystem of that other language. Reason 2: Language X is less popular than the boostrapping language, which makes its ecosystem vulnerable. If a vulnerable language with a tiny ecosystem is boostrapped using ready-made binaries of itself, those binaries are likely only available from one site. If those are infected, it's game over. Reason 3: Language X development does not develop the host language (such as C). C is not of interest to the Language X project, and can be conservatively used, so that Language X will build fine with a 20 year old GCC. Even if a bad actor attacks Language X by getting malicious code upstreamed into GCC, that attack won't appear in old versions of GCC. (By the same token, Language X written in Language X could also use a conservative dialect of Language X for boostrapping, so that a many-years-old implementation of it with a widely known SHA-256 can be used.)

danybittel 3y ago

So, you are arguing that It's easier to download the source and compile it yourself than downloading the compiler?

kazinator 3y ago
The goal is to boostrap the compiler. So we need two things: the source code, and the compiler which understands it. If the compiler which understands it is in some widely used language, things are easier because we likely already have it. If we don't have it, it's likely nicely packaged. We also likely have it on a platform where that language we are trying to build has not been ported. That gives us a big head start in porting.
nilamo 3y ago
If the compiler exists only as a binary, and you cannot compile it yourself, how can you be sure what the compiler is actually doing?
- pjmlp 3y ago
  As proven by Ken Thompson, unless you are a security guru, having source code won't help.

nickpsecurity 3y ago

I’ll add one more benefit to using languages like C or Java for output or building. That is their tooling ecosystem. Almost all research in static analysis, code generation, etc was for these languages. Using them means you can use all their tooling. Talent, too, for contributions and reviewers.

galdosdi 3y ago

> If your language is written in itself, and you make changes to it, you're changing the language in which your language is written. Mistakes in this can cause hair-pulling bugs. Fair, but, hold on, shouldn't you just make sure that if you're working on version 1.2 of your language, you compile it using your compiler from a stable 1.1 build? (other than maybe in tests where you need to test new features etc) This is not a problem specific to programming languages, but to any other system that uses part of itself