6 ms·
While this doesn't so much apply to libcurl (but see below), there is a third alternative to "write everything in C" or "write everything in <some other safer l
by rwmj 10y ago
While this doesn't so much apply to libcurl (but see below), there is a third alternative to "write everything in C" or "write everything in <some other safer language>". That is: use a safer language to generate C code.
End users, even those compiling from source, will still only need a C compiler. Only developers need to install the safer language (even Curl developers must install valgrind to run the full tests).
Where can you use generated code?
- For non-C language bindings (this could apply to the Curl project, but libcurl is a bit unusual in that it doesn't include other bindings, they are supplied by third parties).
- To describe the API and generate header files, function prototypes, and wrappers.
- To enforce type checking on API parameters (eg. all the CURL_EASY_... options could be described in the generator and then that can be turned into some kind of type checking code).
- Any other time you want a single source of truth in your codebase.
We use a generator (written in OCaml, generating mostly C) successfully in two projects: https://github.com/libguestfs/libguestfs/tree/master/generator https://github.com/libguestfs/libguestfs/tree/master/generat... https://github.com/libguestfs/hivex/tree/master/generator https://github.com/libguestfs/hivex/tree/master/generator
- chii 10y ago> generate C code. how is that different from just writing it in another language? End users who need to compile will be able to regardless of the generated C code, but the end users who need to do a _little_ modification will be given ugly generated C code! Seems stictly worse to me...
- fiedzia 10y agoThe difference is that you don't need compiler for this language. There are many hardware platforms that only come with C compiler.
- tyingq 10y agoFor something like curl, where the library is as popular as the command line tool, preserving the C ABI compatibility is probably the strongest reason.
- simias 10y agoRust could expose a C ABI while keeping safe internals. The interface itself would be unsafe of course. There are a few things that rust doesn't handle natively (like varargs functions IIRC) but other than that you could probably write a Rurl that would be completely backward compatible with Curl.
- tyingq 10y agoYes...the post above me was talking about Ocaml. Similar arguments for not redoing curl in Go.
- steveklabnik 10y agoTo be clear, Rust itself does not have varargs, but can handle them with the C ABI.
- Manishearth 10y agoWell, we can call into vararg functions, but not define them. Since vararg functions have the same ABI as the function with only one of the vararg one idea I've always had is to write a macro that lets you write a one-arg function and have it desugar via asm hax.
- wtetzner 10y agoYou do still need a compiler for the language. It's just that the target language is C, instead of assembly.
- rwmj 10y agoIn the libguestfs generator (first link above) the generated C code is required to be completely readable. It must look like it was written by hand (albeit by a programmer who is impossibly consistent and perfect). So reading the generated C code is fine. Modifying the generated code is of course not fine except for tiny test hacks, but we also include in the generated code comments reflecting where in the generator the code comes from.
- IncRnd 10y agoI've created a number of code generators in my projects. Invariably, developers say exactly what you just wrote, "how do I modify the generated code"? The answer is not to modify the generated code. Modify the input to the code generator to make changes. Even when I output a warning to this effect, that all modifications to target code will get overwritten, not to check the target code into version control, the source code is already checked into version control - invariably developers modify the target code right under the comment that says not to, then they check it into version control. They then wonder why there are bugs, and their modified target code no longer works after the target code gets regenerated after the next build.
- deong 10y agoIt's almost as though you can't solve the problem of programmers making errors by having a different set of programmers whom you tell to not make errors.
- IncRnd 10y agoThe issue had nothing to do with programmers. The impetus wasn't that programmers make errors but to solve the problem of repeatability. Many instances of issues can be solved once. There is no need to recreate the solution a number of times if it is already solved. A code generator allows one to focus on the actual meta-problem, which is often smaller and easier to solve.
- KuiN 10y ago> generate C code. Programmatically generating C code not without problems. How can you prove that the C you're generating is free from problems solved by the safer language? Cloudbleed came from computer generated C code: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/ https://blog.cloudflare.com/incident-report-on-memory-leak-c....
- patrec 10y agoNo, it didn't. See quote from the author of Ragel in the comments: There is no mistake in ragel generated code. What happened was that you turned on EOF actions without appropriate testing. The original author most certainly never intended for that. He/She would have known it would require extensive testing. Legacy code needs to be tested heavily after changes. It should have been left alone. PLEASE PLEASE PLEASE take some time to ensure the media doesn't print things like this. It's going to destroy me. You guys have most certainly benefitted from my hard work over the years. Please don't kill my reputation!
- deong 10y agoWell, the general point still applies. The bug occurred using code that was written in a safe language and compiled to C. It's just that there are multiple ways for that to go wrong. The generator could have had a bug -- it's software, so it almost certainly does. Or, as in this case, the user didn't use it correctly. Either way, the idea that you can write code in a safe language and compile to C to eliminate the type of bugs that C allows isn't true. Are such errors less likely? Possibly so, but they're not categorically eliminated. It becomes a risk assessment exercise rather than a simple thing that everyone should do. Note that it also opens the door to Java-style problems, where once the generator becomes ubiquitous, it becomes the most valuable target for exploit-hunting because a vulnerability in the generator gets the keys to all the houses.
- mbel 10y agoYou are arguing that no language X is safer than writing program manually in Y when program in X is compiled to Y. Because compiler from X to Y may have bugs. Therefore no code written in Rust (X) executed on x86 CPU (Y) is safer than manually written x86 assemby, because Rust compiler (and LLVM) may have errors. And well, we can actually go deeper. There is CPU frontend that is generating micro code, which may have bugs. There is also CPU backend which is executing micro code, which also may have bugs. All in all there is no hope in programming. There might be bugs everywhere so you can never be sure what your program does.
- mushiake 10y agoFFTW[0] is also written like that (generator written in OCaml emitting C). [0]http://www.fftw.org/ http://www.fftw.org/
- ndesaulniers 10y agoI wonder how cloudflare feels about that? Ragel