19 ms·

Translating All C to Rust (TRACTOR)

• steveklabnik 2y ago
  See also https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view https://sam.gov/opp/1e45d648886b4e9ca91890285af77eb7/view
  • thesuperbigfrog 2y ago
    Direct link to Proposer's Day info [PDF]: https://sam.gov/api/prod/opps/v3/opportunities/resources/files/3f5caa5af9da4785b335c37c16e2beda/download?&token= https://sam.gov/api/prod/opps/v3/opportunities/resources/fil... "The purpose of this event is to provide information on the TRACTOR technical goals and challenges, address questions from potential proposers, and provide an opportunity for potential proposers to consider how their research may align with the TRACTOR program objectives."
• torgian 2y ago
  [flagged]
• PreInternet01 2y ago
  The one link for those who think that 'Rewrite it All in Rust' will, well, settle any debates: https://github.com/rust-lang/miri/ https://github.com/rust-lang/miri/
  • calebpeterson 2y ago
    Genuine question: Would you mind explaining to a dev that doesn’t know much (anything) about Rust, how does this settle any debate?
    • jerf 2y ago
      I believe it goes something like, "I have constructed a strawman that Rust claims that all code written in it is automatically safe by all conceivable definitions of safe, but look, ha ha, here's something that detects unsafe code in Rust!", and I don't mean "code marked in unsafe blocks". It's a concatenation of several logical fallacies in a row; equivocation, straw manning, binary thinking about safety, several others. It's hard to pick the main one, but I'd go with the dominant problem being a serious case of binary thinking about what "safety" is. Of course, if the commentor is using anything other than Idris for all their programming, they're probably not actually acting on their own accusations.
      • PreInternet01 2y ago
        [flagged]
        • commodoreboxer 2y ago
          > This repository demonstrates that, when using 'safe' Rust, there are still double-digits cases where you may still encounter dread-pirate-UB. No it doesn't. Miri is for unsafe code. There's no UB in safe Rust by design. Any UB caused without unsafe is considered a bug to be fixed.
          • PreInternet01 2y ago
            [flagged]
            • nequo 2y ago
              > ... in generally safe Rust. Just to find agreement about the terminology, wouldn't we call all code that is not inside an unsafe block "safe?" If so, then adding "generally" is superfluous, right? If not, then how is "generally safe" different from "not inside an unsafe block?"
        • jerf 2y ago
          I didn't expect you to outright confirm that you are using the "solve all programming problems ever" strawman, but, err, thanks for the proof I guess. I thought maybe I went a bit overboard in the reading between the lines but I guess I nailed it.
      • marcosdumay 2y ago
        > Of course, if the commentor is using anything other than Idris I'm sure the Idris compiler has bugs somewhere too. If the OP actually programs, they are violating their rationale (I'm quite sure assembly or assembled binary aren't ok either).
    • mrweiden 2y ago
      From the original post > It’s not enough to rely on bug-finding tools From the Miri github: > Miri is an Undefined Behavior detection tool for Rust.
      • keybored 2y ago
        Darpa is already ahead of you all with the hedging: > The preferred approach is to use “safe” programming languages “Safe”. Terms and conditions may apply.
      • Sharlin 2y ago
        There is no contradiction. The fact that UB-finding tools alone are not sufficient doesn't mean they aren't useful even with a safe(r) language. In other words, from "safer languages are necessary" it does not follow that "safer languages are sufficient".
    • PreInternet01 2y ago
      Well, the general 'Rewrite All in Rust' consensus is that it solves all general programming problems, ever. Yet, the linked repository shows a huge list of cases in which simple, documented use of Rust can cause Undefined Behavior (a.k.a. 'UB') Pretty much every argument of Rust advocates against C/C++ boils down to either 'but memory safety' or 'but UB'. Yet there are many convincing counter-arguments that boil down to 'but CompCert' or similar, and, as the linked repository shows, there might be at least some truth in there?
      • steveklabnik 2y ago
        No serious person claims that Rust solves every problem ever. Also, many people cite things like Cargo as a reason to prefer Rust over C and C++, as well as other things. UB is a big part of it, of course, but it isn’t the only thing.
        • galangalalgol 2y ago
          I selected it for performance reasons myself, the UB protection was a nice benefit that was expected, cargo wasn't expected and is extremely nice coming from the cmake,conan,vcpkg and duct tape world I came from.
        • PreInternet01 2y ago
          > No serious person claims that Rust solves every problem ever No, but there are a lot of people claiming that Rust cannot ever have any problems. Just look at this thread. I merely linked to MIRI, and am currently at, like, -10 just for that. Lots of people claiming that it just applies to 'unsafe Rust': is that true or not? Regardless of anything else: can you, as a Rust community leader, please state clearly: is UB in generally safe Rust possible or not?
          • steveklabnik 2y ago
            No, people are not claiming Rust cannot have any problems. UB is not possible in safe Rust, by design. The root cause of UB is always in unsafe code. Miri is useless if your code is 100% safe Rust. The only exception to this is bugs in the compiler, of which there are a few. They’ll be fixed.
    • leftyspook 2y ago
      It is a tool for checking that your unsafe code doesn't cause UB. It doesn't really settle anything, but the commenter uses it as a gotcha to say "rust is no better than C, because you still can compile code that contains UB".
    • spease 2y ago
      They are claiming that because code in ‘unsafe’ blocks in Rust can have undefined behavior, that the language is no safer than C. This does not settle the debate because unsafe is rarely needed for a typical Rust program. In addition, the presence of an unsafe block also alerts the reader that the set of possible errors is greatly increased for that part of the code and more careful auditing is needed. It’s a little like saying traffic lights are useless because emergency responders need to drive through them sometimes, so we should just leave intersections completely unsignaled and expect drivers to do better. Rust is by default restrictive and requires you to explicitly make it unsafe, C/++ are by default unsafe and require you to explicitly make them restrictive.
  • keybored 2y ago
    You linked an interpreter for some kind of internal compiler representation that the Rust compiler uses. What on Earth do you mean?
    • nequo 2y ago
      It's the old trope that some Rust code uses unsafe blocks so all Rust code is as unsafe as C.
      • keybored 2y ago
        Of course. I should have expected the Nirvana Fallacy. :)
      • melling 2y ago
        I don’t know Rust but even if the Rust is just as unsafe in certain blocks, simply being translated to Rust removes a lot of corporate resistance to adopt the language. Getting people to adopt a new language can be a lot of work. I remember people claiming they missed headers files in Swift so they wanted to stick with Objective C.
    • PreInternet01 2y ago
      > What on Earth do you mean? That documented use of safe Rust can easily lead to UB, which this infernal 'internal compiler representation' demonstrates. I'm not even sure what is even remotely confusing about that?
      • woodruffw 2y ago
        Miri is a MIR interpreter aimed at unsafe Rust, not safe Rust. Using the fact that it operats on an internal representation is a very weird swipe; almost all static and dynamic analysis tools work on some kind of IR or decomposed program representation.
      • commodoreboxer 2y ago
        > Miri is an Undefined Behavior detection tool for Rust. It can run binaries and test suites of cargo projects and detect unsafe code that fails to uphold its safety requirements. > ... detect unsafe code that fails ... Show me the documented safe Rust code that causes UB without using any unsafe blocks outside of the standard library.
  • deleted 2y ago
    [deleted]
  • the8472 2y ago
    The trophy cases in miri are about bugs in unsafe code. Yes, you can write UB with unsafe code. This should not be news. And miri is a blessing. There even is a known case where someone found a bug in C by translating it to rust and then running it through miri.
• mike_hearn 2y ago
  That sounds ... hard. Especially as idiomatic Rust as written by skilled programmers looks nothing like C, and most interesting code is written in C++ anyway. Isn't it equivalent to statically determining the lifetimes of all allocations in the C program, including those that are implemented using custom allocators or which cross into proprietary libraries? There's been a lot of research into this sort of thing over the years without much success. C/C++ programs can do things like tie allocation lifetimes to what buttons a user clicks, without ref counting or other mechanisms to ensure safety. It's not a good idea, but, they can do it. The other obvious problem with trying to write such a static analysis is that the programs you're analyzing are by definition buggy and the lifetimes might not make sense (if they did, they wouldn't have memory safety holes and wouldn't need to be replaced). The only research I've seen on this problem of statically detecting what lifetimes should be does assume the code being analyzed is actually correct to begin with. I guess you could try and aim for a program that detects where lifetimes can't be worked out and asks the developer for help though.
  • mattgreenrocks 2y ago
    Projects are termed DARPA-hard for a reason.
  • woodruffw 2y ago
    It's very hard; DARPA likes to fund hard things[1] :-). This isn't, however, DARPA's first foray into automatic program translation, or even automatic translation into Rust[2]. [1]: https://www.urbandictionary.com/define.php?term=DARPA%20hard https://www.urbandictionary.com/define.php?term=DARPA%20hard [2]: https://c2rust.com/ https://c2rust.com/
    • the_snooze 2y ago
      DARPA is basically a state-sponsored VC that optimizes for completely different things. Instead of looking for 100x financial returns, they want technical advantages for the United States. The "moat" is the hardness of developing and operationalizing those technologies first.
• deleted 2y ago
  [deleted]
• niemandhier 2y ago
  Is this supposed to be automatic ? And if so wouldn’t any Programm that can automatically port c to rust, by necessity contain all the functionality to make the c code itself safe?
  • gpm 2y ago
    I don't think a reasonable reading of the statement implies "fully automated", at which point the answer to the question is no. Obviously some C code isn't just "not verifiable correct" but "actually wrong in a memory unsafe way". That code isn't going to be automatically translated without human intervention because, how could it be, there is no correct equivalent code. The tooling is going to have to have an escape hatch where it says "I don't know what this code is meant to do, and I know it isn't meant to do what it does do (violate promises to the compiler), help me human". On a theoretical level it's not possible for that escape hatch to only be used when undefined behaviour does occur (rices theorem). On a practical level it's probably not even desirable to try because obtuse enough code shouldn't just be blindly translated. So what I imagine the tooling ends up looking like is an interactive tool that does the vast majority of the work for you, but is guided by a human, and ultimately as a result of that human guidance doesn't end up with exactly equivalent code, just code that serves the same purpose.
• TinkersW 2y ago
  Good luck with that..also shouldn't the target be C++ to Rust? Is there really that much pure C still being written?
  • surfingdino 2y ago
    IoT, embedded systems still use it. There's loads of them.
  • riku_iki 2y ago
    AGI may find much simpler, more robust/performant and safe language.
• nanolith 2y ago
  I'm personally not a fan of "rewrite the world in Rust" mentality, but that being said, if one is planning to port a project to a new language or platform, mechanical translation is a poor means of doing so. Spend the time planning better architecture and designing a better software system, and find a way to replace it piece by piece. Don't build a castle in the sky, because it will never reach the ground. If you've decided to use Rust for this system, that's fine. But, write Rust. Don't try to back-port C into Rust. I think a far better and more mature process is to update C to modern C and use a model checker such as CBMC to verify memory, resource, and integer math safety. One gets the same safety as a gradual Rust rewrite, but the code base, knowledge base, and developers can be maintained.
  • pdimitar 2y ago
    > I'm personally not a fan of "rewrite the world in Rust" mentality There is no such mentality anywhere. There is a ton of software that's much better off left alone in a dynamic language, or a statically typed language with a garbage collector (like Golang). Good engineers understand the idea of using the right tool for the job. The push is to start reducing those memory safety CVEs because they have been proven to be a real problem, many times over. > mechanical translation is a poor means of doing so Agreed. If we could automatically and reliably translate C/C++ to Rust it would have been done already. > Spend the time planning better architecture and designing a better software system, and find a way to replace it piece by piece. OK, I am just saying that somewhere along that process people might get a bout of confidence and tell themselves "oh, we're doing C much better now, we no longer write memory safety bugs, can't we stop here?" and they absolutely will. Cue another hilarious buffer overflow CVE 6 months later. > I think a far better and more mature process is to update C to modern C and use a model checker such as CBMC to verify memory, resource, and integer math safety. A huge investment. If you are going to do that then you might as well just move to Rust. > One gets the same safety as a gradual Rust rewrite Maybe, but that sounds fairly uncertain or far from a clear takeaway to me.
    • nanolith 2y ago
      > A huge investment. If you are going to do that then you might as well just move to Rust. People say that, but the people who say this rarely have any practical experience using CBMC. It's very straight-forward to use. I could teach a developer to use it reliably, on practical software, in a month.
• deleted 2y ago
  [deleted]
• deepsun 2y ago
  They didn't explain why they've chosen Rust. There are a lot of memory-safe languages besides Rust, especially in application-level area (not systems-level like Rust).
  • woodruffw 2y ago
    There are a lot of memory safe languages; there are fewer that have (1) marginal runtime requirements, (2) transparent interop/FFI with existing C codebases, (3) enable both spatial and temporal memory safety without GC, and (4) have significant development momentum behind them. Rust doesn't have to be unique among these qualifications, but it's currently preeminent.
    • deepsun 2y ago
      Yes, but you assume all their projects need all 4 of these. I like Rust, but it's a bad choice for many areas (e.g. aforementioned application-level code). I'd expect serious decisions to at least take that into account.
      • woodruffw 2y ago
        I’m not assuming anything of the sort. These are just properties that make Rust a nice target for automatic translation of C programs; there are myriad factors that guarantee that nowhere close to 100% of programs (C, application level, or otherwise) won’t be suitable for translation.
  • galangalalgol 2y ago
    If you have your cross hair on c, then you want a language that can do whatever c does. That makes the list of memory safe languages a lot shorter.
  • oconnor663 2y ago
    Apart from runtime/embedded requirements, there's the big question of how you represent what C is doing in other languages that don't have interior pointers and pointer casting. For example, in C I might have a `struct foo*` that aliases the 7th element of a `struct foo[]` array. How do you represent that in Java or Python? I don't think you can use regular objects or regular arrays/lists from either of those languages, because you need assignments through the pointer (of the whole `struct foo`, not just individual field writes) to affect the array. Even worse, in C I might have a `const char*` that aliases the same element and expects every write to affect its bytes. To model all this you'd need some Frankenstein, technically-Turing-complete, giant-bytestring-that-represents-all-of-memory thing that wouldn't really be Java or Python in any meaningful sense, wouldn't be remotely readable or maintainable, and wouldn't be able to interoperate with any existing libraries. In Rust you presumably do all of that with raw pointers, which leaves you with a big unsafe mess to clean up over time, and I imagine a lot of the hard work of this project is trying to minimize that mess. But at least the mess that you have is recognizably Rust, and incremental cleanup is possible.
• thibran 2y ago
  Porting the Linux kernel to 100% Rust should be the benchmark for AGI. ... and when done, please port SQLite too :)
  • 0cf8612b2e1e 2y ago
    I am fully in the RIIR koolaid, but SQLite would be near the absolute bottom of my prioritization list. Care to explain? SQLite is extensively tested, has requirements to run on ~every platform, be backwards compatible, and has a relatively small blast radius if there is a C derived bug. There is much more fertile ground in any number of core system services (network, sudo, dns, etc)
    • eric-p7 2y ago
      Not a small blast radius. There are an estimated 1 trillion active deployed SQLite instances: https://news.ycombinator.com/item?id=29461127 https://news.ycombinator.com/item?id=29461127
      • xbar 2y ago
        Fair. But perhaps it has a narrow attack surface.
• commodoreboxer 2y ago
  A lot of people are reading this as a call or demand to translate all C and C++ code to Rust, but (despite the catchy project name), I don't read the abstract in that way. There are two related but separate paragraphs. 1. C and C++ just aren't safe enough at large. Even with careful programming and good tooling, so many vulnerabilities are caused by their unsafe by default designs. Therefore, as much code as possible should be translated to or written in "safe" languages (especially ones that guarantee memory safety). 2. We are funding and calling for software to translate existing C code into Rust. It's not a consensus to rewrite the world in Rust. It's a consensus to migrate to safe languages, which Rust is an example of, and a program that targets Rust in such migration.
  • akira2501 2y ago
    > or written in "safe" languages So when those languages have 'unsafe' constructs what are the rules going to be around using those? Without a defining set of rules to use here you're just going to end up right back where you started. > to migrate to safe languages, which Rust is an example of Rust has a safe mode. It is _not_ a safe language. To do anything interesting you will require unsafe blocks. This will not get you very much. Meanwhile you have tons of garbage collected languages that don't even let the programmer touch pointers. Why aren't those considered? The reason is performance. And because Rust programmers "care" so much about performance you're not ever going to solve the fundamental problem with that language. Do you want performance or safety? You can't have both.
    • selfmodruntime 2y ago
      > It is _not_ a safe language. To do anything interesting you will require unsafe blocks. This is largely untrue. You can use proven abstractions over 99% of cases that would require unsafe.
    • timeon 2y ago
      > To do anything interesting you will require unsafe blocks. This will not get you very much. This is not true.
• 0xbadcafebee 2y ago
  Every tool has its own specific quirks. Over many years of using a tool, "expertise" is the intimate knowledge of those quirks and how to use that tool most effectively. Changing tools requires you to gain expertise again. You're going to be less proficient in the new tool for a long time, and make a lot of mistakes. Considering we already know how to make C/C++ programs memory safe, it's bizarre that people would ditch all of their expertise, and the years and years of perfecting the operation of those programs, and throw all that out the window because they can't be bothered to use a particular set of functions [that enforce memory safety]. If you're going to go to all of the trouble to gain expertise in an entirely new tool, plus porting a legacy program to the new tool, I think you need a better rationale than "it does memory safety now". You should have more to show for your efforts than just that, and take advantage of the situation to add more value.
  • wffurr 2y ago
    But even proficient C and C++ programmers continue to produce code with memory safety issues leading to remote code execution exploits. This argument doesn’t hold up to the actual experience of large C and C++ projects.
    • deleted 2y ago
      [deleted]
    • 0xbadcafebee 2y ago
      They aren't trying to prevent them. It's trivial to prevent them if you actually put effort into it; if you don't, it's going to be vulnerable. This is true of all security concerns.
      • woodruffw 2y ago
        "You aren't trying hard enough" isn't a serious approach to security: if it was, we wouldn't require seatbelts in cars or health inspections in restaurants. (It's also not clear that they aren't trying hard enough: Google, Apple, etc. have billions of dollars riding on the safety of their products, but still largely fail to produce memory-safe C and C++ codebases.)
• jcalvinowens 2y ago
  This isn't some "pie in the sky" thing, Immunant has a working C to Rust transpiler and it's really interesting: https://github.com/immunant/c2rust https://github.com/immunant/c2rust
  • steveklabnik 2y ago
    Their work was also previously sponsored by DARPA, though I do not know if it was under this program or something else.
    • woodson 2y ago
      It must have been a different program, as this one hasn’t started yet, but perhaps by another program by the same program manager.
  • Animats 2y ago
    I've tried that thing. The Rust that comes out is terrible. It converts C into a set of Rust function calls which explicitly emulate C semantics by manipulating raw pointers. It doesn't even convert C arrays to a Vec. It's a brute-force transliteration, not a translation. I and someone else ran this on a JPEG 2000 decoder that sometimes crashed with a bad memory reference. The Rust version crashed with the same bad memory reference. It's bug-compatible. What comes out is totally unreadable and much bigger than the original C code. Manual "refactoring" of that output is hopeless.
    • marcosdumay 2y ago
      Any automatic translation is bug-compatible with the original. Did you expect it to divine some requirements? It still leave you with Rust code that you can improve piecewise. The only question is if something like it is better than FFI calling the C code.
      • bornfreddy 2y ago
        > Any automatic translation is bug-compatible with the original. Did you expect it to divine some requirements? That would be useless when translating C to Rust. Yes, I would expect the tool to point out the flaws in the original memory handling and only translate the corrected code. This is far from easy, since some information (intent) is missing, but a good coder could do it on decent codebases. The question is, can an automated tool do it too? We'll see.
• jll29 2y ago
  Difficult: most C programs I know would convert to one single large "unsafe" block... One might argue that re-writing from scratch is the safer option; and a re-write is also an opportunity to do things differently (read: improve the architecture by using what one has learned), despite the much-feared "second system" syndrome. But nothing wrong with spending some research dollars towards tooling for "assisted legacy rewrites". DARPA and her sister IARPA fund step innovation (high risk, high reward), and this is an area where good things can come potentially come from.
• deleted 2y ago
  [deleted]
• jdblair 2y ago
  I'm really surprised this can work at all in any automated way. You can't just make a line-by-line transcription of a typical c program into rust. Pointers and aliasing are ubiquitous in c programs, concepts that rust explicitly prevents. You have to rethink many typical constructs at a high level to rewrite a c program in rust, unless you wrap the whole thing in "unsafe."
  • alex_suzuki 2y ago
    I wonder about this as well, especially im code bases that make heavy use of macros.
  • ip26 2y ago
    For a naive newcomer - could you go line by line, wrap the whole thing in “unsafe”, compile to an identical binary, and then slowly peel away the “unsafe” while continuing to validate equivalence? That would at least get you to as much rust as possible, and then let engineers tackle rethinking just those concepts.
    • selfmodruntime 2y ago
      There are "warts" with unsafe Rust that would make this feat very difficult. Aliasing rules still apply.
    • rolph 2y ago
      you need to create a transpiler philosophy. transform CtoASM, then ASMtoRust. what you need to avoid is incompatibilites between different high level languages with a low level intermediary so you arent stuck attempting to convert high level hardware abstraction directly to another high level hardware abstraction.
    • jcranmer 2y ago
      Converting C to legal (unsafe) Rust is quite possible; there is indeed already a tool that does this (https://github.com/immunant/c2rust https://github.com/immunant/c2rust). The problem you run into is that the conversion is so pedantically correct that the resulting code is useless. The result retains all of the problems that the C code has, and is so far from idiomatic Rust that it's easier to toss the code and start from scratch. Progressive lifting on unsafe Rust to safe Rust is a very difficult order, and the tool I mentioned had a tool to do that... which is now abandoned and unmaintained. At the end of the day, the chief issue with converting to safe Rust is not just that you have to copy semantics over, but you also have to recover a lot of high-level preconditions. Turning pointers into slices is perhaps the easiest task of the lot; given the very strict mutability rules in Rust, you also have to work out when and where to insert things like Cell or Rc or Mutex or what have you, as well as building out lifetime analysis. And chances are the original code doesn't get all these rules right, which is why there are bugs in the first place. Solving that problem is the goal of this DARPA proposal, or perhaps more accurately, determining how feasible it is to solve that problem automatically. Personally, I think the better answer is to have a semi-automated approach, where users provide as input the final Rust struct layouts (and possibly parts of the API, to fix lifetime issues), and the tool automates the drudgery of getting the same logic ported to that mapping.
• Animats 2y ago
  It's good to see DARPA pushing on this. It's a hard problem, but by no means impossible. Translating to safe Rust, though, is going to be really tough. There's a C to Rust translator now, but what comes out is horrible Rust, which just rewrites C pointer manipulation as unsafe Rust struct manipulation. The result is less maintainable than the original. So what would it take to actually do this right? The two big problems are 1) array sizes, and 2) non-affine pointer usage. Pointer arithmetic is also hard, but rare. Most pointer arithmetic can be expressed as slices. Every array in C has a size. It's just that the compiler doesn't know what it is. Where is this being discussed in detail?
  • steveklabnik 2y ago
    > Where is this being discussed in detail? In my understanding, this is a call for proposals to do the work, there is no detailed discussion yet. That will come when there's actual responses to this call.
    • Animats 2y ago
      Right, there's a call, and a project day with an in-person meeting coming up.
  • jcranmer 2y ago
    I once tried to use c2rust as a starting point for rustification of code and... it's not even good at that. The code is just too freakishly literal to the original C semantics that you can't even take the non-pointery bits and strip off the unsafe block and use that as a basis. (To give you a sense, it translates something like a + 1 to a.unwrapped_add(1i32), and my recollection is that for (int i = 0; i < 10; i++) gets helpfully turned into a while loop instead of a for loop). In general, the various challenges that all need to be solved that aren't solved yet are: a) when is integer overflow intentional in the original code so that you know when to use wrapping_op instead of regular Rust operators? b) how to convert unions into Rust enums c) when pointers are slices, and what corresponds to the length of the slice d) convert pointers to references, and know when they're mutable or const references e) work out lifetime annotations where necessary f) know when to add interior mutability to structs g) wrap things in Mutex/RwLock/etc. for multithreaded access We're a very long way from having full-application conversion workable, and that might be sufficiently difficult that it's impossible.
• sans-seraph 2y ago
  I have been aware of this proposed initiative for some time and I find it interesting that it is now becoming public. It is a very ambitious proposal and I agree that this level of ambition is appropriate for DARPA's mission and I wish them well. As a Rust advocate in this domain I have attempted to temper the expectations of those driving this proposal with due respect to the feasibility of automatic translation from C to Rust. The fundamental obstacle that I foresee remains that C source code contains less information than Rust source code. In order to translate C code to Rust code that missing information must be produced by someone or something. It is easy to prove that it is impossible to infallibly generate this missing information for the same reason that scaling an image to make it larger cannot infallibly produce bits of information that were not captured by the original image. Instead we must extrapolate (invent) the missing information from the existing source code. To extrapolate correctly we must exercise judgement and this is a fallible process especially when exercised in large quantities by unsupervised language models. I have proposed solutions that I believe would go some way towards addressing these problems but I will decline to go into detail. Ultimately I will say that I believe that it is possible for this project to achieve a measure of success, although it must be undertaken with caution and with measured expectations. At the same time it should be emphasized it is also possible that no public result will come of this project and so I caution those here against reading too much into this at this time. In particular I would remind everyone that the government is not a singular entity and so I would not interpret this project as a blanket denouncement against C or vice versa as a blanket blessing of Rust. Each agency will set its own direction and timelines for the adoption of memory-safe technologies. For example NIST recommends Rust as well as Ada SPARK in addition to various hardened dialects of C/C++.
  • steveklabnik 2y ago
    > As a Rust advocate in this domain I have attempted to temper the expectations of those driving this proposal Thank you!
  • pfdietz 2y ago
    How does it relate to the CRAM effort at Grammatech? https://cpp-rust-assisted-migration.gitlab.io/blog/ https://cpp-rust-assisted-migration.gitlab.io/blog/
• simon_void 2y ago
  a) if every C program could be translated into an equivalent safe Rust program, that would mean that each C program is as safe as the safe Rust equivalent. b) since there are C programs that are open to memory currption in a way safe Rust isn't, this corruptability would need to be translated into partially unsafe Rust. Congrats, you now have a corruptible Rust program, what's the point again?? c) so DARPA must be trying to fix/change what the program is doing when switching to Rust. So how to discern what behaviour is intended and which is not? Doesn't this run directly into the undecidability/uncomputability of the halting problem!?!
  • Arnavion 2y ago
    >Doesn't this run directly into the undecidability/uncomputability of the halting problem!?! The programmer gets to decide. DARPA does not expect the translator program to autonomously output a perfect Rust program. It just wants a "high degree of automation towards translating legacy C to Rust" (from the sam.gov link in the submission, emphasis mine).
  • im3w1l 2y ago
    Memory corruption is undefined behavior and means the compiler is free to do anything it wants. Anything it wants... and that includes doing something entirely safe and reasonable. If you write out of bounds, the compiler is allowed to shut the program down in a controlled manner. It's allowed to transparently resize the array for you. Etc. Hence a rust translation can do these things.
    • fch42 2y ago
      You "anything it wants" folks really annoy me a little. If the compiler can, compile-time, detect that code is prone to memory corruption, it can warn the developer. If it can't detect it at compile time, will and shall it add some sort of magic signal handler heuristic to determine whether a segfault occurred due to a runtime-provable specific instance of memory corruption and hence format your harddrive, while for runtime-indeterminable kinds it'd rather fry cpu core seven preemptively ? But that behaviour changes in the next version to blink sos on the network cable leds ? I mean, it were cool if compilers used their "freedom" here to output nagging messages "the mem-safe UB brigade told you so, told you so, told you so ...". The fact they don't tells me, at least, that compiler developers follow Postel's law - be strict at what you emit but lenient at what you process. They're reasonable people. Not some sort of crusader out there to get you in the most excruciatingly painful ways. Undefined behaviour isn't unreasonable behaviour.
• verdverm 2y ago
  Russ Cox gave a GopherCon talk on the effort to automatically convert the Go compiler from C to Go (in the early days). Lots of interesting IRL issues / solutions in there. https://www.youtube.com/watch?v=QIE5nV5fDwA https://www.youtube.com/watch?v=QIE5nV5fDwA iirc, they were able to transpile 90%+ (without AI) and manually did the rest
• poikroequ 2y ago
  > Those involved with the oversight of C and C++ have pushed back, arguing that proper adherence to ISO standards and diligent application of testing tools can achieve comparable results without reinventing everything in Rust. If you stick to extremely stringent coding practices and incorporate third party static verification tools that require riddling your code with proprietary situations, then sure, you can achieve comparable results with C/C++. Or you can just use Rust.
  • dtx1 2y ago
    It's quite hilarious to see the push back rust gets by the c/c++ community. Obviously their decades of hard work and experience to work with those languages are overriding their reasoning circuits. Who in their right mind would defend a language that has such major and obvious design flaws if a genuine alternative is there.
    • constantcrying 2y ago
      Many of the most widely used languages have obvious major design flaws. (JavaScript is one obvious candidate, python is another. How did a language which has no built-in floating point type become the number one language for numerical analysis?) The real question is what tradeoffs you are making and what you are gaining. Rust makes certain memory safety guarantees about the program at compile time, but at the same time it disallows perfectly safe constructions, which can exist in C++, as well.
      • brigadier132 2y ago
        I think DARPA is making the right decision about choosing Rust as the language for low level systems programming. For national security related matters you'd definitely want the certainty Rust brings. The reason I personally chose Rust as my go to language for low level programming is that despite learning systems programming in college I pretty much never used it outside of school. Meaning I didn't have any of that knowledge that c and c++ programmers had built up over years of experience. So I decided that instead of having to deal with the unknown skill deficiencies in writing concurrent software and memory management I'd rather just have a compiler scream at me. I don't regret the decision. Also, I remember writing an async TCP implementation in college with c++ using boost. Rust tooling is just so far ahead of that.
• tonetegeatinst 2y ago
  I am a total C shill....I'll admit it. I'm just starting to learn it and the only reason I picked it was I wanted a low level language that most systems run. That said, while I can acknowledge the benefits of memory safety, I would personally choose zig over rust. All things considered I know you can do some safety check for C using the compiler, and that helps reduce the odds of memory issues. Idk what rust mail libraries look like( are they even called that?) But I know C's standard library's have made learning stuff easier. Is their any way to know if your libraries in rust are using unsafe code? Will that just spit out compile time errors?
• aaron695 2y ago
  [dead]
• gosub100 2y ago
  slight tangent, but I think it would be amazing if AI could write device drivers. Full-featured GPU drivers for, say, OpenBSD. What does it need? Probably the state machines of the GPUs, how to enable various modes, how to feed data in/out of the device at intended speed, how to load shaders. Why can't AI learn to do that? Its reward could be getting past the initialization and getting to the default state of the driver. It could be trained on hundreds of GPU drivers, not only for the minutiae of how to load values into the control registers, but the bigger picture of what it actually means.
  • constantcrying 2y ago
    >Why can't AI learn to do that? Because it isn't magic. >It could be trained on hundreds of GPU drivers Do you know what an AI trained on hundreds of books looks like? Even with millions of books it can not write a coherent chapter, much less an entire book. This is a genuinely terrible idea. It is exactly the thing AI is bad at, high degree of accuracy over long stretches of output.
    • gosub100 2y ago
      > Because it isn't magic. So graphics drivers are magic? They weren't designed in a determinate way, with a known interface and specifications?
      • constantcrying 2y ago
        No, AI isn't magic. LLMs are bad at exactly those things which you need to make a GPU driver. Extremely high accuracy over long distances. AI totally falls apart when trying to write a novel, how could it write a GPU driver?
        • gosub100 2y ago
          They are excellent at stealing artists' content. I don't think it will be long for them to steal content from authors.
• sumanthvepa 2y ago
  I program in C++ and am very happy to do so. Modern C++ is very safe and actually fun to program in. It gives me enormous expressivity, extraordinary performance and safety when I need it. I'm not building space shuttles, I'm building 3D experiences, so I'm not terribly concerned about crashes. But even for me, I've not run into a memory corruption bug in recent memory (10-15 years.) Bash C/C++ all you want. I'm happy to keep using it to my advantage.
  • tomrod 2y ago
    What is the learning curve for newbies to avoid critical segfaults? If you still have to walk a tightrope to get code across the board, wouldn't all benefit from a plankway with guardrails instead? I'm not dissing C or C++ in any way. I've used it. But I recognize there are some major footguns that aren't easy to avoid, causing a much longer learning curve than necessary to get things built. Rust at least seems determined to address them, good or bad!
    • sumanthvepa 2y ago
      To a first approximation, avoid using raw pointers. They should almost never be needed in application code. Use C++'s standard library facilities for smart pointers and containers instead. They are masterpieces of engineering, and work extremely well.
      • tomrod 2y ago
        Why avoid using defaults? That's literally the part of an unnecessarily long learning curve needed to learn C/C++ that rust helps with.
  • offbynull 2y ago
    I program in modern C++ as well (C++23). I disagree with both "very safe" and "fun". Even with 23 there are an innumerable number of footguns throughout both the language and the standard library. Debugging code is also a mess. Good luck getting anything done without paying for an IDE, and even then it can be a struggle.
• ks2048 2y ago
  If we have smart AIs to write code, find bugs, and write tests - doesn’t that mean we can ditch the “safe” languages and go back to C? Thats mostly a joke. But AI-hardened-C seems like it could be much better than current-human-only-C.
  • rectang 2y ago
    It’s not any more of a joke than the hee-haw nonsense that using an LLM to translate working C code into something else will yield a result with fewer bugs.
  • constantcrying 2y ago
    Why would AI be competent at finding bugs? Most non-trivial bugs I find are about unexpected interactions between distinct pieces of code. Seems totally unfeasible for a llm to be good at.
• andrewstuart 2y ago
  My experience of AI as a coding assistant: for Python - awesome for golang - awesome for JavaScript - awesome for Zig - not awesome, AI doesn't get it, maybe training data set too small for Rust - terrible - AI really doesn't get how it works, especially the hard bits
  • bsder 2y ago
    If you can cobble your program together by copy/pasting Stack Overflow snippets, an AI tends to be useful. Your list reflects that.
• nielsbot 2y ago
  Wonder if they could also/instead change the compiler similarly to Apple: https://support.apple.com/guide/security/memory-safe-iboot-implementation-sec30d8d9ec1/web https://support.apple.com/guide/security/memory-safe-iboot-i...
• TZubiri 2y ago
  If you translate the code from C to Rust automatically. Isn't C still the source code? Just transpiled to Rust as an intermediate before asm?
  • rectang 2y ago
    Indeed. Think of the AI stage as “llmcc”, or maybe “lsdcc” if you want to emphasize the hallucination problem.
  • Gibbon1 2y ago
    Yeah. I'm not sure how transpiling to rust is that much different than using the various standard analysis tools. And back porting things like the counted by attribute. Also the lowest hanging fruit in C would be adding the ability to box and unbox fat pointers to objects.
  • geor9e 2y ago
    There's no direct translation back and forth between unsafe C to safe Rust tho, and theres infinite memory safe interpretations possible. If you're only interested in a black box executable, where certain tests pass, then I suppose you could just save the C and delete the Rust. But, the Rust has more information (that can't be deduced deterministically from the C). C would first get transpiled to unsafe Rust. Then some intelligence (A.I. or human) would get rid of all the unsafe keywords by making new design decisions that affect how the executable works inside. Each intelligence will do it differently. New edge case tests might give different outputs depending when you transpiled the C. It'd be better to save the Rust and make future changes without worrying if the latest A.I. will make the same design decisions at the prior A.I. each time you compile.
• constantcrying 2y ago
  I don't see this working. There are abstractions in C which are not replicable in Rust, without major changes. In C, having two separate data structures which carry an identical pointer and are writing to it is a common occurrence. This can not be trivially replicated in rust and will need some reasonably clever intervention.
  • rectang 2y ago
    I expect you can get something which compiles and is both “Rust” and “AI”, making it doubly buzzword-compliant and justifying the research project. What you won’t get is an output which is less buggy, or a process which automatically generates the program structure which yields Rust’s reliability.
• skywhopper 2y ago
  This is a terrible idea. In order to get rid of one specific class of bugs, you want to risk introducing logic errors and performance issues and make the code harder to maintain. Not to mention that this quote is incredibly scary. This is someone we are trusting to make this decision? "You can go to any of the LLM websites, start chatting with one of the AI chatbots, and all you need to say is 'here's some C code, please translate it to safe idiomatic Rust code,' cut, paste, and something comes out, and it's often very good, but not always," said Dan Wallach, DARPA program manager for TRACTOR, in a statement.
• 29athrowaway 2y ago
  Here they should take a gradual approach. 1. Create a tool that scores code depending on translation difficulty. 2. Automatically translate all code that can be directly translated. 3. Refactor the remaining C code into C or C++ code that is easier to translate. 4. Create a tool that suggests a translation but have a human review the changes. 5. Finally refactor everything else by hand. You're welcome, DARPA.
• Olumde 2y ago
  This initiative presupposes the apps are no longer under development. What happens if an app you are developing gets translated to a language you do not know?
• PaulHoule 2y ago
  Whatever happened to Ada?
  • wffurr 2y ago
    It languished in government work behind a wall of extremely expensive compilers and contractors. Never heard anyone suggest RiiA - Rewrite it in Ada.
    • nvy 2y ago
      GCC contains `gnat` which is a libre Ada compiler. I think Ada has a lot of technical merit but it's just not fashionable the way Rust is, for lots of uninteresting reasons.
      • PaulHoule 2y ago
        I remember Ada getting pushed in a time when there were many in the computer industry that were pushing Pascal as both a systems and a teaching language. Ada was a lot like Pascal which I think caused an immediate violent reaction in some people. (e.g. the implementers of every other programming language were pissed that BASIC was so hegemonic but they never asked "Why?" or if their alternatives were really any better) In the early 1980s, microcomputer implementations such as UCSD Pascal were absolutely horrific in terms of performance plus missing the features you'd need to do actual systems programming work. In the middle of the decade you saw Turbo Pascal which could compile programs before you aged to death and also extended Pascal sufficiently to compete with C. But then you had C, and the three-letter agencies were still covering up everything they knew about buffer overflows.
        • pjmlp 2y ago
          I wish Apple had stayed with Object Pascal, instead of having a couple of folks pushing for MPW, and later on PowerPlant. It could have pushed for more Pascal adoption. Then again, Borland also did their own mess, when decided enterprise should be their customer focus, not the small developer communities.
• rpoisel 2y ago
  I think we have to take that literally: They only translate C code to Rust. Not C++.
• sim7c00 2y ago
  i like the idea but i struggle to see how one can go about doing 'safe' disk reads, having 'safe' ways to manage global resources in kernel land (page tables, descriptor tables etc) and a lot of other stuff. perhaps if those devices also have rust in their firmware they can reply safely?? genuinely curious because i went back to C from rust in my OS. i could not figure it out (maybe i am not a darpa level engineer but i did work at a similar place doing similar things). id be excited if this gets solved. rust is a lot more comfy for higher level kernel stuff.
• ristos 2y ago
  I get the idea of moving to more memory safety, but the whole "rewrite everything in Rust" trend feels really misguided, because if you're talking about being able to trust code and code safety: - Rust's compiler is 1.8 million lines of recursively compiled code, how can you or anyone know that what was written is actually trustworthy? Also memory safety is just a very small part of being able to actually trust code. - C compiles down to straightforward assembly, almost like a direct translation, so you can at least verify that smaller programs that you write in C actually do compile down to assembly you expect, and compose those smaller programs into larger ones. - C has valgrind and ASAN so it's at least possible to write safe code with code coding discipline, and plenty of software has been able to do this for decades. - A lot of (almost all) higher level programming languages are written in C, which means that those languages just need to make sure they get the compiler and GC right, and then those languages can be used for general purpose, scripting, "low level" high level code like Go or OCaml, etc. - There are many C compilers and only one Rust compiler, and it's unclear whether it'll really be feasible to have more than one Rust compiler due to the complexity of the language. So you're putting a lot of trust into a small group of people, and even if they're the most amazing, most ethical people, surely if a lot of critical infra is based on Rust they'll get targeted in some way. - Something being open source doesn't mean it's been fully audited. We've seen all sorts of security vulnerabilities cause a world a hurt for a lot of people that came from all open source code, and often very small libraries that could actually be much easier to audit than lines with millions of lines of code. - Similarly, Rust does not translate to straightforward assembly, and again would seem to be impossible to do given the complexity of the language. - There was an interesting project I came across called CompCert, which aims to have a C compiler that's formally verified (in Coq) to translate into the assembly you expect. Something like a recursively compiled CompCert C -> OCaml -> Coq -> CompCert would be an interesting undertaking, which would make OCaml and Coq themselves built on formally verified code, but I'm not sure if that'll really work and I suspect it's too complicated. - I think Rust might be able to solve some of these problems if they have a fully formally verified thing, and the formally verified thing is itself formally verified, and the compiler was verified by that thing, and then you know that you can trust the whole thing. Still, the level of complexity and the inability to at least manually audit the core of it makes me suspect it's too complicated and would still be based on trust of some sort. - I still think that static analysis and building higher level languages on top of C is a better approach, and working on formal verification from there, because there are really small C compilers like tinycc that are ~50k LOCs, which can be hand verified. You can compile chibi-scheme with tinycc, for example, which is also about ~50k LOCs of C, and so you get a higher level language from about 100k LOCs (tcc and chibi), which is feasible for an ordinary but motivated dev to manually audit to know that it's producing sound assembly and not something wonky or sketchy. Ideally we should be building compilers and larger systems that are formally verified, but I think the core of whatever the formally verified system is has to be hand verifiable in some way in order to be trustworthy, so that you can by induction trust whatever gets built up from that, and I think that would need to require a straightforward translation into assembly, with ideally open source ISA and hardware, and a small enough codebase to be manually audited like the tinycc and chibi-scheme example I gave. - Worst case everyone kind of shrugs it all off and just trusts all of these layers of complexity, which can be like C -> recursively compiled higher level lang -> coffeescript-like layer on top -> framework, which is apparently a thing now, and just hope that all of these layers of millions of lines of code of complexity don't explode in some weird way, intentionally or unintentionally. - Best case of the worst case is that all of our appliances are now "smart" appliances, and then one day they just transform into robots that start chasing you around the house, all the while the Transformers cartoon theme is playing in the background while, which would match up nicely with the current trend of everything being both terrifying and hilarious in a really bizarre way.
• deleted 2y ago
  [deleted]
• plasticeagle 2y ago
  If 1) Rust contains no memory bugs 2) C can be automatically translated to it Then all memory bugs can be fixed automatically, which is almost certainly untrue. This task is very likely completely impossible in the general case.
  • warkdarrior 2y ago
    Since you did not specify that you wish to preserve all behaviors of the C code, there are trivial solutions to this problem. For example, one could replace all dynamic memory allocations with fixed buffers (set at translation time), and reject all inputs that do not fit in those buffers.
• deleted 2y ago
  [deleted]
• ksp-atlas 2y ago
  Technically, Zig has this functionality built in via translate-c, but it's designed for reading by a C compiler, not a human
  • deepsun 2y ago
    Well, the main idea is memory-safety. Zig is certainly better, but not as memory-safe. PS: Java or even JavaScript are memory-safe :)
    • pjmlp 2y ago
      Yes, Zig's memory safety is just like using Modula-2 from 1978. Definitly better than plain old C, but not what is being looked for here.
• kernal 2y ago
  I'm working on something similar that just wraps the C code in an Unsafe block.
• nickpsecurity 2y ago
  I think this is indirectly a great argument for automated, test generation or equivalence checking. The reason is that these translations might change the function of the code. Automated testing would show whether or not that happened. It also reveals many bugs. So, they should solve total, automated testing first. Maybe in parallel. Then, use it for equivalence checks.
• pizlonator 2y ago
  Or you could just use Fil-C.
• luke-stanley 2y ago
  Surely this could be better pitched to researchers as just another AI benchmark, a bit like ARC Prize? ;) There could be some exiting C projects that are already public, with tests for feedback during development iteration and some holdout tests, and some holdout projects too with a leaderboard and prizes. For preferences about converted code quality, both automated assesment and human preferences could be ranked with Elo? Kaggle is made for this sort of thing I think? I'm sure Google Deepmind and others have some MCTS agents that could do a great job with a bit of effort.
  • woodson 2y ago
    And like with most other competition/benchmark, the result is likely optimizing for the benchmark and not the wider goal ;-). It’s difficult to get a serious effort without people trying to game the benchmark.
    • luke-stanley 2y ago
      People could try but it would not help with the withheld datasets so much, and it would be possible to add more to it. If the withheld data was closed, and only available to an assessment system, gaming that would be pretty difficult. Scale.com's SEAL Leaderboards take a similar approach. The ARC Prize still exists too, and it's waiting for winners. See https://scale.com/leaderboard https://scale.com/leaderboard
• 29athrowaway 2y ago
  Why not Ada or Zig?
  • Ygg2 2y ago
    Zig is not safe, it's a C with a better templating system (comptime). Ada, is not popular enough, is my guess. To be fair, writing everything in Ada Spark would make code way more secure, simply because you'd need to write your pre-condition, invariants and post-conditions upfront, and prove they hold, but no one seems to want to think about lifetimes, let alone think about programming in more mathematical terms.
• llm_trw 2y ago
  I'm reminded of Darpa's old plans for Ada. I expect we'll see the same issues come up as the last time they tried this.
  • pjmlp 2y ago
    The problem with Ada, was the price of the compilers, and that the few UNIX vendors that bothered with Ada, like Sun, it was an additional license on top of the already expensive SunOS/Solaris Developer SDK. Thus the push for C and C++, alongside security certifications, where those languages feel like using Ada with a C like syntax. https://www.perforce.com/blog/kw/NASA-rules-for-developing-safety-critical-code https://www.perforce.com/blog/kw/NASA-rules-for-developing-s... https://yurichev.com/mirrors/C/JPL_Coding_Standard_C.pdf https://yurichev.com/mirrors/C/JPL_Coding_Standard_C.pdf https://misra.org.uk/ https://misra.org.uk/ Nowadays we live in a world where developers refuse to pay for their tools like other professionals, but hey, Rust is free beer, not like the several millions per seat licenses used by Ada vendors, of whom there are still 7 vendors in business.
• infamouscow 2y ago
  Anyone interested in this should apply, but also look into one of the small software consultants that does a lot of government contracting. Those consultants will likely also be involved in this and more potential opportunities to work on this. Also, the private sector pays (much) better and you'll have liaisons to handle most of the bureaucratic nonsense that accompanies a government job, especially within the morass of the DoD. Going into this without being prepared for immense political nonsense will not be effective.
• rustcleaner 2y ago
  Ah! Now's my chance to cheaply break into the dev field by becoming an expert Rust-all-the-C fairy who can flutter into high liability industries and get paid big bucks rebuilding the wheel into a memory-safe wheel! (I'm only being half-facetious, I fear I may never break in!)
• b20000 2y ago
  “ the software engineering community has reached a consensus” … hahaha no sorry, I don’t think so Your priority should be to learn how to build better software and not force a new language onto people. do you remember the age old saying about nature and fools?
  • stkdump 2y ago
    I guess it is a consensus like `goto considered harmful` or `numbering should start at zero`, which is not a perfect consensus, but as much of a consensus as you can reach for such a disparate community.
• Sytten 2y ago
  Would be nice if they could first hire all the smart engineers (that Mozilla laid off) to continue working on the language itself. Async is still a half finished mess even for people that use it every day. And that is my main annoyance but there are many (trait specialization, orphan rule limits, HKT, etc.)
• Thro4l31 2y ago
  The problems come with maintaining the translated code bases: 1. A code base written in C and a team of C engineers that have a good mental model of the code base to be able to maintain it. 2. An automatically translated Rust code base. Potentially (I'd say probably, but that is just my gut feeling) harder to read and understand than the original one. 3. Now you need a team of Rust engineers that have a good mental model of the code base that was generated. If you already have that team of Rust engineers, I'd rather let them rewrite the code manually as they can improve it and have the correct mental model from the start.
• Aissen 2y ago
  Isn't weird that they don't mention the already-existing open-source project they funded, c2rust? And no mention of the company behind it, Immunant, either.
• tpoacher 2y ago
  Is this part of the ongoing debate about farmers wanting to be able to fix their own TRACTORs? /s
• account42 2y ago
  > the software engineering community has reached a consensus lol
• anticristi 2y ago
  It sounds to me near-impossible to convert C or C++ into as-safe-as-possible Rust code, because the original intention of the developer are missing. However, I wonder if some clever generative AI could be taught to recognize sufficient C programming patterns in relevant code bases to make the problem tractable.