4 ms·

Hard for humans. But it's DARPA, is it hard for AI? Image classification used to be hard also, today cars drive themselves. I'd say it's good timing.

childintime 2y ago
In my experience claude.ai has near perfect grasp of what a program (that fits its window) is written to do. It can already make a program in another language that can do the same. What this means is that the cost of a full rewrite is going to come down dramatically over the next few years. This is an excellent example of government action I like to see, as it isn't about favoritism or the swamp dynamics. Just provide a target, a bounty and no or low barriers to entry. This challenge does push Rust out in front of everybody. That's a mixed blessing. I hope this challenge gets modified to not specify the target language, but instead the requirement of memory and type safety. Rust is likely an intermediate stop on the way to something better, and it shouldn't matter if that language is called Rust 2.0 or something else.
mike_hearn 2y ago
Well, Claude 3.5 can do translation from one language to another in a fairly competent manner if the languages are close enough. I've used it for that task myself with success (Java -> JavaScript). But, this isn't just about rewriting code from one language to another. It's about reverse engineering complex information out of the code, which may not be immediately visible in it, and then finding a way to make it "safe" according to Rust's type system. Where's the training data for that? It'd be really hard even for skilled humans. Personally I think the most pragmatic way to make C/C++ memory safe quicker is one of two approaches: 1. Incrementally. Make std::vector[] properly bounds checked (still not done even in chrome!), convert allocations to allocations that know their own size and do bounds checking e.g. https://issues.chromium.org/issues/40285824 https://issues.chromium.org/issues/40285824 2. Or, go the whole hog and use runtime techniques like garbage collection and runtime bounds checks. A good example of approach (2) is Managed Sulong, which extends the JVM to execute LLVM bitcode directly whilst exposing to the C/C++/FORTRAN a virtualized Linux syscall interface. The whole piece of code can be sandboxed with permissions, and memory safety errors are caught at runtime. The compiler tries to optimize out as many bounds checks as possible. The interesting thing about this approach is it doesn't require big changes to the source code (as long as it's already been ported to Linux), which means the work of making something safe can be done by teams independent of the original authors. In practice "rewrite it in Rust" will usually mean a fork, which introduces lots of complicated technical, cultural and economic issues. Managed Sulong is also a research project and has a bunch of problems to solve, for instance it needs to lose the JITC dependency and go fully AOT compiled (doable, there's no theoretical issue with it and much of the needed infra already exists). And performance/memory usage can always be improved of course, it regresses vs the original C. But those are "just" systems engineering problems, not rewrite-the-world and solve-static-analysis problems. Disclosure: I do work part time at Oracle Labs which developed Managed Sulong, but I don't work on it.
- TinkersW 2y ago
  std::vector [] has had bounds checking since forever if you set the correct compiler flag. Since they aren't using it this is a choice, presumably they prefer the speed gain.
  - mike_hearn 2y ago
    You mean _GLIBCXX_DEBUG? It's got some issues. Linux only, it doesn't always work [1] and it's all or nothing. What's really needed is the ability to selectively opt-out on a per-instantiation level so very hot paths can keep the needed performance whilst all the rest gets opted into safety checks. Microsoft has this: https://learn.microsoft.com/en-us/cpp/standard-library/safe-libraries-cpp-standard-library?view=msvc-170 https://learn.microsoft.com/en-us/cpp/standard-library/safe-... but it doesn't seem to actually make std::vector[] safe. It's frustrating that low hanging fruit like this doesn't get harvested. [1] "although there are precondition checks for some string operations, e.g. operator[], they will not always be run when using the char and wchar_t specializations (std::string and std::wstring)."
    - TinkersW 2y ago
      With MSVC you can use _CONTAINER_DEBUG_LEVEL=1 to get a fast bounds check that can be used in release builds. Or just use it in development to catch errors.
      - mike_hearn 2y ago
        Interesting thanks. Seems the reason I couldn't find anything on that is because it's internal only and not a feature you're actually meant to use? https://github.com/microsoft/STL/issues/586 https://github.com/microsoft/STL/issues/586 > We talked about this at the weekly maintainer meeting and decided that we're not comfortable enough with the (lack of) design of this feature to begin documenting it for wide usage.
        pjmlp 2y ago
Calavar 2y ago
> today cars drive themselves You can attach about a hundred asterisks to that. If anything, I think self the failure to hit L5 driving after billions of dollars and millions of man hours invested is probably reflective of how automatic C to Rust translation will go. We'll cruise 90% of the way, but the last 10% will prove insurmountable with current technology. Think about the number of C programs in the wild that rely on compiler-specific or libc-specific or platform-specific behavior, or even undefined behavior plus the dumb luck of a certain brittle combination of {compiler version} ∩ {libc version} ∩ {linker version} ∩ {build flags} emitting workable machine code. There's a huge chunk of C software where there's not enough context within the source itself (or even source plus build scripts) to understand the behavior. It's not even clear that this is a solvable problem in the abstract. None of that is to say that DARPA shouldn't fund this. Research isn't always about finding an industrial strength end product; the knowledge and expertise gained along the way is important too.
- psychoslave 2y ago
  Ok, but if it's like 90% of small projects can use it as direct no pain bridge, that can be a huge win. Even if it's "can handle well 90%" of the transition for any project, this is still interesting. Unlike cars on the road, most code transition project out there doesn't need to be 100% fine to provide some useful value.
  - 0cf8612b2e1e 2y ago
    Even if every project can only be 90% done, that’s a huge win. Best would be if it could just wrap the C equivalent code into an unsafe block which would be automatically triaged for human review. Just getting something vaguely Rust shaped which can compile is the first step in overcoming the inertia to leave the program in its current language.
    - swiftcoder 2y ago
      c2rust exists today, and pretty much satisfies this. I've used it to convert a few legacy math libraries to unsafe rust, and then been able to do the unsafe->safe refactor in the relative comfort of the full rust toolset (analyser + IDE + tests) There is real utility in slowly fleshing out the number of transforms in a tool like c2rust that can recognise high-level constructs in C code and produce idiomatic safe equivalents in rust
eesmith 2y ago
As a reminder, DARPA funded self-driving car research since at least the 1980s with the Autonomous Land driven Vehicle (ALV) project, plus the DARPA Grand Challenges, and more.

childintime 2y ago

In my experience claude.ai has near perfect grasp of what a program (that fits its window) is written to do. It can already make a program in another language that can do the same. What this means is that the cost of a full rewrite is going to come down dramatically over the next few years. This is an excellent example of government action I like to see, as it isn't about favoritism or the swamp dynamics. Just provide a target, a bounty and no or low barriers to entry. This challenge does push Rust out in front of everybody. That's a mixed blessing. I hope this challenge gets modified to not specify the target language, but instead the requirement of memory and type safety. Rust is likely an intermediate stop on the way to something better, and it shouldn't matter if that language is called Rust 2.0 or something else.

mike_hearn 2y ago

Well, Claude 3.5 can do translation from one language to another in a fairly competent manner if the languages are close enough. I've used it for that task myself with success (Java -> JavaScript). But, this isn't just about rewriting code from one language to another. It's about reverse engineering complex information out of the code, which may not be immediately visible in it, and then finding a way to make it "safe" according to Rust's type system. Where's the training data for that? It'd be really hard even for skilled humans. Personally I think the most pragmatic way to make C/C++ memory safe quicker is one of two approaches: 1. Incrementally. Make std::vector[] properly bounds checked (still not done even in chrome!), convert allocations to allocations that know their own size and do bounds checking e.g. https://issues.chromium.org/issues/40285824 https://issues.chromium.org/issues/40285824 2. Or, go the whole hog and use runtime techniques like garbage collection and runtime bounds checks. A good example of approach (2) is Managed Sulong, which extends the JVM to execute LLVM bitcode directly whilst exposing to the C/C++/FORTRAN a virtualized Linux syscall interface. The whole piece of code can be sandboxed with permissions, and memory safety errors are caught at runtime. The compiler tries to optimize out as many bounds checks as possible. The interesting thing about this approach is it doesn't require big changes to the source code (as long as it's already been ported to Linux), which means the work of making something safe can be done by teams independent of the original authors. In practice "rewrite it in Rust" will usually mean a fork, which introduces lots of complicated technical, cultural and economic issues. Managed Sulong is also a research project and has a bunch of problems to solve, for instance it needs to lose the JITC dependency and go fully AOT compiled (doable, there's no theoretical issue with it and much of the needed infra already exists). And performance/memory usage can always be improved of course, it regresses vs the original C. But those are "just" systems engineering problems, not rewrite-the-world and solve-static-analysis problems. Disclosure: I do work part time at Oracle Labs which developed Managed Sulong, but I don't work on it.

TinkersW 2y ago
std::vector [] has had bounds checking since forever if you set the correct compiler flag. Since they aren't using it this is a choice, presumably they prefer the speed gain.
- mike_hearn 2y ago
  You mean _GLIBCXX_DEBUG? It's got some issues. Linux only, it doesn't always work [1] and it's all or nothing. What's really needed is the ability to selectively opt-out on a per-instantiation level so very hot paths can keep the needed performance whilst all the rest gets opted into safety checks. Microsoft has this: https://learn.microsoft.com/en-us/cpp/standard-library/safe-libraries-cpp-standard-library?view=msvc-170 https://learn.microsoft.com/en-us/cpp/standard-library/safe-... but it doesn't seem to actually make std::vector[] safe. It's frustrating that low hanging fruit like this doesn't get harvested. [1] "although there are precondition checks for some string operations, e.g. operator[], they will not always be run when using the char and wchar_t specializations (std::string and std::wstring)."
  - TinkersW 2y ago
    With MSVC you can use _CONTAINER_DEBUG_LEVEL=1 to get a fast bounds check that can be used in release builds. Or just use it in development to catch errors.
    - mike_hearn 2y ago
      Interesting thanks. Seems the reason I couldn't find anything on that is because it's internal only and not a feature you're actually meant to use? https://github.com/microsoft/STL/issues/586 https://github.com/microsoft/STL/issues/586 > We talked about this at the weekly maintainer meeting and decided that we're not comfortable enough with the (lack of) design of this feature to begin documenting it for wide usage.
      - pjmlp 2y ago

Calavar 2y ago

> today cars drive themselves You can attach about a hundred asterisks to that. If anything, I think self the failure to hit L5 driving after billions of dollars and millions of man hours invested is probably reflective of how automatic C to Rust translation will go. We'll cruise 90% of the way, but the last 10% will prove insurmountable with current technology. Think about the number of C programs in the wild that rely on compiler-specific or libc-specific or platform-specific behavior, or even undefined behavior plus the dumb luck of a certain brittle combination of {compiler version} ∩ {libc version} ∩ {linker version} ∩ {build flags} emitting workable machine code. There's a huge chunk of C software where there's not enough context within the source itself (or even source plus build scripts) to understand the behavior. It's not even clear that this is a solvable problem in the abstract. None of that is to say that DARPA shouldn't fund this. Research isn't always about finding an industrial strength end product; the knowledge and expertise gained along the way is important too.

psychoslave 2y ago
Ok, but if it's like 90% of small projects can use it as direct no pain bridge, that can be a huge win. Even if it's "can handle well 90%" of the transition for any project, this is still interesting. Unlike cars on the road, most code transition project out there doesn't need to be 100% fine to provide some useful value.
- 0cf8612b2e1e 2y ago
  Even if every project can only be 90% done, that’s a huge win. Best would be if it could just wrap the C equivalent code into an unsafe block which would be automatically triaged for human review. Just getting something vaguely Rust shaped which can compile is the first step in overcoming the inertia to leave the program in its current language.
  - swiftcoder 2y ago
    c2rust exists today, and pretty much satisfies this. I've used it to convert a few legacy math libraries to unsafe rust, and then been able to do the unsafe->safe refactor in the relative comfort of the full rust toolset (analyser + IDE + tests) There is real utility in slowly fleshing out the number of transforms in a tool like c2rust that can recognise high-level constructs in C code and produce idiomatic safe equivalents in rust

eesmith 2y ago

As a reminder, DARPA funded self-driving car research since at least the 1980s with the Autonomous Land driven Vehicle (ALV) project, plus the DARPA Grand Challenges, and more.