5 ms·

Show HN: Copper – Open-source robotics in Rust with deterministic log replay

161 points by gbin 2y ago

a_t48 2y ago
Nice release - deterministic replay isn't easy - I still need to go back and pull our implementation forward to match the new features in the core framework. I'm not sure if you've seen Basis, but we're somewhat "competing" in the same space. You're all in on Rust, we're C++ with bindings to other languages (eventually). I don't see this as a bad thing - more tools in the space is a win, eventually the industry will settle on something better than ROS.
- AlotOfReading 2y ago
  Having maintained something similar myself, how has your experience been with exact determinism in C++? C++ compilers tend to be a bit of a free-for-all when it comes to bit determinism, whereas the rustc folks are very good about spending the design time to preemptively avoid many of the issues you'd need workarounds for in other languages.
  - a_t48 2y ago
    My goal is determinism specifically around message timings/execution in replay. We don't currently advertise runtime determinism, we also don't promise fixes for determinism in your code. For message timing, the algorithm should be stable - it generates a timeline for when messages should be executed by each message handler, and will pause further executions to wait on handlers that "should have" finished, but haven't due to running too slow. Ties are broken by the handler name or data source name. I wrote a bit about it here https://basisrobotics.tech/2024/09/02/determinism/ https://basisrobotics.tech/2024/09/02/determinism/ but didn't go deep into the implementation. If your own code isn't deterministic you will have differences, but this eliminates the biggest classes of nondeterminism when running tests (threading timings, message handling races). I don't think Copper claims to offer more than this either, but I can't speak for them.
pryelluw 2y ago
Such a great two sentence intro to the project: “ Copper is a user-friendly runtime engine for creating fast and reliable robots. Copper is to robots what a game engine is to games.” Other projects should take notice.
- deleted 2y ago
  [deleted]
- gbin 2y ago
  Oh wow thanks, I have to tell you something... It is really hard to explain what a piece of middleware is and why it is important :)
amacneil 2y ago
It’s great to see more innovation in the robotics framework space, it’s sorely needed. Deterministic log replay is a killer feature to have baked in from the start - many autonomous vehicle & robotics companies have invested years of effort to achieve that nirvana, while the most popular robotics framework today (ROS) makes it borderline impossible.
- jeffreygoesto 2y ago
  True. All of it. https://edms.etas.com https://edms.etas.com for example.
the__alchemist 2y ago
Tangent this got me thinking about: Why are the robotics and embedded communities generally separate, with little overlap in people and tech? For example, this project seems unrelated to the rust embedded tooling. And the components categories are also unfamiliar, e.g. the IMUs and ADCs in the readme, using a SBC vice a MCU etc. Another angle: `ROS` and `RTOS` share letters, but not much else! I gather that robotics is a fusion of embedded and mechanical engineering; I refer to the former.
- gbin 2y ago
  It almost feels like we would need an "in between" category. If I paint a broad brush: - embedded is all about latency, low bandwidth - computers are all about bandwidth with often terrible latencies. A modern robot needs both low latency and high bandwidth.
  - the__alchemist 2y ago
    I suppose this lends itself to a hybrid scenario. I suspect most mature robotics projects would have both types of computation. Likely bare metal or RTOS as the default, with delegation to GPOS computer for high bandwidth, and high computation tasks such as CV. I also imagine distributed computing (for example: over CAN or another bus) would be a useful pattern.
- ModernMech 2y ago
  Typically with robots you need more power than what embedded provides. We build on top of commodity x86 systems a lot of the time. Other times we use SOCs like the Nvidia Jetson line, which also doesn't require a lot of embedded skills; you get a full operating system and a userspace so it's more like programming on a typical unix-based system. You find more embedded in the automation side of things. Wearables, IOT, always connected devices -- things that are too small and low powered to afford an OS.
  - AlotOfReading 2y ago
    There's a lot of embedded skills in pretty much any hardware platform. Maybe this is selection bias from being an embedded-focused person in robotics, but there's plenty of jobs out there that need a toolchain wrangler who can deal with cost optimization.
cchance 2y ago
If this pics up support for like a open-library of assorted components and people shared them it has a lot of potential especially if the sim env and replays continues to be improved. I love the idea and looks gerat!
cpgxiii 2y ago
This seems to suffer a bit from the same problem that affects a lot of "new" robotics frameworks: the beginnings are simple, and it's easy at that stage to think that everything else is over-complicated. The comparison with ROS 2 is a bit questionable. Comparing a single-process-only (Copper) approach using shared memory with a multi-process system (ROS 2) using default DDS settings really isn't comparing the same thing. There are ways to make the ROS 2 system much faster if you're willing to be limited to similar constraints (single process with components, or local-system shared-memory transport) but most people don't because those constraints are very limiting in practical applications.
- gbin 2y ago
  Note that Copper can be distributed too, we do support iceoryx2 for example if you need to split it in 2 or more processes. We have ~100x less latency and ~12x faster logging also because we have adopted a data oriented architecture: the tasks outputs are written back to back in memory, all the IOs are linear (we could log to a block device straight, we don't even need a filesystem). I am not sure it is possible to touch this with ROS just because of its "everything is asynchronous" design pattern. So the question is more about those limits in practical applications: do you have use cases where you absolutely need every single component deployed as a kind of micro service in a robot?
- rcxdude 2y ago
  I will say the default network-transparency of ROS is probably more of a long-term hindrance than a help to applications built on top of it. A lot of projects would get a lot further if they were designed from the outset with clear boundaries where network communication was required (and what can be done to deal with what that implies), and it would be nice if components intended to be used in those systems were designed to be composed with that in mind, instead of requiring a network-based pub-sub system with not particularly great latency or reliability guarantees, even if it has a mode where those are tightened up a lot.
- patrick451 2y ago
  I disagree that the ROS2 comparison is questionable. If ROS want's to be seen as performant, they need to use a performant middleware by default. You shouldn't have to fiddle endlessly with it to get reasonable latencies. Having worked on both large ROS systems and large monolithic robotics systems, the monolith wins hands down. The ROS paradigm that every node is it's own process is frankly insane.
rgovostes 2y ago
Do I understand that at each tick, every task outputs a single message, and can depend on the outputs of other tasks? I.e., nodes and topics are merged into tasks, executed in topological order?
- gbin 2y ago
  Exactly. Technically Copper is a compiler taking your graph with those constraints in mind and building a game loop out of it.
  - BWStearns 2y ago
    This reminds me, I've been messing around with Bevy recently and have been wondering if ECS systems might be suited to hardware. On the surface it seems like a good ergonomic fit. Not sure if ECS is a performance/determinism killer but since they're used in gaming where performance is often critical I imagine it's surmountable?
    - gbin 2y ago
      I believe that at least the initial version of ISAAC from Nvidia was based on an ECS. I don't think it is a determinism killer, this is often a well ordered pipeline within a game loop. In terms of architecture though, it looks like it makes a ton of sense inside something like a tracker (ie the entity is basically a track) but for other parts like a vision pipeline, sensor fusion etc... I don't see how it helps. Tell me if I am missing something. Talking about Bevy, fitting Copper within Bevy to build this little simulation example happened super naturally: Copper is a System querying the Entities within the virtual world after each simulation tick.
      - BWStearns 2y ago
        Very cool about ISAAC. I didn't know. As far as how it helps, I was mostly thinking from a dev ergonomics perspective. For things like ROS/mavlink it can sometimes be hard (for me) to think through how all the systems are interacting but for whatever reason ECSs feel like a natural way to think about systems with simultaneousish inputs and outputs.
leonheld 2y ago
> Unlike a game engine we use a data oriented approach to minimize latency and maximize throughput. Don't you mean "like"? I thought game engines were all about data oriented approach.
- gbin 2y ago
  Indeed thanks. "Not unlike" :)
ericyd 2y ago
I know nothing about robotics. Is deterministic log replay really a differentiating factor? My naive assumption would be that this is table stakes for pretty much any software.
- chippiewill 2y ago
  ROS is probably the biggest robotics framework and doesn't have deterministic playback. It's important for safety critical to be sure, but you can get surprisingly far without it.
elmerfuddwalz 2y ago
[dead]

161 points by gbin 2y ago

Nice release - deterministic replay isn't easy - I still need to go back and pull our implementation forward to match the new features in the core framework. I'm not sure if you've seen Basis, but we're somewhat "competing" in the same space. You're all in on Rust, we're C++ with bindings to other languages (eventually). I don't see this as a bad thing - more tools in the space is a win, eventually the industry will settle on something better than ROS.

AlotOfReading 2y ago
Having maintained something similar myself, how has your experience been with exact determinism in C++? C++ compilers tend to be a bit of a free-for-all when it comes to bit determinism, whereas the rustc folks are very good about spending the design time to preemptively avoid many of the issues you'd need workarounds for in other languages.
- a_t48 2y ago
  My goal is determinism specifically around message timings/execution in replay. We don't currently advertise runtime determinism, we also don't promise fixes for determinism in your code. For message timing, the algorithm should be stable - it generates a timeline for when messages should be executed by each message handler, and will pause further executions to wait on handlers that "should have" finished, but haven't due to running too slow. Ties are broken by the handler name or data source name. I wrote a bit about it here https://basisrobotics.tech/2024/09/02/determinism/ https://basisrobotics.tech/2024/09/02/determinism/ but didn't go deep into the implementation. If your own code isn't deterministic you will have differences, but this eliminates the biggest classes of nondeterminism when running tests (threading timings, message handling races). I don't think Copper claims to offer more than this either, but I can't speak for them.

pryelluw 2y ago

Such a great two sentence intro to the project: “ Copper is a user-friendly runtime engine for creating fast and reliable robots. Copper is to robots what a game engine is to games.” Other projects should take notice.

deleted 2y ago
[deleted]
gbin 2y ago
Oh wow thanks, I have to tell you something... It is really hard to explain what a piece of middleware is and why it is important :)

amacneil 2y ago

It’s great to see more innovation in the robotics framework space, it’s sorely needed. Deterministic log replay is a killer feature to have baked in from the start - many autonomous vehicle & robotics companies have invested years of effort to achieve that nirvana, while the most popular robotics framework today (ROS) makes it borderline impossible.

jeffreygoesto 2y ago
True. All of it. https://edms.etas.com https://edms.etas.com for example.

the__alchemist 2y ago

Tangent this got me thinking about: Why are the robotics and embedded communities generally separate, with little overlap in people and tech? For example, this project seems unrelated to the rust embedded tooling. And the components categories are also unfamiliar, e.g. the IMUs and ADCs in the readme, using a SBC vice a MCU etc. Another angle: `ROS` and `RTOS` share letters, but not much else! I gather that robotics is a fusion of embedded and mechanical engineering; I refer to the former.

gbin 2y ago
It almost feels like we would need an "in between" category. If I paint a broad brush: - embedded is all about latency, low bandwidth - computers are all about bandwidth with often terrible latencies. A modern robot needs both low latency and high bandwidth.
- the__alchemist 2y ago
  I suppose this lends itself to a hybrid scenario. I suspect most mature robotics projects would have both types of computation. Likely bare metal or RTOS as the default, with delegation to GPOS computer for high bandwidth, and high computation tasks such as CV. I also imagine distributed computing (for example: over CAN or another bus) would be a useful pattern.
ModernMech 2y ago
Typically with robots you need more power than what embedded provides. We build on top of commodity x86 systems a lot of the time. Other times we use SOCs like the Nvidia Jetson line, which also doesn't require a lot of embedded skills; you get a full operating system and a userspace so it's more like programming on a typical unix-based system. You find more embedded in the automation side of things. Wearables, IOT, always connected devices -- things that are too small and low powered to afford an OS.
- AlotOfReading 2y ago
  There's a lot of embedded skills in pretty much any hardware platform. Maybe this is selection bias from being an embedded-focused person in robotics, but there's plenty of jobs out there that need a toolchain wrangler who can deal with cost optimization.

cchance 2y ago

If this pics up support for like a open-library of assorted components and people shared them it has a lot of potential especially if the sim env and replays continues to be improved. I love the idea and looks gerat!

cpgxiii 2y ago

This seems to suffer a bit from the same problem that affects a lot of "new" robotics frameworks: the beginnings are simple, and it's easy at that stage to think that everything else is over-complicated. The comparison with ROS 2 is a bit questionable. Comparing a single-process-only (Copper) approach using shared memory with a multi-process system (ROS 2) using default DDS settings really isn't comparing the same thing. There are ways to make the ROS 2 system much faster if you're willing to be limited to similar constraints (single process with components, or local-system shared-memory transport) but most people don't because those constraints are very limiting in practical applications.

gbin 2y ago
Note that Copper can be distributed too, we do support iceoryx2 for example if you need to split it in 2 or more processes. We have ~100x less latency and ~12x faster logging also because we have adopted a data oriented architecture: the tasks outputs are written back to back in memory, all the IOs are linear (we could log to a block device straight, we don't even need a filesystem). I am not sure it is possible to touch this with ROS just because of its "everything is asynchronous" design pattern. So the question is more about those limits in practical applications: do you have use cases where you absolutely need every single component deployed as a kind of micro service in a robot?
rcxdude 2y ago
I will say the default network-transparency of ROS is probably more of a long-term hindrance than a help to applications built on top of it. A lot of projects would get a lot further if they were designed from the outset with clear boundaries where network communication was required (and what can be done to deal with what that implies), and it would be nice if components intended to be used in those systems were designed to be composed with that in mind, instead of requiring a network-based pub-sub system with not particularly great latency or reliability guarantees, even if it has a mode where those are tightened up a lot.
patrick451 2y ago
I disagree that the ROS2 comparison is questionable. If ROS want's to be seen as performant, they need to use a performant middleware by default. You shouldn't have to fiddle endlessly with it to get reasonable latencies. Having worked on both large ROS systems and large monolithic robotics systems, the monolith wins hands down. The ROS paradigm that every node is it's own process is frankly insane.

rgovostes 2y ago

Do I understand that at each tick, every task outputs a single message, and can depend on the outputs of other tasks? I.e., nodes and topics are merged into tasks, executed in topological order?

gbin 2y ago
Exactly. Technically Copper is a compiler taking your graph with those constraints in mind and building a game loop out of it.
- BWStearns 2y ago
  This reminds me, I've been messing around with Bevy recently and have been wondering if ECS systems might be suited to hardware. On the surface it seems like a good ergonomic fit. Not sure if ECS is a performance/determinism killer but since they're used in gaming where performance is often critical I imagine it's surmountable?
  - gbin 2y ago
    I believe that at least the initial version of ISAAC from Nvidia was based on an ECS. I don't think it is a determinism killer, this is often a well ordered pipeline within a game loop. In terms of architecture though, it looks like it makes a ton of sense inside something like a tracker (ie the entity is basically a track) but for other parts like a vision pipeline, sensor fusion etc... I don't see how it helps. Tell me if I am missing something. Talking about Bevy, fitting Copper within Bevy to build this little simulation example happened super naturally: Copper is a System querying the Entities within the virtual world after each simulation tick.
    - BWStearns 2y ago
      Very cool about ISAAC. I didn't know. As far as how it helps, I was mostly thinking from a dev ergonomics perspective. For things like ROS/mavlink it can sometimes be hard (for me) to think through how all the systems are interacting but for whatever reason ECSs feel like a natural way to think about systems with simultaneousish inputs and outputs.

leonheld 2y ago

> Unlike a game engine we use a data oriented approach to minimize latency and maximize throughput. Don't you mean "like"? I thought game engines were all about data oriented approach.

gbin 2y ago
Indeed thanks. "Not unlike" :)

ericyd 2y ago

I know nothing about robotics. Is deterministic log replay really a differentiating factor? My naive assumption would be that this is table stakes for pretty much any software.

chippiewill 2y ago
ROS is probably the biggest robotics framework and doesn't have deterministic playback. It's important for safety critical to be sure, but you can get surprisingly far without it.

elmerfuddwalz 2y ago

[dead]