6 ms·

Pub/Sub is a very natural fit for robotics. Companies do write ROS out of their production stacks when they get the resources. But they don’t replace pub/sub ar

Pub/Sub is a very natural fit for robotics. Companies do write ROS out of their production stacks when they get the resources. But they don’t replace pub/sub architecture. ROS provides a plug-in backplane that allows you to innovate in one area while leveraging existing components for parts that are not your differentiator.

cheeselip420 3y ago
pub/sub is NOT a natural fit for robotics - you want bounded timing and generally for error-handling you want to know what happened as a result of a message being published. ROS introduced "commands" and "actions" to attempt to work around this, but its all just shit piled on shit. Just make a multi-threaded app, and call functions. If you need to distribute over multiple CPUs, then go ahead and do some IPC. But pub/sub is NOT an architecture. Its a soup of tightly coupled fragments of functionality. It falls apart very quickly in the real world.
- moglito 3y ago
  Are you aware that ros service calls are RPCs (not based on pub/sub like actions)? Furthermore, if you use nodelets (http://wiki.ros.org/nodelet http://wiki.ros.org/nodelet) you get zero copy communication between your algorithms. So I actually think that ROS has the facilities to address the needs you describe. I'd be interested, though, in your suggestions on what a real architecture for robotics looks like in your mind? I still remember the time before ROS, 20 years ago, when each robotics team had to designate a sub-team just for building and maintaining the middleware. That was a waste of time and effort. But you seem to suggest that we go back to that? ROS might not be perfect but it's so much better than anything else that exists. It's also open source and we can all work together to make it better rather than reinventing the wheel each time.
  - cheeselip420 3y ago
    Services and Actions are built on top of pubsub (with separate Request/Response vs Goal/Feedback/Result topics respectively). At least w/ ROS1 - I'm not sure if ROS2 improved things here... Nodelets are also a disaster, which is why ROS2 kinda fixed this by decoupling nodes and processes. When you're just starting, ROS can be nice for prototyping - you get a batteries-included platform that can do some SLAM and simple motion planning. But as you start adding new features, you need to figure out how to add those features over multiple nodes. This coordination overhead can quickly bring your system to its knees, or at least make it extremely difficult to debug and troubleshoot when things go wrong. No one should be building or maintaining middleware. Build robots. Read your sensor data, build a model of the world, decide what to do, then send commands to your control systems. This is the hard part of robotics. ROS solves the easiest part of robotics (plumbing and process management) in the shittiest possible way.
  - snovv_crash 3y ago
    The async nature of pubsub makes it great for isolating your part of the system, but moves the complexity to the system integration instead. Actually deploying a ROS based system is about as difficult as rewriting the whole thing from scratch as a monolith. Every time something goes over a pubsub it's like using a GOTO, except your debugger can't actually follow it, and you don't even know who was listening (or who wasn't that should have been) or what the downstream effects were. It makes it impossible to properly debug a system, because it isn't deterministic so you can never be sure if you've actually handled all of the edge cases, since there is a temporal component to the state that can't be reproduced. A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates. The reason ROS has traction is because of university labs, who just need something to work once to be able to publish their paper or write their dissertation. In industry the reliability requirements are much higher, and despite the intensive efforts from the ROS community to "industralize" ROS via adoption of DDS, there seemed to be little understanding that the message protocol wasn't the reason the industry uptake was so low.
    - dagar 3y ago
      > A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates. This is how I've structured a control system, but using a pub/sub system to share data across those boundaries (and log/inspect in general). "Nodes" that share some resource or fundamentally run back to back based on the data flow can live in the same thread. Higher rate components (eg inner loop controller) live in a thread with a much higher priority. All of this is event driven, deterministic, and testable. If you have more details about the system you're imagining I'd love to learn more, because so far I don't see what's incompatible with what you've described.
      - snovv_crash 3y ago
        In general PX4 is better than ROS in this respect, but IMO still leans too heavily on queues. A bit of feedback there: - more should be done by setting up constrained functionality-specific data, and then simply calling a function with just that data. Right now a lot of things are passed state they don't need just because it is part of the message they receive. This makes the code dependencies way harder to separate, because you effectively share function signatures (messages) between different modules. Of secondary concern is the extra memory bandwidth from the extra data passed around, and not being able to pass by const& due to the async. - lots of things don't need updating at all until they tick over. If you don't update it frequently it has old data. You can try to make sure it works with the latest data by updating it frequently, but that of course has big overhead. I don't see any decent way of making this work unless you either 1) set up global threadsafe state so everything can access it, which is bad from a dependency and locking perspective, or 2) just call functions synchronously with exactly the data they need. - The issue with message queues, beyond the need for messages to generalize as mentioned above, is that often components need multiple different messages from different sources to perform their tasks. This means every component needs to keep a local copy of the data they need, to translate this async data back into a synchronous paradigm when it runs. In fact, beyond the message passing itself, pretty much everything needs to do its work in a synchronous paradigm, so why even add the async stuff in the middle to begin with? Once the messiness of different sensors' reporting rates is consolidated into the EKF state, and external commands and directives and brought in, after that point pretty much everything could be synchronous. No overhead, no timing issues, no "which message has the data I need", no "why do we have 3 different variants of the same message with slightly different data and which one should I use", etc. As long as you log the initial input data it should still be able to do replays for reproduction of realtime behaviour, but a) better testing can be done because you can actually tell what every subsystem needs to run correctly just by looking at its function signature, and b) its much easier to refactor (and understand) because of the same properties of the function signature describing all of the inputs and outputs.
- chippiewill 3y ago
  Actually I think it's more complicated than that. Pub/sub is a natural fit for robotics in a prototyping sense, but it's a poor fit for real-time systems which makes it harder to productionise something based on ROS. Especially safety critical systems. "Just" a multi-threaded app and calling functions isn't really a good replacement. For something really basic, sure, but the modularity that ROS introduces, and the core principle of having serialisable recordable messages delivers a lot of value once you start building large systems - particularly those that can span multiple physical machines. But pub-sub is _too_ general as a concurrency model, if ROS had adopted something more constrained that was provably deterministic and was amenable to real time analysis then a lot of its concerns would go away.
  - snovv_crash 3y ago
    It would also help to have compile-time guarantees about which nodes are running. But I always like to say, why spend 5ns calling a function when you could spend 1ms waiting for a context switch or 50ms for the next event loop to tick over? Even modern microcontrollers are blindingly fast, and the majority of the time they spend these days is because of our brain-dead architecture decisions, not due to the underlying problem that actually needs solving.
    - dagar 3y ago
      > It would also help to have compile-time guarantees about which nodes are running. I haven't had a chance to try it yet, but in theory I like the idea of something like Zenoh Flow to describe the larger data flow graph. https://zenoh.io/blog/2023-02-10-zenoh-flow/ https://zenoh.io/blog/2023-02-10-zenoh-flow/ > But I always like to say, why spend 5ns calling a function when you could spend 1ms waiting for a context switch or 50ms for the next event loop to tick over? I think your context switch timing is off by several orders of magnitude, but regardless these things aren't one extreme or the other. For sharing data across threads, to an external system, or logging and visualization I still like pub/sub (and I've seen more than my share of horrible abuse), but it definitely shouldn't be treated as one size fits all.
      - 3y ago
- dbcurtis 3y ago
  A single multi-threaded app is a toy-sized robot, not a product with a large dev team behind it. The single-app approach doesn’t scale up very well.
  - cheeselip420 3y ago
    Sure it does. I've seen teams of >100 engineers working on largely monolithic robotic codebases across several teams/companies (volumes in the 10s of thousands). Beyond that, robotics isn't all that special. There are other domains which have soft-realtime requirements and huge scale. AAA video games are a multi-$100-billion industry, which largely ship single-process systems with extreme performance on par with state-of-the-art perception and motion planning use-cases. AR/VR is another area where this is interesting - https://twitter.com/ID_AA_Carmack/status/1497098684546400260 https://twitter.com/ID_AA_Carmack/status/1497098684546400260 - Carmack has alluded to this issue multiple times.
    - rektide 3y ago
      It'll be interesting to see how/when wasm shows up in vr & robotics. Being able to have some shared memory you pass around between safely sandboxed worklets still has any of the same coordination problems. But it at least has much much much less multiprocess jank.
      - snovv_crash 3y ago
        What does the sandbox really gain you? Is it anything that couldn't just be handled at the OS level?
        rektide 3y ago
        It let's you safely run lots of different code in a single processs, without paying the context switching cost. Quests's jank problem is probably in part that multiprocess architecture has to spawn new processes & change priorities of what's running & handle expensive context switches between processes elegantly, which is hard. Not only is non-monolithic code sometimes harder to reason about, it has all kinds of performance boundaries that are very painful to cut through. Read through Pipewire changelogs & how they've worked so hard to make best use of kernel primitives with less and less overhead, look how hard they've worked to get near Jack levels of efficiency. Great example of the slow march of multi-process optimization. Sandboxes let you collocate different processes inside the same process. You still have many of the same architectural challenges, but the ability to spawn new code & run it has much much lower overhead. It creates the possibility to make way more multi-process like architectures than we can explore now.

cheeselip420 3y ago

pub/sub is NOT a natural fit for robotics - you want bounded timing and generally for error-handling you want to know what happened as a result of a message being published. ROS introduced "commands" and "actions" to attempt to work around this, but its all just shit piled on shit. Just make a multi-threaded app, and call functions. If you need to distribute over multiple CPUs, then go ahead and do some IPC. But pub/sub is NOT an architecture. Its a soup of tightly coupled fragments of functionality. It falls apart very quickly in the real world.

moglito 3y ago
Are you aware that ros service calls are RPCs (not based on pub/sub like actions)? Furthermore, if you use nodelets (http://wiki.ros.org/nodelet http://wiki.ros.org/nodelet) you get zero copy communication between your algorithms. So I actually think that ROS has the facilities to address the needs you describe. I'd be interested, though, in your suggestions on what a real architecture for robotics looks like in your mind? I still remember the time before ROS, 20 years ago, when each robotics team had to designate a sub-team just for building and maintaining the middleware. That was a waste of time and effort. But you seem to suggest that we go back to that? ROS might not be perfect but it's so much better than anything else that exists. It's also open source and we can all work together to make it better rather than reinventing the wheel each time.
- cheeselip420 3y ago
  Services and Actions are built on top of pubsub (with separate Request/Response vs Goal/Feedback/Result topics respectively). At least w/ ROS1 - I'm not sure if ROS2 improved things here... Nodelets are also a disaster, which is why ROS2 kinda fixed this by decoupling nodes and processes. When you're just starting, ROS can be nice for prototyping - you get a batteries-included platform that can do some SLAM and simple motion planning. But as you start adding new features, you need to figure out how to add those features over multiple nodes. This coordination overhead can quickly bring your system to its knees, or at least make it extremely difficult to debug and troubleshoot when things go wrong. No one should be building or maintaining middleware. Build robots. Read your sensor data, build a model of the world, decide what to do, then send commands to your control systems. This is the hard part of robotics. ROS solves the easiest part of robotics (plumbing and process management) in the shittiest possible way.
- snovv_crash 3y ago
  The async nature of pubsub makes it great for isolating your part of the system, but moves the complexity to the system integration instead. Actually deploying a ROS based system is about as difficult as rewriting the whole thing from scratch as a monolith. Every time something goes over a pubsub it's like using a GOTO, except your debugger can't actually follow it, and you don't even know who was listening (or who wasn't that should have been) or what the downstream effects were. It makes it impossible to properly debug a system, because it isn't deterministic so you can never be sure if you've actually handled all of the edge cases, since there is a temporal component to the state that can't be reproduced. A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates. The reason ROS has traction is because of university labs, who just need something to work once to be able to publish their paper or write their dissertation. In industry the reliability requirements are much higher, and despite the intensive efforts from the ROS community to "industralize" ROS via adoption of DDS, there seemed to be little understanding that the message protocol wasn't the reason the industry uptake was so low.
  - dagar 3y ago
    > A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates. This is how I've structured a control system, but using a pub/sub system to share data across those boundaries (and log/inspect in general). "Nodes" that share some resource or fundamentally run back to back based on the data flow can live in the same thread. Higher rate components (eg inner loop controller) live in a thread with a much higher priority. All of this is event driven, deterministic, and testable. If you have more details about the system you're imagining I'd love to learn more, because so far I don't see what's incompatible with what you've described.
    - snovv_crash 3y ago
      In general PX4 is better than ROS in this respect, but IMO still leans too heavily on queues. A bit of feedback there: - more should be done by setting up constrained functionality-specific data, and then simply calling a function with just that data. Right now a lot of things are passed state they don't need just because it is part of the message they receive. This makes the code dependencies way harder to separate, because you effectively share function signatures (messages) between different modules. Of secondary concern is the extra memory bandwidth from the extra data passed around, and not being able to pass by const& due to the async. - lots of things don't need updating at all until they tick over. If you don't update it frequently it has old data. You can try to make sure it works with the latest data by updating it frequently, but that of course has big overhead. I don't see any decent way of making this work unless you either 1) set up global threadsafe state so everything can access it, which is bad from a dependency and locking perspective, or 2) just call functions synchronously with exactly the data they need. - The issue with message queues, beyond the need for messages to generalize as mentioned above, is that often components need multiple different messages from different sources to perform their tasks. This means every component needs to keep a local copy of the data they need, to translate this async data back into a synchronous paradigm when it runs. In fact, beyond the message passing itself, pretty much everything needs to do its work in a synchronous paradigm, so why even add the async stuff in the middle to begin with? Once the messiness of different sensors' reporting rates is consolidated into the EKF state, and external commands and directives and brought in, after that point pretty much everything could be synchronous. No overhead, no timing issues, no "which message has the data I need", no "why do we have 3 different variants of the same message with slightly different data and which one should I use", etc. As long as you log the initial input data it should still be able to do replays for reproduction of realtime behaviour, but a) better testing can be done because you can actually tell what every subsystem needs to run correctly just by looking at its function signature, and b) its much easier to refactor (and understand) because of the same properties of the function signature describing all of the inputs and outputs.
chippiewill 3y ago
Actually I think it's more complicated than that. Pub/sub is a natural fit for robotics in a prototyping sense, but it's a poor fit for real-time systems which makes it harder to productionise something based on ROS. Especially safety critical systems. "Just" a multi-threaded app and calling functions isn't really a good replacement. For something really basic, sure, but the modularity that ROS introduces, and the core principle of having serialisable recordable messages delivers a lot of value once you start building large systems - particularly those that can span multiple physical machines. But pub-sub is _too_ general as a concurrency model, if ROS had adopted something more constrained that was provably deterministic and was amenable to real time analysis then a lot of its concerns would go away.
- snovv_crash 3y ago
  It would also help to have compile-time guarantees about which nodes are running. But I always like to say, why spend 5ns calling a function when you could spend 1ms waiting for a context switch or 50ms for the next event loop to tick over? Even modern microcontrollers are blindingly fast, and the majority of the time they spend these days is because of our brain-dead architecture decisions, not due to the underlying problem that actually needs solving.
  - dagar 3y ago
    > It would also help to have compile-time guarantees about which nodes are running. I haven't had a chance to try it yet, but in theory I like the idea of something like Zenoh Flow to describe the larger data flow graph. https://zenoh.io/blog/2023-02-10-zenoh-flow/ https://zenoh.io/blog/2023-02-10-zenoh-flow/ > But I always like to say, why spend 5ns calling a function when you could spend 1ms waiting for a context switch or 50ms for the next event loop to tick over? I think your context switch timing is off by several orders of magnitude, but regardless these things aren't one extreme or the other. For sharing data across threads, to an external system, or logging and visualization I still like pub/sub (and I've seen more than my share of horrible abuse), but it definitely shouldn't be treated as one size fits all.
    - 3y ago
dbcurtis 3y ago
A single multi-threaded app is a toy-sized robot, not a product with a large dev team behind it. The single-app approach doesn’t scale up very well.
- cheeselip420 3y ago
  Sure it does. I've seen teams of >100 engineers working on largely monolithic robotic codebases across several teams/companies (volumes in the 10s of thousands). Beyond that, robotics isn't all that special. There are other domains which have soft-realtime requirements and huge scale. AAA video games are a multi-$100-billion industry, which largely ship single-process systems with extreme performance on par with state-of-the-art perception and motion planning use-cases. AR/VR is another area where this is interesting - https://twitter.com/ID_AA_Carmack/status/1497098684546400260 https://twitter.com/ID_AA_Carmack/status/1497098684546400260 - Carmack has alluded to this issue multiple times.
  - rektide 3y ago
    It'll be interesting to see how/when wasm shows up in vr & robotics. Being able to have some shared memory you pass around between safely sandboxed worklets still has any of the same coordination problems. But it at least has much much much less multiprocess jank.
    - snovv_crash 3y ago
      What does the sandbox really gain you? Is it anything that couldn't just be handled at the OS level?
      - rektide 3y ago
        It let's you safely run lots of different code in a single processs, without paying the context switching cost. Quests's jank problem is probably in part that multiprocess architecture has to spawn new processes & change priorities of what's running & handle expensive context switches between processes elegantly, which is hard. Not only is non-monolithic code sometimes harder to reason about, it has all kinds of performance boundaries that are very painful to cut through. Read through Pipewire changelogs & how they've worked so hard to make best use of kernel primitives with less and less overhead, look how hard they've worked to get near Jack levels of efficiency. Great example of the slow march of multi-process optimization. Sandboxes let you collocate different processes inside the same process. You still have many of the same architectural challenges, but the ability to spawn new code & run it has much much lower overhead. It creates the possibility to make way more multi-process like architectures than we can explore now.