8 ms·

Robotics is really really really really hard. So let's turn the entire thing into a system of distributed microservices, using CORBA/DDS for pubsub... It'd be

Robotics is really really really really hard. So let's turn the entire thing into a system of distributed microservices, using CORBA/DDS for pubsub... It'd be like if you designed a video game so that the physics engine (motion planning) happened in one process, while rendering (perception) happened in another process and the world state needed to be exchanged via pubsub. Why do we keep doing this to ourselves?

blensor 3y ago
Multiple reasons, here are my two cents: * ROS nodes usually do ( or did back when we used ) pretty resource intensive tasks so they are generally distributed over multiple machines * Individual nodes can have quite diverse library requirements ( specific library versions of obscure libraries ) that don't go together, so you want to decouple them
- snovv_crash 3y ago
  Only in academia. In industrial settings sending stuff over a network between machines is too unreliable for realtime operation, and oftentimes also too expensive. Also in industrial environments if you can't install things together, they're not of a standard that you're happy to deploy to production.
dbcurtis 3y ago
Pub/Sub is a very natural fit for robotics. Companies do write ROS out of their production stacks when they get the resources. But they don’t replace pub/sub architecture. ROS provides a plug-in backplane that allows you to innovate in one area while leveraging existing components for parts that are not your differentiator.
- cheeselip420 3y ago
  pub/sub is NOT a natural fit for robotics - you want bounded timing and generally for error-handling you want to know what happened as a result of a message being published. ROS introduced "commands" and "actions" to attempt to work around this, but its all just shit piled on shit. Just make a multi-threaded app, and call functions. If you need to distribute over multiple CPUs, then go ahead and do some IPC. But pub/sub is NOT an architecture. Its a soup of tightly coupled fragments of functionality. It falls apart very quickly in the real world.
  - moglito 3y ago
    Are you aware that ros service calls are RPCs (not based on pub/sub like actions)? Furthermore, if you use nodelets (http://wiki.ros.org/nodelet http://wiki.ros.org/nodelet) you get zero copy communication between your algorithms. So I actually think that ROS has the facilities to address the needs you describe. I'd be interested, though, in your suggestions on what a real architecture for robotics looks like in your mind? I still remember the time before ROS, 20 years ago, when each robotics team had to designate a sub-team just for building and maintaining the middleware. That was a waste of time and effort. But you seem to suggest that we go back to that? ROS might not be perfect but it's so much better than anything else that exists. It's also open source and we can all work together to make it better rather than reinventing the wheel each time.
    - cheeselip420 3y ago
      Services and Actions are built on top of pubsub (with separate Request/Response vs Goal/Feedback/Result topics respectively). At least w/ ROS1 - I'm not sure if ROS2 improved things here... Nodelets are also a disaster, which is why ROS2 kinda fixed this by decoupling nodes and processes. When you're just starting, ROS can be nice for prototyping - you get a batteries-included platform that can do some SLAM and simple motion planning. But as you start adding new features, you need to figure out how to add those features over multiple nodes. This coordination overhead can quickly bring your system to its knees, or at least make it extremely difficult to debug and troubleshoot when things go wrong. No one should be building or maintaining middleware. Build robots. Read your sensor data, build a model of the world, decide what to do, then send commands to your control systems. This is the hard part of robotics. ROS solves the easiest part of robotics (plumbing and process management) in the shittiest possible way.
    - snovv_crash 3y ago
      The async nature of pubsub makes it great for isolating your part of the system, but moves the complexity to the system integration instead. Actually deploying a ROS based system is about as difficult as rewriting the whole thing from scratch as a monolith. Every time something goes over a pubsub it's like using a GOTO, except your debugger can't actually follow it, and you don't even know who was listening (or who wasn't that should have been) or what the downstream effects were. It makes it impossible to properly debug a system, because it isn't deterministic so you can never be sure if you've actually handled all of the edge cases, since there is a temporal component to the state that can't be reproduced. A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates. The reason ROS has traction is because of university labs, who just need something to work once to be able to publish their paper or write their dissertation. In industry the reliability requirements are much higher, and despite the intensive efforts from the ROS community to "industralize" ROS via adoption of DDS, there seemed to be little understanding that the message protocol wasn't the reason the industry uptake was so low.
markisus 3y ago
I used to use ROS 1 for work. It is incredibly over-engineered. Somewhere in the code base there is a three level hierarchy wrapping a shared pointer to double. The actual address is configured using an xml file. The purpose? To “abstract” the commanded torque ultimately sent to a motor.
facontidavide 3y ago
I have been doing robotics for 20 years, and the "trend" to adopt a distributed architecture was already popular 15 years ago. ROS just adopted and continued that trend. I think you are missing some important points: - a large amount of data can be transferred asynchronously. Pub/Sub is a great pattern to achieve that; the cases when you need a synchronous interface are the minority. - a robotic software has MANY components, you need an approach that incentivize decoupling and lousily-coupled interfaces. - this lousily-coupled architecture has been a catalyst for innovation and cooperation, because it makes it easier for people to share their code as "building blocks" of a larger system. Is there a considerable overhead? Sure! But it is much more complicated to write thread-safe code in a huge, monolithic application.
- lqr 3y ago
  I worked on a robotics project with a large team using ROS1. The loose coupling is pernicious: It's easy for everyone to work on their own ROS node in isolation and avoid testing the integrated system. There's no compiler to help you find all the clients of a node, etc. Loose coupling is good if you mainly use open-source packages and modify one component for your research paper. I'm not sure it's a net positive when building most components yourself. A big multithreaded program can still use queues instead of locks to share data between threads.
  - spieswl 3y ago
    So many of these comments sound like engineering process flaws. Everyone's working on their own ROS nodes in isolation....with no thought to eventual integration testing? It doesn't seem like that is a fundamental shortcoming specific to ROS.
    - lqr 3y ago
      Of course, but ROS makes it easier to slip into this flawed process. I prefer tools that help you avoid bad engineering processes.
ragebol 3y ago
I've been using ROS for 10+ years, in both university and professional settings. The hard parts were never in the middleware (ROS), but stuff like perception, world modelling, decision making. ROS gives you an ecosystem of decent options to choose from. And improve on if you need to at all. The standardized interfaces and IPC make it very easy to plug, at runtime, eg. a different localization algorithm into the system.
- mchusma 3y ago
  I've also used ROS a lot (but not in the last 2 years), and +1 your comment. The main gripe with ROS is poor documentation, and people's over-reliance on coding in the simulator. Not the architecture. The biggest challenge IMO is actyally getting the perception, world modeling, decision making work reliably in the real world, and this is where AI should provide a big boost.
  - ragebol 3y ago
    > over-reliance on coding in the simulator Heh, yes. At some RoboCup@Home competition (IIRC, Mexico City in 2012 so a while back), a team came in super confident, saying everything worked and they were going to win. Most of it worked ok in the lab, and well in the simulator too, they said. Didn't make it to the 2nd stage I think. And RoboCup is still not exactly 'the wild' for robots.
ModernMech 3y ago
Because robots are naturally distributed and asynchronous realtime systems .
- cheeselip420 3y ago
  "Naturally?" lol while(true) { read sensors update world model decide what to do act } You should only deviate from this when you have a specific reason (concurrency, libraries, IPC, etc). You can attach a debugger. You can deterministically play sensor data through and get great reproducibility for end-to-end testing. Starting with a distributed system is a handicap 80% of the time.
  - toxik 3y ago
    Decision making and sensor reading happen at vastly different timescales.
    - cheeselip420 3y ago
      You could have a lidar coming in at 15Hz, a camera at 30Hz, odometry at 60 or 100Hz - but typically you'll want to plan within that same range, at least for navigation (20-50Hz). "Vastly different" is a bit of a stretch. Also - we have used queues to deal with different time scales for a really long time. It works fine here too. For higher-level behaviors around grasping or manipulation, your point is super valid though. I suppose I'm mostly focusing on navigation-type tasks.
      - ModernMech 3y ago
        You aren’t thinking broad enough. Algorithms can run at megahertz, sensors can run at 10s of kilohertz to 10s of Hz, control loops can run at 5Hz. Remote database calls can run of course much longer than that, and then you have very long range planning tasks that can cycle days or weeks depending on deployment. I’d say that’s quite the range. And you mention queues, yes exactly. Abstract a little more and you get pub sub. Abstract a little more and you have the actor model, which is a lovely way of building resilient, reliable, fault tolerant systems — exactly what we want out of robots.
sargun 3y ago
I actually talked to someone in industrial robotics, and message passing / message buses are really common there. All of their controllers are loosely coupled, because each unit might be from a different vendor, upgraded at a different time, etc... It also had the nice property that their systems all had multiple supervisors, and if the supervisor detected a bad state by monitoring the bus passively, it could stop all of controllers.

blensor 3y ago

Multiple reasons, here are my two cents: * ROS nodes usually do ( or did back when we used ) pretty resource intensive tasks so they are generally distributed over multiple machines * Individual nodes can have quite diverse library requirements ( specific library versions of obscure libraries ) that don't go together, so you want to decouple them

snovv_crash 3y ago
Only in academia. In industrial settings sending stuff over a network between machines is too unreliable for realtime operation, and oftentimes also too expensive. Also in industrial environments if you can't install things together, they're not of a standard that you're happy to deploy to production.

dbcurtis 3y ago

Pub/Sub is a very natural fit for robotics. Companies do write ROS out of their production stacks when they get the resources. But they don’t replace pub/sub architecture. ROS provides a plug-in backplane that allows you to innovate in one area while leveraging existing components for parts that are not your differentiator.

cheeselip420 3y ago
pub/sub is NOT a natural fit for robotics - you want bounded timing and generally for error-handling you want to know what happened as a result of a message being published. ROS introduced "commands" and "actions" to attempt to work around this, but its all just shit piled on shit. Just make a multi-threaded app, and call functions. If you need to distribute over multiple CPUs, then go ahead and do some IPC. But pub/sub is NOT an architecture. Its a soup of tightly coupled fragments of functionality. It falls apart very quickly in the real world.
- moglito 3y ago
  Are you aware that ros service calls are RPCs (not based on pub/sub like actions)? Furthermore, if you use nodelets (http://wiki.ros.org/nodelet http://wiki.ros.org/nodelet) you get zero copy communication between your algorithms. So I actually think that ROS has the facilities to address the needs you describe. I'd be interested, though, in your suggestions on what a real architecture for robotics looks like in your mind? I still remember the time before ROS, 20 years ago, when each robotics team had to designate a sub-team just for building and maintaining the middleware. That was a waste of time and effort. But you seem to suggest that we go back to that? ROS might not be perfect but it's so much better than anything else that exists. It's also open source and we can all work together to make it better rather than reinventing the wheel each time.
  - cheeselip420 3y ago
    Services and Actions are built on top of pubsub (with separate Request/Response vs Goal/Feedback/Result topics respectively). At least w/ ROS1 - I'm not sure if ROS2 improved things here... Nodelets are also a disaster, which is why ROS2 kinda fixed this by decoupling nodes and processes. When you're just starting, ROS can be nice for prototyping - you get a batteries-included platform that can do some SLAM and simple motion planning. But as you start adding new features, you need to figure out how to add those features over multiple nodes. This coordination overhead can quickly bring your system to its knees, or at least make it extremely difficult to debug and troubleshoot when things go wrong. No one should be building or maintaining middleware. Build robots. Read your sensor data, build a model of the world, decide what to do, then send commands to your control systems. This is the hard part of robotics. ROS solves the easiest part of robotics (plumbing and process management) in the shittiest possible way.
  - snovv_crash 3y ago
    The async nature of pubsub makes it great for isolating your part of the system, but moves the complexity to the system integration instead. Actually deploying a ROS based system is about as difficult as rewriting the whole thing from scratch as a monolith. Every time something goes over a pubsub it's like using a GOTO, except your debugger can't actually follow it, and you don't even know who was listening (or who wasn't that should have been) or what the downstream effects were. It makes it impossible to properly debug a system, because it isn't deterministic so you can never be sure if you've actually handled all of the edge cases, since there is a temporal component to the state that can't be reproduced. A better system would take ideas from game engine design and realtime system execution budgets, with cascaded controllers on separate threads with dedicated compute resources for components that need higher update rates. The reason ROS has traction is because of university labs, who just need something to work once to be able to publish their paper or write their dissertation. In industry the reliability requirements are much higher, and despite the intensive efforts from the ROS community to "industralize" ROS via adoption of DDS, there seemed to be little understanding that the message protocol wasn't the reason the industry uptake was so low.

markisus 3y ago

I used to use ROS 1 for work. It is incredibly over-engineered. Somewhere in the code base there is a three level hierarchy wrapping a shared pointer to double. The actual address is configured using an xml file. The purpose? To “abstract” the commanded torque ultimately sent to a motor.

facontidavide 3y ago

I have been doing robotics for 20 years, and the "trend" to adopt a distributed architecture was already popular 15 years ago. ROS just adopted and continued that trend. I think you are missing some important points: - a large amount of data can be transferred asynchronously. Pub/Sub is a great pattern to achieve that; the cases when you need a synchronous interface are the minority. - a robotic software has MANY components, you need an approach that incentivize decoupling and lousily-coupled interfaces. - this lousily-coupled architecture has been a catalyst for innovation and cooperation, because it makes it easier for people to share their code as "building blocks" of a larger system. Is there a considerable overhead? Sure! But it is much more complicated to write thread-safe code in a huge, monolithic application.

lqr 3y ago
I worked on a robotics project with a large team using ROS1. The loose coupling is pernicious: It's easy for everyone to work on their own ROS node in isolation and avoid testing the integrated system. There's no compiler to help you find all the clients of a node, etc. Loose coupling is good if you mainly use open-source packages and modify one component for your research paper. I'm not sure it's a net positive when building most components yourself. A big multithreaded program can still use queues instead of locks to share data between threads.
- spieswl 3y ago
  So many of these comments sound like engineering process flaws. Everyone's working on their own ROS nodes in isolation....with no thought to eventual integration testing? It doesn't seem like that is a fundamental shortcoming specific to ROS.
  - lqr 3y ago
    Of course, but ROS makes it easier to slip into this flawed process. I prefer tools that help you avoid bad engineering processes.

ragebol 3y ago

I've been using ROS for 10+ years, in both university and professional settings. The hard parts were never in the middleware (ROS), but stuff like perception, world modelling, decision making. ROS gives you an ecosystem of decent options to choose from. And improve on if you need to at all. The standardized interfaces and IPC make it very easy to plug, at runtime, eg. a different localization algorithm into the system.

mchusma 3y ago
I've also used ROS a lot (but not in the last 2 years), and +1 your comment. The main gripe with ROS is poor documentation, and people's over-reliance on coding in the simulator. Not the architecture. The biggest challenge IMO is actyally getting the perception, world modeling, decision making work reliably in the real world, and this is where AI should provide a big boost.
- ragebol 3y ago
  > over-reliance on coding in the simulator Heh, yes. At some RoboCup@Home competition (IIRC, Mexico City in 2012 so a while back), a team came in super confident, saying everything worked and they were going to win. Most of it worked ok in the lab, and well in the simulator too, they said. Didn't make it to the 2nd stage I think. And RoboCup is still not exactly 'the wild' for robots.

ModernMech 3y ago

Because robots are naturally distributed and asynchronous realtime systems .

cheeselip420 3y ago
"Naturally?" lol while(true) { read sensors update world model decide what to do act } You should only deviate from this when you have a specific reason (concurrency, libraries, IPC, etc). You can attach a debugger. You can deterministically play sensor data through and get great reproducibility for end-to-end testing. Starting with a distributed system is a handicap 80% of the time.
- toxik 3y ago
  Decision making and sensor reading happen at vastly different timescales.
  - cheeselip420 3y ago
    You could have a lidar coming in at 15Hz, a camera at 30Hz, odometry at 60 or 100Hz - but typically you'll want to plan within that same range, at least for navigation (20-50Hz). "Vastly different" is a bit of a stretch. Also - we have used queues to deal with different time scales for a really long time. It works fine here too. For higher-level behaviors around grasping or manipulation, your point is super valid though. I suppose I'm mostly focusing on navigation-type tasks.
    - ModernMech 3y ago
      You aren’t thinking broad enough. Algorithms can run at megahertz, sensors can run at 10s of kilohertz to 10s of Hz, control loops can run at 5Hz. Remote database calls can run of course much longer than that, and then you have very long range planning tasks that can cycle days or weeks depending on deployment. I’d say that’s quite the range. And you mention queues, yes exactly. Abstract a little more and you get pub sub. Abstract a little more and you have the actor model, which is a lovely way of building resilient, reliable, fault tolerant systems — exactly what we want out of robots.

sargun 3y ago

I actually talked to someone in industrial robotics, and message passing / message buses are really common there. All of their controllers are loosely coupled, because each unit might be from a different vendor, upgraded at a different time, etc... It also had the nice property that their systems all had multiple supervisors, and if the supervisor detected a bad state by monitoring the bus passively, it could stop all of controllers.