6 ms·

> The joysticks are connected to a pilot hydraulic circuit, which proportionally moves the cylinders in the main hydraulic circuit which ultimately moves the ex

> The joysticks are connected to a pilot hydraulic circuit, which proportionally moves the cylinders in the main hydraulic circuit which ultimately moves the excavator joints I've actually spent a decent amount of time running an excavator, as my Dad owns a construction / road building company. It was a great summer job! An important note about the pilot hydraulics is that they _provide feedback to the operator_. I would encourage any system that moves these controls on behalf of a remote human operator or AI to add strain gauges or some other way to measure this force feedback so that this data isn't lost. The handful of "drive by wire" pieces of equipment that my Dad or other skilled operators in my family have ran were universally panned, because the operators are isolated from this feedback and have a harder time telling when the machine is struggling or when their inputs are not sufficiently smooth. In the automotive world, skilled drivers have similar complaints about fully electronic steering or braking systems, as opposed to traditional vacuum or hydraulic boosting approaches where your foot still has a direct hydraulic connection to the brake pads.

jeffbee 1y ago
My car with its drive-by-wire brakes has a brake feedback simulator that gives the driver the kind of feeling associated with power-boosted hydraulic brakes. This is by far the most expensive single component in the car. Arguably these are just expensive accommodations for human flaws. A self-driving car wouldn't need them. Can't the self-driving system act directly on data like pressure, flow, and displacement?
- 01HNNWZ0MV43FF 1y ago
  Maybe it doesn't matter for a car because feeling the car's motion tells you most of what you need to know. A car is not meant to touch anything but the road, in normal conditions. I think steering is the only case where force feedback is very important for a car - In the winters up here, I can feel the steering go loose when I hit a patch of ice. I imagine an excavator, meant to touch and dig through things, and lift things, benefits from force feedback for the same reason VR would. Have you played those VR sword games? BeatSaber works great because you're cutting through abstract blobs that offer no resistance. But the medieval sword-slashing games feel weird because your sword can't impact your opponent. I saw a video recently of a quadcopter lifting heavy objects. When it's overloaded, it can't maneuver because all its spare power is spent generating lift to maintain altitude. If the controls had force feedback, the copter's computer could tell you "I'm overloaded, I can't move" by putting maximum resistance on the sticks.
  - jashmota 1y ago
    Interestingly, we had some people try out VR teleop: https://x.com/Scobleizer/status/1970245161306464667 https://x.com/Scobleizer/status/1970245161306464667 https://x.com/jash_mota/status/1969091992140304703 https://x.com/jash_mota/status/1969091992140304703 I think force feedback is key for small excavators, but not really true for 25+ tons excavators. Hence how easy it is for operators to accidentally kill someone with it.
- jashmota 1y ago
  That's indeed what we're trying to test to the extreme - to see how far we could go with just vision. We haven't done heavy excavation workflows yet, but we have some early success with some excavation workflows with just vision input and joystick action output (even without joint angle feedback!). We're betting on having really huge data with compact observation input and experiment to see if it holds water. If not we can always dial it down and add more sensors/feedback.
- cyberax 1y ago
  There are no drive-by-wire brakes in the US or Europe for regular cars. Your car's actuator moves the piston that is mechanically linked to your pedal. So even if the electric system fails completely, you can still actuate the brakes.
  - jeffbee 1y ago
    [flagged]
  - seabrookmx 1y ago
    It depends on how you define brake by wire, but the one I'm referencing is the C8 Corvette and it's "eBoost" system. This isn't a purely electronic system like throttle by wire is, but it does mean there is no longer a linear relationship between pedal pressure and brake pad pressure. And my point about isolating the driver from feedback still holds true.
  - kruador 1y ago
    Toyota's hybrids, at least, have valves in the hydraulic system. If everything is working, the driver's pedal is isolated from the physical pistons. Pressing the pedal instead moves a 'stroke simulator' (a cylinder with a spring in it), and the pressure is measured with a transducer. The Brake ECU tries to satisfy as much braking demand through regenerative braking as possible, applying the rear brakes to keep balance and front brakes if you brake too hard, requesting more braking than can be generated or the battery can absorb. If there's a failure of the electrical supply to the brake ECU, or another fault condition occurs, various valves then revert to their normally-open or normally-closed positions to allow hydraulic pressure from the pedal through to the brake cylinders, and isolate the stroke simulator. Because the engine isn't constantly running and providing a vacuum that can be used to assist with brake force, the system also includes a 'brake accumulator' and pump to boost the brake pressure. Reference: https://pmmonline.co.uk/technical/blue-prints-insight-into-the-toyota-brake-system/ https://pmmonline.co.uk/technical/blue-prints-insight-into-t... I don't know for certain, but I would assume that other hybrids and EVs have similar systems to maximise regenerative braking.
opwieurposiu 1y ago
Yes, and you also get feedback from your butt as the machine tips and wobbles, particularly on smaller machines. Hearing the engine straining helps also. Often you can not clearly see what you are digging, this feedback lets you know if you are running into a rock or something. One big advantage would be cameras mounted on the boom and rear view cameras, as many machines have obstructed views.
- jashmota 1y ago
  We're indeed streaming the audio and have haptic feedback. My hypothesis is the seat vibration isn't as helpful. It's sub optimal and operators would be far more productive without it. We would do a paper when we have enough data on this. We're also putting more cameras but streaming a lot of them at once is tricky.
  - Redster 1y ago
    Humbly, have you used excavators of varying sizes on uneven ground? I have and would suggest it's more important than you might think. But if you've operated them, you might know better than I. Also, teleoperation is likely to produce lower-quality operation data than hooking up to locally operated excavators. Just a thought.
    - jashmota 1y ago
      I might be less experienced than you - I've operated upto 38 tons on maximum 15 degrees incline. I wasn't moving tracks a lot when I did that. Would like to hear what scenarios you'be been in and how would you describe your experience? Maybe I could try those out to learn more!
  - fragmede 1y ago
    The problem is how do you quantify it and metricize it and get some numbers to compare? Because it's absolutely invaluable after you get to a level of mastery with a machine that you're sitting in that the seat moves in. You just feel it, you can't explain it in words, you're just one with the machine. Your butt really needs to feel that in order for that to really happen. Can you do it without it? Absolutely. Does it make it worse to not have that? Also true. It's extra effort, it's extra cost, and it makes things better for the actual users of the product, but it'll never show up as something you can measure. Do you want to sell Microsoft Teams to the executives or do you want to give joy to the people who actually use the product?
jashmota 1y ago
You're right! This is exactly why we like to do mechanical actuation - we are able to achieve bilateral telepresence, which essentially gives the torque (haptic) feedback over the internet! So on small excavators, you can absolutely feel the resistance. We also stream the engine audio, which tells you how hard the hydraulic pump is working. Operators like our system for these reasons :) I'd like to get a chance to talk to you and your Dad to get feedback. How do I reach you? My email is contact at useflywheel dot ai
- IgorPartola 1y ago
  Not my industry at all. I am curious if something like this is an opportunity for a whole new type of controls and feedback. Since the operator doesn’t have to be in the excavator physically they could take on any position: standing, sitting, lying down, etc. Instead of sending haptic feedback to the joystick it could be sent to a vibrating wrist band. You could hook up the equivalent of a Nintendo Power Glove to have the scoop operated by the operator simulating scooping action. Turning the excavator can be controlled by the operator turning their head and moving it around can be done by the operator walking on an infinite treadmill. Motor strain can be done via color of light or temperature rather than sound. You could have a VR helmet that can also show you a birds eye view from a companion drone, overlay blueprints, show power and water lines, measure depth, etc. I don’t know if it is possible but maybe you could even measure soil composition somehow to show large rocks, soft soil that is dangerous to drive over, inclination angles where the excavator is about to drive, etc. I imagine skilled operators prefer familiar controls but perhaps there are interesting improvements unlocked by exploring alternatives. It might also fundamentally change how accessible it is for non-professionals to use these machines. I rented an excavator from Home Depot a few years ago to dig a foundation and the learning curve was not shallow. I wonder if a more “natural” interface would help keep people safer.
  - jashmota 1y ago
    These are really really interesting thoughts around the remote teleop interface. A few things you mentioned have been on my mind. Teleop UX is underdeveloped and its affects is underestimated, and it could turn out to be a huge thing for us and humanoid companies if autonomy is harder than it seems now. I don't believe construction equipment operation today is optimal. We just go with it as that's what we have. It's far from intuitive and easy for a newbie to start operating to give some context and requires hours of practice. There would be a bit of training to be done for remote teleop as well. Might as well make it easy to interface and improve on the existing experience, which hasn't really evolved for years and decades! I'd like to have a chat with you if you're up for it: contact at useflywheel dot ai
roamerz 1y ago
I have a 20 ton Takeuchi and I don’t recall any feedback in the controls at all. The feedback I use is from the seat and sounds of the machine - well besides of course visual of course. I cannot imagine this being useful to me unless the virtual operators cab closely mimicked an actual machine. It would have to have audio from the machine and be on a platform that tilted relative to the real thing. It would also need 270 degrees of monitors with a virtual mirror to see behind. On the front monitor, minimally, would need the to see vertically up and down too. I also imagine all of this would be more useful to seasoned operators who can do most things on excavators in their sleep (definitely not me lol)
- jashmota 1y ago
  The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision. So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it. That's on the vision side. We also stream engine audio, and do haptic feedback. Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?
  - AlotOfReading 1y ago
    Just a suggestion from someone who's worked on industrial robots and autonomous vehicles, but I think you're underselling a lot of difficulties here. Skilled humans have a tendency to fully engage all of their senses during a task. For example, human drivers use their entire field of vision at night even though headlights only illuminate tiny portions of their FoV. I've never operated an excavator, but I would be very surprised if skilled operators only used the portion of their vision immediately around the bucket and not the rest of it for situational awareness. That said, UI design is a tradeoff. There's a paper that has a nice list of teleoperation design principles [0], which does talk about single windows as a positive. On the other hand, a common principle in older aviation HCI literature is the idea that nothing about the system UI should surprise the human. It's hard to maintain a good idea of the system state when you have it resizing windows automatically. The hardest thing is going to be making really good autonomous safety layers. It's the most difficult part of building a fully autonomous system anyway. The main advantage of teleop is that you can [supposedly] sidestep having one. [0] https://doi.org/10.1109/THMS.2014.2371048 https://doi.org/10.1109/THMS.2014.2371048

jeffbee 1y ago

My car with its drive-by-wire brakes has a brake feedback simulator that gives the driver the kind of feeling associated with power-boosted hydraulic brakes. This is by far the most expensive single component in the car. Arguably these are just expensive accommodations for human flaws. A self-driving car wouldn't need them. Can't the self-driving system act directly on data like pressure, flow, and displacement?

01HNNWZ0MV43FF 1y ago
Maybe it doesn't matter for a car because feeling the car's motion tells you most of what you need to know. A car is not meant to touch anything but the road, in normal conditions. I think steering is the only case where force feedback is very important for a car - In the winters up here, I can feel the steering go loose when I hit a patch of ice. I imagine an excavator, meant to touch and dig through things, and lift things, benefits from force feedback for the same reason VR would. Have you played those VR sword games? BeatSaber works great because you're cutting through abstract blobs that offer no resistance. But the medieval sword-slashing games feel weird because your sword can't impact your opponent. I saw a video recently of a quadcopter lifting heavy objects. When it's overloaded, it can't maneuver because all its spare power is spent generating lift to maintain altitude. If the controls had force feedback, the copter's computer could tell you "I'm overloaded, I can't move" by putting maximum resistance on the sticks.
- jashmota 1y ago
  Interestingly, we had some people try out VR teleop: https://x.com/Scobleizer/status/1970245161306464667 https://x.com/Scobleizer/status/1970245161306464667 https://x.com/jash_mota/status/1969091992140304703 https://x.com/jash_mota/status/1969091992140304703 I think force feedback is key for small excavators, but not really true for 25+ tons excavators. Hence how easy it is for operators to accidentally kill someone with it.
jashmota 1y ago
That's indeed what we're trying to test to the extreme - to see how far we could go with just vision. We haven't done heavy excavation workflows yet, but we have some early success with some excavation workflows with just vision input and joystick action output (even without joint angle feedback!). We're betting on having really huge data with compact observation input and experiment to see if it holds water. If not we can always dial it down and add more sensors/feedback.
cyberax 1y ago
There are no drive-by-wire brakes in the US or Europe for regular cars. Your car's actuator moves the piston that is mechanically linked to your pedal. So even if the electric system fails completely, you can still actuate the brakes.
- jeffbee 1y ago
  [flagged]
- seabrookmx 1y ago
  It depends on how you define brake by wire, but the one I'm referencing is the C8 Corvette and it's "eBoost" system. This isn't a purely electronic system like throttle by wire is, but it does mean there is no longer a linear relationship between pedal pressure and brake pad pressure. And my point about isolating the driver from feedback still holds true.
- kruador 1y ago
  Toyota's hybrids, at least, have valves in the hydraulic system. If everything is working, the driver's pedal is isolated from the physical pistons. Pressing the pedal instead moves a 'stroke simulator' (a cylinder with a spring in it), and the pressure is measured with a transducer. The Brake ECU tries to satisfy as much braking demand through regenerative braking as possible, applying the rear brakes to keep balance and front brakes if you brake too hard, requesting more braking than can be generated or the battery can absorb. If there's a failure of the electrical supply to the brake ECU, or another fault condition occurs, various valves then revert to their normally-open or normally-closed positions to allow hydraulic pressure from the pedal through to the brake cylinders, and isolate the stroke simulator. Because the engine isn't constantly running and providing a vacuum that can be used to assist with brake force, the system also includes a 'brake accumulator' and pump to boost the brake pressure. Reference: https://pmmonline.co.uk/technical/blue-prints-insight-into-the-toyota-brake-system/ https://pmmonline.co.uk/technical/blue-prints-insight-into-t... I don't know for certain, but I would assume that other hybrids and EVs have similar systems to maximise regenerative braking.

opwieurposiu 1y ago

Yes, and you also get feedback from your butt as the machine tips and wobbles, particularly on smaller machines. Hearing the engine straining helps also. Often you can not clearly see what you are digging, this feedback lets you know if you are running into a rock or something. One big advantage would be cameras mounted on the boom and rear view cameras, as many machines have obstructed views.

jashmota 1y ago
We're indeed streaming the audio and have haptic feedback. My hypothesis is the seat vibration isn't as helpful. It's sub optimal and operators would be far more productive without it. We would do a paper when we have enough data on this. We're also putting more cameras but streaming a lot of them at once is tricky.
- Redster 1y ago
  Humbly, have you used excavators of varying sizes on uneven ground? I have and would suggest it's more important than you might think. But if you've operated them, you might know better than I. Also, teleoperation is likely to produce lower-quality operation data than hooking up to locally operated excavators. Just a thought.
  - jashmota 1y ago
    I might be less experienced than you - I've operated upto 38 tons on maximum 15 degrees incline. I wasn't moving tracks a lot when I did that. Would like to hear what scenarios you'be been in and how would you describe your experience? Maybe I could try those out to learn more!
- fragmede 1y ago
  The problem is how do you quantify it and metricize it and get some numbers to compare? Because it's absolutely invaluable after you get to a level of mastery with a machine that you're sitting in that the seat moves in. You just feel it, you can't explain it in words, you're just one with the machine. Your butt really needs to feel that in order for that to really happen. Can you do it without it? Absolutely. Does it make it worse to not have that? Also true. It's extra effort, it's extra cost, and it makes things better for the actual users of the product, but it'll never show up as something you can measure. Do you want to sell Microsoft Teams to the executives or do you want to give joy to the people who actually use the product?

jashmota 1y ago

You're right! This is exactly why we like to do mechanical actuation - we are able to achieve bilateral telepresence, which essentially gives the torque (haptic) feedback over the internet! So on small excavators, you can absolutely feel the resistance. We also stream the engine audio, which tells you how hard the hydraulic pump is working. Operators like our system for these reasons :) I'd like to get a chance to talk to you and your Dad to get feedback. How do I reach you? My email is contact at useflywheel dot ai

IgorPartola 1y ago
Not my industry at all. I am curious if something like this is an opportunity for a whole new type of controls and feedback. Since the operator doesn’t have to be in the excavator physically they could take on any position: standing, sitting, lying down, etc. Instead of sending haptic feedback to the joystick it could be sent to a vibrating wrist band. You could hook up the equivalent of a Nintendo Power Glove to have the scoop operated by the operator simulating scooping action. Turning the excavator can be controlled by the operator turning their head and moving it around can be done by the operator walking on an infinite treadmill. Motor strain can be done via color of light or temperature rather than sound. You could have a VR helmet that can also show you a birds eye view from a companion drone, overlay blueprints, show power and water lines, measure depth, etc. I don’t know if it is possible but maybe you could even measure soil composition somehow to show large rocks, soft soil that is dangerous to drive over, inclination angles where the excavator is about to drive, etc. I imagine skilled operators prefer familiar controls but perhaps there are interesting improvements unlocked by exploring alternatives. It might also fundamentally change how accessible it is for non-professionals to use these machines. I rented an excavator from Home Depot a few years ago to dig a foundation and the learning curve was not shallow. I wonder if a more “natural” interface would help keep people safer.
- jashmota 1y ago
  These are really really interesting thoughts around the remote teleop interface. A few things you mentioned have been on my mind. Teleop UX is underdeveloped and its affects is underestimated, and it could turn out to be a huge thing for us and humanoid companies if autonomy is harder than it seems now. I don't believe construction equipment operation today is optimal. We just go with it as that's what we have. It's far from intuitive and easy for a newbie to start operating to give some context and requires hours of practice. There would be a bit of training to be done for remote teleop as well. Might as well make it easy to interface and improve on the existing experience, which hasn't really evolved for years and decades! I'd like to have a chat with you if you're up for it: contact at useflywheel dot ai

roamerz 1y ago

I have a 20 ton Takeuchi and I don’t recall any feedback in the controls at all. The feedback I use is from the seat and sounds of the machine - well besides of course visual of course. I cannot imagine this being useful to me unless the virtual operators cab closely mimicked an actual machine. It would have to have audio from the machine and be on a platform that tilted relative to the real thing. It would also need 270 degrees of monitors with a virtual mirror to see behind. On the front monitor, minimally, would need the to see vertically up and down too. I also imagine all of this would be more useful to seasoned operators who can do most things on excavators in their sleep (definitely not me lol)

jashmota 1y ago
The way I think about this - we should not have multi screens. Human field of vision is 60 degrees for central and about 120 degrees binocular. The bucket of the excavator is way narrower than this which means actual task doesn't require wide vision. So if we are able to have really good autonomous safety layers to ensure safe movements, and dynamically resize remote teleop windows, you'd make the operator more efficient. So while we stream 360 degree view, we get creative in how we show it. That's on the vision side. We also stream engine audio, and do haptic feedback. Takeuchi are interesting! Rare ones to have blades even on bigger sizes - is that why you got one?
- AlotOfReading 1y ago
  Just a suggestion from someone who's worked on industrial robots and autonomous vehicles, but I think you're underselling a lot of difficulties here. Skilled humans have a tendency to fully engage all of their senses during a task. For example, human drivers use their entire field of vision at night even though headlights only illuminate tiny portions of their FoV. I've never operated an excavator, but I would be very surprised if skilled operators only used the portion of their vision immediately around the bucket and not the rest of it for situational awareness. That said, UI design is a tradeoff. There's a paper that has a nice list of teleoperation design principles [0], which does talk about single windows as a positive. On the other hand, a common principle in older aviation HCI literature is the idea that nothing about the system UI should surprise the human. It's hard to maintain a good idea of the system state when you have it resizing windows automatically. The hardest thing is going to be making really good autonomous safety layers. It's the most difficult part of building a fully autonomous system anyway. The main advantage of teleop is that you can [supposedly] sidestep having one. [0] https://doi.org/10.1109/THMS.2014.2371048 https://doi.org/10.1109/THMS.2014.2371048