3 ms·

I've long wondered why the DARPA challenge for automated cars is necessary when the code could be tested using video games for orders of magnitude less money.

stonemetal 17y ago
I have friend who works at NASA in sim and test. Let me assure you simulations don't have to cost an order of magnitude less when the gov is involved. All joking aside you are talking about custom designed systems. How would you design a sim that works with whatever wonky sensor setup a team can come up with? Two isn't pre-challenge VandV up to the individual teams? They should have simed it long before it got to the competition if they were going to. The competition is to prove that it works in real life\bragging rights. It isn't nearly as cool to say my simulated robot is faster than your simulated robot.
CGamesPlay 17y ago
In theory there is no difference between theory and practice. In practice, however...
- nazgulnarsil 17y ago
  crude simulations don't invalidate theories. i hate anti-scientific quips like that.
  - agravier 17y ago
    The issue is that - success on a simulator does not mean being able to "scale up" to real life - success IRL would does not imply success on some simulator The link is particularly weak for crude simulations of multi-agent, dynamical, stochastic, partially observable, real-time worlds.
  - scott_s 17y ago
    I don't consider that quip anti-scientific at all. I take it as a reminder that you must always test your theories against reality, because the ways that reality and theory differ might be greater than you realized. To address your original question, the realities of controlling an actual car are probably the hard problems. Dealing with things such as the inertia of the car and tires slipping on the road are probably both required to solve the problem, and difficult to simulate accurately. Simulations only simulate what we know about. It's often the things we didn't anticipate that make the difference between success and failure.
    - nazgulnarsil 17y ago
      let me put it another way then. can the darpa challengers code EVEN handle a simulation? if not then why waste money going ahead with the full project?
      - carterschonwald 17y ago
        that presumes an unreasonable amount of homogeneity in both the programming languages of the entrants and the sensor platforms of their vehicles. The entrants may very well do that inhouse for testing purposes, but since Darpa is merely responsible running the competition and determining if the prize award is merited, why on earth would they do something that would unduly constrain the entrants?
      - calambrac 17y ago
        I'm sure it never occurred to anyone on these teams to try out their code in a simulated environment. I'm 100% positive their development model was to crash car after car after car until they got it working. I can only imagine how expensive some of those bug hunts got. Come on, seriously?
        nazgulnarsil 17y ago
        ya, srsly. From what I've seen they are hacking together the hardware and then constantly updating the code to get just barely working solutions, oftentimes in the middle of the testing. If some teams are running a fairly complex car physics simulator with simulated input from each of the types of sensors they are using this would be the first I've heard of it.
        calambrac 17y ago
        30 seconds on Google: http://www.torctech.com/company/news/jaus-standard-emerges-darpa-urban-challenge-final-event http://www.torctech.com/company/news/jaus-standard-emerges-d...
        nazgulnarsil 17y ago
  - rosser 17y ago
    Crude simulations may or may not invalidate theories, but they're notoriously bad at weeding out ultimately inviable lines of investigation. The raw, real world is messy and unpredictable in a way a simulation can never be, and until something from the DARPA Challenge or its ilk can handle meatspace, we got no business trusting meat with it. Not to say simulations are bad things; they're just of limited utility. Beyond that point, it does get expensive. I think that's part of the reason for the Challenge in the first place.
  - jacquesm 17y ago
    In theory, there is no difference between theory and practice, but in practice there is. I'm quite sure that if there is one set of people that know this by heart then it is the military. Real world experience will bring out the trouble much faster than any amount of simulation. Use simulations to help you design, validate your design in the real world. Then update your simulation to reflect the lessons you learned during your real world tests and iterate. Have a look at armadillo aerospace, John Carmack is as versed as anybody in that field with simulations and STILL they find out almost every trial they do that the real world is not as clean as the simulated one.
    - scott_s 17y ago
      My preferred phrasing is "The only difference between practice and theory is that, in theory, there is no difference between practice and theory."
    - jwhitlark 17y ago
      Indeed. Just take a look at Murphy's laws of combat: http://www.murphys-laws.com/murphy/murphy-war.html http://www.murphys-laws.com/murphy/murphy-war.html
      - marvin 17y ago
        Those are great, thanks :)
  - JabavuAdams 17y ago
    It's not anti-scientific, it's an old engineering adage. It's the unknown unknowns that will get you.
dmm 17y ago
Building the vehicles with the associated sensor and control systems were just as important as the AIs controlling I suppose.
maximilian 17y ago
Given perfect information, like from a video game (with access to the object model, etc), the DARPA challenges aren't really that difficult. You just have to plan a path and then stay on it. Its the noise of the real world that is infinitely challenging. You have to incorporate probabilities into your decision making, which is much more difficult. Jumping a mario around is much more AI intensive because the amount of solutions is much higher. A car is fixed to roads and has a map. Mario might know where everything is, but knowing what to do is much more difficult.

stonemetal 17y ago

I have friend who works at NASA in sim and test. Let me assure you simulations don't have to cost an order of magnitude less when the gov is involved. All joking aside you are talking about custom designed systems. How would you design a sim that works with whatever wonky sensor setup a team can come up with? Two isn't pre-challenge VandV up to the individual teams? They should have simed it long before it got to the competition if they were going to. The competition is to prove that it works in real life\bragging rights. It isn't nearly as cool to say my simulated robot is faster than your simulated robot.

CGamesPlay 17y ago

In theory there is no difference between theory and practice. In practice, however...

nazgulnarsil 17y ago
crude simulations don't invalidate theories. i hate anti-scientific quips like that.
- agravier 17y ago
  The issue is that - success on a simulator does not mean being able to "scale up" to real life - success IRL would does not imply success on some simulator The link is particularly weak for crude simulations of multi-agent, dynamical, stochastic, partially observable, real-time worlds.
- scott_s 17y ago
  I don't consider that quip anti-scientific at all. I take it as a reminder that you must always test your theories against reality, because the ways that reality and theory differ might be greater than you realized. To address your original question, the realities of controlling an actual car are probably the hard problems. Dealing with things such as the inertia of the car and tires slipping on the road are probably both required to solve the problem, and difficult to simulate accurately. Simulations only simulate what we know about. It's often the things we didn't anticipate that make the difference between success and failure.
  - nazgulnarsil 17y ago
    let me put it another way then. can the darpa challengers code EVEN handle a simulation? if not then why waste money going ahead with the full project?
    - carterschonwald 17y ago
      that presumes an unreasonable amount of homogeneity in both the programming languages of the entrants and the sensor platforms of their vehicles. The entrants may very well do that inhouse for testing purposes, but since Darpa is merely responsible running the competition and determining if the prize award is merited, why on earth would they do something that would unduly constrain the entrants?
    - calambrac 17y ago
      I'm sure it never occurred to anyone on these teams to try out their code in a simulated environment. I'm 100% positive their development model was to crash car after car after car until they got it working. I can only imagine how expensive some of those bug hunts got. Come on, seriously?
      - nazgulnarsil 17y ago
        ya, srsly. From what I've seen they are hacking together the hardware and then constantly updating the code to get just barely working solutions, oftentimes in the middle of the testing. If some teams are running a fairly complex car physics simulator with simulated input from each of the types of sensors they are using this would be the first I've heard of it.
        calambrac 17y ago
        30 seconds on Google: http://www.torctech.com/company/news/jaus-standard-emerges-darpa-urban-challenge-final-event http://www.torctech.com/company/news/jaus-standard-emerges-d...
        nazgulnarsil 17y ago
- rosser 17y ago
  Crude simulations may or may not invalidate theories, but they're notoriously bad at weeding out ultimately inviable lines of investigation. The raw, real world is messy and unpredictable in a way a simulation can never be, and until something from the DARPA Challenge or its ilk can handle meatspace, we got no business trusting meat with it. Not to say simulations are bad things; they're just of limited utility. Beyond that point, it does get expensive. I think that's part of the reason for the Challenge in the first place.
- jacquesm 17y ago
  In theory, there is no difference between theory and practice, but in practice there is. I'm quite sure that if there is one set of people that know this by heart then it is the military. Real world experience will bring out the trouble much faster than any amount of simulation. Use simulations to help you design, validate your design in the real world. Then update your simulation to reflect the lessons you learned during your real world tests and iterate. Have a look at armadillo aerospace, John Carmack is as versed as anybody in that field with simulations and STILL they find out almost every trial they do that the real world is not as clean as the simulated one.
  - scott_s 17y ago
    My preferred phrasing is "The only difference between practice and theory is that, in theory, there is no difference between practice and theory."
  - jwhitlark 17y ago
    Indeed. Just take a look at Murphy's laws of combat: http://www.murphys-laws.com/murphy/murphy-war.html http://www.murphys-laws.com/murphy/murphy-war.html
    - marvin 17y ago
      Those are great, thanks :)
- JabavuAdams 17y ago
  It's not anti-scientific, it's an old engineering adage. It's the unknown unknowns that will get you.

dmm 17y ago

Building the vehicles with the associated sensor and control systems were just as important as the AIs controlling I suppose.

maximilian 17y ago

Given perfect information, like from a video game (with access to the object model, etc), the DARPA challenges aren't really that difficult. You just have to plan a path and then stay on it. Its the noise of the real world that is infinitely challenging. You have to incorporate probabilities into your decision making, which is much more difficult. Jumping a mario around is much more AI intensive because the amount of solutions is much higher. A car is fixed to roads and has a map. Mario might know where everything is, but knowing what to do is much more difficult.