4 ms·
Custom Unreal Engine simulator, simulating agents with NVidia Physx and publishing sensors through GStreamer. GStreamer has sinks and sources for ROS, and tenso
by rozgo 7y ago
Custom Unreal Engine simulator, simulating agents with NVidia Physx and publishing sensors through GStreamer. GStreamer has sinks and sources for ROS, and tensorflow elements for inferencing. We package this all into NVidia Docker for scalable simulations. Setup is similar for training and inference. The core framework is a streaming engine with stream combinators that enable reasoning about spatio-temporal data streams. Where each datum is related to a point in space and time. The goal is for tensors to be the streaming primitives, but the pipeline is still fragile, it’s a challenge to keep all this working with so many core technologies changing constantly (UE4 + Physx + Cuda + CuDNN + Tensorflow + …). We train robots in simulation.
Most of our tools are built in Rust. Several of those are for creating(cleaning) data streams out of datasets. They are converted into tensors or ROS messages.
- nravic 7y agoThis is really cool, what are you training?
- rozgo 7y agoQuadcopters, railway vehicles and perching devices.
- eigenvalue 7y agoHow well does the training in the simulator transfer over to real world environments? Do the models require fine tuning on real life data, or do they immediately work? I would imagine they would get confused by things like wheel/tracks slipping a bit on the ground versus in the Unreal simulation.
- rozgo 7y agoTraditional simulations don’t translate well. Usually one has to choose a fidelity level; controls, navigation, exploration, mission, etc… We are having more success with models that can tolerate variable dynamics. Simulations don't need to be realistic, only consistent. We then reinforce models with an infinite number of simulation universes and rules, such that reality is just another sample. We don’t have the resources for end-to-end training, and our initial tests did not yield good results. But chaining smaller specific-purpose networks within traditional control and planning systems is looking really good.
- ericjang 7y agoThis sounds pretty advanced! What lab do you work in?
- rozgo 7y agoOut of Central America. https://vertexstudio.co/ https://vertexstudio.co/
- uoaei 7y agoI want to be where you are. How can I get in contact with you?
- rozgo 7y agoI'm building these labs/teams in Latin America, with a few remote people from around the world. Ping me at: alex.rozgo @ vertexstudio.co