It looks like the image from the server and control information to the server is sent through the VNC protocol. Other information such as the reward signal from

It looks like the image from the server and control information to the server is sent through the VNC protocol. Other information such as the reward signal from the environment server is sent through a WebSockets protocol using JSON: https://github.com/openai/universe/blob/master/doc/protocols.rst https://github.com/openai/universe/blob/master/doc/protocols... You should be able to implement this protocol for your environment and run a VNC server for the rest. A new class for the client representing your environment can be based on this: https://github.com/openai/universe/blob/master/universe/envs/vnc_gtav.py https://github.com/openai/universe/blob/master/universe/envs... Then register the class with OpenAI Gym: https://github.com/openai/universe/blob/master/universe/__init__.py#L1495 https://github.com/openai/universe/blob/master/universe/__in... After creating the environment using gym.make you need to add information about your remote in the call to configure: env = gym.make('gtav.SaneDriving-v0') env.configure(remotes="vnc://localhost:vnc_port+rewarder_port") https://github.com/openai/gym/blob/master/gym/core.py#L234 https://github.com/openai/gym/blob/master/gym/core.py#L234 https://github.com/openai/universe/blob/master/universe/envs/vnc_env.py#L137 https://github.com/openai/universe/blob/master/universe/envs... This is only based on a cursory reading, but it should be possible to use custom environments with OpenAI Universe as it is today.