8 ms·
UnrealCV
- johnhenry 10y agoI was hoping that this would be someone's really cool resume, but this is interesting, nonetheless.
- AtheistOfFail 10y agoWas actually hoping for an alternative to OpenCV that wasn't created in the 90s.
- bluefinity 10y agolibccv perhaps?
- zo1 10y agoDlib has a decent set of features, a few even better to OpenCV in my opinion. Though, at the end of the day, there is no reason why you can't use both.
- mynewtb 10y agoWhat problems do you see with opencv?
- AtheistOfFail 10y agoNothing, actually, it's a quality library, just old. I can think of a few algorithms that have changed or been created since the 1990s so I have to assume there's something that can be done to optimize it. There's also a severe lack of "in-browser" CV solutions (which could be fun), instead opting for binding OpenCV in the server-side.
- ojii 10y agoSame, I expected it to be someones resume built in the unreal engine.
- fnordsensei 10y agoSame here. I was thinking of how long it would take us before the inflation in fancy CVs and resumés would force recruitment agencies to invest in rendering farms and full-body VR suits in order to read job applications.
- TorKlingberg 10y agoI don't mean to criticise you personally, because I have done the same thing, but: Comments about what someone thought the title meant are very common and not interesting in any way.
- johnhenry 10y agoI disagree. Most of the reason that people click on anything on this site is directly because of the title, so if I have particular expectations that the title fails to meet, then it's fair game for commenting. I don't think it's fair to assert that what you find common and uninteresting is irrelevant to someone else.
- Endy 10y agoAww, I don't get to create a walkthrough of my resume in Unreal? I'd love to let recruiters shoot my bosses. (nevermind that the last Unreal I played was in 1998.)
- namaemuta 10y agoHahaha thought the same, some kind of 3d-ish resume.
- soverance 10y agoI could actually make this, and it sounds like a cool idea. But I'm wary about putting the time into it. My concern is how many recruiters would actually download and run a 1 or 2 GB application simply to see my resume, when they instead could visit my website and watch some quick videos. My guess? None.
- TrevorJ 10y agothere's an experimental HTML5 feature so you could ostensibly embed it online.
- AndrewKemendo 10y agoThis is a great example of how we (collectively) are using virtual worlds as "dreams" for Deep learning. In the Deep Vision world we as a group are trying to segment, classify and reinforce our NN training on labeled real world data. The challenge is, it's a very manual process to label data - specifically images. The more that we can do inside the computer, for example automatically labeling pixels inside an image, without having to acquire and label real world data (or making it easier to do with real world data) the easier and faster training becomes that can be applied to real world use cases. The trick is making the virtual world match the real world as closely as possible so that the nets we make are accurate representations of real world scenarios.
- bonoboTP 10y agoIt's not just for deep learning, it's for any computer vision research. Synthetic datasets have been used in the past too, but this makes it much more accessible.
- AndrewKemendo 10y agoIndeed... However CV is becoming more and more Deep Net based.
- deleted 10y ago[deleted]
- alex_hirner 10y agoIn order to transfer learnings from virtual worlds to the real world more efficiently, I imagine a module that translates real world footed to an abstract representation that is the same as for rendered footage. In other words, you would train a NN on input from a footage of a walkthrough of a real building to output rendered footage from the same path (3D data from architects is already there). The middle layer of this quasi autoencoder then is the basis to train fully simulated tasks, e.g. autonomous vehicles. In a way, it would be similar to colorization of b/w footage. Would that scale training data?
- 10y ago
- zitterbewegung 10y agoThis would be cool if openai would participate in it.
- djhworld 10y agoCan someone explain what this is for? Is it so that computer vision researchers can create scenes in UE4 that can then be fed into their application?
- deleted 10y ago[deleted]
- dharma1 10y agoYou could use it to say, auto create large labeled data sets of images and their corresponding depth maps. Or generate large labeled data sets of objects and their segmentation masks. Or for training an autonomous agent (say self driving car) in a simulation environment
- qiuwch 10y agoExactly. That's the goal for this project.
- dharma1 10y agoMany thanks for your work, will be putting it to use! Please consider integrating it with https://gym.openai.com/ https://gym.openai.com/ for easy reinforcement learning tasks. Some similar efforts (though not with open source engines) http://download.visinf.tu-darmstadt.de/data/from_games/ http://download.visinf.tu-darmstadt.de/data/from_games/ https://arxiv.org/abs/1608.01745 https://arxiv.org/abs/1608.01745
- emanuelev 10y agoYep, possibly. This approach has been successfully exploited in real-time SLAM for instance [1][2][3] [1]http://link.springer.com/chapter/10.1007%2F978-3-642-33786-4_17#page-1 http://link.springer.com/chapter/10.1007%2F978-3-642-33786-4... [2]http://www.doc.ic.ac.uk/~ahanda/VaFRIC/iclnuim.html http://www.doc.ic.ac.uk/~ahanda/VaFRIC/iclnuim.html [3]http://robotvault.bitbucket.org/ http://robotvault.bitbucket.org/
- dharma1 10y agoLove it. Anybody use unreal engine with OpenAI gym yet?
- ktta 10y agoAnyone know of something similar for Unity? Unity has a bit of a steep learning curve of you want to do stuff like importing external data and working with outside programs
- qiuwch 10y agoThe CVC lab and Virtual KITTI use Unity in their projects. They might have some internal tools. The advantage of UE4 is open-source, so that we can fully understand what the engine is doing.
- bashtian 10y agoI thought about something like this for autonomous cars. The game engine could simulate all the sensors in a car. Then it would be possible to use this as training data. Would be fun to create a racing game out of it to let different algorithm compete against each other. Does this already exist? Google should have something like this, but probably not freely available.
- gondolgames 10y agoAdasWorks is doing this, check the videos 'Data Generation Tool - Annotation' and 'Real-time Data Simulator' towards the bottom of this article (text is in Hungarian): http://index.hu/tech/2016/09/13/obudan_epul_az_autoipar_nokiaja/ http://index.hu/tech/2016/09/13/obudan_epul_az_autoipar_noki...
- zawarudo 10y agoI, like many others most likely, thought this was gonna be a cool way to present your resume, or someone with a demo of theirs in Unreal Engine... Still, I was not disappointed by the plugin.
- malux85 10y agoDoes anybody know if there's a plugin for unreal, like this, but supports creating geometry inside the game? If not, is there a way of generating levels with code?
- qiuwch 10y agohow do you want to create geometry? like loading an obj file and change its position? Unreal.js might worth a try.
- TrevorJ 10y agoI don't think the plugin needs to directly support those features, as they are already in the engine natively. The engine supports generating meshes dynamically at runtime: https://wiki.unrealengine.com/Procedural_Mesh_Generation https://wiki.unrealengine.com/Procedural_Mesh_Generation It is also fairly easy to generate random or pseudo random levels using blueprint (Unreal's visual scripting API) or C++ If you have any other questions I'd be happy to help.
- malux85 10y agoThanks for your response, great info. The reason I ask is that I have a large dataset that I would like to visualize. The ideal situation would be: I could use python to create a giant flat plane, and then iterate over the dataset to generate the level (which initially would just be quads on the 2d plane). The dataset would be too large to do in-engine in realtime, and the quads are static - I would like to generate the level as a 1 off process, and then have Unreal do any optimisations that it can by "compiling" the level (e.g. frustum culling, quad tree calculation etc) I might have some of those optimizations wrong since my knowledge of engines is still stuck in the Quake 1 era, but you get the gist :) Do you think this is possible? If this tech doesn't exist, I would pay someone to develop it for me....
- TrevorJ 10y agoanything is possible :) How many total datapoints are you trying to represent at once, ballpark? My first instinct is that it would probably be most efficient to feed the data into Unreal and build the physical representation there VS importing a large monolithic 3D model, because then you could make use of instancing, distance culling, etc. This would also mean you could animate the objects separately as well, and manipulate scale, color and the like in realtime. Again, probably depends heavily on the specifics. In general though, modern gaming GPU's can handle just an insane number of polygons so either way, you should be able to represent a ton of datapoints. As far as getting data into the engine, it supports .csv importing where you can define your own row types etc: https://docs.unrealengine.com/latest/INT/Gameplay/DataDriven/ https://docs.unrealengine.com/latest/INT/Gameplay/DataDriven... I believe there's also a SQL lite support module for the engine as well, if you want to deal with really massive sets of data, though the documentation there is sparse. My contact info is in my profile, feel free to hit me up if you have any other questions, happy to help.
- kevinmgranger 10y agoI thought this was going to be Unreal Curriculum Vitae, and it was going to be a playable resumé. I'm slightly disappointed.
- highd 10y agoThis is a brilliant idea, IMO, and a real game changer (in principle, not necessarily tied to the implementation). Without this, CV problems are largely only limited by how much data you can get. If you can simulate arbitrary scene configurations it's now possible to solve almost any non-human CV problem, even ones with sparse data. For example, in security applications you have basically zero data on the sorts of anomalous events you want to trigger on. If you can simulate example cases of those things that means you can actually start applying supervised machine learning.
- qiuwch 10y agoMany researchers have done interesting work using synthetic data (a lot of them are mentioned in this post). This tool just tries to make the creation of virtual world (which can be used by computer vision) easier.