12 ms·
Have We Forgotten about Geometry in Computer Vision?
- cmontella 10y agoI think there's still a lot of room for exploiting geometry. In particular, it leads to very simple models that require little training, and most importantly are completely transparent in how they work. I did some work on a robotic wheelchair that exploited the prevalence of poll-like objects in the environment (trees, parking meters, street lamps) to localize. The model just looked for cylindrical objects and matched them against a map that we generated a priori. Deep models are best suited when you don't know what features are important in your model. For us, it was straight forward with the radius and orientation of the poll object, so a deep model would have probably been the wrong approach.
- krallja 10y agoI love the idea of using poles as your fiducials.
- trevyn 10y agoThis is a good observation. My personal opinion is that using many smaller networks as components of a larger, manually engineered system could be a fruitful approach for these complex problems. End-to-end deep learning is the holy grail -- and may ultimately happen -- but I don't think computation in silicon semiconductors is going to cut it, and we don't yet know what the next computational substrate will be. Did you use a smallish convolutional model for extracting pole position, orientation, and radius from image data, or another approach?
- cmontella 10y agoIf you're interested, here is a link to the initial paper: http://vader.cse.lehigh.edu/publications/fsr12_mon_per.pdf http://vader.cse.lehigh.edu/publications/fsr12_mon_per.pdf
- Stanleyc23 10y agoNot that forgotten, right? Isn't SLAM almost pure geometry? I don't think stuff like LSD-SLAM or any structure from motion stuff has deep learning built in.
- AndrewKemendo 10y agoCorrect about the current state of SLAM. However the next wave for computer vision focused tracking and mapping is heavily based on ML - though there isn't a single combined system that gives robust SLAM/PTAM level results yet.
- logicallee 10y agowhy would anyone downvote the parent comment (by andrewkemendo) given that they seem to speak with the authority of an expert in the field. thanks for your comment.
- osdf 10y agoThere seem to be some nice developments into this direction recently? Something like the SfMNet (https://arxiv.org/abs/1704.07804 https://arxiv.org/abs/1704.07804) or the Correlation Filter Net (http://www.robots.ox.ac.uk/~luca/cfnet.html http://www.robots.ox.ac.uk/~luca/cfnet.html) for tracking.
- rasz 10y agoMost (all?) SLAM approaches are naive and produce point clouds. True geometry approach would result in purely 3d vector output. Kinect, Tango, Hololens all fail at recognizing flat surfaces and straight lines.
- robotresearcher 10y agoThere exist research SLAM systems that use lines, planes[1] and other geometric objects as features, and thus provide non-point geometric reconstructions. Hell, there's even a demo using entire pieces of office furniture as the localization features [2]. [1] https://www.youtube.com/watch?v=KOG7yTz1iTA https://www.youtube.com/watch?v=KOG7yTz1iTA [2] https://www.youtube.com/watch?v=tmrAh1CqCRo https://www.youtube.com/watch?v=tmrAh1CqCRo
- richard___ 10y agoI have a hunch an understanding of classical control will also lead to much better results than the deep RL stuff people are doing for ML based control
- throwaway87423 10y agoClassical control often restricts itself to Linear systems, and non-linear control is next to useless for non drift-free systems (if you can compute the Lie algebra in the first place). SOS relaxations won't scale, and all bets are off when dealing with contacts.
- tnecniv 10y agoSo what's your suggestion when dealing with any non-trivial system? All those issues you mention are true but also active research areas.
- kordless 10y agoWho here visualizes, but does not have strong spatial skills?
- state_less 10y agoI'd be keen on doing or learning about a mapping from spatial tree to 2d raster and have the inverse (dual) mapped back to a spatial tree via a deep learning model, though I'm not sure how you'd represent the shapes and transform matrices. Maybe some sort of matrix stream like char2vec? A nice property of such an experiment, you could generate your training data via permutations of the spatial trees. You may have to accept multiple valid answers, since one could correctly say, the ball is to the left of the car, or the car is to the right of the ball.
- latently 10y ago"However, these models are largely big black-boxes. There are a lot of things we don’t understand about them." This describes geometry as well as it describes deep learning.
- aqsalose 10y agoerr, no. The model is a "black box" if the only thing we have is the input and the output, and only little intuition how the model produces output from the input. We have spent at least a couple of thousand years studying geometry; we know geometry quite well. Let me demonstrate with a stupidly simple geometric model. Suppose (for the sake of the argument) that we have simple image input, consisting only simple solid geometrical structures. Say, solid 2-d circles of one color, on the background of different color. From high school geometry, we know that we know everything there's to know about a circle when we know its location on x-y plane and its radius. We could easily come with a parametric model for fitting circles in the pixel image data of circular objects. (For example, we could minimize difference in 2-norm between data image and image corresponding to a set of circles, [x_i,y_i,r_i], i=1..n ). This kind of descriptive parametric model would be particularly easy to understand: model structure consists nothing but representations of circles! (But of course, it wouldn't be particularly interesting model; it would apply to simple images consisting of circles only). Alternatively we could work out the mathematics bit more, and come up with something like Hough transform to find circular shapes. Still nothing mysterious about it: https://en.wikipedia.org/wiki/Circle_Hough_Transform https://en.wikipedia.org/wiki/Circle_Hough_Transform However, my point is: We could also train a neural network to find circles in the images of our example. It might be good at it. However, understanding how the circle representations are encoded in the final trained network certainly would not be as easy than in our nice parametric model. Some realistic applications of "simple" geometric models would be active contours / snakes ( https://en.wikipedia.org/wiki/Snake_(computer_vision) https://en.wikipedia.org/wiki/Snake_(computer_vision) ) or (stretching the meaning of the word 'geometry') the various traditional edge detection algorithms that have been around long time. Or read the post, in which the author describes how they utilized projection geometry model to account for camera positions and orientation, or for stereographic images. We know how the geometry of stereographic vision works: we don't need waste resources to train a network to learn inscrutable model for it. Deep learning is useful when we need models for things complicated enough that we don't know how to model them. (For example, model that tells us "is there a dog in this image".)
- santaclaus 10y ago> In computer vision, geometry describes the structure and shape of the world. Specifically, it concerns measures such as depth, volume, shape, pose, disparity, motion or optical flow. Isn't optimization geometry? DL is just shifting the requisite geometric insights to a different level of abstraction. Take that knowledge of geometry, revolutionize non-convex optimization, and let all fields that build on the base abstraction benefit.
- deepnotderp 10y agoDL's optimization is pretty simple SGD. But I would suggest that the type of optimization in deep learning is actually a lot more like geometry than calculus. This was more of a hunch until recently, but some results such as the successful application of tropical geometry and less exotic Riemannian geometry to the analysis of DL loss surfaces are showing some evidence for that hypothesis.
- ouid 10y ago"not many problems that aren't solved by deep learning" The problems that deep learning doesn't solve aren't solved by anything else either. There's still lots of computer vision that we cannot do.
- taeric 10y agoThis is making the odd assumption that our previous abstractions should play well with our current abstractions of pixels. My feeling is that the "perfect" abstraction of reality to geometry is actually a very high order function that we don't fully understand. An easy example is parallel lines aren't always parallel, even though that is a common geometry affordance. So, that our current toolset does not play well with our previous one should not be that surprising. Would it be nice if they did? Yes. But it would also be nice if Newtonial physics played well with quantum physics. Why? Tough to answer. Not impossible, but above my ability to understand.
- electronvolt 10y agoI think you're kind of missing the author's point. > My feeling is that the "perfect" abstraction of reality to geometry is actually a very high order function that we don't fully understand. You don't need a perfect abstraction of reality, though. All you need to do is get close enough to catch up with humans or outperform them and you've solved most of the hard problems in computer vision. Fortunately--at human scales, which are the only ones you need to care about for computer vision, reality behaves closely enough to a pure Euclidean space that you're fine. :) The author's primary argument seems slightly more nuanced, too, than just "It'd be nice if we could use old techniques". Basically, they're claiming that if you can build geometric understanding into the ML model, you will get significantly better results that just naively plugging and chugging away with raw data. That's an empirical claim that can be validated by researchers--either it will give a significantly improved performance on well defined problems (stereo vision, etc.) or it won't. Vision is one of the research areas that have developed pretty good benchmarks over the years. :)
- taeric 10y agoBut that is exactly my point. Geometric models are the old tools we had. The new tools are ML Models. It would be nice if they both worked together. But there is nothing to say that they should. Nor is it obvious that it would be beneficial. Your point that this is testable, though, is important. I fully agree with that and was not intending to dismiss the idea. Just because I am not as confident as the author, does not mean that I am right. :) (I'd accept that I am likely not right.)
- trevyn 10y agoHowever, these models are largely big black-boxes. There are a lot of things we don’t understand about them. I'm getting kind of sick of this "deep learning is a black box" trope, because it's really not true anymore. Yes, it's a black box if you just use "some data and 20 lines of code using a basic deep learning API" as mentioned in the article. But if you spend some time understanding the architecture of networks and read the latest literature, it's pretty clear how they function, how they encode data, and how and why they learn what they do. Because neural networks are so dramatically more effective than they used to be, in so many domains, it's true that we don't yet have a good understanding of optimal ways to build, train, and optimize networks. But that is exactly why there is so much excitement -- because there is a lot to discover, and a lot of progress that can be made quickly. I agree with the author that fundamental physical and geometric approaches are still relevant and useful, and have been somewhat ignored recently, but the fact remains: If you and I as individuals want to maximize our personal impact, and capture as much value as we can while working on interesting problems, deep learning is an excellent field in which to do that. It's kind of like we just discovered a nice vein of gold, and the silver miners are like "yeah, but we don't know much about that vein of gold and how long it will last." Which is true, but in the meantime, there's a lot of easy money to be made, and ultimately, both types of resources are important and synergistic.
- electronvolt 10y ago> I'm getting kind of sick of this "deep learning is a black box" trope, because it's really not true anymore. That's fair/probably true. I think there's two things that drive that--one, lack of a widely shared deep understanding of the field[0] (and not really needing a deep understanding to get good results--as both you and the author pointed out), and two, the fact that it feels like cheating, compared to the old ways of doing things. :P [0] When the advice on getting a basic understanding is "read a textbook, then read the last 5 years of papers so that you aren't hopelessly behind", there just isn't going to be widespread understanding.
- trevyn 10y agoFair. How about an excellent 4-minute YouTube video to get a basic understanding? :) https://www.youtube.com/watch?v=AgkfIQ4IGaM https://www.youtube.com/watch?v=AgkfIQ4IGaM
- arketyp 10y agoMy apprehension was that the computer vision community has been suffering some serious cognitive dissonance lately because here they spent all these years mapping problems to feature spaces of manageable dimensionality, backed by theory saying that proper assumptions must be made to reduce the search space; and then comes these deep nets, hardly tailored to the problems, and out-performs algorithms with decade old history of fine-tuning. Despite this, I don't think anyone disputes the potential of a good set of assumptions. Instead I think what deep learning has thought us is that we should reconsider what these assumptions should be. While geometry might well be the first kind of language a toddler learns to think in, this should probably not be confused with the rigorous geometry of Euclid. Quite possibly we have some spatial relationships such as the affine transformations hard-coded in our brain at birth, but this does not mean, for instance, that one is therefore necessarily ever able to to draw a house in correct perspective.
- trevyn 10y agoExcellent point -- our brains and deep networks seem to learn a kind of "fuzzy geometry" that is in some ways more robust than numerically correct geometry, and this also allows us to spend more cycles on higher-level abstractions.
- snovv_crash 10y agoMy problem with our brains and NN is exactly this "fuzzy geometry" you speak of. Using precise geometry, if we have a pair of cameras equivalent in layout to eyes we can reconstruct a world with mm-precise accuracy. That is something that cannot be done with neural networks (artificial or biological), instead you get a sort of semantic tagging of "green sofa here, table here, bed there". But if afterwards you want to use this model to know if the sofa will fit into the alcove (or the car into the garage), the NN systems will be wildly unreliable.
- trevyn 10y agoLike anything, the more tools in your toolbox, the more problems you can solve! I love that humans seem to be innately wired to constantly create new and better tools.
- MichailP 10y agoThis seems like a perfect thread to ask a question that is itching me for some time. Is anyone aware of deep learning approaches or some combo with geometry algorithms for mesh generation? Something like quad, hex or tetrahedral meshing aided with deep learning?
- froindt 10y agoInterested in where you're going with this. What are the advantages you are seeking out? I've done some computational geometry with slice data from meshes, but have been operating off of STL files, not a feature-based format.
- MichailP 10y agoJust interested in simple mesh algorithms. I feel as if machine learning could help but almost no one is going in this direction. Higher order quad and hex meshing is not a solved problem, and could help in a bunch of numerical physics algorithms.
- andreyk 10y agoI have a feeling people are going to reply without reading this through and assume the author poses a Deep Learning vs Classic CV sort of argument, in a deep-learning-is-overrated sort of way. Whereas it seems to me he is merely saying Deep Learning should be informed by Classic CV. "I think we’re running out of low-hanging fruit, or problems we can solve with a simple high-level deep learning API. Specifically, I think many of the next advances in computer vision with deep learning will come from insights to geometry." And it's true. A lot of the low-hanging fruit have been gotten, and stuff like SLAM is not about to be done wholly by deep learning (probably). There are a lot of problems that require more insight and analysis than 'throw a deep network at a large dataset'. As he concludes "learning complicated representations with deep learning is easier and more effective if the architecture can be structured to leverage the geometric properties of the problem." And thinking about what the architecture should is basically the hard bit in deep learning.
- amelius 10y agoThe sad parts, for me are: 1. Thinking about a deep learning architecture is really a different kind of research. It requires much guesswork, a lot of trial and error, and lots of waiting for training cycles to complete. For me this type of work is much less interesting than engineering a solution, even if the results are better. 2. Deep learning is quickly becoming a commodity. One can download libraries to do almost anything with neural networks. I fear that deep learning will become the new "web development" of the 90s. Everybody can do it (this is not really true just yet, but I suspect we are rapidly reaching an asymptote where it is true).
- TheSpiceIsLife 10y agoWhat is sad about 2? Other than reminiscing about the halcyon days of the web (which is a forever moving target of 'how it was when I grew up'), it's the present day internet, technically speaking, a marvel. What's exciting about deep learning becoming a commodity isn't seeing what everybody can do with it, but what the will be built standing on it's shoulders.
- 10y ago
- Eerie 10y agoCheck this out "Learning 3D Models from a Single Still Image" https://www.youtube.com/watch?v=bWbEsDbfayc https://www.youtube.com/watch?v=bWbEsDbfayc "3-D Reconstruction from a Single Still Image" http://cs.stanford.edu/people/asaxena/reconstruction3d/ http://cs.stanford.edu/people/asaxena/reconstruction3d/ "Make3D: Convert your still image into 3D model" http://make3d.cs.cornell.edu/ http://make3d.cs.cornell.edu/
- antman 10y agoWhat is the cost/benefit in this? Is it appropriate to divert resources to older techniques because we haven't yet figured how to do it in the new way? Yes we can score a few points but that is an advancement only to an academic setting. We used to need PhDs to do simple Computer Vision applications and object recognition in a meaningful matter was a far away dream. Now a child can make an application on its Raspberry Pi, because the new way of doing things is generalizable. You don't need to spend huge amounts of time redoing things just to get to the basics and reach the state of the art as it was 20 years ago. Should we reintroduce the old ways for the quick wins, or should we divert our research resources to try to solve unsolved problems? The GPU will get cheaper the cloud GPUs will be cheaper. So this is the state of things today. If somebody wants to advertise his paper's submission to a conference good for him but it should not be presented as an important advancement that should be the new way of doing things. Because it isn't. When we decide that its feasible to send robots to the planets or even build robots on them, building chessboards to do rectification of the cameras should not be one of their tasks. Disclaimer: I had horse in this race too, I was on the losing side of the deep learning argument, I was wrong, I got over it.
- josefx 10y ago> What is the cost/benefit in this? Is it appropriate to divert resources to older techniques because we haven't yet figured how to do it in the new way? I too would like to know why cars have wheels when helicopters have shown that rotors work well. We should have abandoned wheels ages ago since they are clearly old and with that inferior. > Now a child can make an application on its Raspberry Pi, because the new way of doing things is generalizable. And that app will have issues deciding if it sees a couch or a leopard. Or do you expect a child to correctly train its neural net? > The GPU will get cheaper the cloud GPUs will be cheaper. Why again do we have multiple algorithms for sort when a few nested for loops would do? Maybe smart algorithms scale better than hardware ever could. > building chessboards to do rectification of the cameras should not be one of their tasks. And yet we have mars probes with a build in color table. Why do you hate geometry?
- antman 10y ago
- osdf 10y agoThere are several nice references within the article. I think the gvnn, Neural Network Library for Geometric Computer Vision, (https://arxiv.org/abs/1607.07405 https://arxiv.org/abs/1607.07405) should also be mentioned.
- real-hacker 10y agoI am wondering if a properly structured deep net can capture these Geometry features with a few layers, if these features are relevant to the problem? Is this reflecting our human arrogance that we simply know better than AI neural networks?
- faragon 10y agoSure it can. The problem is to validate that e.g. the net really knows geometry, and not that it is just working for problems similar to the ones in the training. Unless the training can be validated, i.e. formally proven that after some specific training the net "knows" covering all the cases, you'll have to complement neural nets with classic computer vision, no matter how good you think your model is.
- deleted 10y ago[deleted]
- martijn_himself 10y agoApologies for asking a computer vision related but off-topic question: does anyone know where to start if I wanted to track moving objects (players) in a (sports) video? Is there a 'general purpose' open-source library out there anyone would recommend or is this not 'trivial' to implement (I presume non of this is trivial stuff)?
- adrianN 10y agoHave you looked at OpenCV?
- martijn_himself 10y agoI have only just started looking at the problem and (coincidentally) found this thread on the front page of HN so I couldn't resist asking the question! I will look into OpenCV, is it the go-to libary for these kind of problems?
- anjc 10y agoIt isn't just a case of deep learning vs stereo geometry, you can have many cameras informing/improving depth analysis. I've been out of the loop for while, but I'd be mind-blown if people were happy to rely on deep learning for depth reconstructions in domains like...self-driving vehicles, versus provably accurate, fault tolerant systems with multiple cameras + geometry. And what are the hardware requirements typically like for DNN with high resolution images for real time reconstruction? Are we well past the stage where it's real time and accurate?
- phkahler 10y ago>> For example, one of my favourite papers last year showed how to use geometry to learn depth with unsupervised training. I've been saying that LIDAR is a hack for some time. People don't need it and neither should computers.
- pasta 10y agoLIDAR is not a hack. It measures depth even when objects are not moving. With vision we can estimate depth, but sometimes we need movement for that.
- musesum 10y agoSo, LIDAR is best solution for an autonomous vehicle when it is not moving?
- nercht12 10y ago>> LIDAR is not a hack. Depends on what one considers a "hack". Epistemologically, yes, it is a hack because it already assumes a 3D world identifiable via such means, and therefore may make the AI incapable of "reconstructing" in its "brain" more dimensions. Whereas if the AI has to learn the "hard way", it may be able to project that analogously to other ideas, and could "speak" of them in the same terms we do when speaking about dimensions. Of course, if we're only looking for productivity from the AI, then philosophical "correctness" is a needless endeavor. ... But it might be more fun.
- osi 10y agoI was just reading https://blog.openai.com/adversarial-example-research/ https://blog.openai.com/adversarial-example-research/ and it struck me that an understanding of geometry might help. For the examples and scenarios cited, the geometries of the two items would be different (sometimes markedly, sometimes less, like washer-vs-safe).
- tmsldd 10y agoIf you can afford data and computing power, dpl is the way to go... you don't need to think much to get the thing done quickly.. but in my eyes , having a suitable mathematical description to a particular problem always give you useful insights with respect to the problem, variables and its relationships and this is actually what most of us are looking for..
- duality 10y agoThe fact that convolutional neural nets are used in vision is significant. The convolutional structure encodes the geometry.
- krosaen 10y agoThis reminded me of a cool paper I came across recently, Spatial Transformer Networks [1], a good example of how knowledge of geometry helps frame the problem more effectively, allowing the network to learn how to e.g rotate objects into a canonical orientation before identifying them. [1] https://arxiv.org/abs/1506.02025 https://arxiv.org/abs/1506.02025