9 ms·
Google open sources key piece of Street View technology
- jacquesm 14y agoNeat, what a pity that real hacker news gets under-appreciated!
- jlapenna 14y agoThe video in the link is awesome. So that's how Google figures out roads and stuff!
- mierle 14y agoThis is a key component of Street View. We're really happy to have released this. When we started writing Ceres, there was nothing available like it. It's state of the art; a real contribution to computer vision and other fields which need large scale nonlinear least squares. Part of the reason I worked on this in my spare time (my real job at Google is something else) was to get it integrated in libmv (http://code.google.com/p/libmv http://code.google.com/p/libmv) and subsequently integrated into Blender. This opens the door to many sophisticated tracking features, like autocalibration, rolling shutter, multi-shot solves, tripod solves, planar tracking, and more. EDIT: Changed link title to make it more provocative
- mierle 14y agoThere's some more discussion over at Reddit: http://www.reddit.com/r/programming/comments/t1mo7/ceres_solver_googles_large_scale_nonlinear_least/ http://www.reddit.com/r/programming/comments/t1mo7/ceres_sol...
- vibrunazo 14y agoWhat exactly would you use this in Blender for? As a library for someone to build a script that would turn 2d photos into 3d objects or something? Could you expand on that?
- ot 14y agoI guess that one application would be camera tracking to implement automatic compositing of 3d models into videos.
- mierle 14y agoBlender got integrated matchmoving / tracking with the 2.61 release, by way of libmv (which I am the BDFL of). A critical part of many tasks in 3D reconstruction and tracking is a solid, flexible nonlinear minimizer. Since Ceres is so easy to model problems with, is so fast, and is appropriately licensed (unlike some alternatives), it is an enabler for these more sophisticated tracking operations. Note that Blender's next open movie, http://mango.blender.org http://mango.blender.org, is live-action done with Blender's tracking tools. Ceres is going to be used for that as well.
- yellowbkpk 14y agoWhen you say "a key component of Street View", which part is it a key component of? Snapping vehicle track data to existing roads? 3D model computation for scene-to-scene jumps? Both? If you're snapping track data to roads, how do you deal with areas where you have no existing data? Do you take into account aerial imagery? </openstreetmap-nerd jealous="true">
- mierle 14y agoUnfortunately we can't talk much about this; sorry.
- est 14y agoGoogle Street View was filmed at 30fps video format. You can generate a 2D route map directly from it without GPS or when GPS is not available. I took some notes during the StreetView Q&A on reddit two years ago: http://blog.est.im/archives/554 http://blog.est.im/archives/554
- davidjohnstone 14y agoIt would be nice if we could use it, or (better) if Google exposed the ability in the Maps API, to fit paths to the most likely road, just like is being done in the demo video on the project page.
- mierle 14y agoThe roads are not an input to the optimization shown in the examples. How this works is left as an exercise for the reader.
- ZeroGravitas 14y agoSince it mentions sensor fusion, is this not just* combining the data from the GPS, compass, accelerometers (and any other sensor input from the car e.g. comparing photos, IR, wifi signals) to smooth out any irregularities in a single sensor and making better guesses where they all agree. If I'm right then the demo video is cheating a bit by only showing the GPS input initially and the rest of the info seems to come from thin air, which might be why people think it's working with the underlying street data. Dead reckoning with a compass and accelerometer and/or wheel odometer/tachograph might also have produced a similarly wonky line, but then you have two different wonky lines to average together to get a better approximation of the true path. Note this supposition is based almost entirely on doing the first 3 weeks of the Udacity course CS373: Programming a Robotic Car, but it seems to fit quite neatly with the stuff they talk about i.e. a car/robot moving, guessing how far it's moved, taking sensor snapshots of its environment then cleverly combining the data to figure out accurately where it is. Off-topic: scrubbing back and forth in the blog video (to get a better idea of what was happening) seemed incredibly smooth for me, is this a side effect of Youtube/Firefox using HTML5 video? (edit: trying it out with Flash, I get a fancier, but less useful pop-up when I try to scrub, a tech limitation of Flash or just a design decision?) * not meaning to underestimate the hard work and genius that underlies this technology, but the same concept is used in such everyday items as Wii remotes and suchlike.
- beagle3 14y agois any geometry point? I mean, when you have a loop (say, around the manhattan grid), then I can understand how everything works when you have one hard(er) constraint such as, "the two farthest points visited are 2nd street at avenue C and 112th street at 12th avenue" - and get everything else aligned accordingly. But if no external geometry constraint is involved (roads being the most abundant, but not the only type of external geometry constraint), then this reader has been unsuccessful with the exercise.
- follower 14y agoThanks for adding your note about integrating this into Blender--that makes the news much more relevant to my interests. :) I was looking forward to integration of motion tracking into Blender and had a small play with the feature when released. Now I just need to make the time to play some more...
- fgcallari 14y agoYeah!!
- mierle 14y agoTook long enough but we finally made it!
- julienr 14y agoGood job !
- strebler 14y agoWhat exactly is the operation being done in the video? My best guess is a noisy set of GPS coordinates / panoramas being repositioned via bundle adjustment on the street view panoramic captures. Or something.
- sandwichmaker 14y agoIn the video, no vision/bundle adjustment is used, just GPS, IMU and other sensors.
- leegao 14y agoI think I can sort of see what's going on, we're trying to fit the data of some GPS position taken from a walk around the park or something similar, and fit it to some linear combination of basis of some "street functions" to minimize its two-norm, where the "street function" corresponding to each street for some point (x,y) is some measure of how far away from the street that point is. For example, in the subproblem of only one-way streets that we can assume to be composed of one or more segments of straight lines per street, we take a simple street-function $\phi_i(x,y)$ corresponding to street i to be the norm of the projection of <x,y> onto that street. Furthermore we also add into the system some "smoothing-function" to ensure that the overall shape of the final path is doable, for example constraining the distance between successive points. Next, we solve the argmin of the norm equation for each point so that each point is now moved to some linear combination of the streets, and truncate all but the most significant street basis, and rerun until we get to some acceptable tolerance. Maybe?
- ot 14y ago> Automatic differentiation This is extremely cool. The documentation seems to be a WIP on this, can anyone comment on what kind of cost functions are supported by automatic differentiation? EDIT: it would be also interesting to see how this compares to Theano [1], which also does symbolic differentiation and can JIT to native code and GPU. It is a largely more generic framework, but I'm not sure how well it can handle large sparse problems. [1] http://deeplearning.net/software/theano/ http://deeplearning.net/software/theano/
- sandwichmaker 14y agoYou can define define arbitrary functions. You are only limited by the operator overloads defined in jet.h. Let us know if you think something is missing.
- ot 14y agoI guess you mean arbitrary expressions, just using arithmetic operations and the overloaded functions (I see that abs, log, sin, cos, ... are supported). What happens if I try to use an if-then or a for loop depending on one of the variables?
- sandwichmaker 14y agoWe have conditionals overloaded which allow you to compare with other jets, and doubles, which boil down to a comparison with the "value" part of the jet. So conditionals are not a problem. Loops I would stay away from, since they would destroy the inlining. But if you have an example of a dynamic loop, we would like to hear about it.
- mierle 14y agoIt's not just expressions; it's any computation you normally would do with doubles. Take a look at the simple bundle adjustment example: http://code.google.com/p/ceres-solver/source/browse/examples/simple_bundle_adjuster.cc http://code.google.com/p/ceres-solver/source/browse/examples... In particular, search for "SnavelyReprojectionError". In this case there are no loops or if statements, but they work fine too. The only issue with branches is that you can make the cost function discontiguous, which can cause issues. The way this works is that you write your cost functions templated on T. For pure-cost evaluation, this is a double. For cost and jacobian, T is replaced with the "Jet" object, found in jet.h. However, you never see this; Ceres does the substitution for you. There are some caveats; for example, sqrt(0) doesn't work on autodiff objects since the derivative is undefined. You have to use a taylor approximation or similar if the argument to sqrt is exactly zero. If you want to see some more complicated code that runs with autodiff, check out the included rotation.h header. We use that internally for 3D reconstruction type cost functions: http://code.google.com/p/ceres-solver/source/browse/include/ceres/rotation.h http://code.google.com/p/ceres-solver/source/browse/include/...
- slig 14y agoIn the video they figure out the road inside a tunnel, with no GPS data. How does it do that?
- mierle 14y agoFusing sensor data (IMU, etc). You can formulate this as a (extremely) giant sparse least squares problem.
- cft 14y agoThat's no Instagram, no Facebook, this is technology, not Hollywoodisation of Silicon Valley
- bonch 14y agoI guess when Google wants to counteract negative publicity, such as supervisors knowing full well that Street View was sniffing wifi data, they just talk about open source again to make techies forget all about it.
- pkh80 14y agoIt'd be great if this is just the start of a trend where Google supports open source mapping tech that would make projects like OSM more viable (for reasons why its not currently viable see my other posts.) Its a bit like Google being the biggest supporter of Firefox, even though they are also a competitor (Chrome.) Though Google supporting OSM is more like Microsoft supporting OpenOffice, since Google has such a large monopoly in maps right now.
- jetsnoc 14y agoThank you for your contribution. It's great to see a giant, who's gained so much from open source contribute back in a significant and meaningful way! For those interested it appears it was open sourced under the "New BSD license."
- rmc 14y agoGoogle has often contributed code to the open source world (like this). However they don't AFAIK release any data. Google drove cars around the world photographing everything. But they claim copyright on it all. If they allowed derivative works of it, then Open Street Map could bloom and benefit massively.
- rmc 14y agoDownvotes because I point out how Google can donate data, not just code? ☹
- sparky 14y agoAwesome! I just filed a few bugs with patches that let it build out-of-the-box on Ubuntu. There didn't seem to be much in this space outside of levmar ( http://www.ics.forth.gr/~lourakis/levmar/ http://www.ics.forth.gr/~lourakis/levmar/ ); levmar is great for what it does, but some friendly competition is most welcome from my perspective. It's really surprising how broadly applicable non-linear least squares is.
- sandwichmaker 14y agoThank you for the bugs and the fixes.
- mierle 14y agoWorking on integrating fixes now...
- mierle 14y agoDone!
- EvilTerran 14y agoWow. Released, then community-sourced patches offered and integrated, all in ten hours. Sometimes open source makes me feel all warm & fuzzy inside. :)
- fgcallari 14y agoTrust me if I say that this baby's performance on large problems is to good'ole levmar what Ferrari is to tricyle. Or rather, don't, just benchmark them and see for yourself ;-) Kier, Sameer and Co. have done an amazing job here.
- eliben 14y agoFrom a cursory look - that's a nicely organized, formatted and documented piece of code. Well done.
- joelthelion 14y agoThis is much more than a key piece of Street View. It's an awesome building block for thousands of very diverse AI applications. Great contribution!
- JoeCamel 14y agoJohnny Lee has written a short post about Ceres: http://procrastineering.blogspot.se/2012/05/today-sameer-agarwal-and-keir-mierle-as.html http://procrastineering.blogspot.se/2012/05/today-sameer-aga... "If I find the time, I might try to post some tutorials on using Ceres. Because I believe this is one of the most powerful tools in modern engineering, and no one ever taught it to me in undergrad or high school. It's like the difference between doing long division by hand and then being handed a calculator."