6 ms·

Google Earth Reverse Engineering

steve19 8y ago
This is amazing. Would it be practical to download an entire town or small city?
- retroplasma 8y ago
  Can be done, although high LOD takes a while and the current implementation might eat up all memory. You can run it in parallel though. Shard by octants :)
automatoney 8y ago
I'm so glad that this has been made, I tried working on something like this a while ago but gave up after struggling for a bit. At the time I was surprised it didn't exist yet. I'm wondering though if Google'll eventually patch things so this doesn't work, or if they're not too bothered by people grabbing data like this.
- paganel 8y ago
  In the “old”, initial days (circa 2005-2006) they weren’t bothered by this, I remember they released the GMaps public API back then as a result (among other factors) of some people reverse engineering the public-facing interface. If I’m not mistaken one of the first projects that was using the reverse-engineered version (before the API was made public) was the former Chicago Crime website, one of the first Django installations.
  - mtmail 8y ago
    The book 'Never Lost Again' (2018) tells the keyhole/Google Maps story line. https://www.goodreads.com/book/show/35820393-never-lost-again https://www.goodreads.com/book/show/35820393-never-lost-agai... Actually I left with the impression nobody involved had a good plan what Google Maps should be after launch. API was an afterthought.
arthurcolle 8y ago
this is sick. how did you figure out the numeric schemes and even have the idea to look where you did?
- retroplasma 8y ago
  Mainly lived inside Chrome's dev tools for many nights. Its deobfuscator and debugger is very handy. Still scratched my head lots of times along the way. Some interesting stuff is still missing, e.g. no idea yet how they get octant paths from geo coordinates.
  - dTal 8y ago
    That's kind of funny that Google wrote both the obfuscated code and the deobfuscator.
    - lima 8y ago
      The obfuscation isn't necessarily a deliberate attempt at hindering reverse engineering, but a result of their optimizing JS compiler and Protobuf-to-JSON-bridge.
  - JZL003 8y ago
    What do you mean by deobfuscator? It's de-minifier that just makes it pretty-printed (with the "{}" icon?)
userbinator 8y ago
I hope this gets mirrored somewhere else, because I don't trust GitHub to not cave in to legal demands despite RE being legal regardless of EULA in parts of the world, and the extra attention this gets may otherwise kill it.
- kodablah 8y ago
  From what I've seen, GitHub will only respond to a DMCA request for takedown, and Google very rarely files those (I think I saw one in the GitHub DMCA notice repo for an entire internal published source repo once, could be misremembering). Google doesn't usually fight with lawyers instead of tech (e.g. making scraping more difficult). Not saying you shouldn't fork or fear, but GitHub and Google are generally good actors compared to others.
  - cronix 8y ago
    Yes, but MS now owns Github and MS might have a different viewpoint in terms of copyright than GH has traditionally had.
    - kuhhk 8y ago
      Why do you suspect that?
      - feikname 8y ago
        Not OP, but I'd say because IIRC it's now a different CEO.
        cronix 8y ago
        MS has traditionally been a software company, and has a LOT of court experience over the last 40 years regarding copyright and lawsuits when it comes to code and IP. I just think they might have a different viewpoint than GH has traditionally had, since GH's business is completely different than traditional MS businesses. The post I was responding to was talking about GH in the past and what they might do based on that. I'm just saying there's a recently new owner who might look at it differently, so past behavior might not be the best indicator for this since it's being run by a different corporation now.
jpatokal 8y ago
"Reverse engineering" seems a bit pointless when Google has open sourced the code for Google Earth Enterprise: https://github.com/google/earthenterprise https://github.com/google/earthenterprise (Update: The client was not open sourced, just the server.) However, this doesn't give you carte blanche to use Google's own imagery: https://maps.google.com/help/terms_maps.html https://maps.google.com/help/terms_maps.html
- userbinator 8y ago
  I wonder why the latest release is over 800MB compressed... I doubt that includes the actual imagery either.
  - oh_sigh 8y ago
    It looks like they include install files for all 3rd party dependencies like postgres, python, openssl, etc.
- retroplasma 8y ago
  That project seems to use some other kind of API. It does not handle real octree/3d data AFAIK. I've only seen quadtree and terrain elevation stuff there.
- tylerl 8y ago
  That's an interesting TOS. I've never before seen "fair use" called out as an explicit allowance in terms of copyright restrictions by the part of the copyright holder. It's not entirely clear to me what that would even imply.
  - widforss 8y ago
    It may have implications in countries where fair use is not a thing.
  - userbinator 8y ago
    I suspect they're trying to give more reassurance to "personal, non-commercial use is allowed as long as you don't hit our servers too hard" --- which seems to be their stance on a lot of their other services too.
walrus01 8y ago
Once you have an OBJ file, you can 3D print it: https://all3dp.com/1/obj-file-format-3d-printing-cad/ https://all3dp.com/1/obj-file-format-3d-printing-cad/ If you want a 3D model of your office building or something.
- TacticalTable 8y ago
  Do you have a better guide to this? The model seems to be infinitely thin, which makes it unprintable in all the slicers I've tried.
pieterhg 8y ago
Love this, but I'm getting an error: "DeprecationWarning: Buffer() is deprecated due to security and usability issues" Opened an issue: https://github.com/retroplasma/earth-reverse-engineering/issues/3 https://github.com/retroplasma/earth-reverse-engineering/iss...
- anc84 8y ago
  That's not an error, just a warning. You can proceed using it just fine.
k9s9 8y ago
NASA's WorldWind is a similar project for those interested - https://github.com/NASAWorldWind/WebWorldWind https://github.com/NASAWorldWind/WebWorldWind docs - https://worldwind.arc.nasa.gov/web/ https://worldwind.arc.nasa.gov/web/

This is amazing. Would it be practical to download an entire town or small city?

retroplasma 8y ago
Can be done, although high LOD takes a while and the current implementation might eat up all memory. You can run it in parallel though. Shard by octants :)

I'm so glad that this has been made, I tried working on something like this a while ago but gave up after struggling for a bit. At the time I was surprised it didn't exist yet. I'm wondering though if Google'll eventually patch things so this doesn't work, or if they're not too bothered by people grabbing data like this.

paganel 8y ago
In the “old”, initial days (circa 2005-2006) they weren’t bothered by this, I remember they released the GMaps public API back then as a result (among other factors) of some people reverse engineering the public-facing interface. If I’m not mistaken one of the first projects that was using the reverse-engineered version (before the API was made public) was the former Chicago Crime website, one of the first Django installations.
- mtmail 8y ago
  The book 'Never Lost Again' (2018) tells the keyhole/Google Maps story line. https://www.goodreads.com/book/show/35820393-never-lost-again https://www.goodreads.com/book/show/35820393-never-lost-agai... Actually I left with the impression nobody involved had a good plan what Google Maps should be after launch. API was an afterthought.

arthurcolle 8y ago

this is sick. how did you figure out the numeric schemes and even have the idea to look where you did?

retroplasma 8y ago
Mainly lived inside Chrome's dev tools for many nights. Its deobfuscator and debugger is very handy. Still scratched my head lots of times along the way. Some interesting stuff is still missing, e.g. no idea yet how they get octant paths from geo coordinates.
- dTal 8y ago
  That's kind of funny that Google wrote both the obfuscated code and the deobfuscator.
  - lima 8y ago
    The obfuscation isn't necessarily a deliberate attempt at hindering reverse engineering, but a result of their optimizing JS compiler and Protobuf-to-JSON-bridge.
- JZL003 8y ago
  What do you mean by deobfuscator? It's de-minifier that just makes it pretty-printed (with the "{}" icon?)

userbinator 8y ago

I hope this gets mirrored somewhere else, because I don't trust GitHub to not cave in to legal demands despite RE being legal regardless of EULA in parts of the world, and the extra attention this gets may otherwise kill it.

kodablah 8y ago
From what I've seen, GitHub will only respond to a DMCA request for takedown, and Google very rarely files those (I think I saw one in the GitHub DMCA notice repo for an entire internal published source repo once, could be misremembering). Google doesn't usually fight with lawyers instead of tech (e.g. making scraping more difficult). Not saying you shouldn't fork or fear, but GitHub and Google are generally good actors compared to others.
- cronix 8y ago
  Yes, but MS now owns Github and MS might have a different viewpoint in terms of copyright than GH has traditionally had.
  - kuhhk 8y ago
    Why do you suspect that?
    - feikname 8y ago
      Not OP, but I'd say because IIRC it's now a different CEO.
      - cronix 8y ago
        MS has traditionally been a software company, and has a LOT of court experience over the last 40 years regarding copyright and lawsuits when it comes to code and IP. I just think they might have a different viewpoint than GH has traditionally had, since GH's business is completely different than traditional MS businesses. The post I was responding to was talking about GH in the past and what they might do based on that. I'm just saying there's a recently new owner who might look at it differently, so past behavior might not be the best indicator for this since it's being run by a different corporation now.

jpatokal 8y ago

"Reverse engineering" seems a bit pointless when Google has open sourced the code for Google Earth Enterprise: https://github.com/google/earthenterprise https://github.com/google/earthenterprise (Update: The client was not open sourced, just the server.) However, this doesn't give you carte blanche to use Google's own imagery: https://maps.google.com/help/terms_maps.html https://maps.google.com/help/terms_maps.html

userbinator 8y ago
I wonder why the latest release is over 800MB compressed... I doubt that includes the actual imagery either.
- oh_sigh 8y ago
  It looks like they include install files for all 3rd party dependencies like postgres, python, openssl, etc.
retroplasma 8y ago
That project seems to use some other kind of API. It does not handle real octree/3d data AFAIK. I've only seen quadtree and terrain elevation stuff there.
tylerl 8y ago
That's an interesting TOS. I've never before seen "fair use" called out as an explicit allowance in terms of copyright restrictions by the part of the copyright holder. It's not entirely clear to me what that would even imply.
- widforss 8y ago
  It may have implications in countries where fair use is not a thing.
- userbinator 8y ago
  I suspect they're trying to give more reassurance to "personal, non-commercial use is allowed as long as you don't hit our servers too hard" --- which seems to be their stance on a lot of their other services too.

walrus01 8y ago

Once you have an OBJ file, you can 3D print it: https://all3dp.com/1/obj-file-format-3d-printing-cad/ https://all3dp.com/1/obj-file-format-3d-printing-cad/ If you want a 3D model of your office building or something.

TacticalTable 8y ago
Do you have a better guide to this? The model seems to be infinitely thin, which makes it unprintable in all the slicers I've tried.

pieterhg 8y ago

Love this, but I'm getting an error: "DeprecationWarning: Buffer() is deprecated due to security and usability issues" Opened an issue: https://github.com/retroplasma/earth-reverse-engineering/issues/3 https://github.com/retroplasma/earth-reverse-engineering/iss...

anc84 8y ago
That's not an error, just a warning. You can proceed using it just fine.

k9s9 8y ago

NASA's WorldWind is a similar project for those interested - https://github.com/NASAWorldWind/WebWorldWind https://github.com/NASAWorldWind/WebWorldWind docs - https://worldwind.arc.nasa.gov/web/ https://worldwind.arc.nasa.gov/web/