11 ms·
How big is the Google Earth database?
- TimGremalm 11y ago"So our final estimate for the total size of the Google Earth database is 3,017 TB or approximately 3 Petabytes!"
- comboy 11y agoWhy do they have to estimate it? Aren't they able to check exact size of their database(s)?
- giarc 11y agoThe author does not work for Google. >Timothy has been using Google Earth since 2004 when it was still called Keyhole before it was renamed Google Earth in 2005 and has been a huge fan ever since. He is a programmer working for Red Wing Aerobatx and lives in Cape Town, South Africa.
- kyberias 11y agoI'm sorry to say I laughed out loud after reading your comment, but I actually wondered about the same thing while reading the article. :) You have to admit it would be hilarious if they didn't know the size of the database or wouldn't be able to calculate it easily.
- Thaxll 11y agoIt's not the case here but it's actually possible that in some cases it's just an estimate ( MYSQL InnoDB ).
- stuff4ben 11y agoI think a better question is "how" is all of that data stored? One large Netapp? JBOD's all over the world? And how does that data make it into our browsers so quickly?
- andrewstuart2 11y agoIf you're really interested, I highly recommend reading the SRE book Google has released. It's got a ton of insight into both their business practices and technical infrastructure. [1] http://www.amazon.com/Site-Reliability-Engineering-Production-Systems/dp/149192912X http://www.amazon.com/Site-Reliability-Engineering-Productio...
- themartorana 11y agoAnd how long until I have 3 PB of storage in my whatever replaces SSDs, and we all laugh at how at one time that was considered a lot of storage?
- agumonkey 11y agoJust when you'll have 1TB ram and 128-core CPUs.
- valarauca1 11y agoSo an Oracle T4 Ultra-SPARC server?
- gkya 11y agoHow would this be more useful than a machine w/ 4gigs of ram and 4 cores? Most of the time the hungriest process on a personal machine is the browser, and most of the human time it uses is wire(less) bound, so even if one had the machine you describe, he'd hardly notice.
- qaq 11y agoLook at how many threads your browser is using and this at a time when very few sites are using web workers.
- nxzero 11y agoHuge database aside, I would be more curious to know what is the most valuable data/metadata in the database. Anyone have any thoughts?
- jws 11y agoThe list of areas they are required to obfuscate?
- mike_hearn 11y agoValuable as in cost? Aerial imagery in general is very expensive, especially from satellites. That's why the Earth/Maps team don't have an open protocol (or one reason). They aren't allowed to just give the imagery away for free: it's licensed specifically for that application. If you try and download the entire dataset there are systems in place that will fight you, to defend the rights of the imagery companies.
- ufmace 11y agoI would tend to think it's the fine Maps detail on every road in every country that Navigation works in - all of the details of every single intersection, how many lanes, what turns you can make from where, etc. The maps and images are expensive, but they already exist and can be bought, if you're willing to pay the price. That detail info doesn't even exist anywhere, at least not in any kind of consistent and reliable way. The only way to get it is to employ a virtual army of drivers to drive every single road in every country, record a ton of info about them all, and figure out how to store it all in a searchable format. And then keep doing it forever, because those roads all change periodically, and often nobody every documents how or where or when.
- ant6n 11y agoI find it odd that they have to estimate this.
- wrg 11y ago>1024 TB So... 1 PB?
- elbigbad 11y agoThere are actually an infinite number of numbers greater than 1024. For the sake of brevity, I'll refrain from listing them though.
- johnjreiser 11y agoFor some interesting historical perspective, here's what Microsoft and partners had to come up with in 1998 to support Terraserver and it's 3TB of imagery. http://research.microsoft.com/pubs/68552/msr_tr_98_17_terraserver.pdf http://research.microsoft.com/pubs/68552/msr_tr_98_17_terras...
- mmanfrin 11y agoNeat that mapping Terra required them to build a computer capable of storing terabytes.
- andylynchnz 11y agoNot a coincidence - it was built to demonstrate the scalability of their technology to Terabyte datasets.
- mmanfrin 11y agoThe 'terra' I was referring to was earth :]
- sievebrain 11y agoThey seem to have forgotten that the imagery database is calculated at multiple zoom levels. If a country has high rest imagery for the max zoom, it also has scaled down versions of that imagery for all the higher altitude image sets. So that would expand things quite a bit. There are also height maps, which were not counted, and metadata.
- brianwawok 11y agoWould it? If full detail zoom took 1 MB... that same area zommed out 2x would take 256KB.. zoomed out by 2 is 64 KB... zoomed out by 2x is 16 KB... Don't know all the zoom levels, but for a rough guess the zoomed in level should be pretty close to the total (within 30% or so).
- Bjartr 11y agoDepends on if it's precalculated or dynamically generated from the full res and then cached.
- Retric 11y agoClients are clearly doing some scaling. So worst case is clearly less than 35% overhead for multiple zoom levels. PS: Sum (0 to N) of (1/4)^N = 1.33333...
- brianwawok 11y agoThanks for adding the maths I did the lazy guess ;)
- JoBrad 11y agoHe talks about the height of the camera, and loading additional images, in the first post of the series.
- hackbinary 11y agoAs interesting as this is, the author should make it clearer on his main landing page that his site is not affiliated with Google. http://www.gearthblog.com/about http://www.gearthblog.com/about > This blog is not officially affiliated with Google. Google Earth Blog is dedicated to sharing the best news, interesting sights, technology, and happenings for Google Earth.
- kzrdude 11y agoIt's really not needed. This comment is entirely my opinion. Which is also redundant to say.
- IshKebab 11y agoNow estimate streetview!
- chris_va 11y agoNow multiply by the number of serving data centers, and disk replication.
- Bedon292 11y agoThis actually seems very low to me. The archive at DigitalGlobe is over 80PB of data [1], and was growing at over 2PB a year [2] before the latest satellite went up. There are more bands in the imagery than RGB, so it is larger because of that. But somewhere Google is storing the original resolution data. Aerial imagery is at an even higher resolution, so even larger per square kilometer. They also are holding different resolution data for different zoom levels. They do not resample aerial imagery past certain zoom levels, they use a different source. At approximately 3225x3225 pixels per square kilometer (31cm pixels in Worldview 3), that is approximately a 10 megapixel tiff, per square kilometer. Which I believe is roughly 35MB. 496Million square kilometers of land on earth not including antartica. Brings this over 17PB of raw data. And that is not including any historical data either. Before starting caching the different zoom levels of png's that are served out. I know not all of the world is covered at that high a resolution, but there are spots with even higher resolution to make up for it. Plus historical data. And plenty of water included in the images not counted either. I highly doubt they are holding anything less than 30PB. [1] https://www.digitalglobe.com/platforms/gbdx https://www.digitalglobe.com/platforms/gbdx [2] http://www.networkcomputing.com/big-data/how-digitalglobe-handles-2-petabytes-satellite-data-yearly/767935503 http://www.networkcomputing.com/big-data/how-digitalglobe-ha...
- kbenson 11y ago> 10 megapixel tiff, per square kilometer. Which I believe is roughly 35MB. I'm sure they are using a storage medium (at a second level of compression) that provides better space efficiency than that. TIFF supports both lossy and lossless compression through extensions, but for storing source data that is used relatively infrequently (relative being the key term here), it might make sense to use a storage compression geared very heavily for space efficiency. It's not like Google doesn't have the computational power or ability to process in parallel efficiently.
- DonHopkins 11y agoI wonder if they use wavelet compression for all those aerial photos of ocean water.
- 11y ago
- erroneousfunk 11y agoMinor point: Can people stop using 3D pie charts? The "satellite imagery" slice in the chart they show is slightly less than a third of the "historical imagery (satellite)" slice, but it looks like it's about half because of how the chart is rotated. "3D imagery" is supposed to be slightly larger than the "Historical imagery (aerial)" slice, but instead, it looks smaller! There's just no reason to use this type of visualization.
- peeters 11y agoWow, I had no idea how creepily awesome Google's 3D imagery had gotten. It used to only be skyscrapers in main cities. Now I can basically see in my own window.
- JoBrad 11y agoI get part of the "creep" factor, but anyone walking down the street has a much higher resolution image of your house - and with real-time "video", to boot! Maybe people just don't think about that very often?
- shubb 11y agoThis is the imaginary database but Google knows a lot more than that. There are also vectors for the roads, with a link between that data and streetview. You can click on the map and google will tell you that you're at 24 Burgess Street, and show you a photo, so it has a link between all that information. I suspect they also fuse this with information from phones. Do the phones contribute to live traffic stats? Or the information about when a shop is busiest (it will show you a bar chart of when a gym or store is busy). Is that done based on knowing when android users are at that location? What about all the Wifi access points it knows the location of? In some ways, this is much harder to deal with than the image data because although it's smaller in size, it's denser in information and the links between are more complex.
- rdtsc 11y agoI understand that compared to raster images vector data for roads is negligible. A dinky Garmin car nav system from 2011 I had held continental US road system data in a few gigs only.
- Doctor_Fegg 11y agoOpenStreetMap's vector data for roads (and buildings, landuse, etc.) for the whole world is currently 31GB, in OSM protobuf format: http://planet.osm.org http://planet.osm.org
- duaneb 11y ago> There are also vectors for the roads, with a link between that data and streetview. You can click on the map and google will tell you that you're at 24 Burgess Street, and show you a photo, so it has a link between all that information. I mean, you can now download your local area's data to your phone from google maps. I suspect this size is negligible compared to the raw 2d/3d/historical data. It would have been nice to include it in their analysis, though.
- SEJeff 11y agoGoogle's purchase of Waze is one of the main ways they get realtime traffic data: http://www.businessinsider.com/how-google-bought-waze-the-inside-story-2015-8 http://www.businessinsider.com/how-google-bought-waze-the-in... For serious drivers, there simply isn't anything on the market (and free!) better than Waze.
- zhte415 11y agoOur AI has developed a propensity for exploring and documenting regions focused on cheese production. We expect a report back in around 4000 years. Depending on results, we may reset you.
- eva1984 11y agoThose are some number flying... But I have to say they dont surprise me that much, by the sheer amount of data. The meat is in efficiently retrieve them, ie index.