8 ms·

I think a better question is "how" is all of that data stored? One large Netapp? JBOD's all over the world? And how does that data make it into our browsers so

I think a better question is "how" is all of that data stored? One large Netapp? JBOD's all over the world? And how does that data make it into our browsers so quickly?

andrewstuart2 11y ago
If you're really interested, I highly recommend reading the SRE book Google has released. It's got a ton of insight into both their business practices and technical infrastructure. [1] http://www.amazon.com/Site-Reliability-Engineering-Production-Systems/dp/149192912X http://www.amazon.com/Site-Reliability-Engineering-Productio...
themartorana 11y ago
And how long until I have 3 PB of storage in my whatever replaces SSDs, and we all laugh at how at one time that was considered a lot of storage?
- agumonkey 11y ago
  Just when you'll have 1TB ram and 128-core CPUs.
  - valarauca1 11y ago
    So an Oracle T4 Ultra-SPARC server?
  - gkya 11y ago
    How would this be more useful than a machine w/ 4gigs of ram and 4 cores? Most of the time the hungriest process on a personal machine is the browser, and most of the human time it uses is wire(less) bound, so even if one had the machine you describe, he'd hardly notice.
    - qaq 11y ago
      Look at how many threads your browser is using and this at a time when very few sites are using web workers.
    - brianwawok 11y ago
      My browser alone uses more than 4GB
      - agumonkey 11y ago
        Not if you use dillo.
        coldtea 11y ago
        And how is that relevant, given that less than 0.1% use it, or wish to use it?
        agumonkey 11y ago
        Sorry, only kidding.
      - JoBrad 11y ago
        I'm not sure that's a bad problem, as long as your OS (and your browser) reallocate memory efficiently when required. There's no need to treat RAM like your hard drive, and keep a certain amount free all of the time. Also, your browser won't use 4GB of RAM if that's all you have.
    - agumonkey 11y ago
      I didn't say that. I just responded to the comment about having PB of data as the norm. You cannot handle such scale without adequate processing power and intermediate RAM. By the way, I'm writing this on a core 2 duo, which is my main computer. So trust having 4 cores already looks like overkill to me :)
      - gkya 11y ago
        Agreed. Actually I sport a Core 2 Duo too (Asus X51RL, "15.4 inches Widescreen Gateway to the World of Entertainment" https://www.asus.com/Notebooks/X51RL/ https://www.asus.com/Notebooks/X51RL/) :) 4/8 cores is useful if one does a lot of things with VMs, but above that it's mostly either a server or an auction.
        agumonkey 11y ago
        For desktop usage, excluding advanced media creation, the only thing a 2006 processor lacks is a decent GPU to delegate rendering stuff.
    - imtringued 11y ago
      Don't underestimate javascript bloat.
      - gkya 11y ago
        I don't underestimate javascript bloat, I blast it with a no-js browser (xombrero).
    - coldtea 11y ago
      >How would this be more useful than a machine w/ 4gigs of ram and 4 cores? Most of the time the hungriest process on a personal machine is the browser, and most of the human time it uses is wire(less) bound, so even if one had the machine you describe, he'd hardly notice. Speak for yourself. Anybody doing 3D, audio work (64 channels all loaded with 2-3 UAD plugins running native), and lots of other things will notice. Even consumers. Advanced 3D gaming of 2030, virtual reality interactive porn, etc. Even something as humble as local speech recognition and AI processing for real time assistant. The thing is whether we can make those cpus/tech -- not whether we can use their power when we make them.
      - gkya 11y ago
        > The thing is whether we can make those cpus/tech -- not whether we can use their power when we make them. We use up so many natural resources in such process that it's not worth it if it is not practically useful. > Speak for yourself. I really don't like this imperative style. You're responding to a stranger who has not said nothing offensive in his comment. > Anybody doing 3D, audio work (64 channels all loaded with 2-3 UAD plugins running native), and lots of other things will notice. I said "a personal machine" in my comment. A machine which is used to browse the web, take notes, watch movies, write documents, etc. 3D and high-performance audio is not every-day, personal computer use, it's a specialised case. > Advanced 3D gaming [...] virtual reality interactive porn [...] local speech recognition and AI processing [...] I recall saying useful.
        eropple 11y ago
        > I really don't like this imperative style. You're responding to a stranger who has not said nothing offensive in his comment. You're projecting your needs on those of a complete stranger. That's going to get some folks hot under the T-shirt. > I recall saying useful. You know what "advanced 3D gaming" also means? It means being able to broaden one's horizons via virtual tourism (and if you think I'm reaching, go use a Vive) without paying tens of thousands of dollars t go be there in person. You know what local speech recognition and AI processing means? It means being able to do all that cool shit Google does without exposing yourself to risk. These might not be things that are useful to you. They are useful to others. You're projecting, and you're kind of being a jerk about it, and that's why you got the response you did. Oh, and while we're at it: 3D unambiguously and without question is "every-day personal computer use," and higher-end audio would be if people actually had the hardware for it. Everyone plays video games now. You might blurf from your very, very high horse that that's not useful, but literally tens of millions of people derive value from that right now and that number's only going to go up. Please don't be That Guy.
    - TylerE 11y ago
      Already there are some games that are using upwards of 10gb of RAM (potentially). More and more games are shipping either as 64bit strongly preferred or even 64bit only.
- BillinghamJ 11y ago
  I'm not entirely sure that will happen. My anecdotal info is that I actually have less data stored locally than I did a couple of years ago. Most of my data is now held online and retrieved on demand.
  - dhimes 11y ago
    Do you think internet networks will be able to keep up?
  - deleted 11y ago
    [deleted]
  - ctrl-j 11y ago
    Most consumers won't be the drivers of this. However gaming, workstations, and enthusiast needs will cause enough demand for storage sizes to go up. Modern AAA games are over 30 GB installed. Applications in general are taking up more and more space locally. An hour of uncompressed 4k video is approximately 1.72 TB. Photos have also grown in size as resolution has gone up. I'm sure there are other industries as well. I know personally between my various hobbies, my SSDs are always full and I have several 4TB drives for storage.
    - greiskul 11y ago
      But for instance, I own several AAA games on my steam account, but I have only some installed at any given time. If I feel the urge to play one that I haven't in a long time, it doesn't take that much to download it again. It's even better that I don't need to worry about save files anymore, they are backed up automatically.
    - mortenlarsen 11y ago
      I am no expert, but I would think "an hour of uncompressed 4k video" would be more like 6TB. Maybe even more. >>> 4096*2304*3*60*3600/1e9 6115.295232 Resolution: 4096*2304 Colors: RGB (3 bytes) Frame rate: 60 Seconds: 3600 (one hour) Even using a lower frame rate like 25 would leave only two bytes for color before we get somewhere near 1.72 TB. >>> 4096*2304*2*25*3600/1e9 1698.69312 How did you estimate/calculate the 1.72TB?
- gnur 11y ago
  Tt will take quite a while, 1TB aught to be enough for everyone. Kidding aside, you can assume that in 10 years storage will be about 1000 times cheaper then it is now, so 3PB of storage would be around the cost of 3TB drive today, around 100$.
  - eterm 11y ago
    How do you figure the 1000 times cheaper? Data storage today certainly isn't 1000 times cheaper than it was a decade ago, I'm not sure it's even 100 times cheaper. For some reason you seem to believe that storage prices will remain the same while we double capacity every 12 months, that isn't even close to reality. (Even if you've misapplied Moore's Law to flash storage, that predicts doubling every 2 years not every year.)
    - gnur 11y ago
      I've looked at the prices between 1980 and 2010, seemed to be getting about 1000 times cheaper every 10 years. And SSD pricing has gone down about 40% every year the last 4 years. Which would come to a price reduction of less, but some breakthrough is bound to be around the corner.
    - toomuchtodo 11y ago
      http://www.mkomo.com/cost-per-gigabyte-update http://www.mkomo.com/cost-per-gigabyte-update
      - coldtea 11y ago
        Don't hold your breath. See the "flat lining" after 2010?
    - JoeAltmaier 11y ago
      According to http://www.statisticbrain.com/average-cost-of-hard-drive-storage/ http://www.statisticbrain.com/average-cost-of-hard-drive-sto... its about 40:1
- coldtea 11y ago
  We've already hit a snag with Moore's law. And storage is not progressing that fast either, for quite longer than that. The thing is, we've lived through the "low hanging fruit" era of the progress curve in IT, and now we're in the diminishing returns stage (increasingly more difficult effort -- e.g. cpu smaller process size issues -- to get ever less interesting improvements). Of course anybody having lived the best and rapidly increasing part of the curve (roughly 1950 to 2005 or so) would think this will go on forever, or that it's some "inevitable" "natural" characteristic of technology to always improve rapidly (which mostly means they're either too optimistic or bad at math). For some areas of tech it's even worse: not only we've reached increasing difficulty in creating machines, factories etc to churn it out, but we've also reached the limits of physical laws involved. So we either need to find totally different processes, or prove Einstein and/or Heisenberg wrong to overcome this (and sure, the first if easier than the latter, but still nowhere as easy as simply creating a new generation of the same process).
  - andrewstuart2 11y ago
    There's also the fundamental limit to the bandwidth that a given consumer can realistically ingest. The richest inputs we have -- our sight -- has been estimated [1] at about 8.75 Mbps. The real driver for bandwidth (and thus local storage) these days is input latency. We're only just beginning to be able to react quickly enough to requests for different input (VR), so the historical workaround has been increasing the amount of data instantaneously available through things like HDTV. You're never fully consuming the bandwidth provided at any instant but you can increase your awareness by constantly looking at different parts of the screen. So again we're fundamentally limited by the speed of light and how quickly data can be retrieved. If we can get better at local caching and query prediction (streaming in data right before it's requested), then we could theoretically reduce massively the amount of site-local storage required. So for supply/demand reasons, I think software combined with HID improvements (better prediction through better sensors) has a significant potential to essentially put the storage industry into maintenance mode. [1] https://www.newscientist.com/article/dn9633-calculating-the-speed-of-sight https://www.newscientist.com/article/dn9633-calculating-the-...
  - smitherfield 11y ago
Joyfield 11y ago
They have data centers around the world. https://www.google.com/about/datacenters/inside/locations/index.html https://www.google.com/about/datacenters/inside/locations/in...
mike_hearn 11y ago
Google developed technologies like BigTable for a reason, you know! You can serve data straight out of a database like BigTable as long as it's done carefully (to keep latency low). And the dataset can be easily replicated around the world in different datacenters to reduce round trip times. I used to work on Google Earth. It's a really cool project. The serving infrastructure is pretty simple though. It's basically just a simple frontend server in front of BigTables (or was). The real magic happens in the data processing pipeline that produces the datasets, and in the client side rendering code. It's not like web search where the serving infrastructure is hugely complex and very much secret sauce. The thing that impresses me the most in recent times about Earth is how the Maps team were able to write an Earth client in Javascript that starts way faster than the desktop client does. The traditional client is a large C++ app that has always struggled with startup time, I think due to a combination of code size and having lots of static constructors that slow down the critical path. And the bulk of the app was written when Keyhole was an independent company selling into government and industry, so startup time didn't matter much. Also, the image processing pipeline is really amazing, because it blends together lots of disparate imagery sets from many different providers and cameras, with different lighting and colour calibrations, whilst still producing something usable. In the early days of Earth you could easily see the seams where different types of images had been stitched together. And if you zoomed you could clearly see the thin strips from the satellite passes. These days you still can see them if you look around, but there is a lot of very clever colour balancing and reblending being done to make the whole thing a lot more seamless and beautiful.
- kuschku 11y ago
  EDIT: Really? Already at -2 after 15 minutes? This is directly on-topic (it directly criticizes the quality of the process that was praised before), it is directly relevant (it provides examples for a counterargument), it is not a personal attack or anything, and it provides sources. Before you downvote, please comment – that’s a lot more helpful to discussion ________________________ > In the early days of Earth you could easily see the seams where different types of images had been stitched together. And if you zoomed you could clearly see the thin strips from the satellite passes. These days you still can see them if you look around, but there is a lot of very clever colour balancing and reblending being done to make the whole thing a lot more seamless and beautiful. Well, I don’t really have to look around: http://i.imgur.com/fhMPHVm.png http://i.imgur.com/fhMPHVm.png Wherever I look around my home town, it’s blurry, stitched together, with bad color balancing. Try taking a look at the areas where it touches the open sea, it’s even worse. In addition to the imagery taking half an hour to load even anything that’s not totally blurry (which can’t be my connection as all competitors are instant), and the missing 3D in a city of 300k, the imagery is (a) not updated since 2005, (b) has constant edges and stripes everywhere, (c) is – due to not being updated – missing several districts already. I’ve complained about this on several occasions before to Google employees who promised to fix it, for example here [1]. I have to be honest, I’m more than disappointed with Google Maps and Earth. Even here maps imagery (linked below in comparison) is more up to date, has higher resolution, and is blended a lot better. Google, as always, seems to only (or mostly) care about the US. There, even most small towns have perfectly clear 3D imagery from the current year. It’s just another punch in the face in a long row of "US-only!". The Internet. Connecting the ~~world~~ US. ________________________ [1] https://www.reddit.com/r/legaladvice/comments/3rer0e/google_maps_directing_folks_on_to_our_private/cwog5ow?context=2 https://www.reddit.com/r/legaladvice/comments/3rer0e/google_...
buckhx 11y ago
I can speak for how Bing Maps was storing their data before being bought by Uber. They had a single massive bespoke compute site for batch processing. I want to say it was 64k cores and multiple PBs. There was a flood in Colorado one summer and it knocked the imagery site out for a day or two. Uber purchased that site along w/ ~100 engineers. I had left about 6 months before that purchase happened.
gtrubetskoy 11y ago
Consider for comparison that gmail alone probably uses close to 100,000PB (based on my back of the napkin math and some reference I found that there are ~900M users).

I think a better question is "how" is all of that data stored? One large Netapp? JBOD's all over the world? And how does that data make it into our browsers so quickly?

andrewstuart2 11y ago

If you're really interested, I highly recommend reading the SRE book Google has released. It's got a ton of insight into both their business practices and technical infrastructure. [1] http://www.amazon.com/Site-Reliability-Engineering-Production-Systems/dp/149192912X http://www.amazon.com/Site-Reliability-Engineering-Productio...

themartorana 11y ago

And how long until I have 3 PB of storage in my whatever replaces SSDs, and we all laugh at how at one time that was considered a lot of storage?

agumonkey 11y ago
Just when you'll have 1TB ram and 128-core CPUs.
- valarauca1 11y ago
  So an Oracle T4 Ultra-SPARC server?
- gkya 11y ago
  How would this be more useful than a machine w/ 4gigs of ram and 4 cores? Most of the time the hungriest process on a personal machine is the browser, and most of the human time it uses is wire(less) bound, so even if one had the machine you describe, he'd hardly notice.
  - qaq 11y ago
    Look at how many threads your browser is using and this at a time when very few sites are using web workers.
  - brianwawok 11y ago
    My browser alone uses more than 4GB
    - agumonkey 11y ago
      Not if you use dillo.
      - coldtea 11y ago
        And how is that relevant, given that less than 0.1% use it, or wish to use it?
        agumonkey 11y ago
        Sorry, only kidding.
    - JoBrad 11y ago
      I'm not sure that's a bad problem, as long as your OS (and your browser) reallocate memory efficiently when required. There's no need to treat RAM like your hard drive, and keep a certain amount free all of the time. Also, your browser won't use 4GB of RAM if that's all you have.
  - agumonkey 11y ago
    I didn't say that. I just responded to the comment about having PB of data as the norm. You cannot handle such scale without adequate processing power and intermediate RAM. By the way, I'm writing this on a core 2 duo, which is my main computer. So trust having 4 cores already looks like overkill to me :)
    - gkya 11y ago
      Agreed. Actually I sport a Core 2 Duo too (Asus X51RL, "15.4 inches Widescreen Gateway to the World of Entertainment" https://www.asus.com/Notebooks/X51RL/ https://www.asus.com/Notebooks/X51RL/) :) 4/8 cores is useful if one does a lot of things with VMs, but above that it's mostly either a server or an auction.
      - agumonkey 11y ago
        For desktop usage, excluding advanced media creation, the only thing a 2006 processor lacks is a decent GPU to delegate rendering stuff.
  - imtringued 11y ago
    Don't underestimate javascript bloat.
    - gkya 11y ago
      I don't underestimate javascript bloat, I blast it with a no-js browser (xombrero).
  - coldtea 11y ago
    >How would this be more useful than a machine w/ 4gigs of ram and 4 cores? Most of the time the hungriest process on a personal machine is the browser, and most of the human time it uses is wire(less) bound, so even if one had the machine you describe, he'd hardly notice. Speak for yourself. Anybody doing 3D, audio work (64 channels all loaded with 2-3 UAD plugins running native), and lots of other things will notice. Even consumers. Advanced 3D gaming of 2030, virtual reality interactive porn, etc. Even something as humble as local speech recognition and AI processing for real time assistant. The thing is whether we can make those cpus/tech -- not whether we can use their power when we make them.
    - gkya 11y ago
      > The thing is whether we can make those cpus/tech -- not whether we can use their power when we make them. We use up so many natural resources in such process that it's not worth it if it is not practically useful. > Speak for yourself. I really don't like this imperative style. You're responding to a stranger who has not said nothing offensive in his comment. > Anybody doing 3D, audio work (64 channels all loaded with 2-3 UAD plugins running native), and lots of other things will notice. I said "a personal machine" in my comment. A machine which is used to browse the web, take notes, watch movies, write documents, etc. 3D and high-performance audio is not every-day, personal computer use, it's a specialised case. > Advanced 3D gaming [...] virtual reality interactive porn [...] local speech recognition and AI processing [...] I recall saying useful.
      - eropple 11y ago
        > I really don't like this imperative style. You're responding to a stranger who has not said nothing offensive in his comment. You're projecting your needs on those of a complete stranger. That's going to get some folks hot under the T-shirt. > I recall saying useful. You know what "advanced 3D gaming" also means? It means being able to broaden one's horizons via virtual tourism (and if you think I'm reaching, go use a Vive) without paying tens of thousands of dollars t go be there in person. You know what local speech recognition and AI processing means? It means being able to do all that cool shit Google does without exposing yourself to risk. These might not be things that are useful to you. They are useful to others. You're projecting, and you're kind of being a jerk about it, and that's why you got the response you did. Oh, and while we're at it: 3D unambiguously and without question is "every-day personal computer use," and higher-end audio would be if people actually had the hardware for it. Everyone plays video games now. You might blurf from your very, very high horse that that's not useful, but literally tens of millions of people derive value from that right now and that number's only going to go up. Please don't be That Guy.
  - TylerE 11y ago
    Already there are some games that are using upwards of 10gb of RAM (potentially). More and more games are shipping either as 64bit strongly preferred or even 64bit only.
BillinghamJ 11y ago
I'm not entirely sure that will happen. My anecdotal info is that I actually have less data stored locally than I did a couple of years ago. Most of my data is now held online and retrieved on demand.
- dhimes 11y ago
  Do you think internet networks will be able to keep up?
- deleted 11y ago
  [deleted]
- ctrl-j 11y ago
  Most consumers won't be the drivers of this. However gaming, workstations, and enthusiast needs will cause enough demand for storage sizes to go up. Modern AAA games are over 30 GB installed. Applications in general are taking up more and more space locally. An hour of uncompressed 4k video is approximately 1.72 TB. Photos have also grown in size as resolution has gone up. I'm sure there are other industries as well. I know personally between my various hobbies, my SSDs are always full and I have several 4TB drives for storage.
  - greiskul 11y ago
    But for instance, I own several AAA games on my steam account, but I have only some installed at any given time. If I feel the urge to play one that I haven't in a long time, it doesn't take that much to download it again. It's even better that I don't need to worry about save files anymore, they are backed up automatically.
  - mortenlarsen 11y ago
    I am no expert, but I would think "an hour of uncompressed 4k video" would be more like 6TB. Maybe even more. >>> 4096*2304*3*60*3600/1e9 6115.295232 Resolution: 4096*2304 Colors: RGB (3 bytes) Frame rate: 60 Seconds: 3600 (one hour) Even using a lower frame rate like 25 would leave only two bytes for color before we get somewhere near 1.72 TB. >>> 4096*2304*2*25*3600/1e9 1698.69312 How did you estimate/calculate the 1.72TB?
gnur 11y ago
Tt will take quite a while, 1TB aught to be enough for everyone. Kidding aside, you can assume that in 10 years storage will be about 1000 times cheaper then it is now, so 3PB of storage would be around the cost of 3TB drive today, around 100$.
- eterm 11y ago
  How do you figure the 1000 times cheaper? Data storage today certainly isn't 1000 times cheaper than it was a decade ago, I'm not sure it's even 100 times cheaper. For some reason you seem to believe that storage prices will remain the same while we double capacity every 12 months, that isn't even close to reality. (Even if you've misapplied Moore's Law to flash storage, that predicts doubling every 2 years not every year.)
  - gnur 11y ago
    I've looked at the prices between 1980 and 2010, seemed to be getting about 1000 times cheaper every 10 years. And SSD pricing has gone down about 40% every year the last 4 years. Which would come to a price reduction of less, but some breakthrough is bound to be around the corner.
  - toomuchtodo 11y ago
    http://www.mkomo.com/cost-per-gigabyte-update http://www.mkomo.com/cost-per-gigabyte-update
    - coldtea 11y ago
      Don't hold your breath. See the "flat lining" after 2010?
  - JoeAltmaier 11y ago
    According to http://www.statisticbrain.com/average-cost-of-hard-drive-storage/ http://www.statisticbrain.com/average-cost-of-hard-drive-sto... its about 40:1
coldtea 11y ago
We've already hit a snag with Moore's law. And storage is not progressing that fast either, for quite longer than that. The thing is, we've lived through the "low hanging fruit" era of the progress curve in IT, and now we're in the diminishing returns stage (increasingly more difficult effort -- e.g. cpu smaller process size issues -- to get ever less interesting improvements). Of course anybody having lived the best and rapidly increasing part of the curve (roughly 1950 to 2005 or so) would think this will go on forever, or that it's some "inevitable" "natural" characteristic of technology to always improve rapidly (which mostly means they're either too optimistic or bad at math). For some areas of tech it's even worse: not only we've reached increasing difficulty in creating machines, factories etc to churn it out, but we've also reached the limits of physical laws involved. So we either need to find totally different processes, or prove Einstein and/or Heisenberg wrong to overcome this (and sure, the first if easier than the latter, but still nowhere as easy as simply creating a new generation of the same process).
- andrewstuart2 11y ago
  There's also the fundamental limit to the bandwidth that a given consumer can realistically ingest. The richest inputs we have -- our sight -- has been estimated [1] at about 8.75 Mbps. The real driver for bandwidth (and thus local storage) these days is input latency. We're only just beginning to be able to react quickly enough to requests for different input (VR), so the historical workaround has been increasing the amount of data instantaneously available through things like HDTV. You're never fully consuming the bandwidth provided at any instant but you can increase your awareness by constantly looking at different parts of the screen. So again we're fundamentally limited by the speed of light and how quickly data can be retrieved. If we can get better at local caching and query prediction (streaming in data right before it's requested), then we could theoretically reduce massively the amount of site-local storage required. So for supply/demand reasons, I think software combined with HID improvements (better prediction through better sensors) has a significant potential to essentially put the storage industry into maintenance mode. [1] https://www.newscientist.com/article/dn9633-calculating-the-speed-of-sight https://www.newscientist.com/article/dn9633-calculating-the-...
- smitherfield 11y ago

Joyfield 11y ago

They have data centers around the world. https://www.google.com/about/datacenters/inside/locations/index.html https://www.google.com/about/datacenters/inside/locations/in...

mike_hearn 11y ago

Google developed technologies like BigTable for a reason, you know! You can serve data straight out of a database like BigTable as long as it's done carefully (to keep latency low). And the dataset can be easily replicated around the world in different datacenters to reduce round trip times. I used to work on Google Earth. It's a really cool project. The serving infrastructure is pretty simple though. It's basically just a simple frontend server in front of BigTables (or was). The real magic happens in the data processing pipeline that produces the datasets, and in the client side rendering code. It's not like web search where the serving infrastructure is hugely complex and very much secret sauce. The thing that impresses me the most in recent times about Earth is how the Maps team were able to write an Earth client in Javascript that starts way faster than the desktop client does. The traditional client is a large C++ app that has always struggled with startup time, I think due to a combination of code size and having lots of static constructors that slow down the critical path. And the bulk of the app was written when Keyhole was an independent company selling into government and industry, so startup time didn't matter much. Also, the image processing pipeline is really amazing, because it blends together lots of disparate imagery sets from many different providers and cameras, with different lighting and colour calibrations, whilst still producing something usable. In the early days of Earth you could easily see the seams where different types of images had been stitched together. And if you zoomed you could clearly see the thin strips from the satellite passes. These days you still can see them if you look around, but there is a lot of very clever colour balancing and reblending being done to make the whole thing a lot more seamless and beautiful.

kuschku 11y ago
EDIT: Really? Already at -2 after 15 minutes? This is directly on-topic (it directly criticizes the quality of the process that was praised before), it is directly relevant (it provides examples for a counterargument), it is not a personal attack or anything, and it provides sources. Before you downvote, please comment – that’s a lot more helpful to discussion ________________________ > In the early days of Earth you could easily see the seams where different types of images had been stitched together. And if you zoomed you could clearly see the thin strips from the satellite passes. These days you still can see them if you look around, but there is a lot of very clever colour balancing and reblending being done to make the whole thing a lot more seamless and beautiful. Well, I don’t really have to look around: http://i.imgur.com/fhMPHVm.png http://i.imgur.com/fhMPHVm.png Wherever I look around my home town, it’s blurry, stitched together, with bad color balancing. Try taking a look at the areas where it touches the open sea, it’s even worse. In addition to the imagery taking half an hour to load even anything that’s not totally blurry (which can’t be my connection as all competitors are instant), and the missing 3D in a city of 300k, the imagery is (a) not updated since 2005, (b) has constant edges and stripes everywhere, (c) is – due to not being updated – missing several districts already. I’ve complained about this on several occasions before to Google employees who promised to fix it, for example here [1]. I have to be honest, I’m more than disappointed with Google Maps and Earth. Even here maps imagery (linked below in comparison) is more up to date, has higher resolution, and is blended a lot better. Google, as always, seems to only (or mostly) care about the US. There, even most small towns have perfectly clear 3D imagery from the current year. It’s just another punch in the face in a long row of "US-only!". The Internet. Connecting the ~~world~~ US. ________________________ [1] https://www.reddit.com/r/legaladvice/comments/3rer0e/google_maps_directing_folks_on_to_our_private/cwog5ow?context=2 https://www.reddit.com/r/legaladvice/comments/3rer0e/google_...

buckhx 11y ago

I can speak for how Bing Maps was storing their data before being bought by Uber. They had a single massive bespoke compute site for batch processing. I want to say it was 64k cores and multiple PBs. There was a flood in Colorado one summer and it knocked the imagery site out for a day or two. Uber purchased that site along w/ ~100 engineers. I had left about 6 months before that purchase happened.

gtrubetskoy 11y ago

Consider for comparison that gmail alone probably uses close to 100,000PB (based on my back of the napkin math and some reference I found that there are ~900M users).