5 ms·
Google: 'At scale, everything breaks'
- herf 15y agoAm a little bit surprised Google hasn't incorporated SSD more.
- arkitaip 15y agoToo expensive, too prone to failure http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid-state-drive-scale.html http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid...
- wmf 15y agoSSDs are actually cheaper per IOPS. Considering that everything we knew about hard disk reliability was shown to be wrong (by Google), I wouldn't be surprised if the same holds for SSDs.
- sliverstorm 15y agoIt's hard to see SSD's being a clear win on the surface. It would be very interesting though to see if the cost begins to change when you consider power savings (much smaller than hoped, but they do exist), figure out how much they'd save on HVAC in the datacenters, etc.
- wmf 15y agoHaving worked on power-aware hybrid storage for the last two years, the power savings per GB is pretty much zero; there is significant W/IOPS savings but that still doesn't pay for the capital cost. Performance is really the reason to use SSDs.
- sliverstorm 15y agoper GB is an excellent point, I had not considered that.
- chrisbolt 15y agoHaving installed almost 200 Intel SSDs into servers with 0 failures, I doubt his anecdotal evidence. Also compare with Intel's reported annual failure rates: http://www.anandtech.com/show/4244/intel-ssd-320-review http://www.anandtech.com/show/4244/intel-ssd-320-review and a french retailer: http://www.anandtech.com/show/4202/the-intel-ssd-510-review/3 http://www.anandtech.com/show/4202/the-intel-ssd-510-review/...
- rxin 15y agoFor lots of applications, the workloads are either memory only or large sequential IOs. SSDs matter less in both situations.
- clarkbox 15y agothink commodity... have ssd's reached the commodity level yet? from a commodity stand point, i think it would be far more likely that google would use an array of SD cards per node. SSDs are really just an array of SD cards anyway, with a pile of hype and marketing piled on. you would still get most benefits of an SSD. power usage (esp at idle) might even be less than an ssd. google could develop their own wear leveling algorithms, and the rest of the stuff that an SSD controller provides for the internal flash. replacement costs could be less as well over time.
- ajtaylor 15y agoI was at a linux user group meeting recently where a talk was given by a Google sysadmin where he talked about hard drive reliability. Someone asked him about SSDs to which he replied that he couldn't talk about it. My take is that they are definitely trying out SSDs, but either 1) found SSDs provide a huge competitive benefit and don't want to publicly share that knowledge or 2) they simply don't have enough data yet. I'm leaning towards #1.
- Locke1689 15y agoWhere did you get the impression that Google hasn't incorporated SSDs?
- DanielRibeiro 15y agoIt is always a pleasure to read from Urs Hölzle. I've been in touch with some minor spec of his work[1], but I'm always impressed by it. [1] http://research.google.com/pubs/author79.html http://research.google.com/pubs/author79.html
- delinquentme 15y ago"We use tapes, still, in this age because they're actually a very cost-effective way as a last resort for Gmail. " .. what kind of tape drives does Google use?
- dholowiski 15y agoBFT... big freaking tapes. I wonder if it's really tape though. You can get 'tape' backup systems where the cartrige looks just like a tape, but it's actually a special hard drive. I wouldn't be surprised if that's it. Otherwise, I wonder how many miles (of tape) long my gmail inbox is?
- wmf 15y agoGoogle wouldn't buy a VTL when they already have GFS. When they say tape, they must mean tape.
- btilly 15y agoUrs wouldn't have said tape if he didn't mean tape. This brings to mind an amusing anecdote. I once saw a conversation about how many miles of tape are storing Viagra ads. Someone quipped, "The only thing that I know for certain is that the trend is up."
- zinkem 15y agoAccording to one source I found[1], 60 meters of DAT tape holds 1300mb of data. That's 21.67mb per meter. My gmail inbox is 679mb, thats about 31 meters of DAT tape. [1] http://menehune.opt.wfu.edu/Kokua/SGI/007-2861-005/sgi_html/ch04.html#id5197689 http://menehune.opt.wfu.edu/Kokua/SGI/007-2861-005/sgi_html/...
- whatusername 15y agoLTO5 hit's about 1.8GB per metre. Or about 3.6GB compressed.[1] There's a reason that people say tape is still cost effective -- and it's not DAT. [1] http://en.wikipedia.org/wiki/Linear_Tape-Open http://en.wikipedia.org/wiki/Linear_Tape-Open
- clarkbox 15y ago"...most applications don't use [Google File System (GFS)] today. In fact, we're phasing out GFS in favour of the next-generation file system that is very similar, but it's not GFS anymore." what could this be? home grown? an open source project?
- deleted 15y ago[deleted]
- haberman 15y agoColossus: see a slide about halfway through this presentation: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/university/relations/facultysummit2010/storage_architecture_and_challenges.pdf http://static.googleusercontent.com/external_content/untrust...
- veb 15y agoAll the best (and fun) scaling issues is best solved by hacking. Is there really any proper decent methods out there that can actually take a company from 5 users a day to 5 billion without any problems at all? I'm aware that you could perhaps "foresee" it, and get a big data-center, and lots of hardware solutions -- but is this really the solution that's called an 'out of the box' solution? Sounds like when Google gets a problem, they create something to fix that, which is pretty damn cool in my opinion.
- zaphar 15y agoAs a google employee I can say it really is "pretty damn cool".
- mv1 15y agoI love this comment: "The reason why we put it in is not physical data loss, but once in a blue moon you will have a bug that destroys all copies of the online data and your only protection is to have something that is not connected to the same software system." I think that is often overlooked when designing HA storage systems.