6 ms·
Computing at scale, or, how Google has warped my brain
- retube 16y agoWhen I was at CERN 10 years ago, the set up was similar. Instead of Macs the desktops were PCs. The code was Fortran, Perl and c++. All work was on remote Linux boxes (typically exporting a desktop over X), although physical not virtual OS instances. You'd ssh to a gateway IP and be logged into one of the hosts. All data and user accounts were stored on a distributed filesystem AFS. There was job scheduling and management middleware that'd batch process your analysis across hundreds of hosts in the datacentre. "The cloud" has been around for a while, although implementation has changed.
- irahul 16y agoYahoo uses a similar setup. Instead of virtual hosts, the hosts here are physical machines. Search team had a dedicated lab(not sure what's happening now with search migration) and all developers would ssh to their assigned lab machines, fire up their vim/emacs and crank code. Search team has a mixture of windows laptops, macs, linux desktops, linux laptops..and it didn't matter because everyone just ssh'd to a RHEL box and worked. Yahoo used to be a BSD shop, then moved to linux. Search team used both Debian and RHEL in the beginning, then transitioned to RHEL fulltime. I don't understand the obsession with RHEL. In my experience, Debian/Ubuntu will give the same benefits as RHEL sans the cost. The rationale for moving from BSD to linux was practical. The gist was most of the vendors either do not do bsd or bsd was second class citizen. And BSD wasn't as quick as linux in catching up with esoteric hardware. But discarding Debian didn't actually make sense to me.
- retube 16y agoCERN used RH too, 6 when I was there. I believe they've now switched to Scientific Linux. Actually I don't understand why Google would use virtualisation. Why would they split a physical box up? Making many hosts look like one, yes (is such a thing possible?), but not the other way around. And AFS was cool: it was actually a global filesystem that many institutions used. you could cd /, ls and browse to nasa.gov amongst many thousands of others (much was protected!) (cern was mounted on /cern.ch)
- mechanical_fish 16y agoI don't understand why Google would use virtualisation. Why would they split a physical box up? Ease of administration. It turns out to be much easier to have ten virtual machines running one daemon each than one virtual machine running ten daemons. Fewer interactions, cleaner process boundaries (the boundary between one VM and another is pretty thick), easier debugging. Easier to swap out parts without affecting other parts. Much easier scaling. (If you need more servers for Service X, launch more VMs; if your machine needs more RAM to hold those VMs, launch the VMs on a different physical box -- the system doesn't need to be changed to support that because it's already architected to assume separate "machines" for everything.) For more on the subject ask Ezra Zygmuntowicz or his blog and books. I stole all of this material from him.
- retube 16y agothat's a great response, cheers
- gaius 16y agoIt turns out to be much easier to have ten virtual machines running one daemon each It's funny, when you needed one OS instance per network server daemon on Windows in the 90s, everyone laughed at it. Now it counts as cutting edge thought.
- borism 16y agoEveryone laughed at it because at the same time there were OSes available that could run unlimited number of daemons simultaneously. Now it's a matter of choice for system owner. And of course all those single-daemon VMs run inside OS that lets numerous VMs run at the same time.
- silverlake 16y agoWould Solaris containers or Linux VServer work as well here? One would need good management tools, of course.
- 16y ago
- cperciva 16y agoYahoo used to be a BSD shop, then moved to linux. Yahoo still uses BSD. They're not completely BSD, of course; but they're not completely linux either.
- irahul 16y agoYep.Yahoo still uses BSD, but earlier, a lot of work was put on BSD. There were BSD kernel teams, and teams which managed BSD ports. AFAIK, almost all of the new development is on RHEL and there aren't any BSD dedicated teams now. A lot of BSDs are still around - some have been running for last 10 years or so. Some new teams use BSD for their webapp deployment and database servers; can't confirm (no links) but I think the teams were of the opinion MySQL on BSD was more stable. But as said, earlier Yahoo was big time BSD, investing in it and submitting patches, which no longer is the case. Search team did a mass migration to RHEL about 2 years back. BSD is still used here and there as deployment platform. My personal preference would be to go with BSD and save some buck but apparently management views it differently.
- regularfry 16y agoI am reminded of a William Gibson quote: "The future is here, it's just not evenly distributed."
- swah 16y agoHuh? His "dumb terminal" is a Mac laptop that will run on a single machine most less-than-Google-sized tasks...
- mechanical_fish 16y agoEven if you are avoiding Google-sized tasks, there turn out to be advantages to treating your local machine as a dumb terminal. One is that remote machines can be rented instead of being bought. Another is that the configuration of everything becomes easier. You buy new laptops by taking them out of the box, running a handful of app installers, and moving over a SSH key or two and an encrypted password file. Your career as amateur sysadmin might just be over. (If you miss the chance to build your own Linux kernel, of course, there's plenty of work left for you as a cloud sysadmin. But cloud system admin is also usually easier, because machines in data centers come with nifty administration tools, like SANs and dashboards, that benefit from economies of scale.) Security: When you lose the "dumb terminal" you lose no data. You do lose the keys that are on the machine, but you can revoke those. And it is much, much easier to get consistent automated backups on a datacenter box than on a laptop. There are definite problems with the "cloud" approach, such as "who owns the data and how do you trust it is being handled properly", but the advantages are strong these days.
- adambyrtek 16y agoIt's more about convenience than resources. For example I'm very happy that I don't have to carry my laptop with me anymore as I can always reconnect to my virtual machine from home and continue at the same place where I left.
- jedbrown 16y agoForget about complex shared-memory or message passing architectures: that stuff doesn't scale, and is so incredibly brittle anyway (think about what happens to an MPI program if one core goes offline). MPI is not good at fault tolerance, though MPI-3 will help some. MapReduce and Sawzall are well-suited to the problems they were made for: largely independent operations on very large data sets with occasional reductions and weak synchronization. Usually disk is a significant factor and strong scalability is not especially important. But they really can't compete with MPI for other problem domains. Claiming that MPI doesn't scale is demonstrably false, as evidenced by PDE solvers running on 200k - 300k cores (Jaguar, Jugene, etc). This is a comfortable order of magnitude larger than Google is running, and the communication requirements of those algorithms are far more demanding, with essential all-to-all communication as well as low-latency global reductions (for which Blue Gene has dedicated hardware). The need for these low-latency all-to-all operations is actually very fundamental (formally proven). The algorithms also tend to have much stricter synchronization requirements, for example, nothing can be done until everyone gets the result of a dot product, because the next operation needs to happen in a consistent subspace (though MPI has fine asynchronous support). There has been plenty of work trying to make algorithms less synchronous, but weakly synchronous algorithms almost uniformly have worse algorithmic scalability (number of iterations to solution increases by more than a constant factor). And since problem sizes are usually chosen to fit in memory, the network doesn't get a free pass due to the disk being slow. If you care about strong scalability, MapReduce and Sawzall are probably not good choices. Similarly, if your problem domain requires low-latency all-to-all, reductions, or "ghost exchange".
- Retric 16y ago200k - 300k cores... a comfortable order of magnitude larger I would assume Google has well over over 20,000 cores. Or did you mean something else?
- jedbrown 16y agoFrom talking with someone at Google Zurich who is involved in page-rank computations, those jobs run on ~10k to 20k cores. Of course Google has many more distributed around the world, but unless something has changed in the last few months, they are not running single jobs on more than about 20k cores.
- edw519 16y agoSo, printf() is your friend. Log everything your program does, and if something seems to go wrong, scour the logs to figure it out. Disk is cheap, so better to just log everything and sort it out later if something seems to be broken. There's little hope of doing real interactive debugging in this kind of environment, and most developers don't get shell access to the machines they are running on anyway. 1985: Interactive debuggers suck. PRINT() is your friend. 1990: Interactive debuggers have matured. No one uses PRINT() any more. Debugger questions are now even part of technical interviews. 2008: Concurrent processing and cloud computing have made interactive debugging difficult. 2010: printf() is your friend again. Developing software is getting to be like fashion. Keep those old skinny jeans and workarounds in your closet. Sooner or later, they'll be in style again.
- jakevoytko 16y agoAll projects are different. Use the most useful tool for your job, and pay little attention to other developers. They may be using something better, or they may just have different needs.
- tibbon 16y agoThis seems to be even more interesting to compare the Ruby community (from what I've seen is massively test-driven and shuns use puts()), to the use of Python at Google. Of course, most Ruby stuff that I mentioned is Rails based, and there isn't that much concurrency happening and its set up well for testing- but just an observation.
- adambyrtek 16y agoGoogle is extremely test-oriented, every project is covered by a test suite and there is a powerful distributed continuous integration infrastructure in place that executes tests "in the cloud" and doesn't allow to commit code that doesn't pass. Not to forget about the public Google Testing Blog[1] and the famous Testing on the Toilet[2]. People in the Ruby community also use temporary puts/logging statements for exploratory debugging, but of course nobody in right mind commits such code to the repository. [1] http://googletesting.blogspot.com/ http://googletesting.blogspot.com/ [2] http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html http://googletesting.blogspot.com/2007/01/introducing-testin...
- brown9-2 16y agoI do all of my development work on a virtual Linux machine running in a datacenter somewhere -- I am not sure exactly where, not that it matters. I ssh into the virtual machine to do pretty much everything: edit code, fire off builds, run tests, etc. The systems I build are running in various datacenters and I rarely notice or care where they are physically located. I love moments like this when you realize as a developer that the bits that are the fruit of your labor are off executing in some far off physical location you'll never know about.
- global_hero 16y agoThis guy is a total lamer!
- jessriedel 16y ago>The idea that you need a physical machine close by to get any work done is completely out the window at this point. ... I ssh into the virtual machine to do pretty much everything: edit code, fire off builds, run tests, etc. ...Wide-area network latencies are low enough that this works fine for interactive use, even when I'm at home on my cable modem. This situation sounds fine if you're a developer and your job requires only low-bandwidth text streams between you and your physical computer. But for anyone who does anything graphical, or for anyone at home who wants to enjoy media, the bandwidth and latency are big problems. Maybe I should just assume that because this is on HN and written on a developer's blog, context is assumed. But it sure seems like a disconnect between hard programmers and the other 99% of the population.