5 ms·
That's what I thought before I joined. Once I got into the swing of things and started doing actual work, I was like "Goddamnit. We need more employees. Ther
by nostrademons 15y ago
That's what I thought before I joined. Once I got into the swing of things and started doing actual work, I was like "Goddamnit. We need more employees. There's too much work to do!"
The simple, non-confidential answer is that running a global service used by billions of people entails facing a lot of problems, on a daily basis, that most developers will never face. And that makes everything move slower.
Have you ever internationalized an app? It's not just a matter of translating every string in the UI, although that certainly is a pain in the ass. You also need to handle right-to-left text and mirrored UIs for Arabic/Hebrew/Farsi, and English text mixed in with that mirrored text (this is common in Israel) which requires a directionality reset. Did you know that you're not supposed to bold Chinese characters? The typographic convention is to make them red for emphasis - but only Chinese characters, English text embedded in Chinese characters (eg, "President Obama" in a Chinese news article) should still be bolded as necessary, so you can't just use a global CSS rule. You know that several European countries format a million bucks as $1.000.000,00 instead of $1,000,000.00, right? And dates go DD/MM/YYYY instead of MM/DD/YYYY, and your date parsing routines have to understand that as well as your date display routines. Ever dealt with Russian plurals? In English, you know something is plural based on whether it's one or many; in Russian, it's based on whether the last digit of the number ends in 1, 2-4, or 5-0, and sometimes depends upon its position in the sentence. Can you do case-insensitive comparisons in multiple languages?
And every single UI engineer needs to be familiar with these pitfalls. A lot are hidden behind frameworks and utility functions, but then every single UI engineer needs to learn the frameworks and utility functions, and understand the purpose behind the libraries so they know when they're appropriate to use.
Do you worry about efficiency when you program? Most programmers these days grab something like Python or Ruby whenever they need to do something, but doing a complex parsing routine in Python can easily run 100x slower than doing it in C++, and at Google scale that means 100x more machines. There are a bunch of open-source libraries for the type of internationalization problems above - but the ICU libraries can easily pull in 50M or so of data files into RAM. If you're running on 20K machines (number pulled out of a hat), that's a terabyte of additional RAM consumed because you wanted to use an open-source library. Do you care about end-user latency? Most webdevs these days take for granted that they'll have JQuery or Prototype available to deal with cross-browser differences. One of the first things I did when I joined Google was try to convince them to allow JQuery for development, and as part of that I ran an experiment to figure out what the actual user impact would be. It turned out it would've doubled end-user latency of the time, and so we had to keep writing everything from scratch based on browser primitives.
Then there're all the issues surrounding running a massively distributed system. Machines fail, packets get dropped, and you have to handle those failures gracefully. It's impossible to upgrade a distributed system all at once without downtime, so every time you make a change that crosses a component boundary, you need to make the new component backwards-compatible with the old one and run both versions of the code in prod until the whole fleet has been upgraded. Many familiar algorithms that work on one machine don't parallelize effectively, so you're limited to what you can compute. Even if they do, the standard libraries for them usually assume a single flat address space, and so you have to really understand the algorithm and how to convert it to a distributed network of computers, and then reimplement it yourself.
If this sounds exciting, we're still hiring. :-) If it doesn't, well, if your startup gets big you'll have to face these problems anyway. Hopefully you can hire someone who finds them exciting.
- steventruong 15y agoFunny but I was thinking about some of these things a few months back that you are addressing here but more in the form of an API that allows real time conversion of visitors by geo-targeting traffic for all types of sites. So say I run a blog and I write a new blog post that happens to mention $1,000,000.00. It would be nice if a reader from Europe ends up on the blog and is reading it would naturally see a conversion appear next that that number in Euro or something to that affect, but not just currency conversion but in the correct format (similar for other metrics since the US seem to be the rarity when it comes to measuring things in Fahrenheit, inches and feet, miles, etc...).
- jsnell 15y agoThat's all true, but I fear it might be painting a slightly unrealistically positive picture :-) There are also other kinds of reasons for why everything moves slower than one would hope, and they aren't quite as interesting. Some system your project depends got deprecated again, and you need to decide whether to spend the time to migrate it to the replacement now, or hope it continues working until the replacement has been deprecated too. The development configuration files for other teams systems that you need to inherit from to bring up a full test cluster for your own system changed again, causing mysterious errors. Hopefully early on, rather than in the output of a data pipeline that takes 20 hours to run. Or the system your working on is somehow relevant to the outside world. There's a legal/political/PR problem, and actual engineering effort is needed to implement the minimal hack to get rid of the problem as soon as possible. Or your project has inexplicably and apparently without anyone's knowledge been marked as falling under Sarbannes-Oxley, and now some change needs to be explained in great detail to a SOX auditor. Which would of course be much more important than any work you were actually hoping to get done :-) The code base is so massive and there are still so many unexpected dependencies that even with distributed and cached compilation, just the processes responsible for the distribution might be enough to drive your machine to its knees. Of course there's a process in place to try to keep the dependencies from growing out of control, but that's then going to make your life much more difficult if you're working on the inconvenient parts of code nearer the center of the dependency graph, and thus more strictly regulated. Also, there have historically been a lot of misguided projects that in practice failed but somehow didn't die. Usually they will get killed eventually, but they do tend to grow larger and live longer than they should. Maybe that's changed lately. This is, of course, an unrealistically negative view. But at least in my experience these kinds of issues were more significant drags on productivity than the fun kind of issues you listed. Though that could be just because the boring kind of drag is much more noticeable. Google's a great place to work at in any case.