7 ms·
Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're
by nickelcitymario 4y ago
Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS.
Is it a security thing? Are Googlers not allowed to directly control their own computers?
- ptero 4y agoI never worked at Google, but based on my experience elsewhere it is partly security and partly standardization of guaranteed minimum functionality. For example, security aside, as an engineer working on X that depends on internal tools Y, Z and W, I do not particularly care what variant of Linux is under the hood as long as I can run my favorite WM, editor and user apps. But I do want support on Y, Z and W if they misbehave. If I run a non-standard OS or distro I will likely get a lukewarm support because those teams will put debugging problems in "non-standard configurations" as a low priority. Personally, I learned to live with most Linux distros. When I did roll out a non-standard setup I had a standby system in a standard configuration that I can show failures on before asking for help. My 2c.
- dman 4y agoGoogle was the only employer I had where I had root on my machine, so I would say that developers have control over their machine.
- baobob 4y agoWas this pre or post 2009?
- compiler-guy 4y agoI have root on my Google machine right this second.
- sulam 4y agoI have root right now on my Linux and MacOS systems.
- dman 4y agoPost.
- geekbird 4y agoI had root on my Linux desktop box at my university job, but I was in IT, administering a Linux service. Most of my coworkers were using Macs only.
- danpalmer 4y agoGoogler opinion, it feels like the right balance between letting me be in control while automating away the boring stuff that I don't want to deal with.
- wayne 4y agoI've worked at large tech companies where very capable software engineers struggled to install or upgrade their OS, let alone be familiar with the intricacies of Linux. I was surprised at first too, but I do think there's a large class of engineers who think about code who don't think at all about the other stuff, and engineering is broad enough that that's totally fine. Not to mention all the non-engineers, even in tech roles like PM, design, etc.
- drewzero1 4y agoThe way I see it, the engineers aren't being paid to install and repair hardware and operating system stuff. Even if they are perfectly capable, it makes sense to have IT handle that and let the engineers get on with their engineering.
- cactacea 4y agoThis. There's a lot of I shaped people out there that don't have the first clue about anything beyond their day-to-day.
- matheusmoreira 4y agoI think it depends on what the priorites are. I care a lot about my own systems but when I'm working I really don't want to have to care about anything but the activity I'm getting paid for.
- mattnewton 4y agoIt’s for security and for homogeneity, a lot of google is setup around the principle that “works on my machine” is terrible, and also removing needless cleverness. You have root and can run anything you want*, but you have to go out of your way to configure anything differently than others, and the result is (hopefully) it just works the same everywhere. The monorepo also runs only natively on these Linux machines through a magical fuse interface so most development is either using the web ide or ssh-ing into the Linux box if you aren’t sitting in front of it. There were big economies of scale running this way and at least this setup, definitely felt pretty great and efficient I gotta say. * By default, a tool called “Santa” keeps a naughty and nice list of runnable programs but all it took to get a program added to the nice list is any other googler vouching for it in an automated web tool.
- dehrmann 4y agoSanta is open-source: https://github.com/google/santa https://github.com/google/santa
- miggol 4y agoThere probably a lot of brilliant minds at Google who have their very first taste of Linux there. Designers, management types, even developers, who suddenly need a Linux install to run a specific tool or dev build. Having a consistent Distro for them to just turn on and go is very useful. If something doesn't work, they can communicate back to the (well-bearded) developer who sent them that weird build. The developer surely has some crazy esoteric distro on their bring-your-own device, but they also have desktops running the consistent distro available to them all around the office to reproduce the issue on. It's a bridge between worlds. I work in academia and we manage our own Ubuntu spin for this reason, despite everyone we ship it to being very conventionally smart. > if you're smart enough to work at Google, you're smart enough to manage your own OS. I also like to think of myself as "smart" for self-managing most OS and cloud things. But the role of smarts is probably minor compared to the decade+ of experience I have doing it. When it's something I'm good at, I think it's because of intelligence. When it's something I can't do, I assume the people who can have lots of practice.
- mistrial9 4y agothe word is not "smart", the word is "autonomy" and there is none in that network environment. (see post above for real example)
- happyopossum 4y ago> The developer surely has some crazy esoteric distro on their bring-your-own device Nope. If you want access to more than just the basic corp resources, you’re doing it from a fully managed and approved OS on a company owned machine or VM.
- poopypoopington 4y agoSoftware Engineers are worth more to the company when they're writing code, not when they're debugging issues with the OS. It's much more worthwhile to standardize the machines people have and let people focus on doing their actual work more effortlessly.
- duped 4y agoI've worked with Xoogler SWEs who didn't know how to use the command line. They're not hiring for computer literacy.
- bityard 4y ago> Are Googlers not allowed to directly control their own computers? First, if the computer is provided by the company, for company work, they're not "their own computers." They're the company's computers. And I don't know about Google, but yes it is VERY common in most mid- to large-size organizations that employees are not allowed to change things on the machines they use for work. The IT department is usually required by either contracts or regulation to follow certain security standards and many of those prohibit end-user modification of machines. And further, these are often enforced through annual, semi-annual, or quarterly audits. Failure to follow these standards can result in the loss of a sales contract, a critical certification, or fines. So, just like a manufacturing worker is not allowed to modify a machine on the factory floor (even to fix it, or improve it, regardless of their skill to do so), employees are not allowed to modify the organization's computing equipment. And, it's important to note, the IT department generally has NO say in these rules since they are a business or legal requirement. Edited to add: I've also worked in companies where the developers who write the in-house software might be masters of the business problem domain and their programming language, but have exactly ZERO knowledge of hardware or the most basic system administration principles. It doesn't cross their mind that RAM and disk space are actually finite things, for example, and that it makes for a Bad Day when one of their programs starts to consume ALL of it. These are the people who should not really be managing their own operating systems at work.
- mistrial9 4y agoI gave a demo in Mountain View of some excellent viz, on their big screen in some room. I wanted to make some tiny change to (plug something in) edit no not a new device, being polite, I suggested to move a device of theirs already on that network, closer to the overhead projector .. the alarm, consternation and definite NO that rippled through the three quickly-relaying employees there, was literally like an electric current. I was amazed that they couldn't change even the smallest thing, ever. of course I was trying to get my brilliant colleague without a degree, noticed with that code; zero response, like stonewall. Later I read, that year Google got over 1million job applications. weird place
- lupire 4y ago
- thrashh 4y agoTry doing support when everyone has a different setup Or your software has all these customization options Or your product has a bunch of different configurations What was already hard just became a thousand times more painful
- jnwatson 4y agoWe have root on our own (personal) boxes, so we can essentially do what we want (within reason). There's a tremendous value in having a somewhat homogenous environment. We have our own package repo and .deb packages to access internal resources. Even little stuff like editor plugins customized to our internal stuff we have packages for. Plus, we hire a great deal of folks that aren't Linux experts. Have a sane set of defaults gets people going faster. It makes a lot of sense to have our own distro when we're talking this kind of scale.
- raggi 4y ago> Having never worked at a place the size of Google, I'm surprised that their IT team directly manages 100,000 machines. In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS. Smart isn't really a key point here. You can be smart and make choices that solve your immediate needs but may not be good for the company in the long run. The distro provides for some standardization and baseline in among other variance. > Is it a security thing? Yes, one of the forms of standardization are audit solutions that keep track of all kinds of data about the machines. In the time I worked there, I once received outreach because a service I had started on the machine had retained an outgoing port to a third party service provider for an extended period of time. I explained what it was, and I did not get in trouble, but I also shut it down. There are many other forms of audit systems. > Are Googlers not allowed to directly control their own computers? It depends on your role, but many roles have root on their machines. You can of course do just about anything from there, however some things may get undone by periodic scripts if they're defensive (e.g. removal of certain software), or in other cases trigger some kind of outreach as described above. For certain other policies, you may need to file a bug with a business justification to request specific policies - you can also do this for entire teams if it applies to their work.
- dangus 4y ago> if you're smart enough to work at Google, you're smart enough to manage your own OS. No disrespect to you but this is a really naive take on how IT management works. For one thing, not everyone at Google is an OS expert. There are people working for Google in marketing, sales, support, data science, graphic design, hardware design, and other fields where you can't depend on every person to "do the right thing" when it comes to "managing their own OS." And even if they can, is that where you want them spending their time? Manually managing their OS? Don't you want them to do the job they were hired to do instead? Their computer is a work tool, not tinkertown. Basically, the opposite of what you're saying is true: because 100,000 people work at Google, each person represents another way to potentially mess something up and cause problems for the whole company. Assign each employee with probabilities: - Probability of falling for a phishing scam - Probability of installing malware - Probability of an employee missing an announcement regarding OS/security policy - Probability of an employee opening a support ticket with IT These probabilities are very low for people who are computer-savvy, educated, very highly qualified people. The thing is, all you have to do is multiply those probabilities by 100,000 and now you've got a big problem. It's much easier to run with zero IT automation if you've only got 10 people in your company and they all know each other.
- jms703 4y agoMost of the people you listed don't run Linux desktops at Google. Of those that do, most are able to manage their own OS. For your probabilities, you describe many issues that aren't big issues at Google. Phishing credentials is mitigated by mandatory FIDO tokens, binary whitelisting heavily reduces installs of malware, security policy isn't needed on these desktops (really just needed for mobile devices incl laptops).
- TulliusCicero 4y agoMost could, sure. I could do so. But why would I want to? It would take research and investment into something I don't give a shit about. I don't particularly care what Linux variant is being used, as long as it works. The less I have to think about it, the more I can spend time on other, more important stuff.
- deleted 4y ago[deleted]
- yathaid 4y ago"In my (apparently naive) mind, if you're smart enough to work at Google, you're smart enough to manage your own OS." This is a bit like saying if you can operate on human brains, you should absolutely be able to take apart your car and put it back together.
- tryauuum 4y agoI wish people never used analogies. It adds nothing to conversation and you are always inclined to pick the most ridiculous one. Neurosurgeon disassembling and reassembling a whole car, sounds indeed crazy.
- isatty 4y agoI work at google, and I’m also someone who has used and messed around with Gentoo for like 12 years. When it comes to my day job I rather not spend time messing around with my own OS (my workflow aside) and leave it to other people. Multiply that by 100k or so engineers and the productivity boost is insane. We’re root on both desktops and laptops though, I just rather use my time for work and I’m sure my employer would rather I do that too.
- prmoustache 4y ago> I rather not spend time messing around with my own OS (my workflow aside) and leave it to other people. I don't really understand what "messing around mean" in your context. Exotic things like slackware or alpine aside, all major popular distros now offer unattended updates, easy upgrade path between major releases. Your vscode, neovim, jetbrains whatever or emacs do not work differently from one distro to another and since nowadays many external stuff is either distributed as static binaries, containers or flatpacks or appimages everything is quite straightforward and not distro specific.
- dvirsky 4y ago(recent Xoogler here) There is quite a lot of freedom, though if you do custom stuff it's just not supported. I usually replaced the desktop, installed a custom terminal app, etc. There is also a lot of freedom in IDE choice - you can use literally whatever you want, it's just that the support is crappy if you don't use a supported one. And Vim support was IIRC completely volunteer based - there was no vim team, just people working on plugins and building releases as 20% etc. I contributed one plugin which sadly not a lot of people used, but Googlers out there - try BlazeDebugCurrentTest (it's part of the Blaze plugin) or something like that! Actually one of my favorite Google experiences starting out was that I wanted to work with an Apple Magic Keyboard, and to get proper support for it I needed to compile some kernel driver I found on github. I asked IT support if I can do that, and the answer was "you're a SWE, review the code, make sure it looks legit and doesn't contain any security holes, and then it's your responsibility if you mess anything up". Which I did, and it worked just fine. It was in 2018 so it's not like I'm referring to some early days thing.
- adtac 4y agovim support had only gotten better since 2018. With CiderLSP (LSP support for all of google3), it was a breeze navigating the massive codebase.
- dvirsky 4y agoBefore WFH I used CLion for cpp. When we were locked down I found Cider to be annoying (I think mostly because you couldn't tab switch to a a chrome window on MacOS) so I took a couple of days to really master vim and configure all the plugins (and wrote the one plugin I was missing), and it worked just fine for a while, I would just ssh to my workstation and use that. Later on they fixed the tab switching thing so I found myself drifting more towards Cider, then Cider-V came along, and I was back at the office, so I was mostly using that, though by that time I wasn't writing cpp anymore.
- anonquixey 4y agoI am a google engineer, and a fairly successful one. I would absolutely despise it if I had to manage my own OS, just like I despise thinking about my keyboard layout or text editor. Hacker News overrepresents the hacker type which loves the feeling of full control but I'd estimate that at least 33% of engineers, including many very talented ones, don't want that, and only want to focus on the concrete problems they are trying to solve.
- number6 4y agoDid the even more successful engineers liked the experience more or less than you?
- TulliusCicero 4y agoWhy would I want to manage my own Linux distro? I use it to code for work, that's it. > Is it a security thing? Are Googlers not allowed to directly control their own computers? This and simplicity I think. There's definitely some controls over what kind of things you install on the MacBooks as well, though it's actually not that restrictive.
- charcircuit 4y agoIt's also a matter of efficiency. I can spend more time on projects that matter instead of on system administration. The people doing the system administration are more efficient since they don't need to spend extra time figuring out what to do. It's not a matter of being smart enough to do it, but instead it's a matter of time efficiency.
- summerlight 4y agoNot everyone is interested in playing with their own customized OS configurations. Perhaps 90% of the employees would be okay if everything "just works". This is especially true due to lots of internal tooling and figuring out the right configurations for most of them would be a very painful time-consuming process for newcomers.
- ska 4y ago> if you're smart enough to work at Google, you're smart enough to manage your own OS Regardless of the scale, that is a recipe for having your highly capable, highly paid engineers losing hours, days, even worse - to annoying configuration issues etc. Sometimes it's the right thing to do (e.g. very early stage, no IT infra) but if you can have an IT person who is actually good/experienced at this sort of thing sort out the core problems, it's way more efficient. It's astonishing how much collective time can be wasted on "well, it works on my machine" issues. This makes it worth coming up with some sort of systematic way to avoid (mostly) it.
- synicalx 4y ago> if you're smart enough to work at Google, you're smart enough to manage your own OS. Oh boy, have I got some stories to tell you! Although to be fair the question of who "manages" a workstation is not always as simple as just "who has the technical ability to manage it". It's the intersection of who is legally responsible for the device and it's data (this is BY FAR the most important part), who has the time to maintain it (that's often NOT the user), and who technically can maintain it according to whatever standard it has to be maintained to (this gets complicated when you factor in reporting, patching etc). In my experience, a common approach on non-Windows systems is to give the users privileged access to do what they need to do while also monitoring that closely, managing patching for them, and if you have some legal requirement to do so; running some kind of AV for reporting purposes. This tends to strike a happy medium between giving IT/Security peace of mind to some extent while also not totally ruining the developer experience.