11 ms·
Ask HN: What Have You Learned at Google as a Software Engineer?
Hello Googlers (former and current ones),
What have you learned at Google, as a Software Engineer, that you think was interesting and helped you up your game as an engineer? Anything you can share with people outside of Google that can help them improve as engineers?
- sulpiride 7y agoit's also interesting what have someone learned as a google intern
- apollo_ 7y agoI've been at Google for 1.5 years and before that startups. I've learned a lot about working at a large company (getting adjacent teams on board with my ideas, writing compelling design docs, etc). I haven't observed any changes in the quality of my code or ideas.
- zuck9 7y agoYou should replace Google with "big tech company" or FAANG to get more responses.
- mav3rick 7y agoFAANG don't share the same engineering culture.
- mlthoughts2018 7y agoYes they do. In all these places except Amazon, the engineering culture is essentially identical. I’ve worked at 2 FAANG companies and a few startups that had a big influx of ex-Google engineers. In fact it’s a whole process that startups go through to copy Google: monorepo + misguided in-house tooling for it, homebrewed canary deployment strategies, naming things “mon” like “altermon” or “datamon”, etc. It’s a very miserable monoculture.
- yitchelle 7y agoThe engineering culture among FAANG companies is pretty much the same because of the industry that they operating in and the location of the HQ. Expand that to military or automotive, aerospace, in non-US companies and you might start seeing different engineering cultures.
- mav3rick 7y agoEngineering culture at Google is vastly different than Amazon. Even Facebook which is more product oriented and iterates faster than focusing on the "right engineering design" first.
- mlthoughts2018 7y agoFB and Google engineering is basically identical in culture.
- mav3rick 7y agoYes because lots of Google left to join FB. And yet, I've heard everyone say in terms of pure engineering and tooling Google is far ahead. That's still F&G though not AAN.
- akhilcacharya 7y ago> Engineering culture at Google is vastly different than Amazon How so?
- yibg 7y agoI assume the two FAANG you are referring to are the F and G. The rest of the letters aren’t like that. At least no monorepo.
- saagarjha 7y agoOr "mons".
- mlazos 7y agoI agree that there are cultural differences, but culture definitely varies from team to team in any large tech company. I think a job at any of the big tech companies is interchangeable. I hear stories about learning from code reviews, maintaining build systems, and building out new cross team products at all of the big tech firms, I don’t think the lessons learned at google are particularly special.
- amelius 7y agoPerhaps nice to mention here: "Talks at Google" https://talksat.withgoogle.com/ https://talksat.withgoogle.com/ Not restricted to just software engineering, though.
- forgot-my-pw 7y agoI think Talks at Google is mostly not about Google? For the engineering side: https://www.youtube.com/user/GoogleDevelopers https://www.youtube.com/user/GoogleDevelopers
- awinter-py 7y agoA friend who spent a short time there said "you know that tool that you want to build before you start working on your real job? At G, it already exists" This is less true now because a lot of their tooling has been exported either directly or by rumors leading to it getting rebuilt outside. I'm talking containerization / orchestration and protobufs specifically.
- riku_iki 7y ago> At G, it already exists quality of the tool has also significant impact on the end result. Looking at the number of unsuccessful external projects, I bet they also have enough mess inside.
- mav3rick 7y agoRight search , maps , YouTube all run smoothly because of a "mess". Google code is some of the cleanest most organized I've seen.
- aappleby 7y agoEven really smart people sometimes write really dumb code. Including you. :)
- mav3rick 7y agoMy quality of code has been immensely helped by code reviews by senior people. My team also has a good culture of getting feedback on design. I've learnt a lot on designing with security on mind - sandboxes, SELinux etc. I had no clue about them before I joined. Design docs and comments by great engineers like Jeff Dean are openly accessible. Treasure cove of knowledge that I try to dip into everyday.
- octocode 7y agoNot Google but similar. I learned that my motivation is proportional to the quality and amount of free food I receive. This includes willingness to participate in pointless meetings. Note: this is not something I'm particularly proud of or anything. Just an observation.
- jeremysalwen 7y agoEverybody comments about how there are all these amazing tools that help your productivity (which is true), but they don't talk about how the opposing force of organizational complexity ultimately results in "just normal" levels of productivity. They don't cancel each other out exactly though, so some things are way easier (spinning up a huge computation), while some things are way harder (making a small change to shared components).
- bt848 7y agoAbsolutely disagree, to the maximum possible extent. My entire job at Google was to find and fix inefficiency in other people’s projects. I didn’t own any projects! But I submitted changes to foreign code daily and I rarely got any kind of pushback. Working on shared code is so easy at Google because they have a fantastic cross-referenced code browser and search engine, ways to inspect production jobs and configs that anyone can access, and everything is open and visible. By contrast the other large publicly-traded cloud company where I now work has the opposite culture: no code search worth using, “why are you touching my project?” culture, not enough tests to give anyone confidence in a change. The polar opposite of Google.
- BeetleB 7y ago>My entire job at Google was to find and fix inefficiency in other people’s projects. I believe when he said "organizational efficiency" he was not referring to technical inefficiencies, but rather chain of command, how organizations are structured, etc. Not at Google, but that is the case in my company as well. In most of the company, doing extremely great technical work is not going to get you far.
- jeremysalwen 7y agoHmm, I think you interpreted my comment about "making a small change to shared components" differently than I meant it. I am guessing that your changes did not change the semantic behavior of other people's code. e.g. you were replacing one hash-map implementation with a more efficient one. That's something where obviously Google's tools do cancel out the inherent inefficiency of large organizations/code bases. My point was that Google has the level of efficiency of a small organization, despite being a large organization. That the default for big organizations is to "just suck" as you described, makes this remarkable, but I never felt like this resulted in a superhuman level of productivity that you might expect considering how advanced the tools are.
- Zach_the_Lizard 7y agoNot Google, but another Big Tech company. Visibility is very important to getting a promotion at a large company. Selling your work is important. To move up, you must be playing the "choose a good project or team" game for at least 6 months before you try to get promoted. Preferably for a year or more to hit the right checkboxes for multiple cycles. If you fail to do so, you can do absolutely amazing work but rigid processes and evaluation criteria will conspire to defeat you in a promotion committee setting. At least, that's true in my company. From my ex-Google peers it seems to be true there as well. Being in a smaller office means you get fewer of the best projects available to you. Reorgs sometimes steal them. Cancelling projects makes the last half a waste of time from a promotion standpoint. As for what constitutes a good project. It will: * Let you lead it * Have peers at your level + one or two * Work with multiple other teams * Ideally work with multiple teams outside of your group, e.g. you're in, say, a chat app team and get to solve issues for some other app * Good product manager and designers with a well thought out product; it's not fair, but if the project is successful business-wise people will often incorrectly attribute that to your own skills * Have large engineering scope. Committees get confused and sometimes think simple designs to solve complex problems are bad. You want to solve problems that are complex even in terms of the solution (or can be made to sound complex) to check more boxes. * Allows you to solve problems for other teams I haven't optimized for promotion and thats personally hurt me in the quest to get to L6. I very, very close but didn't quite make the leap. Don't ignore the flaws in this committee process. Exploit them. Don't be me.
- skizm 7y agoWhat does "promoted" look like for an software engineer at large tech companies? In my company you're either an individual contributor or tech lead. There aren't really any other rungs on the ladder without going to manager.
- bt848 7y agoAt Google there’s an IC ladder from level 3 (new grads) to 11 (Jeff and Sanjay). Basically it’s a pay scale. Half of engineers are below 5, and 90% are below 6. Edit: pretty sure I was off by one originally. But you get the idea: it’s a triangular organization with a lot of people at the base.
- brianolson 7y agoBig organizational problems can't be fixed by writing better code. Meritocracy sounds nice but you have to learn to play office politics. New products and new features should have a business case (yes, even at Google) so that eventually they can stand and be valued.
- justicezyx 7y agoGoogle showed me the fine engineering at large scale. And the key is the people as they are behind everything. That’s why I changed my view on who was the key person for google success. I used to believe Larry and Sergei, then urs, now I put on Laszlo Bock.
- nessunodoro 7y agohonorable mention: Guidooo!
- bt848 7y agoReally? One thing I learned at Google is that even having Guido on staff can’t make Python work.
- mundanevoice 7y agoElaborate, please?
- bt848 7y agoPython just doesn’t scale. Th language isn’t typed enough to be safe in a large program. Eventually you will iterate over a string or access an attribute of None. And the runtime is too slow. SRE at Google years ago banned new projects in python because the incredible slowness of the tools was extending the time to repair outages.
- aalhour 7y agoCan you share more details? What advice would you give to people to engineer things at scale?
- justicezyx 7y agoI wrote a partially finished article: https://docs.google.com/document/d/10wylnn8qHD4clDyrEIhYyP1FWxzEkf89hfwbmoyS-yI/edit?usp=drivesdk https://docs.google.com/document/d/10wylnn8qHD4clDyrEIhYyP1F... It's not even touching the techniques and approaches yet. I was too lazy. Here a summary of my understanding: * Relentlessly hire the best, train them and give space for them to grow and shine. This is the basis. It's not that a less quality engineer cannot grow, it's just too costly to do that at large scale. * Build the infrastructure to support large scale engineering. The best example is the idea of "warehouse scale computer" that is the data center, embodied in systems like Borg Spanner etc. * Relentless consistency at global scale. Example would be the Google C++ style guide. It's opinionated at its time, but is crucial to ensure a large C++ code base to grow to big size. * Engineering oriented front line management. L6-L8 managers are mostly engineering focused. That's necessary. * Give the super star the super star treatment. Google's engineers are rewarded as strategic asset of the company. Numerous example are made public.
- cmrdporcupine 7y ago8 years; what I've learned -- working in absolutely huge code bases, good C++ style, good code review culture, intellectual humility. What I haven't learned: how to thrive in a huge organization with people more bureaucratically ambitious/motivated than you. How to turn a blind eye to broken waterfall method dysfunction. Starting to get to me. Probably looking for the door soon; p.s. entertaining offers for remote (Canada) work.
- lazyant 7y ago> entertaining offers for remote (Canada) work. want to add an email or something to your HN profile :-)
- cmrdporcupine 7y agoIs it not visible? It's there. Don't see an option to make it public.
- guessmyname 7y agoHere is @cmrdporcupine GitHub [1] if anyone wants to hire him. You’ll find his LinkedIn profile searching his name online. [1] https://github.com/rdaum https://github.com/rdaum
- jorawebdev 7y agoOther than having “Google” on my resume there is nothing special or applicable outside of Google. Most tools are internal, isolated and the choices are restrictive. Management is shitty - micro-management is in full bloom, display lack of management knowledge, skills and there’s plenty of abuse of power. They don’t show their appreciation to what we do. All developers are very competitive. My entire time of over a year in 2 different teams is spent in isolation and self learning without much help or directions. I’m currently actively interviewing outside.
- alephnan 7y agoWe’re you a TVC or a full time employee?
- jorawebdev 7y agoI'm still a TVC. This could be a separate discussion on the scope of how TVCs are treated and it could complement all the recent articles in the media, however, my main focus was on something common, such as tech stack, restrictions and constrictions that will impact you when you go out and work somewhere else.
- mav3rick 7y agoTVCs get many of the same perks full time Googlers do. You can always interview if you want to be a full time.
- kyrra 7y agoAs someone who has worked at multiple companies, currently at Google, you don't tend to leave a company, you leave a manager. Managers are one of your primary interfaces to a company. If those managers are crap, you will have a bad experience.
- scottlegrand2 7y agoIt's bigger than that IMO. I can switch managers at a big company. But if the entire division is inept, it's time to jump companies because it's a tell that the culture is decaying. For me, the only reason I will never go back to GOOG is blind allocation. IMO that's a bug, not a feature.
- dannypgh 7y agoI spent 9 years at Google, just left at the end of July. The biggest thing I grew to appreciate was that for iterating large scale production systems, rollout plans are as important as anything else. A very large change may be cost or risk prohibitive to release at once, but with thought you can subdivide the release into easier to rollout and verify subcomponents, you can usually perform the same work as a series of lower risk well understood rollouts. That's critical enough where it's worth planning for this from the design phases: much as how you may write software differently to allow for good unit tests, you may want to develop systems differently to allow for good release strategies. Google is a large company, and I think there are software engineers learning very different things from what I focused on. I worked on large scale machine learning in ads; someone working on chrome or Android likely learned very different things.
- ithkuil 7y ago(Hi Danny!) Yeah Google has a lot to teach about building and maintaining reliable systems. I noticed that a lot of that, including the name SRE, gets cargo-culted by other companies without really understanding what it is. I also guess that there might be people within Google that don't end up working in projects that invest in reliability (for whatever reason) and so might have had different experience.
- Xlythe 7y agoI've been at Google for almost 4 years, mostly working at startups before that. First and foremost, there's nothing special about Google (compared to any other tech company) and it's best not to evangelize any one company. Learning comes from experience, which means trying things (even if you end up making mistakes) and figuring out the shortcomings of a particular solution. I've seen a lot of bad (and a lot of good) code at Google. What I learned, design-wise, came from a coworker. They planned everything out in advance and would draw out components on a whiteboard and define what they'd do. A few weeks would pass before they'd write any code. In contrast, I would start with writing skeleton classes and intuit the breakpoints for building a new class. We ended up with similar code structure at the end of the day, since we were both designing the architecture (even if our methods were different). But in code reviews, I would focus on if the new code made sense within the scope of the change; are there any clear bugs, code duplication, is there a more simple approach, etc. My coworker always went back to the whiteboard and drew up the components again, making sure the new code fit within the originally defined scope or if a larger change would make more sense. Because my skeleton classes were quickly replaced with implementations, I lost my original frame of reference and only focused on what was in front of me. I could still tell when something was obviously wrong (stop sending me CLs with global state to cross talk between components!), but there was creep in the CLs I approved and the code would slowly get more complicated until I realized it needed to be refactored. There's a tradeoff (like in anything). You can't design your own solution for every code review, as it takes too much time and removes autonomy from your peers. But remembering to step back and look at the big picture was something I needed. There were other things I learned, like structuring code in ways that it's hard to write bugs. For example, you could write a method like... void foo(Callback callback) { if (error1) { callback.onError(); return; } if (error2) { callback.onError(); return; } if (error3) { callback.onError(); return; } doWork(); callback.onSuccess(); } but it's easy to forget to call 'callback.onError()' in a future CL. A small refactor, like... void foo(Callback callback) { if (bar()) { callback.onSuccess(); } else { callback.onError(); } } boolean bar() { if (error1) { return false; } if (error2) { return false; } if (error3) { return false; } doWork(); return true; } has the compiler help you catch mistakes. Unit tests obviously help too, but doing both reduces the number of bugs that slip through. Similar tricks include annotating methods as @Nullable so you don't forget to nullcheck, annotating which thread is calling a method (eg. @FooManagerThread, @UiThread, etc), and doing all the if checks as close to the top of a method as possible so that you only do work if you're in a good state. Oh, and here's one last tip that I only realized needs to be reiterated because most of my coworkers forget it. Validate incoming data! Every API needs a wrapper around the entry point that... * Verifies the caller is allowed to call that method * Verifies the method can be called at this point in time (eg. hackerNewsApi.postComment(threadId, msg) only works if the threadId is valid) * Verifies that the arguments make sense (eg. 'msg' is not empty/null/above the max comment size). And this is needed at every layer (application, server, etc). Trust no one, even if the only caller is supposed to be yourself. Glossary: CL = changelist ~= pull request
- rictic 7y agoAt Google for 9 years, startup before that. Roughly half my time at Google I worked on individual products, and the other half on open source infrastructure (common libraries, compilers, linters, IDE plugins, etc). The biggest thing I've learned is how software changes over time. Previously I thought of a good program as like a crystal. Write it correctly the first time, and it doesn't need to change much, if at all. Sure you might want to add features or fix bugs, but otherwise you write it once and it's good forever, right? Most programs are closer to organisms than crystals. They're adapted to specific niche, and when the world changes around them they need to either adapt or die. This is the biggest gap I notice between senior devs and strong junior devs. Junior devs have mostly built things, released them or turned them in for credit, and then moved on. It's rare to find a junior dev that has maintained an app for the long haul, and it accounts for so much of the advice that I found strange when I was junior: * code is read many more times than it is written * be very mindful of your dependencies, if push comes to shove you could end up owning them * commenting why is often more important than commenting how. it's good to comment about both, but it's much more important to comment why. the source code explains how, but it has nothing about why * it's ok to love your code, but delete it the first chance you get. it will save you and the people around you a lot of time Do you think I'm overselling things? Maybe that's a problem for people who write code in $cursed_language on $bad_platform but your tools favor writing True and Long Lasting Solutions That Stand the Test of Time! Let me give a handful of real world examples: * A major security vulnerability is found in an API that previously seemed safe. Refactor your code not to use it. * Turning on a certain compiler optimization breaks a pattern common in your code. After some experimentation, you discover an ugly fix that you then need to apply to everywhere in the code that uses that pattern. * A compiler/browser/hardware update adds support for a feature that your code has been using a bunch of ugly workarounds to cope without. Do you use the new feature? Do you still support running without the feature? Is it a breaking change to users downstream of your code? How much does it improve things, how hard will it be for your users to update, and how many users do you have? * You need to migrate to the new version of one of your dependencies. Maybe this is just a version bump, maybe you end up needing to review every single line of your app. * Your app for phones and desktops is working great! We want to run it on TVs as well! This means a brand new (and super weird) matrix of features that are and aren't supported, from hardware to OS to UI. In short, the only constant is change. Write your code so that you can go away for six months and forget just about everything about it and still be able to maintain it. Lots of tests, document why decisions were made, and it's like Bruce Lee said, be like water.
- samueljackson 7y agodecades of working for big companies on projects budgeted at billions. Sometimes as part of large teams, other times a three man operation!!! seen it all. Big companies taught me to be a politician and that skills alone were not enough. You'd need to make a strong relationship with everyone and I mean everyone. Insane working hours, a few projects had expectation of 24/7 and had I refused, they'd politely replace me with another without notice. Lots of talented people to do the work under that condition. Had enough of it, not like they paid me millions and when I complained, they'd say "make your own company and do it as you like it to be". Every big company I've been to had that sort of culture. As for small companies, there isn't much to the future. Lots of work to be done for free, always welcome to work for free. Finally here I am, looking for a remote job of sorts that'd allow me to travel without having to be online all day. At least I can enjoy programming and travel.
- mindfulplay 7y ago* Always be learning and doing. Never be stagnant. Even if you are in a very boring team you can still learn and supplement your skills / knowledge on the side. This may or may not help immediately but definitely helps long term. * Really practice open/active listening and critical thinking: there are very smart people and it's really good to assume good intent when they talk. Critical thinking and question is more around 'where is the disconnect? How do I try and actively move towards and see their point of view'. * Questioning authority and being able to say no. * Don't use code reviews as a debate forum but instead as a learning forum. Granted time is of essence, so a timely resolution may be necessary but learning and actively listening through reviews helps your skills as a coder. * Cut through heirarchy and organizational politics to achieve the larger goals for the team, the project and ultimately for Google.
- bt848 7y agoThe most important thing I learned there was that you hire and promote people not so management can tell them what to do, but so they can tell management what to do. I didn’t even realize it at the time but the first post-Google job I had there was some clueless manager trying to set the roadmap and I was like “LOL what is this guy’s problem?” That when I started noticing the reasons that Google succeeds.
- pembrook 7y ago1) All big companies inevitably devolve into Dilbert, no matter how cool they seem from the outside 2) Humans are easily fooled by signaling and vanity metrics, exploit this quirk 3) Depends on the team, but in general working at a big company is 10X easier and more relaxing than working at a startup. You’ll probably never go back to working at a small company after realizing this. 4) Buddying up to the right people will dramatically alter your experience in a big company—meritocracy can never exist when human emotions are involved 5) the difficulty of the interview process is just plain bizarre given how simple the daily jobs of 95% of the engineers are 6) you will have the nagging feeling a high school kid could do most of the (non engineering) jobs in the company with a few months of on the job training 7) due to 5 and 6, unless you are a sociopath, you will feel guilty on a daily basis for building generational wealth while an immigrant contractor struggling to get by dishes out your free food 8) you will learn that at most big companies, the innovation is in the past 9) on the bright side, google does have really cool internal tools and perks 10) after leaving, people will grossly overestimate your intelligence and skill level due to the name google being on your resume (see #2)
- StephenAmar 7y agoCode search is a game changer.
- kyrra 7y agoIf you want to hint of what code search looks like a Google check out code search for chromium. https://cs.chromium.org/ https://cs.chromium.org/ What is internal is more featureful. And it covers the entire code base.
- akhilcacharya 7y agoI'm confused. Is this AST search? How is it otherwise different from Github search or the internal code search my company uses?
- crimsonalucard 7y agoI think I'm a good engineer, but I don't think I have the intelligence to get into google. Most people can't. There's definitely some sort of difference between an engineer who can get in google and one who can't as displayed by the interview. Most companies that interview people outside of FAANG tend to have easier interview questions in my experience. Some even forego algorithms all together. For googlers who worked at companies outside of FAANG or startups. Is there a noticeable difference when working these people? Will you learn more from a google level engineer then you will from someone who can't pass the google interview? What are your thoughts about it? If you think that FAANG engineers are genuinely better, say it. There's no need to be politically correct here, honesty is appreciated.
- opportune 7y agoI’m not sure why many people seem to think this, there are like tens of thousands of engineers at google, and there are other good companies out there that also have tons of engineers, you don’t need to be that special to get a job at google. As long as you can get your first interview all you need to do is be able to solve data structures and algorithms problems and talk about software design. Let’s not put google on a pedestal I will say, those problems do filter out a lot of bad people. But it only ensures a minimum level of competence. It doesn’t mean everybody is insanely smart.
- crimsonalucard 7y agoThe interviews at google are hard enough such that only a fraction of software engineers can make it through. Thus by statistics and logic, the majority of the worlds engineers cannot get past the google interview. This is not something I am thinking, this is a statistical reality. Literally, in order for those engineers to work anywhere else the algorithm questions must be significantly easier in other companies outside of the elite FAANG. This is inline with logic and my observations. My question is... for googlers who have spent time outside of google at a more scrapy startup. Does the interview filter actually create a better class of engineers? Do you learn much more working at google then you would at a company who's engineers typically can't get into google? What's the experience like if there is any? Don't get me wrong. I'm not worshipping googlers or asking for tips. A real dichotomy exists in the interview stage for the few engineers who pass and the majority who don't... I'm just curious if this dichotomy becomes obvious outside of the interview... like on the job? Do you notice a difference when working with a google engineer vs. someone who probably can't ever get into google. It's a complicated question with a complicated answer. I'm not really looking for some humble answer telling me that google engineers aren't that much different from other engineers when the interview essentially disproves this statement by filtering out everyone except for a certain type of person. I want an honest and candid answer.
- opportune 7y agoAll big tech companies are approximately the same, since people job hop a lot and these places have tens of thousands of engineers, it’s basically the same people at all of them anyway. I am not sure why people have these starry eyes views of them
- mav3rick 7y agoNo it's not. Microsoft is way different from Google.
- opportune 7y agoHaving worked at both I can say it’s not that different. But I had a great team at Microsoft
- mav3rick 7y agoI had great teams at both but the bar was consistent across Google unlike Microsoft. And 9/10 engineers will accept offers at Google over Microsoft given both.
- opportune 7y agoOh I totally agree, I guess for me when I think of “work” I don’t really think too much about other people’s engineering ability. It makes a difference but less of one than most people think.
- nailer 7y agoContract Python / JS engineer, working on the largest GAE app (internal, but 2400% RoI so well regarded). - GAE was missing basic functionality that we (as their biggest customer) couldn't get them to work on. The engineer in charge of GAE had other tasks and didn't care for this role. We ended up doing this work ourselves and I'm relatively sure this is now inside the App Engine SDK. - Delivery was incredibly badly managed, I have to shut one one eye when I sneeze due to sinus damage caused by working when ill on a project with global visibility and no cover. - 'Decisions based on data' isn't a thing: if you want an outcome, you research to that outcome or don't depending on how you feel. As others have said having Google on your resume is great.
- admils 7y agoContract Python / JS engineer, working on the largest GAE app (internal, but 2400% RoI so well regarded). - GAE was missing basic functionality that we (as their biggest customer) couldn't get them to work on. The engineer in charge of GAE had other tasks and didn't care for this role. We ended up doing this work ourselves and I'm relatively sure this is now inside the App Engine SDK. - Delivery was incredibly badly managed, I have to shut one one eye when I sneeze due to sinus damage caused by working when ill on a project with global visibility and no cover. - 'Decisions based on data' isn't a thing: if you want an outcome, you research to that outcome or don't depending on how you feel. As others have said having Google on your resume is great.
- deleted 7y ago[deleted]