11 ms·

An ex-Googler's guide to dev tools (2020)

203 points by goranmoomin 4y ago

mandeepj 4y ago
Another Code Review tool discussion from 25 days ago: Show HN: Crocodile - Better code review for GitHub (https://news.ycombinator.com/item?id=31841215 https://news.ycombinator.com/item?id=31841215) Related, but from distant past: https://news.ycombinator.com/item?id=15722849 https://news.ycombinator.com/item?id=15722849 @Dang: Can we add a feature to auto link related threads in an expand\collapse section?
metadat 4y ago
Sourcegraph is one piece of the puzzle but there is still a long way to go to bring google-quality dev experience to the universe that exists outside of Big-G! * Consistent build structure * Rigorous code reviews * Ginormous company-wide shared filesystem holding all code * Point-click-drag report building * Big-query I'm sure I'm missing additional awesome internal aspects of being a developer at Google, and some parts are also horrendously bad.
- mayank 4y ago
  I agree! But — - BigQuery is available on GCP - Bazel is an open source Blaze - google source formatting (for java at least) is open source There are probably more…
  - metadat 4y ago
    Blaze/Bazel actually sucks imo. The only good thing about it is that all of google uses that one piece of shit, making things nice and uniform and consistent. There's a reason it isn't popular outside of elgoog. IRL, every project tends to do a little bit or a lot of its own thing.
    - dilyevsky 4y ago
      It isn’t popular? Like “uber and a dozen of similar sized companies and some major oss projects using it” not popular?
      - metadat 4y ago
        Uber engineering has a few gems but for every 1 gem they've hired 10 or more greedy idiots, so maybe not a great example? If you love fighting with your hands tied behind your back, choose Bazel. Otherwise, be pragmatic: Learn Make, Maven, and Gradle; then you'll be well-equipped for 95-99% of cases. Thankfully pip and npm are as straightforward as it gets.
        ctvo 4y ago
        > Uber engineering has a few gems but for every 1 gem they've hired 10 or more greedy idiots, so maybe not a great example? What an odd sprinkling of something entirely personal. > Otherwise, be pragmatic: Learn Make, Maven, and Gradle; then you'll be well-equipped for 95-99% of cases. There's a time and a place for Bazel. Very large monorepos like those at Pinterest and Uber, with cross dependencies, and written in multiple languages benefit a lot from the remote backend and distributed cache of built artifacts. Make, Maven, and Gradle, even only for JVM based projects seem to not be entirely comparable.
        dilyevsky 4y ago
        Can you explain why you bother with pip and npm if php and plain js already cover 95-99% of cases?
        metadat 4y ago
        Because I do whatever it takes to get the job done. I also appreciate what Python and Javascript offer, there are some amazing libraries and tools tied to those ecosystems.
    - dub 4y ago
      Bazel can be clunky, but not having some bazel equivalent can have very significant costs that are easy to get accustomed to or overlook. Things like engineers losing time wondering why their node dependencies weren't correctly installed, or dealing with a pre-commit check that reminds them they didn't manually regenerate the generated files, or having humans write machine-friendly configuration that's not actually human-friendly because there's no easy way to introduce custom file transformations during the build. Bazel doesn't spark joy for me and I wouldn't say I look forward to using it, but personally I would still always choose it for a codebase that's going to have multiple developers and last a long time. It's vastly easier to go with bazel from the beginning than to wish you could switch to it and realize people have already introduced a million circular dependencies and it's going to be a multi-month or multi-year process to migrate to it.
      - reissbaker 4y ago
        In my experience, Bazel is a net negative for most teams. Pretty much every JS engineer is familiar with npm; a tiny fraction are familiar with Bazel. Ditto with pip, cargo, etc. And it doesn't solve the hard part of the build process, which is distributed builds. Most of the user-perceptible value of Blaze is making builds fast by farming them out to a zillion machines — that's why it's called "blaze," because it's fast! — and Bazel doesn't do that for you. And it's clunky, and you need to teach every new hire how to use it. The juice just isn't worth the squeeze. Just write an adapter in your build infra for the well known tools and be done with it. You'll get much more value putting work into something else, like code review tools, testing, dev environments, staging...
        dilyevsky 4y ago
        There are many remote build backends for bazel available - some open source and some proprietary (EngFlow and Google’s RBE). Even without a build farm it will still be a huge performance boost due to caching - something make and co (recommended in sibling thread) can not do along with a bunch of other stuff that bazel does for you.
        4y ago
    - sleepydog 4y ago
      As a googler, I agree with this. Blaze itself is not that remarkable. What makes it good is that it is the only tool you need to use to build anything, in any language, in Google's monorepo, and that build failures are (usually) aggressively corrected. Plenty of build systems these days support varying levels of hermetic builds. The important thing to copy from Google here is the discipline and consistency around using one build tool and fixing build failures, not the specific tool itself.
  - dub 4y ago
    For those of us stuck on AWS it's sad not having BigQuery, but the thing that really gets me is not having Dataflow Most of industry still seems unaware that no-knobs data query and pipeline systems even exist. If I only had a dollar for every time I saw a PR tweaking the memory settings of some Spark job or hive query that stopped running as the input data grew.... I'd love to see more people write their workflows using the Apache Beam API so they'll have the option to switch to a no-knobs, scalable pipeline engine in the future even if they're not using one today.
- dilyevsky 4y ago
  * Build farm consisting of (tens?) of thousands of servers * Automated presubmit/bisect queues * Well built code review ui and not like github On this topic, most companies will give up on “rigorous code reviews”.
  - brightball 4y ago
    I’m curious if you can think of any tools with a well built code review UI?
    - rockwotj 4y ago
      Our company uses codeapprove.com to get a critique (Google's internal review tool) like experience. It's not all the way there, but miles ahead of GitHub
      - brightball 4y ago
        Thanks, I’ll check it out!
        codeapprove 4y ago
        CodeApprove founder here! If you want to chat, my email is in my HN profile.
        brightball 4y ago
        Thanks!
      - gruez 4y ago
        Based on the screenshots on codeapprove.com it looks similar to gerrit[1]. It also looks like gerrit is used for several google projects (eg. android, chromium), so overall it looks like a potential replacement as well. Are there any features that are in codeapprove.com/critique that aren't in gerrit? [1] https://en.wikipedia.org/wiki/Gerrit_(software) https://en.wikipedia.org/wiki/Gerrit_(software)
- skybrian 4y ago
  I don't think an enormous company-wide filesystem matters when you're not an enormous company and you're not committing to fixing downstream dependencies maintained by other teams (or, often, by nobody) when you break them. Go's approach to versioning and dependencies works better when you don't have or want a monorepo managed by a single organization. Let unmaintained code sit, using the dependencies it was using at the time, until someone decides to maintain it.
- codeapprove 4y ago
  One obvious benefit of the internal tools at Google is that they're all integrated with each other. So any Googler can quickly: 1. Search another team's code 2. Quickly jump into a Cloud IDE to make a change 3. Build and test the code, because everyone uses one build system 4. Send that code off for review and have the review tool automatically select the right reviewers and run the right tests For a simple change (let's say a typo fix) all of the above can happen in a single browser tab in under 5 minutes even if you've never been in that part of the code before. ---- While I'm here a shameless but relevant plug: if you're looking for a Google-like code review experience on GitHub check out https://codeapprove.com https://codeapprove.com
  - zoomzoom 4y ago
    The hardest part of this cycle for most teams are steps 2 & 3, getting a consistent development environment and build system across projects just does not happen. The loop is interrupted by the process of reviewing outdated docs, finding the people with the knowledge, and chatting with them to get a development environment running and helping with deploys to various QA/test environments before merging. At Coherence, we agree that code search and code review, along with automated code intelligence, are a huge step forward. Tools like sourcegraph, graphite, codeapprove are all solving important problems in the cycle. But we believe that the general approach of “building” vs “buying” development infrastructure for these key steps is holding teams back. Check us out at withcoherence.com if these sound like problems your team faces, too! (Disclosure, I’m a cofounder)
bayareabadboy 4y ago
I’ve never worked at google, but if I exclusively used other search engines for the rest of my life, would I, too, be an ex-Googler?
- deleted 4y ago
  [deleted]
- CobrastanJorji 4y ago
  Only if you frequently brought up the fact that you used to use Google.
  - Waterluvian 4y ago
    And in a way where you were convinced it mattered.
- bhawks 4y ago
  Maybe you should try applying for a job on their natural language understanding team.
- resoluteteeth 4y ago
  You can continue to use google and still be an ex-googler as long as you search for people you were previously in a relationship with.
  - monktastic1 4y ago
    You don't even have to search for your exes. You used to use Google. You still do, but you used to, too.
    - dotancohen 4y ago
      That would be like calling your wife "the ex-wife". Though, to be fair, I did once purchase an ex-parrot.
no_wizard 4y ago
I’ve hear a lot about the amazing internal tools at Google but why on earth is GCP such a mess outside Firebase? You’d think they would lead in developer experience from the way people talk about their internal tools but it’s just as bad as AWS: opaque, not enough examples and hard to figure out what services actually do
- scarmig 4y ago
  1) GCP/AWS/Azure etc all do a lot of things. Hundreds of services each? It's tricky to build out a good UX that exposes everything they do; and different customers all have very different workflows. 2) For awhile Google wanted to move more new development onto GCP (an effort called "Google on Google"), but it had limited traction and has been abandoned AFAIK, because internal tooling, infrastructure, and devex was just so much better than what GCP had to offer.
  - no_wizard 4y ago
    1) is a platform problem though and honestly I don’t think it’s that complex to be able to explain in plan English what your services do and who on a high level should use them. It’s Byzantine on purpose I feel like. 2) feels like this just proves my point
  - shadowgovt 4y ago
    Google on Google basically died as a result of a specific attack. A group of malicious actors over the holiday season attempted to compromise all of Tor by running a truly massive number of GCP nodes to succeed in a majority-control attack. Site reliability discovered how positively primitive the tooling was to mitigate such an attack... Not only was the GCP tooling not ready for prime time on an attack of that scale, but they were not able to use the regular tooling that they were accustomed to because GCP instances were encrypted for user privacy reasons. And one of the things that sets Google apart in terms of its corporate design is that site reliability has the leadership authority to tell management all the way up the chain "no."
    - ericbarrett 4y ago
minighost 4y ago
The author is absolutely right. codesearch is amazing. Blaze/bazel is amazing. When will sourcegraph support bazel repos? :-)
- beyang 4y ago
  Ironically, some of the steps that Bazel takes to make builds hermetic make it more difficult to integrate with to pull the information we need out of the build for precise code navigation. But we're working on it :) If this is an area of interest to you, pop into our community Discord and say hello!
  - gravypod 4y ago
    Have you checked out Kythe? It seems possible to use their existing action magic to pull out build data.
- technion 4y ago
  Bazel often comes up as an awesome aspect of Google's tooling but as far as I can see it's open source and hasn't really caught on elsewhere. Is there something I'm missing about why?
  - joatmon-snoo 4y ago
    Bazel by itself is not nearly as useful as Blaze (the Google-internal version), in large part because of how standardized the internal ecosystem was around Blaze. A few reasons (non-exhasutive list): - Blaze defaults to building everything remotely, on a massive farm of build servers. (This farm was sizeable enough to have its own capacity planning teams. Yes, plural.) There are a few startups trying to clone this SaaS for Bazel, but it is - predictably - not easy. - Since all builds were remote, build _telemetry_ was also uploaded and available on the intranet. This meant that if you wanted to ask someone "hey, my build failed, can you help", you could literally send them a link "https://google-build-system/00000-11111-22222-the-rest-of-my-build-id https://google-build-system/00000-11111-22222-the-rest-of-my..." with all the information about your build: the thing(s) you were trying to build, the options you were building with, all the foreign keys you needed to look up build timing data, etc. You could also query company-wide build telemetry data to see what people were building more frequently than others, if specific areas needed engineering investment, etc. - Since builds were remote, they pretty much had to be hermetic and reproducible, which meant platform teams - e.g. C++ team, Java team, Python team, Go team, etc. - could make infra-level changes and test out the effects on various teams. This is a very Google-scale specific need: allow another team to do your infra maintenance for you, i.e. upgrade the Python runtime underneath you. (Yes, this comes with caveats. Google eng tends to live at head, so while there are "my team needs to control our rollout to the latest infra version" concerns, there is a consistent mandate for most teams that "upgrading to the latest infra that we run on" is a priority.) The big place where problems came up was when source code needed to exist _outside_ of google3: open-source repos, for instance, very frequently have to do a delicate dance of figuring out (1) is their primary source of truth going to be inside google3, where the tools/infra are uniform and Better TM, or on github/etc, where they have to configure their own test stack, CI stack, etc., and (2) what are the downstream implications of decision (1).
jsnell 4y ago
(2020) Original discussion: https://news.ycombinator.com/item?id=25217291 https://news.ycombinator.com/item?id=25217291
doodlesdev 4y ago
Related: The new GitHub Code search in testing has already completely changed my workflow, not only allowing me to search code within my codebase but on the whole of GitHub which allows me to find documentation, examples, and other kinds of stuff you'd never find by using Google or any other search engine. It's also great because I know where the code comes from, what's the license, who made it, and in what context. I really hate dependency on GitHub for workflows but Microsoft is nailing the dev tools people are looking for and I think taking a look at what happens inside of big tech dev tooling is always interesting to see what we can build in the wild to improve developers ergonomics.
- ffhhj 4y ago
  Can we finally search those Awesome lists and get decently rendered markdown? https://github.com/sindresorhus/awesome/search?q=markdown https://github.com/sindresorhus/awesome/search?q=markdown
omoikane 4y ago
I wonder if other large companies also have great development experiences, and we just don't hear about those as often? For example, I would have expected Microsoft's internal tools to be at least at parity with what they sell. I have heard certain large companies having very a fragmented development culture (every team does their own thing, no central repository), so maybe Google's tools being particularly refined is the result of most of the company sharing the same tools.
- mmiyer 4y ago
  Amazon's internal tooling has everything described here. I can't say if it's better or worse than Google, but every team uses the same internal tooling for building and testing, code reviews, deployment, issue tracking etc. All code is also searchable in one place. The tooling is generally quite good too IMO.
  - angarg12 4y ago
    Overall I'd say they are above industry average, but quality is all over the place. Some of the tools are old and carry a lot of baggage. Migrations are incredibly hard and there seems to be an underinvestment in modernization and developer experience.
  - vineyardmike 4y ago
    Used both. Amazon is worse than google but only by a bit and where they’re better they’re much better. Blaze>Brazil, but pipelines has no real comparison at G. Metric Portal/Carnaval also come to mind as having no real competition. Meanwhile AWS/CloudFormation or Apollo et al are way more annoying to work with than the much more strict and uniform G alternatives. Anything data related (company wide data query tools, etc like Plx) is something that Amazon just doesn’t have. Amazons approach is to let each team use their own combo of tools, all centered around the Pipelines workflow, while G has one set of tools and everyone uses it the same way (comparatively).
    - shepherdjerred 4y ago
netik 4y ago
All of these words would have more meaning if they were not a shameless plug disguised as a important blog post
- deleted 4y ago
  [deleted]
mupuff1234 4y ago
A bit of a clickbait title given that his LinkedIn profile says he spent 1 month as in intern at Google.
- deleted 4y ago
  [deleted]
- ihateolives 4y ago
  And here I am not having 2 months at company Foo in my CV because it doesn't feel quite right. I left because the position was not what was advertised, and while I did learn some of their processes but not enough to call myself ex-Fooer.
- anonred 4y ago
  Summer interns using “ex-Company” is a clever hack but so infuriating to me. It’s like vacationing in France for 3 months and calling yourself French. There’s a special place for people who use it to be misleading when advertising their products. (A very popular iOS app dev did this many years ago and I’ll never use their app out of principle.)
- smeagull 4y ago
  I'm an ex-Googler too. These days I use bing.
  - chetanbhasin 4y ago
    Out of curiosity, what made you switch from Google to Bing?
    - dogleash 4y ago
      I use bing whenever I need a search that sticks closer to exact keywords instead of google's attempt to know what you want better than you do. Good for when you need to cut through SEO-ed spam that's in the vague area of the topic you're searching for, but your specific topic has a tiny number of people writing about it and they're all on low-profile sites.
sthupukari 4y ago
For code review: At my previous company we used Phacility/Phabricator but unfortunately it's no longer maintained[1]. Both my previous and current companies have since switched to using Graphite[2], which has delivered a pretty compelling stacked PR and code review experience. [1] https://admin.phacility.com/phame/post/view/11/phacility_is_winding_down_operations/ https://admin.phacility.com/phame/post/view/11/phacility_is_... [2] https://graphite.dev/ https://graphite.dev/
- mdaniel 4y ago
  https://graphite.dev/ https://graphite.dev/ > Join the waitlist :-( Since you seem to be in the beta, you may want to let them know their Homebrew tap is pointing to the wrong GH org: https://github.com/withgraphite/homebrew-tap/blob/master/Formula/graphite.rb#L6 https://github.com/withgraphite/homebrew-tap/blob/master/For...
- surajrmal 4y ago
  How does it compare to using Gerrit? It doesn't come with a custom cli wrapper around git, but the code review experience is getting fairly close to critique these days. Self hosting, GitHub wrappers, and paid hosts are all available so there is a lot of flexibility.
- hashar 4y ago
  Phabricator is no more actively maintained but its author (Evan) is still doing some improvements and recently applied a security fix. The latest release https://secure.phabricator.com/w/changelog/2022.15/ https://secure.phabricator.com/w/changelog/2022.15/ There is also a (blessed) community fork at https://we.phorge.it/ https://we.phorge.it/
- n42 4y ago
  Oh my god, it finally happened! I've been waiting for someone to make something like this for ages. The best part is I can use this for my own work without forcing my team to. Thanks for sharing! So far, it seems pretty good.
- mikevm 4y ago
  ReviewBoard is free and very good.
tomthumb 4y ago
There used to be SourceInsight in the MS Windows world.
29athrowaway 4y ago
If you want to have a taste of what those tools look like (kind of) https://source.chromium.org/chromium https://source.chromium.org/chromium https://chromium-review.googlesource.com https://chromium-review.googlesource.com
zeroonetwothree 4y ago
> the dev tools inside Google are the most advanced in the world I think arguably Facebook/Meta has them beat.
- googlryas 4y ago
  What does one have that the other doesn't? It doesn't make sense to me why there would be any real difference because of the cross pollination of employees
  - thurn 4y ago
    I can't immediately think of any really big differences, and the ones that do exist are more down to cultural differences (e.g. Google has a much more involved/rigorous code review process, which is reflected in their tools). Mercurial is definitely better than Perforce/Piper, but IIRC Google now actually has a very similar mercurial frontend wrapper that a lot of employees use.
aaomidi 4y ago
Bazel is pretty broken outside of google. The go bazel lib is still broken with new versions of go.
zx8080 4y ago
Ex-google __intern__.
gumby 4y ago
> The Google diaspora has seeded so many other organizations with... But adapting to programming outside of Google can be tough, especially when you've come to rely on tools you no longer have at your disposal. You also don’t have problems to solve like google’s. Use the tooling appropriate to the task. I find this problem with non-programmer ex googlers as well. They are so used to having so much infrastructure and opaque/screwed up objectives they’ve been given to meet that they often struggle in a startup environment (where the goals are merely “let’s get this button to show up 100% of the time,”, “let’s store the data for our 10k customers”, or “let’s give our phone number to these key customers so if there’s a problem they can call us right away”, or “figure out the most important thing that’s keeping our sign ups from converting and fix it — this week”).
- stingraycharles 4y ago
  Well, Google is a large enterprise, and in my experience most people that primarily worked at enterprises have problems adapting to working for startups, for the exact reasons you describe. I don’t think that’s unique to Google at all.
  - gumby 4y ago
    I don’t think it’s unique to google either except from the standpoint that they have cultivated a reputation for genius and that causes cargo-culting into places where they can’t succeed. I remember my gf working for a company of about 500 ppl (so not a startup) who, shortly after she joined brought in a new CEO, an ex googler. In fact it had been the only place he’d worked. When I met him it was clear he 1 - was quite smart and 2 - had no idea how the company worked or why ppl bought its product. And 16 months later, he was gone, along with the ppl he’d brought into the exec team, and the company was losing even more money. But by that point my gf was long gone.
- deleted 4y ago
  [deleted]
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
beyang 4y ago
Author of the post here. Thought it might be of interest to call out some observations about how the world has changed since I wrote this: 1. Code search is more mainstream. It used to be most of Sourcegraph's sales came from ex-Googlers missing code search, but nowadays, most of our new revenue originates from dev teams just struggling with the challenge of big codebases. 2. Investment in dev tools has only accelerated. A lot of pandemic beneficiaries (virtual events, tele-medicine, at-home exercise equipment, crypto) are now reversing, but dev tools growth continues to be strong. 3. Bazel adoption has grown. Were I writing the post today, I would have had a category for monorepo tooling that included companies like EngFlow and Turborepo. 4. Kubernetes is still dominant for container orchestration, but serverless and managed services seem to be gaining more adoption. Classic tension between customer focus and open standards—will be interesting to see how we trade off dev-ex vs proprietary platform dependency. Since this post was first published, we've also made a lot of advances at Sourcegraph and are looking for people who want to play around with some new experimental search and code intel features. Come say hi in our Discord! https://discord.gg/SSCBGByJeu https://discord.gg/SSCBGByJeu
dinobones 4y ago
Was this person actually a Xoogler…? Who actually writes code in critique, or would say that they use “blaze” to test software..?
- schreiaj 4y ago
  Protobuf was really nice to do in critique, the new vs code version of critique they were rolling out also was a pretty solid Golang environment for the brief bit I did there. But yeah, for the day to day java I was writing IntelliJ was way better.
onion2k 4y ago
Technically this should mean hiring people from Google is a bad idea because they won't be as productive without access to the tools that make them so capable at Google...
- exitheone 4y ago
  They will probably be just as productive as all the other employees but they will be sightly unhappy about how incredibly lacking the non-google tooling is because they know how productive someone could be. Source: Googler but still writing lots of code outside of Google and constantly think to myself "why is this so needlessly hard? Why does nothing integrate well?"
  - onion2k 4y ago
    That translates to "They'll be as good as others, but far more likely to leave" doesn't it? Still not a good signal for hiring.
    - exitheone 4y ago
      I'd rather say that they are far more likely to help improve your tooling and productivity because they have been exposed to Google tooling and know what development _could_ be like. Even if they just do incremental improvements that others would not consider.
sawyna 4y ago
I think Google's tooling works best primarily because it's uniform, not because it's the best for all use cases. Yes, they are efficient but that efficiency is worth it at Google scale and probably not required for everyone. Blaze is cool, but do you want to do it for every small project? It's actually a bit annoying for small projects, the verbosity of it. Granted that it helps optimise a lot of downstream workflows and enables pretty good tooling, but you wouldn't need those in startups not operating at that scale. Startups need speed over accuracy. Google needs accuracy over speed. Source: Acquired from startup to Google.
dijit 4y ago
Relatedly, I was reading software engineering at google and came across the chapter about being able to find things; they have a URL shortener internally that works a little like AOL keywords from back in the day it sounded like. With human readable URLs so things like the employee handbook could easily be found at “go/handbook” or if you were looking for information on blaze then “go/blaze” would link to the docs. So, I was trying to find a URL shortnener which allowed for pretty links similar to what the internal “go/“ site supports. I couldn’t find anything, and it seemed simple enough so I made it: https://github.com/dijit/redirector-rs https://github.com/dijit/redirector-rs
- rincebrain 4y ago
  The reason URL shorteners have mostly gone away, AFAICT, outside of ones that a company provides for its own links only, is that they got used a lot for spam and malware campaigns. I miss the functionality a lot for a number of not malicious use cases, but I don't see a way to avoid that problem at the moment.

203 points by goranmoomin 4y ago

Another Code Review tool discussion from 25 days ago: Show HN: Crocodile - Better code review for GitHub (https://news.ycombinator.com/item?id=31841215 https://news.ycombinator.com/item?id=31841215) Related, but from distant past: https://news.ycombinator.com/item?id=15722849 https://news.ycombinator.com/item?id=15722849 @Dang: Can we add a feature to auto link related threads in an expand\collapse section?

metadat 4y ago

Sourcegraph is one piece of the puzzle but there is still a long way to go to bring google-quality dev experience to the universe that exists outside of Big-G! * Consistent build structure * Rigorous code reviews * Ginormous company-wide shared filesystem holding all code * Point-click-drag report building * Big-query I'm sure I'm missing additional awesome internal aspects of being a developer at Google, and some parts are also horrendously bad.

mayank 4y ago
I agree! But — - BigQuery is available on GCP - Bazel is an open source Blaze - google source formatting (for java at least) is open source There are probably more…
- metadat 4y ago
  Blaze/Bazel actually sucks imo. The only good thing about it is that all of google uses that one piece of shit, making things nice and uniform and consistent. There's a reason it isn't popular outside of elgoog. IRL, every project tends to do a little bit or a lot of its own thing.
  - dilyevsky 4y ago
    It isn’t popular? Like “uber and a dozen of similar sized companies and some major oss projects using it” not popular?
    - metadat 4y ago
      Uber engineering has a few gems but for every 1 gem they've hired 10 or more greedy idiots, so maybe not a great example? If you love fighting with your hands tied behind your back, choose Bazel. Otherwise, be pragmatic: Learn Make, Maven, and Gradle; then you'll be well-equipped for 95-99% of cases. Thankfully pip and npm are as straightforward as it gets.
      - ctvo 4y ago
        > Uber engineering has a few gems but for every 1 gem they've hired 10 or more greedy idiots, so maybe not a great example? What an odd sprinkling of something entirely personal. > Otherwise, be pragmatic: Learn Make, Maven, and Gradle; then you'll be well-equipped for 95-99% of cases. There's a time and a place for Bazel. Very large monorepos like those at Pinterest and Uber, with cross dependencies, and written in multiple languages benefit a lot from the remote backend and distributed cache of built artifacts. Make, Maven, and Gradle, even only for JVM based projects seem to not be entirely comparable.
      - dilyevsky 4y ago
        Can you explain why you bother with pip and npm if php and plain js already cover 95-99% of cases?
        metadat 4y ago
        Because I do whatever it takes to get the job done. I also appreciate what Python and Javascript offer, there are some amazing libraries and tools tied to those ecosystems.
  - dub 4y ago
    Bazel can be clunky, but not having some bazel equivalent can have very significant costs that are easy to get accustomed to or overlook. Things like engineers losing time wondering why their node dependencies weren't correctly installed, or dealing with a pre-commit check that reminds them they didn't manually regenerate the generated files, or having humans write machine-friendly configuration that's not actually human-friendly because there's no easy way to introduce custom file transformations during the build. Bazel doesn't spark joy for me and I wouldn't say I look forward to using it, but personally I would still always choose it for a codebase that's going to have multiple developers and last a long time. It's vastly easier to go with bazel from the beginning than to wish you could switch to it and realize people have already introduced a million circular dependencies and it's going to be a multi-month or multi-year process to migrate to it.
    - reissbaker 4y ago
      In my experience, Bazel is a net negative for most teams. Pretty much every JS engineer is familiar with npm; a tiny fraction are familiar with Bazel. Ditto with pip, cargo, etc. And it doesn't solve the hard part of the build process, which is distributed builds. Most of the user-perceptible value of Blaze is making builds fast by farming them out to a zillion machines — that's why it's called "blaze," because it's fast! — and Bazel doesn't do that for you. And it's clunky, and you need to teach every new hire how to use it. The juice just isn't worth the squeeze. Just write an adapter in your build infra for the well known tools and be done with it. You'll get much more value putting work into something else, like code review tools, testing, dev environments, staging...
      - dilyevsky 4y ago
        There are many remote build backends for bazel available - some open source and some proprietary (EngFlow and Google’s RBE). Even without a build farm it will still be a huge performance boost due to caching - something make and co (recommended in sibling thread) can not do along with a bunch of other stuff that bazel does for you.
        4y ago
  - sleepydog 4y ago
    As a googler, I agree with this. Blaze itself is not that remarkable. What makes it good is that it is the only tool you need to use to build anything, in any language, in Google's monorepo, and that build failures are (usually) aggressively corrected. Plenty of build systems these days support varying levels of hermetic builds. The important thing to copy from Google here is the discipline and consistency around using one build tool and fixing build failures, not the specific tool itself.
- dub 4y ago
  For those of us stuck on AWS it's sad not having BigQuery, but the thing that really gets me is not having Dataflow Most of industry still seems unaware that no-knobs data query and pipeline systems even exist. If I only had a dollar for every time I saw a PR tweaking the memory settings of some Spark job or hive query that stopped running as the input data grew.... I'd love to see more people write their workflows using the Apache Beam API so they'll have the option to switch to a no-knobs, scalable pipeline engine in the future even if they're not using one today.
dilyevsky 4y ago
* Build farm consisting of (tens?) of thousands of servers * Automated presubmit/bisect queues * Well built code review ui and not like github On this topic, most companies will give up on “rigorous code reviews”.
- brightball 4y ago
  I’m curious if you can think of any tools with a well built code review UI?
  - rockwotj 4y ago
    Our company uses codeapprove.com to get a critique (Google's internal review tool) like experience. It's not all the way there, but miles ahead of GitHub
    - brightball 4y ago
      Thanks, I’ll check it out!
      - codeapprove 4y ago
        CodeApprove founder here! If you want to chat, my email is in my HN profile.
        brightball 4y ago
        Thanks!
    - gruez 4y ago
      Based on the screenshots on codeapprove.com it looks similar to gerrit[1]. It also looks like gerrit is used for several google projects (eg. android, chromium), so overall it looks like a potential replacement as well. Are there any features that are in codeapprove.com/critique that aren't in gerrit? [1] https://en.wikipedia.org/wiki/Gerrit_(software) https://en.wikipedia.org/wiki/Gerrit_(software)
skybrian 4y ago
I don't think an enormous company-wide filesystem matters when you're not an enormous company and you're not committing to fixing downstream dependencies maintained by other teams (or, often, by nobody) when you break them. Go's approach to versioning and dependencies works better when you don't have or want a monorepo managed by a single organization. Let unmaintained code sit, using the dependencies it was using at the time, until someone decides to maintain it.
codeapprove 4y ago
One obvious benefit of the internal tools at Google is that they're all integrated with each other. So any Googler can quickly: 1. Search another team's code 2. Quickly jump into a Cloud IDE to make a change 3. Build and test the code, because everyone uses one build system 4. Send that code off for review and have the review tool automatically select the right reviewers and run the right tests For a simple change (let's say a typo fix) all of the above can happen in a single browser tab in under 5 minutes even if you've never been in that part of the code before. ---- While I'm here a shameless but relevant plug: if you're looking for a Google-like code review experience on GitHub check out https://codeapprove.com https://codeapprove.com
- zoomzoom 4y ago
  The hardest part of this cycle for most teams are steps 2 & 3, getting a consistent development environment and build system across projects just does not happen. The loop is interrupted by the process of reviewing outdated docs, finding the people with the knowledge, and chatting with them to get a development environment running and helping with deploys to various QA/test environments before merging. At Coherence, we agree that code search and code review, along with automated code intelligence, are a huge step forward. Tools like sourcegraph, graphite, codeapprove are all solving important problems in the cycle. But we believe that the general approach of “building” vs “buying” development infrastructure for these key steps is holding teams back. Check us out at withcoherence.com if these sound like problems your team faces, too! (Disclosure, I’m a cofounder)

bayareabadboy 4y ago

I’ve never worked at google, but if I exclusively used other search engines for the rest of my life, would I, too, be an ex-Googler?

deleted 4y ago
[deleted]
CobrastanJorji 4y ago
Only if you frequently brought up the fact that you used to use Google.
- Waterluvian 4y ago
  And in a way where you were convinced it mattered.
bhawks 4y ago
Maybe you should try applying for a job on their natural language understanding team.
resoluteteeth 4y ago
You can continue to use google and still be an ex-googler as long as you search for people you were previously in a relationship with.
- monktastic1 4y ago
  You don't even have to search for your exes. You used to use Google. You still do, but you used to, too.
  - dotancohen 4y ago
    That would be like calling your wife "the ex-wife". Though, to be fair, I did once purchase an ex-parrot.

no_wizard 4y ago

I’ve hear a lot about the amazing internal tools at Google but why on earth is GCP such a mess outside Firebase? You’d think they would lead in developer experience from the way people talk about their internal tools but it’s just as bad as AWS: opaque, not enough examples and hard to figure out what services actually do

scarmig 4y ago
1) GCP/AWS/Azure etc all do a lot of things. Hundreds of services each? It's tricky to build out a good UX that exposes everything they do; and different customers all have very different workflows. 2) For awhile Google wanted to move more new development onto GCP (an effort called "Google on Google"), but it had limited traction and has been abandoned AFAIK, because internal tooling, infrastructure, and devex was just so much better than what GCP had to offer.
- no_wizard 4y ago
  1) is a platform problem though and honestly I don’t think it’s that complex to be able to explain in plan English what your services do and who on a high level should use them. It’s Byzantine on purpose I feel like. 2) feels like this just proves my point
- shadowgovt 4y ago
  Google on Google basically died as a result of a specific attack. A group of malicious actors over the holiday season attempted to compromise all of Tor by running a truly massive number of GCP nodes to succeed in a majority-control attack. Site reliability discovered how positively primitive the tooling was to mitigate such an attack... Not only was the GCP tooling not ready for prime time on an attack of that scale, but they were not able to use the regular tooling that they were accustomed to because GCP instances were encrypted for user privacy reasons. And one of the things that sets Google apart in terms of its corporate design is that site reliability has the leadership authority to tell management all the way up the chain "no."
  - ericbarrett 4y ago

minighost 4y ago

The author is absolutely right. codesearch is amazing. Blaze/bazel is amazing. When will sourcegraph support bazel repos? :-)

beyang 4y ago
Ironically, some of the steps that Bazel takes to make builds hermetic make it more difficult to integrate with to pull the information we need out of the build for precise code navigation. But we're working on it :) If this is an area of interest to you, pop into our community Discord and say hello!
- gravypod 4y ago
  Have you checked out Kythe? It seems possible to use their existing action magic to pull out build data.
technion 4y ago
Bazel often comes up as an awesome aspect of Google's tooling but as far as I can see it's open source and hasn't really caught on elsewhere. Is there something I'm missing about why?
- joatmon-snoo 4y ago
  Bazel by itself is not nearly as useful as Blaze (the Google-internal version), in large part because of how standardized the internal ecosystem was around Blaze. A few reasons (non-exhasutive list): - Blaze defaults to building everything remotely, on a massive farm of build servers. (This farm was sizeable enough to have its own capacity planning teams. Yes, plural.) There are a few startups trying to clone this SaaS for Bazel, but it is - predictably - not easy. - Since all builds were remote, build _telemetry_ was also uploaded and available on the intranet. This meant that if you wanted to ask someone "hey, my build failed, can you help", you could literally send them a link "https://google-build-system/00000-11111-22222-the-rest-of-my-build-id https://google-build-system/00000-11111-22222-the-rest-of-my..." with all the information about your build: the thing(s) you were trying to build, the options you were building with, all the foreign keys you needed to look up build timing data, etc. You could also query company-wide build telemetry data to see what people were building more frequently than others, if specific areas needed engineering investment, etc. - Since builds were remote, they pretty much had to be hermetic and reproducible, which meant platform teams - e.g. C++ team, Java team, Python team, Go team, etc. - could make infra-level changes and test out the effects on various teams. This is a very Google-scale specific need: allow another team to do your infra maintenance for you, i.e. upgrade the Python runtime underneath you. (Yes, this comes with caveats. Google eng tends to live at head, so while there are "my team needs to control our rollout to the latest infra version" concerns, there is a consistent mandate for most teams that "upgrading to the latest infra that we run on" is a priority.) The big place where problems came up was when source code needed to exist _outside_ of google3: open-source repos, for instance, very frequently have to do a delicate dance of figuring out (1) is their primary source of truth going to be inside google3, where the tools/infra are uniform and Better TM, or on github/etc, where they have to configure their own test stack, CI stack, etc., and (2) what are the downstream implications of decision (1).

jsnell 4y ago

(2020) Original discussion: https://news.ycombinator.com/item?id=25217291 https://news.ycombinator.com/item?id=25217291

doodlesdev 4y ago

Related: The new GitHub Code search in testing has already completely changed my workflow, not only allowing me to search code within my codebase but on the whole of GitHub which allows me to find documentation, examples, and other kinds of stuff you'd never find by using Google or any other search engine. It's also great because I know where the code comes from, what's the license, who made it, and in what context. I really hate dependency on GitHub for workflows but Microsoft is nailing the dev tools people are looking for and I think taking a look at what happens inside of big tech dev tooling is always interesting to see what we can build in the wild to improve developers ergonomics.

ffhhj 4y ago
Can we finally search those Awesome lists and get decently rendered markdown? https://github.com/sindresorhus/awesome/search?q=markdown https://github.com/sindresorhus/awesome/search?q=markdown

omoikane 4y ago

I wonder if other large companies also have great development experiences, and we just don't hear about those as often? For example, I would have expected Microsoft's internal tools to be at least at parity with what they sell. I have heard certain large companies having very a fragmented development culture (every team does their own thing, no central repository), so maybe Google's tools being particularly refined is the result of most of the company sharing the same tools.

mmiyer 4y ago
Amazon's internal tooling has everything described here. I can't say if it's better or worse than Google, but every team uses the same internal tooling for building and testing, code reviews, deployment, issue tracking etc. All code is also searchable in one place. The tooling is generally quite good too IMO.
- angarg12 4y ago
  Overall I'd say they are above industry average, but quality is all over the place. Some of the tools are old and carry a lot of baggage. Migrations are incredibly hard and there seems to be an underinvestment in modernization and developer experience.
- vineyardmike 4y ago
  Used both. Amazon is worse than google but only by a bit and where they’re better they’re much better. Blaze>Brazil, but pipelines has no real comparison at G. Metric Portal/Carnaval also come to mind as having no real competition. Meanwhile AWS/CloudFormation or Apollo et al are way more annoying to work with than the much more strict and uniform G alternatives. Anything data related (company wide data query tools, etc like Plx) is something that Amazon just doesn’t have. Amazons approach is to let each team use their own combo of tools, all centered around the Pipelines workflow, while G has one set of tools and everyone uses it the same way (comparatively).
  - shepherdjerred 4y ago

netik 4y ago

All of these words would have more meaning if they were not a shameless plug disguised as a important blog post

deleted 4y ago
[deleted]

mupuff1234 4y ago

A bit of a clickbait title given that his LinkedIn profile says he spent 1 month as in intern at Google.

deleted 4y ago
[deleted]
ihateolives 4y ago
And here I am not having 2 months at company Foo in my CV because it doesn't feel quite right. I left because the position was not what was advertised, and while I did learn some of their processes but not enough to call myself ex-Fooer.
anonred 4y ago
Summer interns using “ex-Company” is a clever hack but so infuriating to me. It’s like vacationing in France for 3 months and calling yourself French. There’s a special place for people who use it to be misleading when advertising their products. (A very popular iOS app dev did this many years ago and I’ll never use their app out of principle.)
smeagull 4y ago
I'm an ex-Googler too. These days I use bing.
- chetanbhasin 4y ago
  Out of curiosity, what made you switch from Google to Bing?
  - dogleash 4y ago
    I use bing whenever I need a search that sticks closer to exact keywords instead of google's attempt to know what you want better than you do. Good for when you need to cut through SEO-ed spam that's in the vague area of the topic you're searching for, but your specific topic has a tiny number of people writing about it and they're all on low-profile sites.

sthupukari 4y ago

For code review: At my previous company we used Phacility/Phabricator but unfortunately it's no longer maintained[1]. Both my previous and current companies have since switched to using Graphite[2], which has delivered a pretty compelling stacked PR and code review experience. [1] https://admin.phacility.com/phame/post/view/11/phacility_is_winding_down_operations/ https://admin.phacility.com/phame/post/view/11/phacility_is_... [2] https://graphite.dev/ https://graphite.dev/

mdaniel 4y ago
https://graphite.dev/ https://graphite.dev/ > Join the waitlist :-( Since you seem to be in the beta, you may want to let them know their Homebrew tap is pointing to the wrong GH org: https://github.com/withgraphite/homebrew-tap/blob/master/Formula/graphite.rb#L6 https://github.com/withgraphite/homebrew-tap/blob/master/For...
surajrmal 4y ago
How does it compare to using Gerrit? It doesn't come with a custom cli wrapper around git, but the code review experience is getting fairly close to critique these days. Self hosting, GitHub wrappers, and paid hosts are all available so there is a lot of flexibility.
hashar 4y ago
Phabricator is no more actively maintained but its author (Evan) is still doing some improvements and recently applied a security fix. The latest release https://secure.phabricator.com/w/changelog/2022.15/ https://secure.phabricator.com/w/changelog/2022.15/ There is also a (blessed) community fork at https://we.phorge.it/ https://we.phorge.it/
n42 4y ago
Oh my god, it finally happened! I've been waiting for someone to make something like this for ages. The best part is I can use this for my own work without forcing my team to. Thanks for sharing! So far, it seems pretty good.
mikevm 4y ago
ReviewBoard is free and very good.

tomthumb 4y ago

There used to be SourceInsight in the MS Windows world.

29athrowaway 4y ago

If you want to have a taste of what those tools look like (kind of) https://source.chromium.org/chromium https://source.chromium.org/chromium https://chromium-review.googlesource.com https://chromium-review.googlesource.com

zeroonetwothree 4y ago

> the dev tools inside Google are the most advanced in the world I think arguably Facebook/Meta has them beat.

googlryas 4y ago
What does one have that the other doesn't? It doesn't make sense to me why there would be any real difference because of the cross pollination of employees
- thurn 4y ago
  I can't immediately think of any really big differences, and the ones that do exist are more down to cultural differences (e.g. Google has a much more involved/rigorous code review process, which is reflected in their tools). Mercurial is definitely better than Perforce/Piper, but IIRC Google now actually has a very similar mercurial frontend wrapper that a lot of employees use.

aaomidi 4y ago

Bazel is pretty broken outside of google. The go bazel lib is still broken with new versions of go.

zx8080 4y ago

Ex-google __intern__.

gumby 4y ago

> The Google diaspora has seeded so many other organizations with... But adapting to programming outside of Google can be tough, especially when you've come to rely on tools you no longer have at your disposal. You also don’t have problems to solve like google’s. Use the tooling appropriate to the task. I find this problem with non-programmer ex googlers as well. They are so used to having so much infrastructure and opaque/screwed up objectives they’ve been given to meet that they often struggle in a startup environment (where the goals are merely “let’s get this button to show up 100% of the time,”, “let’s store the data for our 10k customers”, or “let’s give our phone number to these key customers so if there’s a problem they can call us right away”, or “figure out the most important thing that’s keeping our sign ups from converting and fix it — this week”).

stingraycharles 4y ago
Well, Google is a large enterprise, and in my experience most people that primarily worked at enterprises have problems adapting to working for startups, for the exact reasons you describe. I don’t think that’s unique to Google at all.
- gumby 4y ago
  I don’t think it’s unique to google either except from the standpoint that they have cultivated a reputation for genius and that causes cargo-culting into places where they can’t succeed. I remember my gf working for a company of about 500 ppl (so not a startup) who, shortly after she joined brought in a new CEO, an ex googler. In fact it had been the only place he’d worked. When I met him it was clear he 1 - was quite smart and 2 - had no idea how the company worked or why ppl bought its product. And 16 months later, he was gone, along with the ppl he’d brought into the exec team, and the company was losing even more money. But by that point my gf was long gone.
deleted 4y ago
[deleted]

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

beyang 4y ago

Author of the post here. Thought it might be of interest to call out some observations about how the world has changed since I wrote this: 1. Code search is more mainstream. It used to be most of Sourcegraph's sales came from ex-Googlers missing code search, but nowadays, most of our new revenue originates from dev teams just struggling with the challenge of big codebases. 2. Investment in dev tools has only accelerated. A lot of pandemic beneficiaries (virtual events, tele-medicine, at-home exercise equipment, crypto) are now reversing, but dev tools growth continues to be strong. 3. Bazel adoption has grown. Were I writing the post today, I would have had a category for monorepo tooling that included companies like EngFlow and Turborepo. 4. Kubernetes is still dominant for container orchestration, but serverless and managed services seem to be gaining more adoption. Classic tension between customer focus and open standards—will be interesting to see how we trade off dev-ex vs proprietary platform dependency. Since this post was first published, we've also made a lot of advances at Sourcegraph and are looking for people who want to play around with some new experimental search and code intel features. Come say hi in our Discord! https://discord.gg/SSCBGByJeu https://discord.gg/SSCBGByJeu

dinobones 4y ago

Was this person actually a Xoogler…? Who actually writes code in critique, or would say that they use “blaze” to test software..?

schreiaj 4y ago
Protobuf was really nice to do in critique, the new vs code version of critique they were rolling out also was a pretty solid Golang environment for the brief bit I did there. But yeah, for the day to day java I was writing IntelliJ was way better.

onion2k 4y ago

Technically this should mean hiring people from Google is a bad idea because they won't be as productive without access to the tools that make them so capable at Google...

exitheone 4y ago
They will probably be just as productive as all the other employees but they will be sightly unhappy about how incredibly lacking the non-google tooling is because they know how productive someone could be. Source: Googler but still writing lots of code outside of Google and constantly think to myself "why is this so needlessly hard? Why does nothing integrate well?"
- onion2k 4y ago
  That translates to "They'll be as good as others, but far more likely to leave" doesn't it? Still not a good signal for hiring.
  - exitheone 4y ago
    I'd rather say that they are far more likely to help improve your tooling and productivity because they have been exposed to Google tooling and know what development _could_ be like. Even if they just do incremental improvements that others would not consider.

sawyna 4y ago

I think Google's tooling works best primarily because it's uniform, not because it's the best for all use cases. Yes, they are efficient but that efficiency is worth it at Google scale and probably not required for everyone. Blaze is cool, but do you want to do it for every small project? It's actually a bit annoying for small projects, the verbosity of it. Granted that it helps optimise a lot of downstream workflows and enables pretty good tooling, but you wouldn't need those in startups not operating at that scale. Startups need speed over accuracy. Google needs accuracy over speed. Source: Acquired from startup to Google.

dijit 4y ago

Relatedly, I was reading software engineering at google and came across the chapter about being able to find things; they have a URL shortener internally that works a little like AOL keywords from back in the day it sounded like. With human readable URLs so things like the employee handbook could easily be found at “go/handbook” or if you were looking for information on blaze then “go/blaze” would link to the docs. So, I was trying to find a URL shortnener which allowed for pretty links similar to what the internal “go/“ site supports. I couldn’t find anything, and it seemed simple enough so I made it: https://github.com/dijit/redirector-rs https://github.com/dijit/redirector-rs

rincebrain 4y ago
The reason URL shorteners have mostly gone away, AFAICT, outside of ones that a company provides for its own links only, is that they got used a lot for spam and malware campaigns. I miss the functionality a lot for a number of not malicious use cases, but I don't see a way to avoid that problem at the moment.