7 ms·
Sourcegraph is one piece of the puzzle but there is still a long way to go to bring google-quality dev experience to the universe that exists outside of Big-G!
by metadat 4y ago
Sourcegraph is one piece of the puzzle but there is still a long way to go to bring google-quality dev experience to the universe that exists outside of Big-G!
* Consistent build structure
* Rigorous code reviews
* Ginormous company-wide shared filesystem holding all code
* Point-click-drag report building
* Big-query
I'm sure I'm missing additional awesome internal aspects of being a developer at Google, and some parts are also horrendously bad.
- mayank 4y agoI agree! But — - BigQuery is available on GCP - Bazel is an open source Blaze - google source formatting (for java at least) is open source There are probably more…
- metadat 4y agoBlaze/Bazel actually sucks imo. The only good thing about it is that all of google uses that one piece of shit, making things nice and uniform and consistent. There's a reason it isn't popular outside of elgoog. IRL, every project tends to do a little bit or a lot of its own thing.
- dilyevsky 4y agoIt isn’t popular? Like “uber and a dozen of similar sized companies and some major oss projects using it” not popular?
- metadat 4y agoUber engineering has a few gems but for every 1 gem they've hired 10 or more greedy idiots, so maybe not a great example? If you love fighting with your hands tied behind your back, choose Bazel. Otherwise, be pragmatic: Learn Make, Maven, and Gradle; then you'll be well-equipped for 95-99% of cases. Thankfully pip and npm are as straightforward as it gets.
- ctvo 4y ago> Uber engineering has a few gems but for every 1 gem they've hired 10 or more greedy idiots, so maybe not a great example? What an odd sprinkling of something entirely personal. > Otherwise, be pragmatic: Learn Make, Maven, and Gradle; then you'll be well-equipped for 95-99% of cases. There's a time and a place for Bazel. Very large monorepos like those at Pinterest and Uber, with cross dependencies, and written in multiple languages benefit a lot from the remote backend and distributed cache of built artifacts. Make, Maven, and Gradle, even only for JVM based projects seem to not be entirely comparable.
- dilyevsky 4y agoCan you explain why you bother with pip and npm if php and plain js already cover 95-99% of cases?
- metadat 4y agoBecause I do whatever it takes to get the job done. I also appreciate what Python and Javascript offer, there are some amazing libraries and tools tied to those ecosystems.
- dub 4y agoBazel can be clunky, but not having some bazel equivalent can have very significant costs that are easy to get accustomed to or overlook. Things like engineers losing time wondering why their node dependencies weren't correctly installed, or dealing with a pre-commit check that reminds them they didn't manually regenerate the generated files, or having humans write machine-friendly configuration that's not actually human-friendly because there's no easy way to introduce custom file transformations during the build. Bazel doesn't spark joy for me and I wouldn't say I look forward to using it, but personally I would still always choose it for a codebase that's going to have multiple developers and last a long time. It's vastly easier to go with bazel from the beginning than to wish you could switch to it and realize people have already introduced a million circular dependencies and it's going to be a multi-month or multi-year process to migrate to it.
- reissbaker 4y agoIn my experience, Bazel is a net negative for most teams. Pretty much every JS engineer is familiar with npm; a tiny fraction are familiar with Bazel. Ditto with pip, cargo, etc. And it doesn't solve the hard part of the build process, which is distributed builds. Most of the user-perceptible value of Blaze is making builds fast by farming them out to a zillion machines — that's why it's called "blaze," because it's fast! — and Bazel doesn't do that for you. And it's clunky, and you need to teach every new hire how to use it. The juice just isn't worth the squeeze. Just write an adapter in your build infra for the well known tools and be done with it. You'll get much more value putting work into something else, like code review tools, testing, dev environments, staging...
- dilyevsky 4y agoThere are many remote build backends for bazel available - some open source and some proprietary (EngFlow and Google’s RBE). Even without a build farm it will still be a huge performance boost due to caching - something make and co (recommended in sibling thread) can not do along with a bunch of other stuff that bazel does for you.
- 4y ago
- sleepydog 4y agoAs a googler, I agree with this. Blaze itself is not that remarkable. What makes it good is that it is the only tool you need to use to build anything, in any language, in Google's monorepo, and that build failures are (usually) aggressively corrected. Plenty of build systems these days support varying levels of hermetic builds. The important thing to copy from Google here is the discipline and consistency around using one build tool and fixing build failures, not the specific tool itself.
- dub 4y agoFor those of us stuck on AWS it's sad not having BigQuery, but the thing that really gets me is not having Dataflow Most of industry still seems unaware that no-knobs data query and pipeline systems even exist. If I only had a dollar for every time I saw a PR tweaking the memory settings of some Spark job or hive query that stopped running as the input data grew.... I'd love to see more people write their workflows using the Apache Beam API so they'll have the option to switch to a no-knobs, scalable pipeline engine in the future even if they're not using one today.
- dilyevsky 4y ago* Build farm consisting of (tens?) of thousands of servers * Automated presubmit/bisect queues * Well built code review ui and not like github On this topic, most companies will give up on “rigorous code reviews”.
- brightball 4y agoI’m curious if you can think of any tools with a well built code review UI?
- rockwotj 4y agoOur company uses codeapprove.com to get a critique (Google's internal review tool) like experience. It's not all the way there, but miles ahead of GitHub
- brightball 4y agoThanks, I’ll check it out!
- codeapprove 4y agoCodeApprove founder here! If you want to chat, my email is in my HN profile.
- brightball 4y agoThanks!
- gruez 4y agoBased on the screenshots on codeapprove.com it looks similar to gerrit[1]. It also looks like gerrit is used for several google projects (eg. android, chromium), so overall it looks like a potential replacement as well. Are there any features that are in codeapprove.com/critique that aren't in gerrit? [1] https://en.wikipedia.org/wiki/Gerrit_(software) https://en.wikipedia.org/wiki/Gerrit_(software)
- skybrian 4y agoI don't think an enormous company-wide filesystem matters when you're not an enormous company and you're not committing to fixing downstream dependencies maintained by other teams (or, often, by nobody) when you break them. Go's approach to versioning and dependencies works better when you don't have or want a monorepo managed by a single organization. Let unmaintained code sit, using the dependencies it was using at the time, until someone decides to maintain it.
- codeapprove 4y agoOne obvious benefit of the internal tools at Google is that they're all integrated with each other. So any Googler can quickly: 1. Search another team's code 2. Quickly jump into a Cloud IDE to make a change 3. Build and test the code, because everyone uses one build system 4. Send that code off for review and have the review tool automatically select the right reviewers and run the right tests For a simple change (let's say a typo fix) all of the above can happen in a single browser tab in under 5 minutes even if you've never been in that part of the code before. ---- While I'm here a shameless but relevant plug: if you're looking for a Google-like code review experience on GitHub check out https://codeapprove.com https://codeapprove.com
- zoomzoom 4y agoThe hardest part of this cycle for most teams are steps 2 & 3, getting a consistent development environment and build system across projects just does not happen. The loop is interrupted by the process of reviewing outdated docs, finding the people with the knowledge, and chatting with them to get a development environment running and helping with deploys to various QA/test environments before merging. At Coherence, we agree that code search and code review, along with automated code intelligence, are a huge step forward. Tools like sourcegraph, graphite, codeapprove are all solving important problems in the cycle. But we believe that the general approach of “building” vs “buying” development infrastructure for these key steps is holding teams back. Check us out at withcoherence.com if these sound like problems your team faces, too! (Disclosure, I’m a cofounder)