6 ms·

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing.

jeffbee 3y ago
It's not just that. Livegrep isn't just a pale imitation of something inside Google. It's totally unrelated in implementation, capabilities, and use case.
scubbo 3y ago
I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)
- mdaniel 3y ago
  My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?
  - shepherdjerred 3y ago
    Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from the CI/CD to the code review tool to your local builds use the same build configuration with Brazil. All software packages in Brazil are built from source on Amazon's gigantic fleet of build servers. Builds are cached, so even though Amazon builds its own version of Make, Java, etc., these are all built and cached by the build servers and downloaded. A simple Java application at Amazon might have hundreds of dependencies (because you'll need to build Java from scratch), but since this is all cached you don't have to wait very long. Lastly, you have Pipelines which is their internal CI/CD tool which integrates naturally with Brazil + the build fleet. It can deploy to their internal fleet with Apollo, or to AWS Lambda, S3 buckets, etc. In all, everything is just very well integrated. I haven't seen anything come close to what you get internally at Amazon.
    - mdaniel 3y ago
      so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like # in some "app-runtimes" project, they assemble your app's runtime cat > Dockerfile <<FOO FROM public.ecr.aws/runtimes/runtime-y:2.0 ADD https://cache.example/library-x/1.1/library-x-1.1.jar FOO tar -cf - Dockerfile | podman build -t public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 - # then you consume it in your project cat > Dockerfile <<FOO FROM public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 ADD ./app-1.0.jar FOO cat > .gitlab-ci.yml <<'YML' # you can also distribute artifacts other than just docker images # https://docs.gitlab.com/ee/user/packages/package_registry/supported_package_managers.html cook image: stage: package script: # or this https://docs.gitlab.com/ee/topics/autodevops/customize.html#customize-buildpacks-with-cloud-native-buildpacks - podman build -t $CI_REGISTRY_IMAGE . # https://docs.gitlab.com/ee/user/packages/#container-registry is built in - podman push $CI_REGISTRY_IMAGE review env: stage: staging script: auto-devops deploy # for free: https://docs.gitlab.com/ee/ci/review_apps/index.html environment: name: review/${CI_COMMIT_REF_SLUG} url: https://${CI_ENVIRONMENT_SLUG}.int.example on_stop: teardown-review teardown-review: stage: staging script: auto-devops stop when: manual environment: name: review/${CI_COMMIT_REF_SLUG} action: stop ... etc ... YML and then, yadda, yadda, blue-green, incremental rollout <https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml#L81-84 https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/...>, feature flags <https://docs.gitlab.com/ee/operations/feature_flags.html https://docs.gitlab.com/ee/operations/feature_flags.html>, error capture <https://docs.gitlab.com/ee/operations/error_tracking.html#integrated-error-tracking https://docs.gitlab.com/ee/operations/error_tracking.html#in...>, project-managed provisioning <https://docs.gitlab.com/ee/user/infrastructure/iac/#integrate-your-project-with-terraform https://docs.gitlab.com/ee/user/infrastructure/iac/#integrat...>, on call management <https://docs.gitlab.com/ee/operations/incident_management/ https://docs.gitlab.com/ee/operations/incident_management/>, on call runbooks <https://docs.gitlab.com/ee/user/project/clusters/runbooks/index.html https://docs.gitlab.com/ee/user/project/clusters/runbooks/in...> you can orchestrate all that from ~~Slack~~ Chime :-D if you're into that kind of thing https://docs.gitlab.com/ee/ci/chatops/ https://docs.gitlab.com/ee/ci/chatops/
- sdesol 3y ago
  > (I would be _delighted_ for someone to "Umm actually" me by providing a great product!) I think the issue is, nobody would be willing to pay for a good solution since they usually come with a steep maintenance cost. I wouldn't be surprised if the in-house CD team at Amazon were putting out fires every week/month behind the scene.
- xyzzy_plugh 3y ago
  NixOS/nixpkgs is about the closest thing you'll find in the wild. You have to squint a bit, I'll admit.
- tripdout 3y ago
  Is it true that teams don't do branches in source control at all? Just publishing a CR?
  - scubbo 3y ago
    Yep! Well, a branch is created under-the-hood, and you can still check it out locally if you want to - but the standard code contribution workflow doesn't require you to think or work in those terms at all.
- zaphirplane 3y ago
  AWS has services around pipeline and deploy, right ?
alienchow 3y ago
Just left Google a few months ago. My take is that there's a difference between a company that is willing to invest money into EngProd endeavors, and a company that uses SaaS for everything. While I can understand that most companies don't have the financial means to invest heavily into EngProd, the outcome is that the tightly integrated development experience in the former is far superior. Code Search is definitely #2 on the list of things I miss the most.
- fragmede 3y ago
  What's #1? Memegen?
  - voiceblue 3y ago
    The business end of gThanks
  - alienchow 3y ago
    Cheesy answer but, the people.
elevatedastalt 3y ago
You would be better off actually trying to understand those sentiments instead of posting sarcastic replies on HN. The sort of tight knit integration and developer-focus that internal tools at developer-friendly companies like Google has cannot be matched by clobbering together 50 different SaaS products, half of which will probably run out of funding in 3 years. You literally have entire companies just LARPing internal tools that Google has because they are just that good. Glean is literally Moma. There's really nothing like Critique or Buganizer.

jeffbee 3y ago

It's not just that. Livegrep isn't just a pale imitation of something inside Google. It's totally unrelated in implementation, capabilities, and use case.

scubbo 3y ago

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

mdaniel 3y ago
My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?
- shepherdjerred 3y ago
  Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from the CI/CD to the code review tool to your local builds use the same build configuration with Brazil. All software packages in Brazil are built from source on Amazon's gigantic fleet of build servers. Builds are cached, so even though Amazon builds its own version of Make, Java, etc., these are all built and cached by the build servers and downloaded. A simple Java application at Amazon might have hundreds of dependencies (because you'll need to build Java from scratch), but since this is all cached you don't have to wait very long. Lastly, you have Pipelines which is their internal CI/CD tool which integrates naturally with Brazil + the build fleet. It can deploy to their internal fleet with Apollo, or to AWS Lambda, S3 buckets, etc. In all, everything is just very well integrated. I haven't seen anything come close to what you get internally at Amazon.
  - mdaniel 3y ago
    so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like # in some "app-runtimes" project, they assemble your app's runtime cat > Dockerfile <<FOO FROM public.ecr.aws/runtimes/runtime-y:2.0 ADD https://cache.example/library-x/1.1/library-x-1.1.jar FOO tar -cf - Dockerfile | podman build -t public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 - # then you consume it in your project cat > Dockerfile <<FOO FROM public.ecr.aws/app-runtimes/app-1.0-runtime-build:20240410 ADD ./app-1.0.jar FOO cat > .gitlab-ci.yml <<'YML' # you can also distribute artifacts other than just docker images # https://docs.gitlab.com/ee/user/packages/package_registry/supported_package_managers.html cook image: stage: package script: # or this https://docs.gitlab.com/ee/topics/autodevops/customize.html#customize-buildpacks-with-cloud-native-buildpacks - podman build -t $CI_REGISTRY_IMAGE . # https://docs.gitlab.com/ee/user/packages/#container-registry is built in - podman push $CI_REGISTRY_IMAGE review env: stage: staging script: auto-devops deploy # for free: https://docs.gitlab.com/ee/ci/review_apps/index.html environment: name: review/${CI_COMMIT_REF_SLUG} url: https://${CI_ENVIRONMENT_SLUG}.int.example on_stop: teardown-review teardown-review: stage: staging script: auto-devops stop when: manual environment: name: review/${CI_COMMIT_REF_SLUG} action: stop ... etc ... YML and then, yadda, yadda, blue-green, incremental rollout <https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml#L81-84 https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/...>, feature flags <https://docs.gitlab.com/ee/operations/feature_flags.html https://docs.gitlab.com/ee/operations/feature_flags.html>, error capture <https://docs.gitlab.com/ee/operations/error_tracking.html#integrated-error-tracking https://docs.gitlab.com/ee/operations/error_tracking.html#in...>, project-managed provisioning <https://docs.gitlab.com/ee/user/infrastructure/iac/#integrate-your-project-with-terraform https://docs.gitlab.com/ee/user/infrastructure/iac/#integrat...>, on call management <https://docs.gitlab.com/ee/operations/incident_management/ https://docs.gitlab.com/ee/operations/incident_management/>, on call runbooks <https://docs.gitlab.com/ee/user/project/clusters/runbooks/index.html https://docs.gitlab.com/ee/user/project/clusters/runbooks/in...> you can orchestrate all that from ~~Slack~~ Chime :-D if you're into that kind of thing https://docs.gitlab.com/ee/ci/chatops/ https://docs.gitlab.com/ee/ci/chatops/
sdesol 3y ago
> (I would be _delighted_ for someone to "Umm actually" me by providing a great product!) I think the issue is, nobody would be willing to pay for a good solution since they usually come with a steep maintenance cost. I wouldn't be surprised if the in-house CD team at Amazon were putting out fires every week/month behind the scene.
xyzzy_plugh 3y ago
NixOS/nixpkgs is about the closest thing you'll find in the wild. You have to squint a bit, I'll admit.
tripdout 3y ago
Is it true that teams don't do branches in source control at all? Just publishing a CR?
- scubbo 3y ago
  Yep! Well, a branch is created under-the-hood, and you can still check it out locally if you want to - but the standard code contribution workflow doesn't require you to think or work in those terms at all.
zaphirplane 3y ago
AWS has services around pipeline and deploy, right ?

alienchow 3y ago

Just left Google a few months ago. My take is that there's a difference between a company that is willing to invest money into EngProd endeavors, and a company that uses SaaS for everything. While I can understand that most companies don't have the financial means to invest heavily into EngProd, the outcome is that the tightly integrated development experience in the former is far superior. Code Search is definitely #2 on the list of things I miss the most.

fragmede 3y ago
What's #1? Memegen?
- voiceblue 3y ago
  The business end of gThanks
- alienchow 3y ago
  Cheesy answer but, the people.

elevatedastalt 3y ago

You would be better off actually trying to understand those sentiments instead of posting sarcastic replies on HN. The sort of tight knit integration and developer-focus that internal tools at developer-friendly companies like Google has cannot be matched by clobbering together 50 different SaaS products, half of which will probably run out of funding in 3 years. You literally have entire companies just LARPing internal tools that Google has because they are just that good. Glean is literally Moma. There's really nothing like Critique or Buganizer.