3 ms·

Cool project. I'm always wondering though, if those optimizations are really needed or if they are created by engineers that want to have fun and do premature

Cool project. I'm always wondering though, if those optimizations are really needed or if they are created by engineers that want to have fun and do premature optimizations. Ever wondered why a "WebApp" like Uber needs so many engineers and new projects?

barnabee 8y ago
I equally wonder where we'd be if no-one ever indulged in a premature optimisation…
- warp_factor 8y ago
  premature optimization is super fun. I did it a lot and I would advise to engineers to do it if they can. But when I look at companies like Airbnb and Uber for example, I cannot really understand why they require that type of tools. Yes they are big, even very big, but nowhere the size of a Facebook or a Google. Most of those projects seem to be engineers having fun at work, and marketing themselves by publishing cool blog posts.
puzzle 8y ago
I was at Google when this became the standard more than a decade ago (https://www.usenix.org/sites/default/files/conference/protected-files/lisa_2014_talk.pdf https://www.usenix.org/sites/default/files/conference/protec...). For jobs where you had tens or hundreds of replicas, i.e. for services with lots of traffic, it made the rollout of new binaries in Borg much faster, especially if you were deploying to a cluster in Asia and your packages were built and stored in the US. Even in the US, this being before Firehose/Jupiter fabrics, network congestion had had noticeable impact. MPM traffic was low priority, so for large enough binaries (and most of them were, because of static linking) you could easily see a Borg task spend seconds or even tens of seconds in "downloading" state if the borglet hadn't already seen and cached locally that version of the package. It was about time this became available also for Docker and Kubernetes.
thekhatribharat 8y ago
WhatsApp is a good example, they had a small engineering team handling tens of millions of users.

barnabee 8y ago

I equally wonder where we'd be if no-one ever indulged in a premature optimisation…

warp_factor 8y ago
premature optimization is super fun. I did it a lot and I would advise to engineers to do it if they can. But when I look at companies like Airbnb and Uber for example, I cannot really understand why they require that type of tools. Yes they are big, even very big, but nowhere the size of a Facebook or a Google. Most of those projects seem to be engineers having fun at work, and marketing themselves by publishing cool blog posts.

puzzle 8y ago

I was at Google when this became the standard more than a decade ago (https://www.usenix.org/sites/default/files/conference/protected-files/lisa_2014_talk.pdf https://www.usenix.org/sites/default/files/conference/protec...). For jobs where you had tens or hundreds of replicas, i.e. for services with lots of traffic, it made the rollout of new binaries in Borg much faster, especially if you were deploying to a cluster in Asia and your packages were built and stored in the US. Even in the US, this being before Firehose/Jupiter fabrics, network congestion had had noticeable impact. MPM traffic was low priority, so for large enough binaries (and most of them were, because of static linking) you could easily see a Borg task spend seconds or even tens of seconds in "downloading" state if the borglet hadn't already seen and cached locally that version of the package. It was about time this became available also for Docker and Kubernetes.

thekhatribharat 8y ago

WhatsApp is a good example, they had a small engineering team handling tens of millions of users.