5 ms·

> if you didn’t need 100 database shards scattered around the globe, were you even doing real work? Is this actually a prevalent mentality at Google?

yoz-y 5y ago
Personally I've never heard anybody utter something similar.
- crate_barre 5y ago
  Sometimes a pervasive truth is often tacitly accepted and never uttered? Conjecture of course.
- petters 5y ago
  A video by someone named Taliver: Answer to What’s an inside joke among Google employees? by David Seidman https://www.quora.com/What-s-an-inside-joke-among-Google-employees/answer/David-Seidman?ch=15&oid=109049652&share=78cca675&srid=8LtW&target_type=answer https://www.quora.com/What-s-an-inside-joke-among-Google-emp... Pretty old by now, don't know when you started at Google but probably not seen much anymore.
- paulgb 5y ago
  I was only at Google for a couple of years, but I remember “I don't know how to count that low” as a recurring punchline on the internal meme page. I never actually heard someone utter it seriously, but I saw it frequently enough in jokes that I assumed this article was about Google.
fragmede 5y ago
Of course not. Don't be silly. The post even tells you with its title: "I don’t Know How To Count That Low. :)
shadowgovt 5y ago
A running gag at Google is that the infrastructure makes hard problems easier and easy problems harder. There's a new-hire parable centered around the phrase "I just want to serve five terabytes" that demonstrates all the layers of systems and services you have to plug into to get a simple static-data server running reliably, scalably, and with monitoring and support. The parable hints that "The Google way" to solve that problem is to find the service someone already built and use it to vend your data; don't re-invent a wheel. But (a) that means part of your job is now keeping abreast of all of Google's wheels (and there are hundreds of them) and (b) if your work didn't require the creation of a new wheel, was it impactful enough to justify a promotion later?
colonelxc 5y ago
There are a series of jokes in Google about things like this. Things like just wanting to serve a file, or run some small webapp, etc, etc. Google's tooling is geared towards building large, reliable apps. That makes wanting to build some tiny thing seem like a big bother (seemingly lots of overhead). I remember my first hackathon at Google, probably in 2015. My group built a service and deployed it with some automated deployment tools. In "production", it was running 9 instances, spread over 3 continents. A little overkill for a hackathon project. But I have never heard it (in 7 years) seriously.
bestcoder69 5y ago
I'm an outsider, but this book[0] talks about how they think about these questions. The mentality makes sense to me in the sense that they are also trying to scale development processes. Does it make sense for every team to start small with a single VPS running python scripts and then migrate to up to 3-tier architecture, then all the way up to k8s when the feature is a success so needs global scale and availability? Or could the company make k8s easy enough that its trivial to just host v1 there, and back it with a massive multi-tenant db that also works fine for low volume prototypes? 0: https://abseil.io/resources/swe_at_google.2.pdf https://abseil.io/resources/swe_at_google.2.pdf
- sam0x17 5y ago
  Why does everyone cite k8s as the tool for the job when there have been massive problems actually getting k8s to work at scale without constant tweaking and babysitting? In reality this is almost always either traditional load balanced clusters, ala-carte systems like Google App Engine / Elastic Beanstalk or (more so these days) serverless compute. At Google, they use Borg internally which is an early precursor to k8s that is more akin to traditional load balancing.
  - IggleSniggle 5y ago
    Isn't the promise of k8s a kind of dev-friendly serverless-compute engine, but your process(es)/architecture may be running for moments, days, weeks, or years, and spread across environments dynamically? I would expect aspects of such a thing to require tweaking and babysitting, as the demands of the application(s) evolve.
    - sam0x17 5y ago
      Right but the second I have to dictate the number of nodes to scale any further, that contract has been broken imo.
deleted 5y ago
[deleted]
hn_throwaway_99 5y ago
Here is a concrete example where this mentality really hurt Google in the cloud space: App Engine. App Engine was really Google's first foray (or one of the first) into the cloud space. When it was first released, the only persistent store available was the Datastore. The Datastore was/is a great piece of technology, but it was painfully clear that this tool was optimized for Google's use cases and came from the mindset that things need to be infinitely scalable/shardable, e.g. eventual consistency and the fact that you couldn't have indexes with inequalities in different directions, so doing a query that is trivial in a DB (e.g "dateCol > someMinVal AND dateCol < someMaxVal") was annoyingly difficult. So the early days of App Engine were really cool, but it took some time before Google actually made a plain ol' database (their Cloud SQL tech) available. As someone looking in, and as someone who has talked to a lot of ex-Google employees, it was clearly a cultural shift for them to need to focus on standardized enterprise tools that so much of the industry is familiar with, rather than all the tons of custom shit Google has due to their scale.
- macNchz 5y ago
  This makes a lot of sense and totally jives with my brief experience of working on a single App Engine app years ago. It felt like an easy platform to spin up a simple web app at first blush, but was actually such a frustrating experience when trying to do a lot of actual simple web app things.
  - stavros 5y ago
    The Datastore could do simple things well, and hard things well, and both exactly with the same level of effort.
    - Dylan16807 5y ago
      How would you do a range query "well", then? With filtering on your own server definitely not being "well".
  - pas 5y ago
    The same was true of early Firebase. (Which Google acquired.) Sometimes a "general" backend/datastore just sucks independent of scale.
gowld 5y ago
No, and that informs my estimate of the accuracy the other judgments in the story.
efsavage 5y ago
It's a bit tongue-in-cheek, but there's some truth in it at least for anything infrastructure related. If your thing is a success, it almost certainly needs to scale to Very Large numbers, and be reliable/distributed/performant/etc. So if you're proposing something that you expect to be successful but only needs a couple of VMs, it would be an outlier. (I work there)
hbosch 5y ago
I've heard someone from Google talk about how they only think about the "B's and T's" of their projects. B's and T's, as in, Billions of users and Trillions of dollars.
Rebelgecko 5y ago
No, like the phrase "I don't remember how to count that low", I've pretty much only heard it used ironically to poke fun at how infrastructure can complicate simple use cases. Want to host a 1PB dataset, globally available with enough failover to handle a moderately large nuclear war? Easy! Want to throw up a 200KB web page with best-effort reliability? Surprisingly complicated.

yoz-y 5y ago

Personally I've never heard anybody utter something similar.

crate_barre 5y ago
Sometimes a pervasive truth is often tacitly accepted and never uttered? Conjecture of course.
petters 5y ago
A video by someone named Taliver: Answer to What’s an inside joke among Google employees? by David Seidman https://www.quora.com/What-s-an-inside-joke-among-Google-employees/answer/David-Seidman?ch=15&oid=109049652&share=78cca675&srid=8LtW&target_type=answer https://www.quora.com/What-s-an-inside-joke-among-Google-emp... Pretty old by now, don't know when you started at Google but probably not seen much anymore.
paulgb 5y ago
I was only at Google for a couple of years, but I remember “I don't know how to count that low” as a recurring punchline on the internal meme page. I never actually heard someone utter it seriously, but I saw it frequently enough in jokes that I assumed this article was about Google.

fragmede 5y ago

Of course not. Don't be silly. The post even tells you with its title: "I don’t Know How To Count That Low. :)

shadowgovt 5y ago

A running gag at Google is that the infrastructure makes hard problems easier and easy problems harder. There's a new-hire parable centered around the phrase "I just want to serve five terabytes" that demonstrates all the layers of systems and services you have to plug into to get a simple static-data server running reliably, scalably, and with monitoring and support. The parable hints that "The Google way" to solve that problem is to find the service someone already built and use it to vend your data; don't re-invent a wheel. But (a) that means part of your job is now keeping abreast of all of Google's wheels (and there are hundreds of them) and (b) if your work didn't require the creation of a new wheel, was it impactful enough to justify a promotion later?

colonelxc 5y ago

There are a series of jokes in Google about things like this. Things like just wanting to serve a file, or run some small webapp, etc, etc. Google's tooling is geared towards building large, reliable apps. That makes wanting to build some tiny thing seem like a big bother (seemingly lots of overhead). I remember my first hackathon at Google, probably in 2015. My group built a service and deployed it with some automated deployment tools. In "production", it was running 9 instances, spread over 3 continents. A little overkill for a hackathon project. But I have never heard it (in 7 years) seriously.

bestcoder69 5y ago

I'm an outsider, but this book[0] talks about how they think about these questions. The mentality makes sense to me in the sense that they are also trying to scale development processes. Does it make sense for every team to start small with a single VPS running python scripts and then migrate to up to 3-tier architecture, then all the way up to k8s when the feature is a success so needs global scale and availability? Or could the company make k8s easy enough that its trivial to just host v1 there, and back it with a massive multi-tenant db that also works fine for low volume prototypes? 0: https://abseil.io/resources/swe_at_google.2.pdf https://abseil.io/resources/swe_at_google.2.pdf

sam0x17 5y ago
Why does everyone cite k8s as the tool for the job when there have been massive problems actually getting k8s to work at scale without constant tweaking and babysitting? In reality this is almost always either traditional load balanced clusters, ala-carte systems like Google App Engine / Elastic Beanstalk or (more so these days) serverless compute. At Google, they use Borg internally which is an early precursor to k8s that is more akin to traditional load balancing.
- IggleSniggle 5y ago
  Isn't the promise of k8s a kind of dev-friendly serverless-compute engine, but your process(es)/architecture may be running for moments, days, weeks, or years, and spread across environments dynamically? I would expect aspects of such a thing to require tweaking and babysitting, as the demands of the application(s) evolve.
  - sam0x17 5y ago
    Right but the second I have to dictate the number of nodes to scale any further, that contract has been broken imo.

deleted 5y ago

[deleted]

hn_throwaway_99 5y ago

Here is a concrete example where this mentality really hurt Google in the cloud space: App Engine. App Engine was really Google's first foray (or one of the first) into the cloud space. When it was first released, the only persistent store available was the Datastore. The Datastore was/is a great piece of technology, but it was painfully clear that this tool was optimized for Google's use cases and came from the mindset that things need to be infinitely scalable/shardable, e.g. eventual consistency and the fact that you couldn't have indexes with inequalities in different directions, so doing a query that is trivial in a DB (e.g "dateCol > someMinVal AND dateCol < someMaxVal") was annoyingly difficult. So the early days of App Engine were really cool, but it took some time before Google actually made a plain ol' database (their Cloud SQL tech) available. As someone looking in, and as someone who has talked to a lot of ex-Google employees, it was clearly a cultural shift for them to need to focus on standardized enterprise tools that so much of the industry is familiar with, rather than all the tons of custom shit Google has due to their scale.

macNchz 5y ago
This makes a lot of sense and totally jives with my brief experience of working on a single App Engine app years ago. It felt like an easy platform to spin up a simple web app at first blush, but was actually such a frustrating experience when trying to do a lot of actual simple web app things.
- stavros 5y ago
  The Datastore could do simple things well, and hard things well, and both exactly with the same level of effort.
  - Dylan16807 5y ago
    How would you do a range query "well", then? With filtering on your own server definitely not being "well".
- pas 5y ago
  The same was true of early Firebase. (Which Google acquired.) Sometimes a "general" backend/datastore just sucks independent of scale.

gowld 5y ago

No, and that informs my estimate of the accuracy the other judgments in the story.

efsavage 5y ago

It's a bit tongue-in-cheek, but there's some truth in it at least for anything infrastructure related. If your thing is a success, it almost certainly needs to scale to Very Large numbers, and be reliable/distributed/performant/etc. So if you're proposing something that you expect to be successful but only needs a couple of VMs, it would be an outlier. (I work there)

hbosch 5y ago

I've heard someone from Google talk about how they only think about the "B's and T's" of their projects. B's and T's, as in, Billions of users and Trillions of dollars.

Rebelgecko 5y ago

No, like the phrase "I don't remember how to count that low", I've pretty much only heard it used ironically to poke fun at how infrastructure can complicate simple use cases. Want to host a 1PB dataset, globally available with enough failover to handle a moderately large nuclear war? Easy! Want to throw up a 200KB web page with best-effort reliability? Surprisingly complicated.