26 ms·

I don’t know how to count that low

deleted 5y ago
[deleted]
brrrrrm 5y ago
> if you didn’t need 100 database shards scattered around the globe, were you even doing real work? Is this actually a prevalent mentality at Google?
- yoz-y 5y ago
  Personally I've never heard anybody utter something similar.
  - crate_barre 5y ago
    Sometimes a pervasive truth is often tacitly accepted and never uttered? Conjecture of course.
  - petters 5y ago
    A video by someone named Taliver: Answer to What’s an inside joke among Google employees? by David Seidman https://www.quora.com/What-s-an-inside-joke-among-Google-employees/answer/David-Seidman?ch=15&oid=109049652&share=78cca675&srid=8LtW&target_type=answer https://www.quora.com/What-s-an-inside-joke-among-Google-emp... Pretty old by now, don't know when you started at Google but probably not seen much anymore.
  - paulgb 5y ago
    I was only at Google for a couple of years, but I remember “I don't know how to count that low” as a recurring punchline on the internal meme page. I never actually heard someone utter it seriously, but I saw it frequently enough in jokes that I assumed this article was about Google.
- fragmede 5y ago
  Of course not. Don't be silly. The post even tells you with its title: "I don’t Know How To Count That Low. :)
- shadowgovt 5y ago
  A running gag at Google is that the infrastructure makes hard problems easier and easy problems harder. There's a new-hire parable centered around the phrase "I just want to serve five terabytes" that demonstrates all the layers of systems and services you have to plug into to get a simple static-data server running reliably, scalably, and with monitoring and support. The parable hints that "The Google way" to solve that problem is to find the service someone already built and use it to vend your data; don't re-invent a wheel. But (a) that means part of your job is now keeping abreast of all of Google's wheels (and there are hundreds of them) and (b) if your work didn't require the creation of a new wheel, was it impactful enough to justify a promotion later?
- colonelxc 5y ago
  There are a series of jokes in Google about things like this. Things like just wanting to serve a file, or run some small webapp, etc, etc. Google's tooling is geared towards building large, reliable apps. That makes wanting to build some tiny thing seem like a big bother (seemingly lots of overhead). I remember my first hackathon at Google, probably in 2015. My group built a service and deployed it with some automated deployment tools. In "production", it was running 9 instances, spread over 3 continents. A little overkill for a hackathon project. But I have never heard it (in 7 years) seriously.
- bestcoder69 5y ago
  I'm an outsider, but this book[0] talks about how they think about these questions. The mentality makes sense to me in the sense that they are also trying to scale development processes. Does it make sense for every team to start small with a single VPS running python scripts and then migrate to up to 3-tier architecture, then all the way up to k8s when the feature is a success so needs global scale and availability? Or could the company make k8s easy enough that its trivial to just host v1 there, and back it with a massive multi-tenant db that also works fine for low volume prototypes? 0: https://abseil.io/resources/swe_at_google.2.pdf https://abseil.io/resources/swe_at_google.2.pdf
  - sam0x17 5y ago
    Why does everyone cite k8s as the tool for the job when there have been massive problems actually getting k8s to work at scale without constant tweaking and babysitting? In reality this is almost always either traditional load balanced clusters, ala-carte systems like Google App Engine / Elastic Beanstalk or (more so these days) serverless compute. At Google, they use Borg internally which is an early precursor to k8s that is more akin to traditional load balancing.
    - IggleSniggle 5y ago
      Isn't the promise of k8s a kind of dev-friendly serverless-compute engine, but your process(es)/architecture may be running for moments, days, weeks, or years, and spread across environments dynamically? I would expect aspects of such a thing to require tweaking and babysitting, as the demands of the application(s) evolve.
      - sam0x17 5y ago
        Right but the second I have to dictate the number of nodes to scale any further, that contract has been broken imo.
- deleted 5y ago
  [deleted]
- hn_throwaway_99 5y ago
  Here is a concrete example where this mentality really hurt Google in the cloud space: App Engine. App Engine was really Google's first foray (or one of the first) into the cloud space. When it was first released, the only persistent store available was the Datastore. The Datastore was/is a great piece of technology, but it was painfully clear that this tool was optimized for Google's use cases and came from the mindset that things need to be infinitely scalable/shardable, e.g. eventual consistency and the fact that you couldn't have indexes with inequalities in different directions, so doing a query that is trivial in a DB (e.g "dateCol > someMinVal AND dateCol < someMaxVal") was annoyingly difficult. So the early days of App Engine were really cool, but it took some time before Google actually made a plain ol' database (their Cloud SQL tech) available. As someone looking in, and as someone who has talked to a lot of ex-Google employees, it was clearly a cultural shift for them to need to focus on standardized enterprise tools that so much of the industry is familiar with, rather than all the tons of custom shit Google has due to their scale.
  - macNchz 5y ago
    This makes a lot of sense and totally jives with my brief experience of working on a single App Engine app years ago. It felt like an easy platform to spin up a simple web app at first blush, but was actually such a frustrating experience when trying to do a lot of actual simple web app things.
    - stavros 5y ago
      The Datastore could do simple things well, and hard things well, and both exactly with the same level of effort.
      - Dylan16807 5y ago
        How would you do a range query "well", then? With filtering on your own server definitely not being "well".
    - pas 5y ago
      The same was true of early Firebase. (Which Google acquired.) Sometimes a "general" backend/datastore just sucks independent of scale.
- gowld 5y ago
  No, and that informs my estimate of the accuracy the other judgments in the story.
- efsavage 5y ago
  It's a bit tongue-in-cheek, but there's some truth in it at least for anything infrastructure related. If your thing is a success, it almost certainly needs to scale to Very Large numbers, and be reliable/distributed/performant/etc. So if you're proposing something that you expect to be successful but only needs a couple of VMs, it would be an outlier. (I work there)
- hbosch 5y ago
  I've heard someone from Google talk about how they only think about the "B's and T's" of their projects. B's and T's, as in, Billions of users and Trillions of dollars.
- Rebelgecko 5y ago
  No, like the phrase "I don't remember how to count that low", I've pretty much only heard it used ironically to poke fun at how infrastructure can complicate simple use cases. Want to host a 1PB dataset, globally available with enough failover to handle a moderately large nuclear war? Easy! Want to throw up a 200KB web page with best-effort reliability? Surprisingly complicated.
cushychicken 5y ago
Does it seem a little strange that a startup would not hire someone who had solved a harder problem (big concurrent sharded DB system) for not knowing exactly how to solve an easier problem (keep the database in memory)? OP seems to imply that is some kind of arrogance to believe that. Seems like, if you could figure out how to crack the first kind of problem, you could probably figure out how to crack the second. Maybe it's dismissive to say this, but it seems like this was a problem with the hiring org, not the author.
- BeetleB 5y ago
  There is a legitimate fear of employees overengineering. If a candidate spins an AWS service for something a simple Python script could do, I too would not hire.
  - rudian 5y ago
    Same here. As a consultant I often find large “enterprise-style” and I get paid to make them a tenth of the size and with much less redirection. Simple is better.
- jwolfe 5y ago
  Is a big concurrent sharded DB necessarily a "harder problem" when it's provided to you by another team at the company to use? The skills you need just seem different to me. In the large case, the more important skills are going to be the higher level reasoning of what is a good shard size, how do we avoid hotspotting, what is an appropriate way to partition for the set of queries we want to run. None of those really help you solve the smaller problem. You're also going to likely be relying on infrastructure-provided solutions like automatic backups. In the small case, you're going to need to worry about things like those backups. Then there's the skill mentioned in the interview -- it seems like this person had never had to actually think through how much space N rows of a particular schema were going to take up in memory, or how much order of magnitude QPS you can get out of (e.g.) a MySQL server. You're also going to be optimizing to different peculiarities of what is fast to do on this different database technology, which sometimes does not translate directly. In a smaller company (more so in older days, but still in some places) you may also need to deal with the operational challenges of buying and installing a bigger box if it's called for.
deleted 5y ago
[deleted]
tantalor 5y ago
> when it was several orders of magnitude lower than when I should even have begun worrying about that Really confused by this phrase. What does it mean?
- arsome 5y ago
  Their database was likely measured in megabytes not gigabytes or terabytes. I have a similar situation... it's awesome, I can afford to back the entire production DB up in its entirety once per hour for months on end.
- DreadY2K 5y ago
  I read it as "the database could increase in size by several orders of magnitude and still fit in memory"
- rcpt 5y ago
  I had it to read it a few times to. I think they were trying to say that database was so small that obviously you wouldn't need to think about distributed systems. I'd be surprised if that was the reason they failed the interview.
  - shadowgovt 5y ago
    That's a common way I see well-educated candidates fail our interviews. What happens is that they only have N minutes to do the whole interview... If they burn twenty minutes of that time gaming out whether the DB needs redundancy, multi-tier storage, and synchronization, they don't have time to actually show me the algorithm that will generate the data to go in the database, and that's what I'm hiring them for. "Is the database the right shape" is rarely a problem at our scale.
    - jrochkind1 5y ago
      Have you considered giving them a hint, that you primarily are looking for the algorithm that will generate the data?
spicybright 5y ago
> "I didn’t know how to count that low, but now that it matters I am learning" This is the sign a good engineer.
fragmede 5y ago
TBH author sounds like a bit of a broccoli from their time at Google. The public sees this same attitude "in the wild" though, with Kubernetes, which is a thoroughly inappropriate solution in many cases (but is absolutely right in some). But once you've bitten that bug, if there's a computer problem, step 1 is going to always be "first, set up a kubernetes cluster because I've forgotten how to computer without that". If, during an interview, someone couldn't move past "well, I'd just use kubernetes" or give other possible solutions then I could see myself questioning how appropriate they are for a role and consider other candidates.
- jstarks 5y ago
  What is a broccoli?
  - ghostbrainalpha 5y ago
    I am also interested in this expression. Google searching turns up nothing, except maybe a reference to marijuana. So maybe 'broccoli' is their way of saying craziness? Like being on Google drugs. But obviously that's not a perfect fit. Maybe its a misspelling of something else?
    - deleted 5y ago
      [deleted]
    - deleted 5y ago
      [deleted]
    - guyzero 5y ago
      No one in Google uses this as an expression. It's close to something people actually do reference, but it's not described like this.
  - yaacov 5y ago
    It’s a reference to an internal Google comedy video mocking these tendencies (one of the characters is an anthropomorphic broccoli)
_dwt 5y ago
Oooof, SAS (in the author's dad anecdote). If they get hooked young, it's expensive hell to get them to quit. I'm not sure how that colors by analogy my view of the main point of the article, but it's tempting to reach for: yes, if you get addicted to expensive (and/or proprietary) complexity, you sometimes lose sight of how to build simple solutions to simple problems. I'm not sure that's an argument in favor of the intellectual superiority of those who build the complexity...
- siver_john 5y ago
  It reminded me of my days in college where professors encouraged us to use Matlab. A variety of reasons made me not like it but one of them for sure was the realization that the moment I left college Matlab would no longer be "free" to me. Yes, I know Octave exists, no it can't do everything Matlab can do and a lot of the programs I saw used needed Matlab specific things.
- madrox 5y ago
  I'm a huge SAS fan, but after college it fell out of my toolkit for the same reason as the author: license fees. It's a shame, because I found SAS far and away to be the best tool for data cleaning and querying. Plus, it's got great readability...especially over R, though that may be a contentious opinion. I'm honestly surprised and disappointed to see how little SAS has adapted to competition. I'm sure they'll milk their existing customers for another generation, but it'll kill their business on a long enough timeline.
  - garmaine 5y ago
    Likewise I absolutely love Mathematica, and have a personal license I make good use of. I would never, ever use it in a business or consulting context though as the license fees are just way too high.
  - wlesieutre 5y ago
    I had the misfortune of needing to install SAS on my computer (fiancee has a license through grad school) and the whole install process is built around a complicated licensing scheme that ties any of a hundred optional components together into a massive assembly depending on exactly how much money they were able to squeeze out of you. There was a problem with our license file which meant we got through 5-10 screens of installer and then hit "The SID file you selected is not intended for the order you are deploying because the Tech Support Site Number values do not match. Select a SID file with a matching tech support site number." And then two weeks later we were finally able to get a correct license file from the IT department. I can't say how much of that was IT's fault or if it really took two weeks to get the fixed license from SAS, but either way I'd be happier to never touch it again.
missblit 5y ago
> I didn’t know the limit because it had been many years since I’d had a problem that could be solved with a DB small enough to be held in its entirety in memory. And they were right to fail me for that: the fact that I was good at solving strictly more difficult problems didn’t matter because I didn’t know how to solve the easier ones they actually had. No way it's "right" to fail an interviewee because they wondered outloud if a dataset can fit in memory. A candidate should be able to estimate the answer, but beyond that it's trivia that can be researched as needed -- indeed even asking the question is a positive signal not a negative one (it's not just people with "google scale" background who reach for over-complicated solutions).
- leephillips 5y ago
  “No way it's "right" to fail an interviewee because [she] wondered outloud if a dataset can fit in memory.” That made no sense to me, either. My reaction was that she was fortunate not to work there (but further down she mentions that she wound up there anyway).
- sam0x17 5y ago
  I also came in to comment on this excerpt that failing someone over something so trivial is literally insane.
  - blueplanet200 5y ago
    I was mostly in agreement. I suppose it depends on the scale of the data. Fitting 1000 short rows in memory is something I'd expect anyone to know off the top of their head just works. It really depends on the specifics of the question and the seniority of the role...
    - woko 5y ago
      It is clear by the mention of "several orders of magnitude" that we are talking about a database which fits in a few MB of disk space on a machine with a few GB of RAM. If the interviewee has never worked with a database which would fit in the memory of the machine, and if the company only deals with such databases, then it is clear that it was not a good fit. Failing the interview does not mean that the interviewee is bad, just that they don't know how to count that low, and thus were not the person for the job.
alecbz 5y ago
I always heard "I don't know how to count that low" as more of a tongue-in-cheek joke about Google's scale, not necessarily a sign of smug superiority, or that work at smaller scales "didn't matter". Like if someone was wondering "is it wasteful to store this data as a 1 GiB file, maybe we should compress it/structure it differently", someone might say "I forgot how to count that low", meaning, "that's a small enough amount of space that it's not worth it to worry about optimizing it".
- TulliusCicero 5y ago
  > I always heard "I don't know how to count that low" as more of a tongue-in-cheek joke about Google's scale, not necessarily a sign of smug superiority, or that work at smaller scales "didn't matter". This is exactly correct. I dunno where the author's coming from, it sounds like they might be one of those people who frequently misinterprets others' intent.
- ergocoder 5y ago
  I was at Google, and I didn't see anyone using it with superiority complex.
- nybble41 5y ago
  1 GiB? That's nothing. I just finished reformatting 280 GiB of Google Takeout data from .tgz format into a 130 GiB SquashFS filesystem (both for space reasons and to allow efficient random access). It would have been nice to be able to do that on the server without downloading 150 GB of extra data first, plus another 30+ GiB because one of the downloads failed partway through and couldn't be resumed due to the download URLs having very short expiration times relative to the file sizes. Apparently 180 GiB of data transfer and 150 GiB of storage doesn't register as worthy of optimization for Google, but it's quite a bit for your typical home user.
  - Seirdy 5y ago
    I'd imagine that the inconvenience of handling takeout data is beneficial for Google's business interests, so there isn't a big incentive to improve the situation.
neural_thing 5y ago
As a data scientist, I like to use the most accessible tool that can get the job done without pain. Can I do it in Google Sheets? Awesome! I'll do that. Can I do that in SQL? That's great too! Will I have to use pandas on a cloud instance? OK, I'll do that. Do I absolutely have to use some complex pipeline? Then I will do that. There are benefits to using tools that many people can understand.
- spicybright 5y ago
  Right tool for the right job. It's some of the simplest wisdom that's the hardest to follow :)
- ImaCake 5y ago
  Same approach I take in a research group in computational biology. Our datasets are often around the ~1-10 million rows range so those get processed in R or the command line. But often the summary analysis fits into a single page in excel. I personally make my notes in markdown, but when I share them with others I convert to word docs or html because it’s not fair to make others deal with my raw text when it is so simple to format it for them.
mst 5y ago
Lighter weight UI version available here: https://www.greaterwrong.com/posts/koGbEwgbfst2wCbzG/i-don-t-know-how-to-count-that-low https://www.greaterwrong.com/posts/koGbEwgbfst2wCbzG/i-don-t...
another-dave 5y ago
From the title, I was expecting an article on counter-offering in salary negotiations
mrfusion 5y ago
How come folks don’t make an open source SAS interpreter? Or maybe a translator to Python?
- CRConrad 5y ago
  There is -- or at least used to be -- something called, IIRC, "Data Step". As I understood it, that was an implementation of the base SAS data-set manipulation language. Can't recall if it was OSS or just Freeware, nor if it had any of the statistical PROCs. GIYF, as we used to say (before we knew better).
croes 5y ago
Is this mindset the reason why Google makes Android and Chrome worse and doesn't care about the users?
wly_cdgr 5y ago
This makes me think of how big tech job postings always talk about scale and impact in a way that makes it sound like they know that most of the day to day work there is boring and pointless as shit and scale is the only thing they can think of to try to convince you that you want to do it
madrox 5y ago
This is a very systems-engineering way of talking about over-engineering, and possibility a better way of talking about over-engineering in a world where it's just as easy to spin up MySQL as it is to spin up DynamoDB on AWS. Quite often the largest material difference for engineers these days is cost of operation, and we're not often trained to add that to our calculus when designing systems.
dekhn 5y ago
I just wish google would release the brocolliman video where he says 'I forgot how to count that low'
- nostrademons 5y ago
  It doesn't make sense unless you know the internal systems it's referencing, and Google probably does not want to release the details of those internal systems.
  - jmgao 5y ago
    I was over in Android, where "borgmon" and "readability" were mythical concepts, and it was still hilarious.
tombert 5y ago
I am certainly guilty of this problem. I've pretty much hated MS Word for about 10 years now, more or less since I started being an engineer for a living. I hated its equation editor, I hated how the documents ended up looking, I hated the damn spacing between the letters, every time I used it I always felt like my documents ended up looking like ransom letters. I discovered LaTeX + Pandoc, and it created documents that I thought looked much nicer, and I could use trusty Vim to work with it, so it felt perfect, and I subsequently made fun of people who insisted on using Word. After about two years of doing this, my dad eventually snapped back saying "Tom, we're all really sorry we didn't become software engineers, I guess we'll all quit our jobs and go back to college," which (rightly) made me feel like an asshole. Obviously this stuff was easy to me, I live and breath in text editors and in plain text, but I had to realize that not everyone is trying to create publisher-worthy documents. They want a document that looks good enough to share with some coworkers, or print out for a school report, and for that MS Word will work fine for most use cases. They don't really want to sit there and learn the minutia of how to compile with `pdflatex` and dissect its weird error messages. They didn't really feel the need to finagle with begin and end tags, and that's totally fine, people should use the right tool for the right job. After that I would still recommend LaTeX (particularly if the document required equations), but I became substantially less pushy about it.
- lostlogin 5y ago
  > I've pretty much hated MS Word Do you mean the app, the web or the Teams version? Or is it the way each allows different things, blocks certain actions or displays differently? It is incredible that this mess is the industry standard.
  - tombert 5y ago
    I was referring to the app, whatever was recent in 2011 or so. 2007 I think? I haven't used it in a long time. I always wanted my documents (and particularly equations) to look as nice as the formatting in my textbooks, and it felt like no matter how much finagling I did (which was not an insignificant amount of time being spent), it never looked quite as good. Also, I never fully knew what copy-paste was actually going to do. Was it going to copy the source formatting? Was it going to adopt the style of where I was pasting? Was it going to do neither and just break everything? I was never sure, so I always ended up pasting into Notepad and then re-copying it to guarantee all the formatting was stripped out. And sometimes there would be just a vanilla "I pressed a random keystroke by accident, all the formatting broke, and undo doesn't appear to do anything" issues. This might just be gross incompetence on my end (it wouldn't be the first time) but nearly everyone I've described this issue to has experienced similar problems. Also, I can't remember the details of it, and it's possibly fixed now, but I had issues with the equation editor simply not having all the symbols I needed. In fairness, I have similar complaints about most of the other WYSIWYG word processors, so I don't mean to pick on MS Word specifically (most of these complaints apply to Pages and LibreOffice as well).
mensetmanusman 5y ago
What this is grasping at is that everyone works at a different level of abstraction, and they are all important. We need those that oversee the huge complex tasks (that are simplified by abstractions) just as we need those in the trenches.
stonogo 5y ago
This attitude is why none of the web-native hyperscalers have been able to even come close to competing in the HPC space: you cannot optimize a large-scale problem if you cannot optimize a small-scale problem. The first, last, and only approach from Google, Amazon, and even Azure is horizontal scaling, which is ridiculous because they absolutely employ people who can go deep on anything they might need. Unfortunately, while throwing more hardware at the problem will eventually solve the problem, it gets horrifically expensive in the meantime, and never really makes optimal use of the resources. One day one of these providers will figure this out, and then I'll either need to go work for them or retire... but today, I'm not worried.
ChrisMarshallNY 5y ago
I like messing with "the small stuff." I love "getting my hands dirty," and crafting a direct user experience. I really don't give a damn whether or not anyone thinks that it's "beneath" them.
mrandish 5y ago
This is a perfect catchphrase for a problem I've warned other startup entrepreneurs about. Over-crediting a resume that has a very impressive name brand company on it. It may or may not actually signify much depending on what they actually did, know and how they think. I learned it during the interview process with a candidate for CFO referred by a board member who was impressed by where they used to work. Fortunately, the candidate and I both realized he'd probably be fairly bad at working on the financial challenges ahead of our 35-person-going-to-100-people startup.
colordrops 5y ago
On another similar note, I moved from writing flight software for aerospace to tooling for a media corporation. Others around me made similar moves. Many of us had difficulties realizing that rigor around code quality and infrastructure is a spectrum, a knob you could turn, rather than an absolute standard, and would hold up PRs that didn't demonstrate testing and validation of things like CSS changes to an internal tool. It caused and causes a lot of unecessary heartache.
jollybean 5y ago
In startup parlance, by the time you need to shard your DB you have already solved the hardest business problems. Without being to condescending to our 'inner engineer' - doing things at scale is relatively pedantic compared to building something that people want to pay for in the first place. Yes, some of these problems are hard, we need to respect that - but they are ultimately, mostly problems to be solved - like difficult questions on an exam. You'll find the path through and that's that. It's more akin to operational complexity, just a party of doing business smartly.
zh3 5y ago
AKA Don't sweat the small stuff.
tester756 5y ago
It's like software engineers struggling to write code on a piece of paper? or without IDE?
stickyricky 5y ago
> the fact that I was good at solving strictly more difficult problems didn’t matter because I didn’t know how to solve the easier ones they actually had I refuse to believe Google employees solve "scale" problems every day. Be realistic, you configure systems and make calls to libraries which perform the scaling for you. Yes, the systems and the libraries can be impressive but your glue code is not. That's why you fail the "easy" questions.
temac 5y ago
> the fact that I was good at solving strictly more difficult problems didn’t matter because I didn’t know how to solve the easier ones they actually had The funny thing is: "small" systems are probably not presenting problems strictly less difficult than the ones Google have. Different, with the need of caring about other abstraction, other layers, yes. It's even astonishing the author does not understand that when she writes: "I didn’t know how to solve the easier ones" -- so how is that strictly less difficult.
jprupp 5y ago
It seems to me that there is a bias for complexity among software engineers. When faced with a problem, they seem to be deliberately choosing the most complex possible solution for it. When another programmer runs a problem and proposed solution by me, the most common answer that he gets is: "the solution to that problem can be a lot simpler". Then he thinks about it and comes up with a significantly simpler, better solution on the spot. But I notice a touch of irritation, a hint that he doesn't really want to do it the simple way. It baffles me.
CRConrad 5y ago
> ...Excel could have gotten better, except AFAIK they never fixed the problem with reading genes as dates so they get no benefit of a doubt from me. Not that this is only Excel's fault: Before entering your gene names, select the column they're going in* (by clicking the column header), right-click, select "Format cells" from the pop-up menu, select "Text" from the list on the left of the formatting dialogue, click "OK", and you're done. Now it won't turn text that looks like dates into the numeric representation of a date value. Not exactly rocket surgery. ___ *: Or, if you're just using it for data entry and/or all your data is textual, select the whole sheet by clicking on the grey square in the top left, at the intersection of row numbers and column-name letters.
benley 5y ago
Regarding the phrase "I don't know how to count that low", here's the video where it originated. Frankly I'm astonished this hasn't been shared before, and I hope I don't get in trouble for posting it (it's like 11 years old, surely nobody cares at this point, right?) https://www.youtube.com/watch?v=3t6L-FlfeaI https://www.youtube.com/watch?v=3t6L-FlfeaI

deleted 5y ago

[deleted]

> if you didn’t need 100 database shards scattered around the globe, were you even doing real work? Is this actually a prevalent mentality at Google?

yoz-y 5y ago
Personally I've never heard anybody utter something similar.
- crate_barre 5y ago
  Sometimes a pervasive truth is often tacitly accepted and never uttered? Conjecture of course.
- petters 5y ago
  A video by someone named Taliver: Answer to What’s an inside joke among Google employees? by David Seidman https://www.quora.com/What-s-an-inside-joke-among-Google-employees/answer/David-Seidman?ch=15&oid=109049652&share=78cca675&srid=8LtW&target_type=answer https://www.quora.com/What-s-an-inside-joke-among-Google-emp... Pretty old by now, don't know when you started at Google but probably not seen much anymore.
- paulgb 5y ago
  I was only at Google for a couple of years, but I remember “I don't know how to count that low” as a recurring punchline on the internal meme page. I never actually heard someone utter it seriously, but I saw it frequently enough in jokes that I assumed this article was about Google.
fragmede 5y ago
Of course not. Don't be silly. The post even tells you with its title: "I don’t Know How To Count That Low. :)
shadowgovt 5y ago
A running gag at Google is that the infrastructure makes hard problems easier and easy problems harder. There's a new-hire parable centered around the phrase "I just want to serve five terabytes" that demonstrates all the layers of systems and services you have to plug into to get a simple static-data server running reliably, scalably, and with monitoring and support. The parable hints that "The Google way" to solve that problem is to find the service someone already built and use it to vend your data; don't re-invent a wheel. But (a) that means part of your job is now keeping abreast of all of Google's wheels (and there are hundreds of them) and (b) if your work didn't require the creation of a new wheel, was it impactful enough to justify a promotion later?
colonelxc 5y ago
There are a series of jokes in Google about things like this. Things like just wanting to serve a file, or run some small webapp, etc, etc. Google's tooling is geared towards building large, reliable apps. That makes wanting to build some tiny thing seem like a big bother (seemingly lots of overhead). I remember my first hackathon at Google, probably in 2015. My group built a service and deployed it with some automated deployment tools. In "production", it was running 9 instances, spread over 3 continents. A little overkill for a hackathon project. But I have never heard it (in 7 years) seriously.
bestcoder69 5y ago
I'm an outsider, but this book[0] talks about how they think about these questions. The mentality makes sense to me in the sense that they are also trying to scale development processes. Does it make sense for every team to start small with a single VPS running python scripts and then migrate to up to 3-tier architecture, then all the way up to k8s when the feature is a success so needs global scale and availability? Or could the company make k8s easy enough that its trivial to just host v1 there, and back it with a massive multi-tenant db that also works fine for low volume prototypes? 0: https://abseil.io/resources/swe_at_google.2.pdf https://abseil.io/resources/swe_at_google.2.pdf
- sam0x17 5y ago
  Why does everyone cite k8s as the tool for the job when there have been massive problems actually getting k8s to work at scale without constant tweaking and babysitting? In reality this is almost always either traditional load balanced clusters, ala-carte systems like Google App Engine / Elastic Beanstalk or (more so these days) serverless compute. At Google, they use Borg internally which is an early precursor to k8s that is more akin to traditional load balancing.
  - IggleSniggle 5y ago
    Isn't the promise of k8s a kind of dev-friendly serverless-compute engine, but your process(es)/architecture may be running for moments, days, weeks, or years, and spread across environments dynamically? I would expect aspects of such a thing to require tweaking and babysitting, as the demands of the application(s) evolve.
    - sam0x17 5y ago
      Right but the second I have to dictate the number of nodes to scale any further, that contract has been broken imo.
deleted 5y ago
[deleted]
hn_throwaway_99 5y ago
Here is a concrete example where this mentality really hurt Google in the cloud space: App Engine. App Engine was really Google's first foray (or one of the first) into the cloud space. When it was first released, the only persistent store available was the Datastore. The Datastore was/is a great piece of technology, but it was painfully clear that this tool was optimized for Google's use cases and came from the mindset that things need to be infinitely scalable/shardable, e.g. eventual consistency and the fact that you couldn't have indexes with inequalities in different directions, so doing a query that is trivial in a DB (e.g "dateCol > someMinVal AND dateCol < someMaxVal") was annoyingly difficult. So the early days of App Engine were really cool, but it took some time before Google actually made a plain ol' database (their Cloud SQL tech) available. As someone looking in, and as someone who has talked to a lot of ex-Google employees, it was clearly a cultural shift for them to need to focus on standardized enterprise tools that so much of the industry is familiar with, rather than all the tons of custom shit Google has due to their scale.
- macNchz 5y ago
  This makes a lot of sense and totally jives with my brief experience of working on a single App Engine app years ago. It felt like an easy platform to spin up a simple web app at first blush, but was actually such a frustrating experience when trying to do a lot of actual simple web app things.
  - stavros 5y ago
    The Datastore could do simple things well, and hard things well, and both exactly with the same level of effort.
    - Dylan16807 5y ago
      How would you do a range query "well", then? With filtering on your own server definitely not being "well".
  - pas 5y ago
    The same was true of early Firebase. (Which Google acquired.) Sometimes a "general" backend/datastore just sucks independent of scale.
gowld 5y ago
No, and that informs my estimate of the accuracy the other judgments in the story.
efsavage 5y ago
It's a bit tongue-in-cheek, but there's some truth in it at least for anything infrastructure related. If your thing is a success, it almost certainly needs to scale to Very Large numbers, and be reliable/distributed/performant/etc. So if you're proposing something that you expect to be successful but only needs a couple of VMs, it would be an outlier. (I work there)
hbosch 5y ago
I've heard someone from Google talk about how they only think about the "B's and T's" of their projects. B's and T's, as in, Billions of users and Trillions of dollars.
Rebelgecko 5y ago
No, like the phrase "I don't remember how to count that low", I've pretty much only heard it used ironically to poke fun at how infrastructure can complicate simple use cases. Want to host a 1PB dataset, globally available with enough failover to handle a moderately large nuclear war? Easy! Want to throw up a 200KB web page with best-effort reliability? Surprisingly complicated.

cushychicken 5y ago

Does it seem a little strange that a startup would not hire someone who had solved a harder problem (big concurrent sharded DB system) for not knowing exactly how to solve an easier problem (keep the database in memory)? OP seems to imply that is some kind of arrogance to believe that. Seems like, if you could figure out how to crack the first kind of problem, you could probably figure out how to crack the second. Maybe it's dismissive to say this, but it seems like this was a problem with the hiring org, not the author.

BeetleB 5y ago
There is a legitimate fear of employees overengineering. If a candidate spins an AWS service for something a simple Python script could do, I too would not hire.
- rudian 5y ago
  Same here. As a consultant I often find large “enterprise-style” and I get paid to make them a tenth of the size and with much less redirection. Simple is better.
jwolfe 5y ago
Is a big concurrent sharded DB necessarily a "harder problem" when it's provided to you by another team at the company to use? The skills you need just seem different to me. In the large case, the more important skills are going to be the higher level reasoning of what is a good shard size, how do we avoid hotspotting, what is an appropriate way to partition for the set of queries we want to run. None of those really help you solve the smaller problem. You're also going to likely be relying on infrastructure-provided solutions like automatic backups. In the small case, you're going to need to worry about things like those backups. Then there's the skill mentioned in the interview -- it seems like this person had never had to actually think through how much space N rows of a particular schema were going to take up in memory, or how much order of magnitude QPS you can get out of (e.g.) a MySQL server. You're also going to be optimizing to different peculiarities of what is fast to do on this different database technology, which sometimes does not translate directly. In a smaller company (more so in older days, but still in some places) you may also need to deal with the operational challenges of buying and installing a bigger box if it's called for.

deleted 5y ago

[deleted]

tantalor 5y ago

> when it was several orders of magnitude lower than when I should even have begun worrying about that Really confused by this phrase. What does it mean?

arsome 5y ago
Their database was likely measured in megabytes not gigabytes or terabytes. I have a similar situation... it's awesome, I can afford to back the entire production DB up in its entirety once per hour for months on end.
DreadY2K 5y ago
I read it as "the database could increase in size by several orders of magnitude and still fit in memory"
rcpt 5y ago
I had it to read it a few times to. I think they were trying to say that database was so small that obviously you wouldn't need to think about distributed systems. I'd be surprised if that was the reason they failed the interview.
- shadowgovt 5y ago
  That's a common way I see well-educated candidates fail our interviews. What happens is that they only have N minutes to do the whole interview... If they burn twenty minutes of that time gaming out whether the DB needs redundancy, multi-tier storage, and synchronization, they don't have time to actually show me the algorithm that will generate the data to go in the database, and that's what I'm hiring them for. "Is the database the right shape" is rarely a problem at our scale.
  - jrochkind1 5y ago
    Have you considered giving them a hint, that you primarily are looking for the algorithm that will generate the data?

spicybright 5y ago

> "I didn’t know how to count that low, but now that it matters I am learning" This is the sign a good engineer.

fragmede 5y ago

TBH author sounds like a bit of a broccoli from their time at Google. The public sees this same attitude "in the wild" though, with Kubernetes, which is a thoroughly inappropriate solution in many cases (but is absolutely right in some). But once you've bitten that bug, if there's a computer problem, step 1 is going to always be "first, set up a kubernetes cluster because I've forgotten how to computer without that". If, during an interview, someone couldn't move past "well, I'd just use kubernetes" or give other possible solutions then I could see myself questioning how appropriate they are for a role and consider other candidates.

jstarks 5y ago
What is a broccoli?
- ghostbrainalpha 5y ago
  I am also interested in this expression. Google searching turns up nothing, except maybe a reference to marijuana. So maybe 'broccoli' is their way of saying craziness? Like being on Google drugs. But obviously that's not a perfect fit. Maybe its a misspelling of something else?
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
  - guyzero 5y ago
    No one in Google uses this as an expression. It's close to something people actually do reference, but it's not described like this.
- yaacov 5y ago
  It’s a reference to an internal Google comedy video mocking these tendencies (one of the characters is an anthropomorphic broccoli)

_dwt 5y ago

Oooof, SAS (in the author's dad anecdote). If they get hooked young, it's expensive hell to get them to quit. I'm not sure how that colors by analogy my view of the main point of the article, but it's tempting to reach for: yes, if you get addicted to expensive (and/or proprietary) complexity, you sometimes lose sight of how to build simple solutions to simple problems. I'm not sure that's an argument in favor of the intellectual superiority of those who build the complexity...

siver_john 5y ago
It reminded me of my days in college where professors encouraged us to use Matlab. A variety of reasons made me not like it but one of them for sure was the realization that the moment I left college Matlab would no longer be "free" to me. Yes, I know Octave exists, no it can't do everything Matlab can do and a lot of the programs I saw used needed Matlab specific things.
madrox 5y ago
I'm a huge SAS fan, but after college it fell out of my toolkit for the same reason as the author: license fees. It's a shame, because I found SAS far and away to be the best tool for data cleaning and querying. Plus, it's got great readability...especially over R, though that may be a contentious opinion. I'm honestly surprised and disappointed to see how little SAS has adapted to competition. I'm sure they'll milk their existing customers for another generation, but it'll kill their business on a long enough timeline.
- garmaine 5y ago
  Likewise I absolutely love Mathematica, and have a personal license I make good use of. I would never, ever use it in a business or consulting context though as the license fees are just way too high.
- wlesieutre 5y ago
  I had the misfortune of needing to install SAS on my computer (fiancee has a license through grad school) and the whole install process is built around a complicated licensing scheme that ties any of a hundred optional components together into a massive assembly depending on exactly how much money they were able to squeeze out of you. There was a problem with our license file which meant we got through 5-10 screens of installer and then hit "The SID file you selected is not intended for the order you are deploying because the Tech Support Site Number values do not match. Select a SID file with a matching tech support site number." And then two weeks later we were finally able to get a correct license file from the IT department. I can't say how much of that was IT's fault or if it really took two weeks to get the fixed license from SAS, but either way I'd be happier to never touch it again.

missblit 5y ago

> I didn’t know the limit because it had been many years since I’d had a problem that could be solved with a DB small enough to be held in its entirety in memory. And they were right to fail me for that: the fact that I was good at solving strictly more difficult problems didn’t matter because I didn’t know how to solve the easier ones they actually had. No way it's "right" to fail an interviewee because they wondered outloud if a dataset can fit in memory. A candidate should be able to estimate the answer, but beyond that it's trivia that can be researched as needed -- indeed even asking the question is a positive signal not a negative one (it's not just people with "google scale" background who reach for over-complicated solutions).

leephillips 5y ago
“No way it's "right" to fail an interviewee because [she] wondered outloud if a dataset can fit in memory.” That made no sense to me, either. My reaction was that she was fortunate not to work there (but further down she mentions that she wound up there anyway).
sam0x17 5y ago
I also came in to comment on this excerpt that failing someone over something so trivial is literally insane.
- blueplanet200 5y ago
  I was mostly in agreement. I suppose it depends on the scale of the data. Fitting 1000 short rows in memory is something I'd expect anyone to know off the top of their head just works. It really depends on the specifics of the question and the seniority of the role...
  - woko 5y ago
    It is clear by the mention of "several orders of magnitude" that we are talking about a database which fits in a few MB of disk space on a machine with a few GB of RAM. If the interviewee has never worked with a database which would fit in the memory of the machine, and if the company only deals with such databases, then it is clear that it was not a good fit. Failing the interview does not mean that the interviewee is bad, just that they don't know how to count that low, and thus were not the person for the job.

alecbz 5y ago

I always heard "I don't know how to count that low" as more of a tongue-in-cheek joke about Google's scale, not necessarily a sign of smug superiority, or that work at smaller scales "didn't matter". Like if someone was wondering "is it wasteful to store this data as a 1 GiB file, maybe we should compress it/structure it differently", someone might say "I forgot how to count that low", meaning, "that's a small enough amount of space that it's not worth it to worry about optimizing it".

TulliusCicero 5y ago
> I always heard "I don't know how to count that low" as more of a tongue-in-cheek joke about Google's scale, not necessarily a sign of smug superiority, or that work at smaller scales "didn't matter". This is exactly correct. I dunno where the author's coming from, it sounds like they might be one of those people who frequently misinterprets others' intent.
ergocoder 5y ago
I was at Google, and I didn't see anyone using it with superiority complex.
nybble41 5y ago
1 GiB? That's nothing. I just finished reformatting 280 GiB of Google Takeout data from .tgz format into a 130 GiB SquashFS filesystem (both for space reasons and to allow efficient random access). It would have been nice to be able to do that on the server without downloading 150 GB of extra data first, plus another 30+ GiB because one of the downloads failed partway through and couldn't be resumed due to the download URLs having very short expiration times relative to the file sizes. Apparently 180 GiB of data transfer and 150 GiB of storage doesn't register as worthy of optimization for Google, but it's quite a bit for your typical home user.
- Seirdy 5y ago
  I'd imagine that the inconvenience of handling takeout data is beneficial for Google's business interests, so there isn't a big incentive to improve the situation.

neural_thing 5y ago

As a data scientist, I like to use the most accessible tool that can get the job done without pain. Can I do it in Google Sheets? Awesome! I'll do that. Can I do that in SQL? That's great too! Will I have to use pandas on a cloud instance? OK, I'll do that. Do I absolutely have to use some complex pipeline? Then I will do that. There are benefits to using tools that many people can understand.

spicybright 5y ago
Right tool for the right job. It's some of the simplest wisdom that's the hardest to follow :)
ImaCake 5y ago
Same approach I take in a research group in computational biology. Our datasets are often around the ~1-10 million rows range so those get processed in R or the command line. But often the summary analysis fits into a single page in excel. I personally make my notes in markdown, but when I share them with others I convert to word docs or html because it’s not fair to make others deal with my raw text when it is so simple to format it for them.

mst 5y ago

Lighter weight UI version available here: https://www.greaterwrong.com/posts/koGbEwgbfst2wCbzG/i-don-t-know-how-to-count-that-low https://www.greaterwrong.com/posts/koGbEwgbfst2wCbzG/i-don-t...

another-dave 5y ago

From the title, I was expecting an article on counter-offering in salary negotiations

mrfusion 5y ago

How come folks don’t make an open source SAS interpreter? Or maybe a translator to Python?

CRConrad 5y ago
There is -- or at least used to be -- something called, IIRC, "Data Step". As I understood it, that was an implementation of the base SAS data-set manipulation language. Can't recall if it was OSS or just Freeware, nor if it had any of the statistical PROCs. GIYF, as we used to say (before we knew better).

croes 5y ago

Is this mindset the reason why Google makes Android and Chrome worse and doesn't care about the users?

wly_cdgr 5y ago

This makes me think of how big tech job postings always talk about scale and impact in a way that makes it sound like they know that most of the day to day work there is boring and pointless as shit and scale is the only thing they can think of to try to convince you that you want to do it

madrox 5y ago

This is a very systems-engineering way of talking about over-engineering, and possibility a better way of talking about over-engineering in a world where it's just as easy to spin up MySQL as it is to spin up DynamoDB on AWS. Quite often the largest material difference for engineers these days is cost of operation, and we're not often trained to add that to our calculus when designing systems.

dekhn 5y ago

I just wish google would release the brocolliman video where he says 'I forgot how to count that low'

nostrademons 5y ago
It doesn't make sense unless you know the internal systems it's referencing, and Google probably does not want to release the details of those internal systems.
- jmgao 5y ago
  I was over in Android, where "borgmon" and "readability" were mythical concepts, and it was still hilarious.

tombert 5y ago

I am certainly guilty of this problem. I've pretty much hated MS Word for about 10 years now, more or less since I started being an engineer for a living. I hated its equation editor, I hated how the documents ended up looking, I hated the damn spacing between the letters, every time I used it I always felt like my documents ended up looking like ransom letters. I discovered LaTeX + Pandoc, and it created documents that I thought looked much nicer, and I could use trusty Vim to work with it, so it felt perfect, and I subsequently made fun of people who insisted on using Word. After about two years of doing this, my dad eventually snapped back saying "Tom, we're all really sorry we didn't become software engineers, I guess we'll all quit our jobs and go back to college," which (rightly) made me feel like an asshole. Obviously this stuff was easy to me, I live and breath in text editors and in plain text, but I had to realize that not everyone is trying to create publisher-worthy documents. They want a document that looks good enough to share with some coworkers, or print out for a school report, and for that MS Word will work fine for most use cases. They don't really want to sit there and learn the minutia of how to compile with `pdflatex` and dissect its weird error messages. They didn't really feel the need to finagle with begin and end tags, and that's totally fine, people should use the right tool for the right job. After that I would still recommend LaTeX (particularly if the document required equations), but I became substantially less pushy about it.

lostlogin 5y ago
> I've pretty much hated MS Word Do you mean the app, the web or the Teams version? Or is it the way each allows different things, blocks certain actions or displays differently? It is incredible that this mess is the industry standard.
- tombert 5y ago
  I was referring to the app, whatever was recent in 2011 or so. 2007 I think? I haven't used it in a long time. I always wanted my documents (and particularly equations) to look as nice as the formatting in my textbooks, and it felt like no matter how much finagling I did (which was not an insignificant amount of time being spent), it never looked quite as good. Also, I never fully knew what copy-paste was actually going to do. Was it going to copy the source formatting? Was it going to adopt the style of where I was pasting? Was it going to do neither and just break everything? I was never sure, so I always ended up pasting into Notepad and then re-copying it to guarantee all the formatting was stripped out. And sometimes there would be just a vanilla "I pressed a random keystroke by accident, all the formatting broke, and undo doesn't appear to do anything" issues. This might just be gross incompetence on my end (it wouldn't be the first time) but nearly everyone I've described this issue to has experienced similar problems. Also, I can't remember the details of it, and it's possibly fixed now, but I had issues with the equation editor simply not having all the symbols I needed. In fairness, I have similar complaints about most of the other WYSIWYG word processors, so I don't mean to pick on MS Word specifically (most of these complaints apply to Pages and LibreOffice as well).

mensetmanusman 5y ago

What this is grasping at is that everyone works at a different level of abstraction, and they are all important. We need those that oversee the huge complex tasks (that are simplified by abstractions) just as we need those in the trenches.

stonogo 5y ago

This attitude is why none of the web-native hyperscalers have been able to even come close to competing in the HPC space: you cannot optimize a large-scale problem if you cannot optimize a small-scale problem. The first, last, and only approach from Google, Amazon, and even Azure is horizontal scaling, which is ridiculous because they absolutely employ people who can go deep on anything they might need. Unfortunately, while throwing more hardware at the problem will eventually solve the problem, it gets horrifically expensive in the meantime, and never really makes optimal use of the resources. One day one of these providers will figure this out, and then I'll either need to go work for them or retire... but today, I'm not worried.

ChrisMarshallNY 5y ago

I like messing with "the small stuff." I love "getting my hands dirty," and crafting a direct user experience. I really don't give a damn whether or not anyone thinks that it's "beneath" them.

mrandish 5y ago

This is a perfect catchphrase for a problem I've warned other startup entrepreneurs about. Over-crediting a resume that has a very impressive name brand company on it. It may or may not actually signify much depending on what they actually did, know and how they think. I learned it during the interview process with a candidate for CFO referred by a board member who was impressed by where they used to work. Fortunately, the candidate and I both realized he'd probably be fairly bad at working on the financial challenges ahead of our 35-person-going-to-100-people startup.

colordrops 5y ago

On another similar note, I moved from writing flight software for aerospace to tooling for a media corporation. Others around me made similar moves. Many of us had difficulties realizing that rigor around code quality and infrastructure is a spectrum, a knob you could turn, rather than an absolute standard, and would hold up PRs that didn't demonstrate testing and validation of things like CSS changes to an internal tool. It caused and causes a lot of unecessary heartache.

jollybean 5y ago

In startup parlance, by the time you need to shard your DB you have already solved the hardest business problems. Without being to condescending to our 'inner engineer' - doing things at scale is relatively pedantic compared to building something that people want to pay for in the first place. Yes, some of these problems are hard, we need to respect that - but they are ultimately, mostly problems to be solved - like difficult questions on an exam. You'll find the path through and that's that. It's more akin to operational complexity, just a party of doing business smartly.

zh3 5y ago

AKA Don't sweat the small stuff.

tester756 5y ago

It's like software engineers struggling to write code on a piece of paper? or without IDE?

stickyricky 5y ago

> the fact that I was good at solving strictly more difficult problems didn’t matter because I didn’t know how to solve the easier ones they actually had I refuse to believe Google employees solve "scale" problems every day. Be realistic, you configure systems and make calls to libraries which perform the scaling for you. Yes, the systems and the libraries can be impressive but your glue code is not. That's why you fail the "easy" questions.

temac 5y ago

> the fact that I was good at solving strictly more difficult problems didn’t matter because I didn’t know how to solve the easier ones they actually had The funny thing is: "small" systems are probably not presenting problems strictly less difficult than the ones Google have. Different, with the need of caring about other abstraction, other layers, yes. It's even astonishing the author does not understand that when she writes: "I didn’t know how to solve the easier ones" -- so how is that strictly less difficult.

jprupp 5y ago

It seems to me that there is a bias for complexity among software engineers. When faced with a problem, they seem to be deliberately choosing the most complex possible solution for it. When another programmer runs a problem and proposed solution by me, the most common answer that he gets is: "the solution to that problem can be a lot simpler". Then he thinks about it and comes up with a significantly simpler, better solution on the spot. But I notice a touch of irritation, a hint that he doesn't really want to do it the simple way. It baffles me.

CRConrad 5y ago

> ...Excel could have gotten better, except AFAIK they never fixed the problem with reading genes as dates so they get no benefit of a doubt from me. Not that this is only Excel's fault: Before entering your gene names, select the column they're going in* (by clicking the column header), right-click, select "Format cells" from the pop-up menu, select "Text" from the list on the left of the formatting dialogue, click "OK", and you're done. Now it won't turn text that looks like dates into the numeric representation of a date value. Not exactly rocket surgery. ___ *: Or, if you're just using it for data entry and/or all your data is textual, select the whole sheet by clicking on the grey square in the top left, at the intersection of row numbers and column-name letters.

benley 5y ago

Regarding the phrase "I don't know how to count that low", here's the video where it originated. Frankly I'm astonished this hasn't been shared before, and I hope I don't get in trouble for posting it (it's like 11 years old, surely nobody cares at this point, right?) https://www.youtube.com/watch?v=3t6L-FlfeaI https://www.youtube.com/watch?v=3t6L-FlfeaI