7 ms·

Does it seem a little strange that a startup would not hire someone who had solved a harder problem (big concurrent sharded DB system) for not knowing exactly h

Does it seem a little strange that a startup would not hire someone who had solved a harder problem (big concurrent sharded DB system) for not knowing exactly how to solve an easier problem (keep the database in memory)? OP seems to imply that is some kind of arrogance to believe that. Seems like, if you could figure out how to crack the first kind of problem, you could probably figure out how to crack the second. Maybe it's dismissive to say this, but it seems like this was a problem with the hiring org, not the author.

BeetleB 5y ago
There is a legitimate fear of employees overengineering. If a candidate spins an AWS service for something a simple Python script could do, I too would not hire.
- rudian 5y ago
  Same here. As a consultant I often find large “enterprise-style” and I get paid to make them a tenth of the size and with much less redirection. Simple is better.
jwolfe 5y ago
Is a big concurrent sharded DB necessarily a "harder problem" when it's provided to you by another team at the company to use? The skills you need just seem different to me. In the large case, the more important skills are going to be the higher level reasoning of what is a good shard size, how do we avoid hotspotting, what is an appropriate way to partition for the set of queries we want to run. None of those really help you solve the smaller problem. You're also going to likely be relying on infrastructure-provided solutions like automatic backups. In the small case, you're going to need to worry about things like those backups. Then there's the skill mentioned in the interview -- it seems like this person had never had to actually think through how much space N rows of a particular schema were going to take up in memory, or how much order of magnitude QPS you can get out of (e.g.) a MySQL server. You're also going to be optimizing to different peculiarities of what is fast to do on this different database technology, which sometimes does not translate directly. In a smaller company (more so in older days, but still in some places) you may also need to deal with the operational challenges of buying and installing a bigger box if it's called for.
marcinzm 5y ago
It's a mindset as there's a hundred decisions you make without even thinking deeply about it. Startups need to build just enough to survive but no more since budget (money, time, etc.) is limited. So if you make decisions that are too scalable then you're likely spending "budget" inefficiently.
thrower123 5y ago
Not at all. I'd be leery of somebody who jumped to a big, complicated, expensive solution instead of something simple for a simple problem. Been there, done that, it sucks, and if you're not in a position where you can slap it down early, you end up having to support that poor decision after the person who instigated it has departed.
morelisp 5y ago
I have the somewhat weird inverse problem. I've inherited so many services and such that deal with data in the 100MB-10GB range (much of which could even be streamed!) and all have been written to do ridiculous things to distribute the work. They even manage to sometimes make it not fit at first because they're loading UTF-8 into UTF-32 or quadruple indirecting uninterned strings or whatnot. (You ain't seen nothing until you've seen a Java UUID class implemented as `List<Byte>` - with length 32!) Most of the company looked at me like I was crazy when I said we'd be fine loading our next-gen 2TB datasets directly into RAM with some trivial sharding. I have also worked with people who could cram an unbelievable amount of content into a GBA ROM but struggled to get a 10 node webserver cluster to do 10k/sec. So yeah, I don't think the knowledge automatically works both ways from either side. Some people know how scaling works, but even more only know how to operate at specific scales.
- toast0 5y ago
  > Most of the company looked at me like I was crazy when I said we'd be fine loading our next-gen 2TB datasets directly into RAM with some trivial sharding. FWIW, you can get 2TB of ram on a single node these days. No need to shard.
oh_sigh 5y ago
Without knowing much about the author's time at Google, I'd suspect that they weren't actually responsible for solving the harder problem - they just used a service which did that for them (like spanner, or whatever) and maybe set some configuration values.
_xnmw 5y ago
No it's not strange at all -- I once had a colleague who went on to become a brilliant engineer at Google, but was horrible at the small startup we worked together at, because his solutions were over-engineered, fanciful, totally ignored the day-to-day messy problems we had, and did NOT help further our mission - to make the customer happy.
- laurent123456 5y ago
  > did NOT help further our mission - to make the customer happy I can't help but think he'll fit right in at Google.
zomglings 5y ago
I have inherited overengineered unwieldy architectures from programmers who had solved much harder technical problems than making our product successful. Having to live with this kind of complexity slows you down in ways that you can't even really quantify (because it's difficult to run the required experiments while you're living the situation). A list of issues it causes in web development (for example): 1. Makes it difficult to experiment with new features or different features in a product because every change is now 10 times as involved as it needs to be. 2. Makes it difficult to hire programmers with less experience because you are working with 3 or 4 complicated technologies (caches, job queues, etc.) when a single database server would do. 3. Makes it difficult to debug problems because you have to identify bugs across $(n \choose 2)$ microservice boundaries (where $n$ is the number of microservices in your architecture), each boundary involves a different set of components to enable communication (REST APIs, GraphQL APIs, Redis, RabbitMQ, Elasticsearch, etc.) each of which perhaps made sense locally but which certainly do not make sense as part of the whole system. Apologies if I triggered anyone with this comment.
- morelisp 5y ago
  More than overengineering, misplaced engineering. We have a lot of stuff that "scales" and would not solve a real problem if it didn't "scale", but there is also an equally good solution not in the domain of the person who was tasked with writing it. So for example, we were distributing a "large" (~100e6) multi-dimensional lookup table to ~30 servers to do some transformation job. The engineer who did it didn't like to think about data formats, so they used JSON. They didn't like to think about cache friendliness, so it was just maps of maps of maps. They didn't look at the actual dataset contents - which was some mostly opaque-to-us customer catalog SKUs - so didn't notice the massive redundancy in it. Implementing their algorithm as a straight line process, yeah, it did really take several hours to run and it was too long and probably required more memory than we wanted to commit. But the solution wasn't "farm it out to a Hadoop cluster", it was "translate the dataset into some DFA-like things and then run some optimization passes on them" which yeah, was slow to "compile" - like 2min - but once compiled could chew through the rest of the job in another 10min, with sub-1G memory. Very big / indirect "everything looks like a nail" situation, combined with lots of people who want to rush for the Big Data / ML / whatever stuff and not fundamentals. And the reasons don't seem to be uniform either. I've seen it from careerists and architecture astronauts, but also bootcamp grads who just weren't properly educated, at startups often early non-technical or technically-adjacent employees trying to help out but don't realize the mess they're making, and sometimes people who just don't want to think very hard.
dragonwriter 5y ago
> Does it seem a little strange that a startup would not hire someone who had solved a harder problem (big concurrent sharded DB system) for not knowing exactly how to solve an easier problem (keep the database in memory)? Actually, it seems they knew how to keep the database in memory, they lacked an intuition as to the scale at which that solution becomes unviable and openly questioned its applicability when it should (in the retrospective opinion of the applicant and presumably the contemporary opinion of the interviewer) have been intuitively obvious. Rejecting for that reason seems a kind of dumb though typical symptom of systems that overemphasize easily-acquired contextual familiarity, leading to preferring inflexible hyperspecialists in narrow problem domains and solution approaches.
temp8964 5y ago
I think "easier" and "harder" are not used appropriately here. Technologies to be used at different scales of size do not always match to the scales of difficulty. Especially when you are the person to use the technologies, not inventing the technologies. Maybe there is a sequence of learning, for example you always learn to manage small databases and then go on to the technics to manage larger databases. But again, the sequence of learning does not always match to the scale of difficulty.
stonemetal12 5y ago
Scaling down and scaling up are two very different problems. It isn't a given that scaling up is a harder problem than scaling down, it rather depends how far down and how far up. If they were looking for a "subject matter expert", rather than someone who could become a SME then failing the interview is entirely expected. The techniques for scaling down are unrelated at best and at worst totally the opposite of the techniques used for scaling up.
garmaine 5y ago
A big concurrent sharded DB system is not a harder problem, it is a different problem. Knowing how to solve problems at one scale doesn't transfer very well to other scales.
Zababa 5y ago
Those are not harder or easier problems, those are different problems.
cratermoon 5y ago
While someone with experience in solving a larger-scale problem could certainly be expected to solve one at a smaller scale, the result might be too costly and complex for a company that is not Google to build and maintain. To use a very rough analogy, if you have a family and need to haul the kids to school and soccer practice, you'd probably want a minivan or small SUV. Sure, you could buy a school bus or minibus and hire someone with a CDL to drive it. That would certainly work, but would it be the right choice?

BeetleB 5y ago

There is a legitimate fear of employees overengineering. If a candidate spins an AWS service for something a simple Python script could do, I too would not hire.

rudian 5y ago
Same here. As a consultant I often find large “enterprise-style” and I get paid to make them a tenth of the size and with much less redirection. Simple is better.

jwolfe 5y ago

Is a big concurrent sharded DB necessarily a "harder problem" when it's provided to you by another team at the company to use? The skills you need just seem different to me. In the large case, the more important skills are going to be the higher level reasoning of what is a good shard size, how do we avoid hotspotting, what is an appropriate way to partition for the set of queries we want to run. None of those really help you solve the smaller problem. You're also going to likely be relying on infrastructure-provided solutions like automatic backups. In the small case, you're going to need to worry about things like those backups. Then there's the skill mentioned in the interview -- it seems like this person had never had to actually think through how much space N rows of a particular schema were going to take up in memory, or how much order of magnitude QPS you can get out of (e.g.) a MySQL server. You're also going to be optimizing to different peculiarities of what is fast to do on this different database technology, which sometimes does not translate directly. In a smaller company (more so in older days, but still in some places) you may also need to deal with the operational challenges of buying and installing a bigger box if it's called for.

marcinzm 5y ago

It's a mindset as there's a hundred decisions you make without even thinking deeply about it. Startups need to build just enough to survive but no more since budget (money, time, etc.) is limited. So if you make decisions that are too scalable then you're likely spending "budget" inefficiently.

thrower123 5y ago

Not at all. I'd be leery of somebody who jumped to a big, complicated, expensive solution instead of something simple for a simple problem. Been there, done that, it sucks, and if you're not in a position where you can slap it down early, you end up having to support that poor decision after the person who instigated it has departed.

morelisp 5y ago

I have the somewhat weird inverse problem. I've inherited so many services and such that deal with data in the 100MB-10GB range (much of which could even be streamed!) and all have been written to do ridiculous things to distribute the work. They even manage to sometimes make it not fit at first because they're loading UTF-8 into UTF-32 or quadruple indirecting uninterned strings or whatnot. (You ain't seen nothing until you've seen a Java UUID class implemented as `List<Byte>` - with length 32!) Most of the company looked at me like I was crazy when I said we'd be fine loading our next-gen 2TB datasets directly into RAM with some trivial sharding. I have also worked with people who could cram an unbelievable amount of content into a GBA ROM but struggled to get a 10 node webserver cluster to do 10k/sec. So yeah, I don't think the knowledge automatically works both ways from either side. Some people know how scaling works, but even more only know how to operate at specific scales.

toast0 5y ago
> Most of the company looked at me like I was crazy when I said we'd be fine loading our next-gen 2TB datasets directly into RAM with some trivial sharding. FWIW, you can get 2TB of ram on a single node these days. No need to shard.

oh_sigh 5y ago

Without knowing much about the author's time at Google, I'd suspect that they weren't actually responsible for solving the harder problem - they just used a service which did that for them (like spanner, or whatever) and maybe set some configuration values.

_xnmw 5y ago

No it's not strange at all -- I once had a colleague who went on to become a brilliant engineer at Google, but was horrible at the small startup we worked together at, because his solutions were over-engineered, fanciful, totally ignored the day-to-day messy problems we had, and did NOT help further our mission - to make the customer happy.

laurent123456 5y ago
> did NOT help further our mission - to make the customer happy I can't help but think he'll fit right in at Google.

zomglings 5y ago

I have inherited overengineered unwieldy architectures from programmers who had solved much harder technical problems than making our product successful. Having to live with this kind of complexity slows you down in ways that you can't even really quantify (because it's difficult to run the required experiments while you're living the situation). A list of issues it causes in web development (for example): 1. Makes it difficult to experiment with new features or different features in a product because every change is now 10 times as involved as it needs to be. 2. Makes it difficult to hire programmers with less experience because you are working with 3 or 4 complicated technologies (caches, job queues, etc.) when a single database server would do. 3. Makes it difficult to debug problems because you have to identify bugs across $(n \choose 2)$ microservice boundaries (where $n$ is the number of microservices in your architecture), each boundary involves a different set of components to enable communication (REST APIs, GraphQL APIs, Redis, RabbitMQ, Elasticsearch, etc.) each of which perhaps made sense locally but which certainly do not make sense as part of the whole system. Apologies if I triggered anyone with this comment.

morelisp 5y ago
More than overengineering, misplaced engineering. We have a lot of stuff that "scales" and would not solve a real problem if it didn't "scale", but there is also an equally good solution not in the domain of the person who was tasked with writing it. So for example, we were distributing a "large" (~100e6) multi-dimensional lookup table to ~30 servers to do some transformation job. The engineer who did it didn't like to think about data formats, so they used JSON. They didn't like to think about cache friendliness, so it was just maps of maps of maps. They didn't look at the actual dataset contents - which was some mostly opaque-to-us customer catalog SKUs - so didn't notice the massive redundancy in it. Implementing their algorithm as a straight line process, yeah, it did really take several hours to run and it was too long and probably required more memory than we wanted to commit. But the solution wasn't "farm it out to a Hadoop cluster", it was "translate the dataset into some DFA-like things and then run some optimization passes on them" which yeah, was slow to "compile" - like 2min - but once compiled could chew through the rest of the job in another 10min, with sub-1G memory. Very big / indirect "everything looks like a nail" situation, combined with lots of people who want to rush for the Big Data / ML / whatever stuff and not fundamentals. And the reasons don't seem to be uniform either. I've seen it from careerists and architecture astronauts, but also bootcamp grads who just weren't properly educated, at startups often early non-technical or technically-adjacent employees trying to help out but don't realize the mess they're making, and sometimes people who just don't want to think very hard.

dragonwriter 5y ago

> Does it seem a little strange that a startup would not hire someone who had solved a harder problem (big concurrent sharded DB system) for not knowing exactly how to solve an easier problem (keep the database in memory)? Actually, it seems they knew how to keep the database in memory, they lacked an intuition as to the scale at which that solution becomes unviable and openly questioned its applicability when it should (in the retrospective opinion of the applicant and presumably the contemporary opinion of the interviewer) have been intuitively obvious. Rejecting for that reason seems a kind of dumb though typical symptom of systems that overemphasize easily-acquired contextual familiarity, leading to preferring inflexible hyperspecialists in narrow problem domains and solution approaches.

temp8964 5y ago

I think "easier" and "harder" are not used appropriately here. Technologies to be used at different scales of size do not always match to the scales of difficulty. Especially when you are the person to use the technologies, not inventing the technologies. Maybe there is a sequence of learning, for example you always learn to manage small databases and then go on to the technics to manage larger databases. But again, the sequence of learning does not always match to the scale of difficulty.

stonemetal12 5y ago

Scaling down and scaling up are two very different problems. It isn't a given that scaling up is a harder problem than scaling down, it rather depends how far down and how far up. If they were looking for a "subject matter expert", rather than someone who could become a SME then failing the interview is entirely expected. The techniques for scaling down are unrelated at best and at worst totally the opposite of the techniques used for scaling up.

garmaine 5y ago

A big concurrent sharded DB system is not a harder problem, it is a different problem. Knowing how to solve problems at one scale doesn't transfer very well to other scales.

Zababa 5y ago

Those are not harder or easier problems, those are different problems.

cratermoon 5y ago

While someone with experience in solving a larger-scale problem could certainly be expected to solve one at a smaller scale, the result might be too costly and complex for a company that is not Google to build and maintain. To use a very rough analogy, if you have a family and need to haul the kids to school and soccer practice, you'd probably want a minivan or small SUV. Sure, you could buy a school bus or minibus and hire someone with a CDL to drive it. That would certainly work, but would it be the right choice?