4 ms·

Wild idea: maybe if more devs had good fundamental knowledge to begin with, the good systems engineering could be done along the way.

SkyPuncher 2y ago
It all takes time, mental energy, etc. Different environments require different tradeoffs. The vast majority of startups will die before their systems engineering becomes a problem.
- sgarland 2y ago
  Constant firefighting because you engineered a pile of shit also takes time and mental energy.
  - dzhiurgis 2y ago
    You are describing worst of both worlds - systems engineering, done poorly. I find good abstractions to be almost maintenance-free.
  - closewith 2y ago
    Only if you let it. None of that is your problem - just consume the task queue. Unless of course you're in a leadership role, in which case it's going to be priority #1,000 in 99.9% of cases.
ketzo 2y ago
And if more systems engineers had more design knowledge, navigating the AWS console wouldn’t be like walking on hot coals. But it’s still $X0 billion/year business! We’re all different at good things, and it’s usually better to lean into your strengths than it is to paper over your weaknesses. We can wish everyone were good at everything, or we can try to actually get things done.
- godelski 2y ago
  > We can wish everyone were good at everything, or we can try to actually get things done. False dichotomy. There's no reason we can't have both. I want to be clear, there's no perfect code or a perfect understanding or any of that. But the complaint here about not knowing /enough/ fundamentals is valid. There is some threshold which we should recognize as a minimum. The disagreement is about where this threshold is, and no one is calling for perfection. But certainly there are plenty who want the threshold to not exist. Be that AI will replace coders or coding bootcamps get you big tech jobs. Zero to hero in a few months is bull.
  - ketzo 2y ago
    It’s not a false dichotomy at all. You only have so many hours in a day. At a startup, it’s very unlikely (certainly not impossible!) that your differentiation will come from very cheap system orchestration - your time is likely better spent on building your product. Minimum knowledge is one thing; minimum time to apply it is another.
    - sgarland 2y ago
      If you had to spend time / VC money learning all of this stuff before you could begin to apply it, I absolutely agree, it's a waste of time. That's not my point. My point is people (by people, I mean "someone interested in tech and is likely to pursue it as a career") can and should learn these things earlier in life such that it's trivial once they're in the workforce. I could go from servers sitting on the ground to racked, imaged, and ready to serve traffic in a few hours, because I've spent the time learning how to do it, and have built scripts and playbooks to do so. Even if I hadn't done the latter, many others have also done so and published them, so as long as you knew what you were looking for, you could do the same.
      - godelski 2y ago
        Yeah this is what I meant as well. Though I'd also argue that on the job returning is essential too. Which should come through multiple avenues. Mentorship from seniors to junior as well as allowing for time to learn on the job. I can tell you from having been an aerospace engineer you'd be given this time. And from what I hear, in the old days you'd naturally get this time any time you hit compile. There's a bunch of sayings from tradesmen that I think are relevant here. And it's usually said by people who take pride in their work and won't do shoddy craftsmanship - measure twice, cut once - there's never time to do it right, but there's always time to do it twice - if you don't have time to do it right when will you have time to do it again? I think the advantage these guys have is that when they do a shit job it's more noticeable. Not only to the builders but anyone else. Unfortunately we work with high abstractions but high skill is the main reason we get big bucks. Unfortunately I think this makes it harder for managers to differentiate high quality from shit. So they'd rather get shit fast than quality a tad slower because all they can differentiate is time. But they don't see the how this is so costly since everything has to be done at least thrice
vaylian 2y ago
I'm a big fan of fundamental knowledge, but I disagree somewhat with your statement. The thing that startups care most about is a product market fit. And finding that fit requires a lot of iteration and throw-away code. Once the dust settles and you have an initial user base, you can start looking into optimizations.
- sgarland 2y ago
  > Once the dust settles and you have an initial user base, you can start looking into optimizations. But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it, and they usually then discover that no one knows how to optimize things other than code (maybe – I've worked with many devs who have no idea how to profile their code, which is horrifying).
  - BoiledCabbage 2y ago
    > But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it, Yes because usually the other option is focus on those things you advocate for up front and then they go out of business before they get a chance to have the problems you're arguing against.
  - habinero 2y ago
    Because that's the right approach for that situation. A core skill in engineering is understanding tradeoffs, and in that case you want speed. Outside of eng, nobody cares if your company has the prettiest, leanest infrastructure in the world. They care about product.
vineyardmike 2y ago
There is such a variety of work environments, and realistically most people learn on the job. Everyone has different skills and knowledge bases. When I was at <FAANG> we didn’t control our infrastructure, there were teams that did it for us. Those guys knew a lot more about the internals of Linux than your average HNer. Getting access to the SSD of the host wasn’t a sys-call away, it was a ticket to an SRE and a library import. It wasn’t about limited knowledge, it was an intentional engineering tradeoff made at a multi-billion dollar infra level. When I worked at <startup>, we spent 1hr writing 50loc and throwing it at AWS lambda just to see if it would work. No thought to long term cost or scalability, because the company might not be there tomorrow, and this is the fastest way to prototype an API in the cloud. When it works, obviously management wants you to hit the “scale” button in that moment and if it costs 50% more, well that’s probably only a few hundred dollars a month. It wasn’t about limited knowledge, but instead an intentional engineering tradeoff when you’re focused on speed and costs are small And there is a whole bunch of companies that exist in between.
- SkyPuncher 2y ago
  This is exactly my experience. Nearly every dev on my team can dive into the details and scale that service effectively, but it’s rarely worth it. If an engineer costs $100/hour, scaling an extra $100/month (or even an extra $1k/month) is generally a no brainer. That money is almost always better served towards shipping product.
chromanoid 2y ago
Premature optimization may hit them hard. Overengineering is imo usually the bigger technical debt and a huge upfront cost as well. Well-thought out plans tend to become a sunken cost fallacy. Making room for changes is hard enough in XP like ways of working. When you have to tell your manager that half a year of careful plans and engineering can be thrown away, because of the new requirements, which emerge from late entry to market, you look like a clown. Plans and complexity usually introduce more risk than less.
- sgarland 2y ago
  Infra should not require much in the way of redoing if it's done correctly. Foundational software's configuration like RDBMS schema, maybe, but I wouldn't classify that as infra per se. Seriously, I'm struggling to figure out how "we have servers that run containers / applications" would need to be redone just because the application changed.
  - chromanoid 2y ago
    Some things that can happen: Product gets canned. Customers want on premise in their data center. Usage spikes are too extreme and serverless is simply the cheapest option. I would always recommend "serverless" monolith first with the option to develop with mocks locally/offline. That's imo the best risk/effort ratio.

SkyPuncher 2y ago

It all takes time, mental energy, etc. Different environments require different tradeoffs. The vast majority of startups will die before their systems engineering becomes a problem.

sgarland 2y ago
Constant firefighting because you engineered a pile of shit also takes time and mental energy.
- dzhiurgis 2y ago
  You are describing worst of both worlds - systems engineering, done poorly. I find good abstractions to be almost maintenance-free.
- closewith 2y ago
  Only if you let it. None of that is your problem - just consume the task queue. Unless of course you're in a leadership role, in which case it's going to be priority #1,000 in 99.9% of cases.

ketzo 2y ago

And if more systems engineers had more design knowledge, navigating the AWS console wouldn’t be like walking on hot coals. But it’s still $X0 billion/year business! We’re all different at good things, and it’s usually better to lean into your strengths than it is to paper over your weaknesses. We can wish everyone were good at everything, or we can try to actually get things done.

godelski 2y ago
> We can wish everyone were good at everything, or we can try to actually get things done. False dichotomy. There's no reason we can't have both. I want to be clear, there's no perfect code or a perfect understanding or any of that. But the complaint here about not knowing /enough/ fundamentals is valid. There is some threshold which we should recognize as a minimum. The disagreement is about where this threshold is, and no one is calling for perfection. But certainly there are plenty who want the threshold to not exist. Be that AI will replace coders or coding bootcamps get you big tech jobs. Zero to hero in a few months is bull.
- ketzo 2y ago
  It’s not a false dichotomy at all. You only have so many hours in a day. At a startup, it’s very unlikely (certainly not impossible!) that your differentiation will come from very cheap system orchestration - your time is likely better spent on building your product. Minimum knowledge is one thing; minimum time to apply it is another.
  - sgarland 2y ago
    If you had to spend time / VC money learning all of this stuff before you could begin to apply it, I absolutely agree, it's a waste of time. That's not my point. My point is people (by people, I mean "someone interested in tech and is likely to pursue it as a career") can and should learn these things earlier in life such that it's trivial once they're in the workforce. I could go from servers sitting on the ground to racked, imaged, and ready to serve traffic in a few hours, because I've spent the time learning how to do it, and have built scripts and playbooks to do so. Even if I hadn't done the latter, many others have also done so and published them, so as long as you knew what you were looking for, you could do the same.
    - godelski 2y ago
      Yeah this is what I meant as well. Though I'd also argue that on the job returning is essential too. Which should come through multiple avenues. Mentorship from seniors to junior as well as allowing for time to learn on the job. I can tell you from having been an aerospace engineer you'd be given this time. And from what I hear, in the old days you'd naturally get this time any time you hit compile. There's a bunch of sayings from tradesmen that I think are relevant here. And it's usually said by people who take pride in their work and won't do shoddy craftsmanship - measure twice, cut once - there's never time to do it right, but there's always time to do it twice - if you don't have time to do it right when will you have time to do it again? I think the advantage these guys have is that when they do a shit job it's more noticeable. Not only to the builders but anyone else. Unfortunately we work with high abstractions but high skill is the main reason we get big bucks. Unfortunately I think this makes it harder for managers to differentiate high quality from shit. So they'd rather get shit fast than quality a tad slower because all they can differentiate is time. But they don't see the how this is so costly since everything has to be done at least thrice

vaylian 2y ago

I'm a big fan of fundamental knowledge, but I disagree somewhat with your statement. The thing that startups care most about is a product market fit. And finding that fit requires a lot of iteration and throw-away code. Once the dust settles and you have an initial user base, you can start looking into optimizations.

sgarland 2y ago
> Once the dust settles and you have an initial user base, you can start looking into optimizations. But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it, and they usually then discover that no one knows how to optimize things other than code (maybe – I've worked with many devs who have no idea how to profile their code, which is horrifying).
- BoiledCabbage 2y ago
  > But people never do. Instead they just scale up, get more funding, rinse and repeat. It isn't until the bill gets silly that anyone bothers to consider it, Yes because usually the other option is focus on those things you advocate for up front and then they go out of business before they get a chance to have the problems you're arguing against.
- habinero 2y ago
  Because that's the right approach for that situation. A core skill in engineering is understanding tradeoffs, and in that case you want speed. Outside of eng, nobody cares if your company has the prettiest, leanest infrastructure in the world. They care about product.

vineyardmike 2y ago

There is such a variety of work environments, and realistically most people learn on the job. Everyone has different skills and knowledge bases. When I was at <FAANG> we didn’t control our infrastructure, there were teams that did it for us. Those guys knew a lot more about the internals of Linux than your average HNer. Getting access to the SSD of the host wasn’t a sys-call away, it was a ticket to an SRE and a library import. It wasn’t about limited knowledge, it was an intentional engineering tradeoff made at a multi-billion dollar infra level. When I worked at <startup>, we spent 1hr writing 50loc and throwing it at AWS lambda just to see if it would work. No thought to long term cost or scalability, because the company might not be there tomorrow, and this is the fastest way to prototype an API in the cloud. When it works, obviously management wants you to hit the “scale” button in that moment and if it costs 50% more, well that’s probably only a few hundred dollars a month. It wasn’t about limited knowledge, but instead an intentional engineering tradeoff when you’re focused on speed and costs are small And there is a whole bunch of companies that exist in between.

SkyPuncher 2y ago
This is exactly my experience. Nearly every dev on my team can dive into the details and scale that service effectively, but it’s rarely worth it. If an engineer costs $100/hour, scaling an extra $100/month (or even an extra $1k/month) is generally a no brainer. That money is almost always better served towards shipping product.

chromanoid 2y ago

Premature optimization may hit them hard. Overengineering is imo usually the bigger technical debt and a huge upfront cost as well. Well-thought out plans tend to become a sunken cost fallacy. Making room for changes is hard enough in XP like ways of working. When you have to tell your manager that half a year of careful plans and engineering can be thrown away, because of the new requirements, which emerge from late entry to market, you look like a clown. Plans and complexity usually introduce more risk than less.

sgarland 2y ago
Infra should not require much in the way of redoing if it's done correctly. Foundational software's configuration like RDBMS schema, maybe, but I wouldn't classify that as infra per se. Seriously, I'm struggling to figure out how "we have servers that run containers / applications" would need to be redone just because the application changed.
- chromanoid 2y ago
  Some things that can happen: Product gets canned. Customers want on premise in their data center. Usage spikes are too extreme and serverless is simply the cheapest option. I would always recommend "serverless" monolith first with the option to develop with mocks locally/offline. That's imo the best risk/effort ratio.