4 ms·

I don't think most people today realize how wasteful modern software engineering is. They think, it's fine to be inefficient, I'll just get a bigger X. And we a

I don't think most people today realize how wasteful modern software engineering is. They think, it's fine to be inefficient, I'll just get a bigger X. And we are now blessed with technology and solutions that make that a possibility, when for the longest time it wasn't. But you don't want to have to optimize when you don't expect to. It takes time away from your features and quality, and is risky. You want to predict when you'll need to grow and how much effort it might take, and control costs. That's why software development should be efficient from the start. By only using what you need, you don't run into situations where you have to optimize to survive. Anyone who says to me "I'll just put everything in PostgreSQL and optimize later" sounds just as naive as the person who says "let's build for scalability before we have any customers". Both are giving themselves problems they don't need, and trying to justify it by thinking it's going to save them time. The former is giving themselves problems in the long term, the latter is giving themselves problems in the short term. Even if these tradeoffs are acceptable to you, they are often unnecessary!

api 3y ago
... then they see the cloud bill. Cloud companies found a way to directly monetize developers' love of complexity and overengineering. It's genius.
- robocat 3y ago
  Developers often love simplicity and underengineering, which can also lead to performance problems. My personal definition of engineering is: choosing good compromises between competing needs.
- ecshafer 3y ago
  This comment will be taught in business schools one day.
  - api 3y ago
    1. Find pathology. 2. Attach toll booth. 3. Profit.
- aleph_minus_one 3y ago
  > Cloud companies found a way to directly monetize developers' love of complexity and overengineering. It's genius. The developers' tastes differ a lot. Consider the extreme counterexamples * the "small code" scene * people who love to port modern games to C64, Amiga, retro consoles, ... The situation is in my opinion rather that quite some managers love complexity and "future-proofness", and thus hire developers who have a similar taste.
- _a_a_a_ 3y ago
  IME it's idiot managers that go for love of complexity and overengineering - though not all mgrs are idiots.
okeuro49 3y ago
> "I'll just put everything in PostgreSQL and optimize later" Sounds good to me!
- 0xbadcafebee 3y ago
  "I'll just drink and eat heavily in my 20s and worry about exercising later" Or you could... not?
  - yjftsjthsd-h 3y ago
    To what benefit? In my (early) twenties, I could do that without consequence. But what benefit is there to not doing it while you can? (I mean, assuming you're doing so at a non dangerous level)
    - richie_adler 3y ago
      I'd ask what's the benefit in drinking, but most people see brain destruction and increased risk of doing something stupid as "fun".
      - JelteF 3y ago
        Alcohol also reduces the level of social awkwardness quite a bit that many people in the software engineering field exhibit. Which can be quite helpful in making new friends or finding a partner. I'd expect a significant percentage of people would not have been with their current partner without alcohol.
    - 0xbadcafebee 3y ago
      > In my (early) twenties, I could do that without consequence I mean, maybe for you, anecdotally. Anecdotally, I had a friend at 23 who almost died due to his unhealthy drinking and his body basically shutting down. Doctors told him he couldn't drink anymore. And studies have shown that any amount of binge drinking/eating is bad for health outcomes long-term. You're actually proving my point... people just hand-waive away warnings about future problems, using cherry-picked "evidence" like "hey my friend binge-drank for 30 years and he's fine", or "my father smoked two packs a day for 50 years and he's still around". Similarly, (ab)use of technology for the wrong thing sets you up for problems later on. Don't do that and you avoid the problems later on. But nobody wants to listen to the doctor telling them to drink less or give up smoking. Just like nobody wants to listen to good advice from experienced technologists. Sigh.
zamadatix 3y ago
A new business already has a laundry list of "I need to ${thing} to survive". Being able to move one such problem a couple of years in the future is already an attractive value before considering most businesses won't last that long anyways (for reasons unrelated to how optimized the database was). I mean if you already have the know how, confidence, and experience to do it right in the same amount of time then duh. If not, is spending precious extra time that early on really going to help you survive long enough to reach where how optimized you infrastructure spend is seems like an important problem for your cash flow? Sometimes I think this then gets conflated with more established businesses where continuing to start every project so heavily this way does tend to create more problems than it solves.
SkyPuncher 3y ago
> Anyone who says to me "I'll just put everything in PostgreSQL and optimize later" sounds just as naive as the person who says "let's build for scalability before we have any customers". Both are giving themselves problems they don't need Both are giving themselves problems, but they are fundamentally different decisions. I will almost always take the "optimize later" approach as there's not always a "later". For most new products, the biggest challenge is finding product market fit. Once you have that, the world is your oyster. I will happily pay a 10% tax after PMF to be 10% better at finding PMF now. If I've found PMF, I will have more revenue, more traction, more budget, etc to fix the problems I created. If I don't have PMF, everything goes in the trash.
- adamnemecek 3y ago
  It's not 10% though, is it.
  - SkyPuncher 3y ago
    10% was just a smallish number. In my experience, it's actually less than 1%. Code typically follows one of two paths: * It's in an unused/underused part of the app. Performance doesn't really matter. * It's in a highly used part of the app. Performance matters, but you'll never be able to predict the features that get layered on top of it. Fix performance as you build those new features.
    - Waterluvian 3y ago
      This is my experience. Get it out there, learn what you don’t yet know about the problem/product space, then improve and optimize. The hard parts, in my experience, are being sufficiently seasoned to know what parts are hard to do later and are worth facing up front, and making sure the decision makers buy into this so you’re not blindsided by the classic “there’s no time for tech debt.”
nijave 3y ago
Especially in small/medium sized businesses, hardware is cheaper than payroll. It's most cost effective to pay people to build features (that drive revenue) while paying a bigger hardware bill than spend engineering hours optimizing hardware usage. At some scale they'll be a break even/shift
hliyan 3y ago
I once ran the daily trade reconciliation of an entire US primary exchange on my laptop, using a PERL script (in 2007). I wrote it in a couple of days, and each day's processing ran in minutes. Ten years later, I witnessed a team spending weeks spinning up spark clusters to perform a job that was significantly smaller. At the time I thought I was the person who was missing some important context. Now I'm beginning to realise that I was not in fact misinformed or insane.
- cutemonster 3y ago
  Maybe you were missing something? You could have been a manager, commanding a team of Spark engineers and an even higher salary?
teaearlgraycold 3y ago
I don’t know if you’ve ever had to build an MVP at an early stage company
numpad0 3y ago
In my fuzzy sentimental memory, software bloat seemed to always self-regulate to 5-10 lines per second of logs or 2-3 screen flickers per minute, whichever applicable. Things go slower and optimizations come about, go faster and another Midleware.js comes to be. My working theory is this is the common denominator max comfortable flickering rate for developers. There's got to be sweet spot for UI, how do I say, fluidity. And that fluid-ness usually aren't controlled for dev tools, tools themselves must evolve to stay at comfortable refresh-rate, by natural selection. That shouldn't be too weird of an idea.
mynameisash 3y ago
> They think, it's fine to be inefficient, I'll just get a bigger X. And we are now blessed with technology and solutions that make that a possibility, when for the longest time it wasn't. Unfortunately, the culture in my group is basically this. We never address things until they're dire inefficiencies; thus, I spend a nontrivial amount of my time addressing workloads that take 10+ hours but need to be faster (and often with minor changes are cut down to 2 hours). Having a culture of "I'll just throw more hardware at it" / "If it takes an hour or ten, I still am going to run it and get the results tomorrow" is costing ridiculous amounts of money and lost time. It's such an uphill battle to try to change that culture.

api 3y ago

... then they see the cloud bill. Cloud companies found a way to directly monetize developers' love of complexity and overengineering. It's genius.

robocat 3y ago
Developers often love simplicity and underengineering, which can also lead to performance problems. My personal definition of engineering is: choosing good compromises between competing needs.
ecshafer 3y ago
This comment will be taught in business schools one day.
- api 3y ago
  1. Find pathology. 2. Attach toll booth. 3. Profit.
aleph_minus_one 3y ago
> Cloud companies found a way to directly monetize developers' love of complexity and overengineering. It's genius. The developers' tastes differ a lot. Consider the extreme counterexamples * the "small code" scene * people who love to port modern games to C64, Amiga, retro consoles, ... The situation is in my opinion rather that quite some managers love complexity and "future-proofness", and thus hire developers who have a similar taste.
_a_a_a_ 3y ago
IME it's idiot managers that go for love of complexity and overengineering - though not all mgrs are idiots.

okeuro49 3y ago

> "I'll just put everything in PostgreSQL and optimize later" Sounds good to me!

0xbadcafebee 3y ago
"I'll just drink and eat heavily in my 20s and worry about exercising later" Or you could... not?
- yjftsjthsd-h 3y ago
  To what benefit? In my (early) twenties, I could do that without consequence. But what benefit is there to not doing it while you can? (I mean, assuming you're doing so at a non dangerous level)
  - richie_adler 3y ago
    I'd ask what's the benefit in drinking, but most people see brain destruction and increased risk of doing something stupid as "fun".
    - JelteF 3y ago
      Alcohol also reduces the level of social awkwardness quite a bit that many people in the software engineering field exhibit. Which can be quite helpful in making new friends or finding a partner. I'd expect a significant percentage of people would not have been with their current partner without alcohol.
  - 0xbadcafebee 3y ago
    > In my (early) twenties, I could do that without consequence I mean, maybe for you, anecdotally. Anecdotally, I had a friend at 23 who almost died due to his unhealthy drinking and his body basically shutting down. Doctors told him he couldn't drink anymore. And studies have shown that any amount of binge drinking/eating is bad for health outcomes long-term. You're actually proving my point... people just hand-waive away warnings about future problems, using cherry-picked "evidence" like "hey my friend binge-drank for 30 years and he's fine", or "my father smoked two packs a day for 50 years and he's still around". Similarly, (ab)use of technology for the wrong thing sets you up for problems later on. Don't do that and you avoid the problems later on. But nobody wants to listen to the doctor telling them to drink less or give up smoking. Just like nobody wants to listen to good advice from experienced technologists. Sigh.

zamadatix 3y ago

A new business already has a laundry list of "I need to ${thing} to survive". Being able to move one such problem a couple of years in the future is already an attractive value before considering most businesses won't last that long anyways (for reasons unrelated to how optimized the database was). I mean if you already have the know how, confidence, and experience to do it right in the same amount of time then duh. If not, is spending precious extra time that early on really going to help you survive long enough to reach where how optimized you infrastructure spend is seems like an important problem for your cash flow? Sometimes I think this then gets conflated with more established businesses where continuing to start every project so heavily this way does tend to create more problems than it solves.

SkyPuncher 3y ago

> Anyone who says to me "I'll just put everything in PostgreSQL and optimize later" sounds just as naive as the person who says "let's build for scalability before we have any customers". Both are giving themselves problems they don't need Both are giving themselves problems, but they are fundamentally different decisions. I will almost always take the "optimize later" approach as there's not always a "later". For most new products, the biggest challenge is finding product market fit. Once you have that, the world is your oyster. I will happily pay a 10% tax after PMF to be 10% better at finding PMF now. If I've found PMF, I will have more revenue, more traction, more budget, etc to fix the problems I created. If I don't have PMF, everything goes in the trash.

adamnemecek 3y ago
It's not 10% though, is it.
- SkyPuncher 3y ago
  10% was just a smallish number. In my experience, it's actually less than 1%. Code typically follows one of two paths: * It's in an unused/underused part of the app. Performance doesn't really matter. * It's in a highly used part of the app. Performance matters, but you'll never be able to predict the features that get layered on top of it. Fix performance as you build those new features.
  - Waterluvian 3y ago
    This is my experience. Get it out there, learn what you don’t yet know about the problem/product space, then improve and optimize. The hard parts, in my experience, are being sufficiently seasoned to know what parts are hard to do later and are worth facing up front, and making sure the decision makers buy into this so you’re not blindsided by the classic “there’s no time for tech debt.”

nijave 3y ago

Especially in small/medium sized businesses, hardware is cheaper than payroll. It's most cost effective to pay people to build features (that drive revenue) while paying a bigger hardware bill than spend engineering hours optimizing hardware usage. At some scale they'll be a break even/shift

hliyan 3y ago

I once ran the daily trade reconciliation of an entire US primary exchange on my laptop, using a PERL script (in 2007). I wrote it in a couple of days, and each day's processing ran in minutes. Ten years later, I witnessed a team spending weeks spinning up spark clusters to perform a job that was significantly smaller. At the time I thought I was the person who was missing some important context. Now I'm beginning to realise that I was not in fact misinformed or insane.

cutemonster 3y ago
Maybe you were missing something? You could have been a manager, commanding a team of Spark engineers and an even higher salary?

teaearlgraycold 3y ago

I don’t know if you’ve ever had to build an MVP at an early stage company

numpad0 3y ago

In my fuzzy sentimental memory, software bloat seemed to always self-regulate to 5-10 lines per second of logs or 2-3 screen flickers per minute, whichever applicable. Things go slower and optimizations come about, go faster and another Midleware.js comes to be. My working theory is this is the common denominator max comfortable flickering rate for developers. There's got to be sweet spot for UI, how do I say, fluidity. And that fluid-ness usually aren't controlled for dev tools, tools themselves must evolve to stay at comfortable refresh-rate, by natural selection. That shouldn't be too weird of an idea.

mynameisash 3y ago

> They think, it's fine to be inefficient, I'll just get a bigger X. And we are now blessed with technology and solutions that make that a possibility, when for the longest time it wasn't. Unfortunately, the culture in my group is basically this. We never address things until they're dire inefficiencies; thus, I spend a nontrivial amount of my time addressing workloads that take 10+ hours but need to be faster (and often with minor changes are cut down to 2 hours). Having a culture of "I'll just throw more hardware at it" / "If it takes an hour or ten, I still am going to run it and get the results tomorrow" is costing ridiculous amounts of money and lost time. It's such an uphill battle to try to change that culture.