5 ms·

At a previous job, there was an argument over a code review where I had done some SQL queries that fixed a problem but were not optimal. The other side were ve

At a previous job, there was an argument over a code review where I had done some SQL queries that fixed a problem but were not optimal. The other side were very much "this won't work for 1000 devices! we will not approve it!" whereas my stance was "we have a maximum of 25 devices deployed by our only customer who is going to leave us next week unless we fix this problem today". One of the most disheartening weeks of my software development life. (that was also the place I had to have a multi-day argument over the precise way to define constants in Perl because they varied in performance except it was a long running mod_perl server process and the constants were only defined at startup and it made absolutely zero difference once it had been running for an hour or more.)

b112 2y ago
I actually like having room for optimization, especially when running my own infra servers included. As an example, I can think of half a dozen things I can currently optimize just in the DB layer, but my time is being spent (sensibly!) in other areas that are customer facing and directly impacting them. So fix what needs to be fixed, but if there was a major load spike due to onboarding of new clients/users, I could in a matter of hours have the DB handling 100x the traffic. That's a nice ace in the back of my pocket. And yes, if I had endless time I'd have resolved all issues.
- zimpenfish 2y ago
  > if there was a major load spike due to onboarding of new clients/users This company was a hardware company with reasonably complex installation procedures - even going full whack, I doubt they could have added more than 20 new devices a week (and even then there'd be a hefty lead time to get the stuff manufactured and shipped etc.)
- nine_k 2y ago
  If that optimization is mere hours of work, I would go for it outright. BTW when you have an overwhelming wave of signups, you are likely to have more pressing and unexpected issues, and badly lack spare hours. Usually serious gains that are postponed would require days and weeks of effort. Maybe mere hours of coding proper, but much longer time testing, migrating data, etc.
  - xandrius 2y ago
    I think that's the pitfall: there are infinite things a skilled developer can do within "mere hours of work". The key is to find which ones are the most effective use of one's limited hours. I developed a small daily game and it has now grown to over 10K DAU, so now I've started going back to refine the low hanging fruits which just didn't make sense to touch when I had just 10s of players a day.
    - Jach 2y ago
      We might all be operating under different ideas of what "matter of hours" means. Often times in a project, at least for me, the range of things that I can start and finish in mere hours is not actually infinite, but rather constrained. Only the simplest and smallest things can be done so fast or faster. So many other things just take longer, at least a day and a half, and can take extra long when other teams are involved. More experienced developers can learn how to break things down into increments, so while an entire big feature might take weeks to do, you're making clear chunks of progress in much smaller units. Those chunks are probably not mostly sized in mere hours pieces, though... You're right that priority matters. Just beware of priority queue starvation. Still, if some newly discovered bug isn't urgent, even if I think it'd be rather easy (under an hour) to fully address, I'd rather not break my current flow, and just keep working on the thing I had earlier decided was highest priority. A lot of the time something will prevent direct progress and break the flow anyway, having smaller items available to quickly context switch to and finish is a good use of those gap times. The "DB handling 100x the traffic" example above isn't quite well defined. I wonder if it's making queries return 100x faster? Or is it making sure the queries return at roughly their current speeds even if there's 100x more traffic? Either way, I can make arguments for doing the work proactively rather than reactively, but I'd at least write down the half a dozen things. Then maybe someone else can do them, and maybe those things can be done in around half a dozen tiny increments of 30 minutes or less each, instead of all at once in hours.
      - zimpenfish 2y ago
        > Or is it making sure the queries return at roughly their current speeds even if there's 100x more traffic? From what I remember, it was this. The DB was MySQL and a whole bunch of stuff would have been less efficient when there were 1000 devices instead of 25. But on the other hand the system was broken and the customer was threatening to cancel everything and fixing the DB stuff was going to take a fair amount of rearchitecting (not least dumping MySQL for something less insane) that we didn't have the time or resources to do in a hurry.
- willvarfar 2y ago
  Usually a good trick is to run small deployments at high logging levels. Then, as soon as there are performance issues, you can dial down the logging and get the hours of respite needed to actually make a bigger algorithmic improvement.
- soco 2y ago
  like != need
renatovico 2y ago
totally agree with your view, the big design upfront moment again, maybe because of the current economy, we need focus more on be profitable in short term, i think it's great and always focus on optimize for now, and test for specs (specs in sense of requirements of customer)
- throwaway48540 2y ago
  I really don't think that's the case. If you ask a CEO/CTO of a startup, he would fire the guy who did the latter instead of the middle approach. Longer term stability and development velocity are very important concerns in engineering management. This is pure inexperience - it wouldn't take too long to setup for an experienced engineer anyways, they probably did it 5 times last month and have a library of knowledge and templates. I can't call a guy who isn't capable of setting up a project this way swiftly without seeing it as a problem "senior".
  - dambi0 2y ago
    What is the middle approach in this scenario? What specifically are the templates you refer to?
    - throwaway48540 2y ago
      The middle approach would be implementing the infrastructure in a basic way that is simplistic, but provides benefit and can be expanded.
      - dambi0 2y ago
        How does that relate to the scenario and what of the templates?
      - zimpenfish 2y ago
        > The middle approach would be implementing the infrastructure That's great if you're implementing but it doesn't really work when you're coming in to an existing infrastructure (or codebase) that other people manage.
meribold 2y ago
> I had to have a multi-day argument over the precise way to define constants in Perl Couldn't you have used whatever the other person was suggesting even if the change was pointless?
- zimpenfish 2y ago
  These days, I would, yeah, but this was a long time ago and I was a lot more invested in not letting nonsense go past without a fight.
  - JohnBooty 2y ago
    Yeah it's one of those things where it's like... ...okay, I can let this nonsense go today with very little impact, but if I do... will the team need to deal with it 100 times in the future? (And more broadly, by silence am I contributing to the evolution of a bad engineering culture ruled by poor understandings?) It is very very difficult to know where to draw the line.
    - Izkata 2y ago
      Three axes in my mind to take into account: Likelihood, impact, and complexity. The article seems to mostly be about likelihood and complexity. I'd place more importance on impact and complexity. It's worth saving a headache later if it's simple to do right now, however unlikely it is, as long as it doesn't create a maintenance burden before that point.
lofaszvanitt 2y ago
This is why everyone uses microservices and React. We don't know how these work, but we are netfligs/farcebook level companiez, so we must.
- mardifoufs 2y ago
  What? How does react fit into that? It's just a web framework that works for any scale basically. If you are a react dev and familiar with react, it's as easy as anything else to use. Now sure if you're not familiar with it you'd have to learn it but that's not an issue with react.
  - 123yawaworht456 2y ago
    ~90% of SPAs could be static HTML with a few kilobytes of vanilla JS sprinkled in for AJAX.
    - soerxpso 2y ago
      And they could also be react apps. What's your point? For a lot of people, a suitable react app is easier to make than the equivalent static HTML page with vanilla JS. The "it should be static HTML with a sprinkle of JS so the filesize is extra small!" guy is the one overengineering, here. The SPA's 20 customers are all using modern web browsers on modern first-world internet connections and don't care or notice that the page is not Stallman-approved.
      - Timber-6539 2y ago
        Stallman-approved pages will happily welcome any and all users visiting your site. Modern SPAs maybe not so much due to the overengineering and mostly unnecessary features.
      - walteweiss 2y ago
        [dead]
    - mardifoufs 2y ago
anal_reactor 2y ago
I'm stunned to realize that most developers just blindly follow whatever are the newest hottest "good practices" and completely ignore the actual goals that code should try to achieve. Literally yesterday I had an argument with a coworker because he wouldn't buy the argument "High-availability costs us $2000 per year extra, downtime costs us $100 per day, so if the database breaks once per year, there's no point having high-availability". Some time ago someone important from upper management announced that we need to pump out features faster to stay afloat as a business, even if they're a bit broken, and two hours later the most senior programmer in my team introduced us to new bureaucracy "because that's the best practice" and he wouldn't understand that what he's doing is directly at odds with the company's goals. Same guy also often accuses me of "not seeing the bigger picture". I gave up trying to reason with coworkers because it's just pain and changes nothing in the long run besides exhausting me.
- mewpmewp2 2y ago
  On the other side of the argument, does $2000 a year seem a lot for high-availability? Does downtime really cost only $100 per day? How was it calculated? How much does your business make? It would seem it should make more than 365 * $100 = $36,500 to be able to be in a position to hire people in the first place. Database downtime would potentially: 1) Break trust with customers. 2) Take away focus from your engs for a day or more which is also a cost. If an eng costs $200 per day for example, and you have 4 engs involved with, it's already a cost of $800, not to mention increase of odds of stress and burn out. And in US engs would cost of course much more than that, I was just picking a more favourable sum. 3) If it's a database that breaks, it's likely to cause further damages due to issues with data which might take indefinite time to fix. Overall in most businesses it would seem a no brainer to pay $2000 a year to have high availability and avoid any odds of a database breaking. It's very little compared to what you pay your engineers.
  - anal_reactor 2y ago
    The whole thing is a side project with 5 customers total. If it dies it'll take a while before anyone notices.
SebFender 2y ago
In my last company that was always the golden rule - manage the present challenge with an eye on the future - but let's get things done.
JohnBooty 2y ago
Ugggggggggggh. My favorite part of the job is scaling/optimization, and my least favorite part of the job is... this. There are a LOT of "engineers" who understand that one thing might be faster than another thing, but lack the chops/wisdom to understand when it actually matters.
- mewpmewp2 2y ago
  Best example I've seen is wanting to optimize a frontend JS for loop that gets triggered maximally only once per page load, runs max 100 times in a loop with complicated data structures that will confuse any reader of that code why it was done like this and not just with a for loop.
  - JohnBooty 2y ago
    It's such a Dunning-Kruger effect. You get legitimately talented engineers who are talented enough to do a legitimately hard thing: optimizing a complex piece of code. So, they don't realize that there's kind of another level of understanding beyond that (when to optimize vs. how to optimize) There's also the fact that some people just like optimizing shit for selfish or even slightly nefarious reasons. Some just like doing it because it's fun. Others like it because it gives them visible and easily quantified "wins." On the rarer and more nefarious side of things, they may enjoy now having "ownership" over a piece of complex code that only they can understand....
    - mewpmewp2 2y ago
      I don't know how to exactly classify this, but based on my experience it feels like a certain type of engs who have only been exposed to this type of way of doing things from when they started, like at uni. It's to do with going through certain type of education, after that certain type of enterprisey environment following very strict rules of design patterns and applying these patterns to absolutely everything they do. They haven't done many quicker projects or side projects to realize that simpler things can work absolutely well enough to realize there can be flexiblity. I think they are good people and outside of work I get along well with them, but during work I do feel frustrated and when I try to explain it to them, I get kind of a West World like response "Doesn't look like anything to me". Like they can't get it, because they only have the uni and enterprise experience, and not experience to build things on their own volition.
OutOfHere 2y ago
With SQL you really should support 10x your current user count, rounded up to the nearest power of 10. If you have 25 devices, this means you should support 250 rounded up to 1000. If you don't, it will come back to bite you really hard and really quickly. This is just something that you learn from experience. Obviously I am not talking about an emergency situation, but about planned design.

b112 2y ago

I actually like having room for optimization, especially when running my own infra servers included. As an example, I can think of half a dozen things I can currently optimize just in the DB layer, but my time is being spent (sensibly!) in other areas that are customer facing and directly impacting them. So fix what needs to be fixed, but if there was a major load spike due to onboarding of new clients/users, I could in a matter of hours have the DB handling 100x the traffic. That's a nice ace in the back of my pocket. And yes, if I had endless time I'd have resolved all issues.

zimpenfish 2y ago
> if there was a major load spike due to onboarding of new clients/users This company was a hardware company with reasonably complex installation procedures - even going full whack, I doubt they could have added more than 20 new devices a week (and even then there'd be a hefty lead time to get the stuff manufactured and shipped etc.)
nine_k 2y ago
If that optimization is mere hours of work, I would go for it outright. BTW when you have an overwhelming wave of signups, you are likely to have more pressing and unexpected issues, and badly lack spare hours. Usually serious gains that are postponed would require days and weeks of effort. Maybe mere hours of coding proper, but much longer time testing, migrating data, etc.
- xandrius 2y ago
  I think that's the pitfall: there are infinite things a skilled developer can do within "mere hours of work". The key is to find which ones are the most effective use of one's limited hours. I developed a small daily game and it has now grown to over 10K DAU, so now I've started going back to refine the low hanging fruits which just didn't make sense to touch when I had just 10s of players a day.
  - Jach 2y ago
    We might all be operating under different ideas of what "matter of hours" means. Often times in a project, at least for me, the range of things that I can start and finish in mere hours is not actually infinite, but rather constrained. Only the simplest and smallest things can be done so fast or faster. So many other things just take longer, at least a day and a half, and can take extra long when other teams are involved. More experienced developers can learn how to break things down into increments, so while an entire big feature might take weeks to do, you're making clear chunks of progress in much smaller units. Those chunks are probably not mostly sized in mere hours pieces, though... You're right that priority matters. Just beware of priority queue starvation. Still, if some newly discovered bug isn't urgent, even if I think it'd be rather easy (under an hour) to fully address, I'd rather not break my current flow, and just keep working on the thing I had earlier decided was highest priority. A lot of the time something will prevent direct progress and break the flow anyway, having smaller items available to quickly context switch to and finish is a good use of those gap times. The "DB handling 100x the traffic" example above isn't quite well defined. I wonder if it's making queries return 100x faster? Or is it making sure the queries return at roughly their current speeds even if there's 100x more traffic? Either way, I can make arguments for doing the work proactively rather than reactively, but I'd at least write down the half a dozen things. Then maybe someone else can do them, and maybe those things can be done in around half a dozen tiny increments of 30 minutes or less each, instead of all at once in hours.
    - zimpenfish 2y ago
      > Or is it making sure the queries return at roughly their current speeds even if there's 100x more traffic? From what I remember, it was this. The DB was MySQL and a whole bunch of stuff would have been less efficient when there were 1000 devices instead of 25. But on the other hand the system was broken and the customer was threatening to cancel everything and fixing the DB stuff was going to take a fair amount of rearchitecting (not least dumping MySQL for something less insane) that we didn't have the time or resources to do in a hurry.
willvarfar 2y ago
Usually a good trick is to run small deployments at high logging levels. Then, as soon as there are performance issues, you can dial down the logging and get the hours of respite needed to actually make a bigger algorithmic improvement.
soco 2y ago
like != need

renatovico 2y ago

totally agree with your view, the big design upfront moment again, maybe because of the current economy, we need focus more on be profitable in short term, i think it's great and always focus on optimize for now, and test for specs (specs in sense of requirements of customer)

throwaway48540 2y ago
I really don't think that's the case. If you ask a CEO/CTO of a startup, he would fire the guy who did the latter instead of the middle approach. Longer term stability and development velocity are very important concerns in engineering management. This is pure inexperience - it wouldn't take too long to setup for an experienced engineer anyways, they probably did it 5 times last month and have a library of knowledge and templates. I can't call a guy who isn't capable of setting up a project this way swiftly without seeing it as a problem "senior".
- dambi0 2y ago
  What is the middle approach in this scenario? What specifically are the templates you refer to?
  - throwaway48540 2y ago
    The middle approach would be implementing the infrastructure in a basic way that is simplistic, but provides benefit and can be expanded.
    - dambi0 2y ago
      How does that relate to the scenario and what of the templates?
    - zimpenfish 2y ago
      > The middle approach would be implementing the infrastructure That's great if you're implementing but it doesn't really work when you're coming in to an existing infrastructure (or codebase) that other people manage.

meribold 2y ago

> I had to have a multi-day argument over the precise way to define constants in Perl Couldn't you have used whatever the other person was suggesting even if the change was pointless?

zimpenfish 2y ago
These days, I would, yeah, but this was a long time ago and I was a lot more invested in not letting nonsense go past without a fight.
- JohnBooty 2y ago
  Yeah it's one of those things where it's like... ...okay, I can let this nonsense go today with very little impact, but if I do... will the team need to deal with it 100 times in the future? (And more broadly, by silence am I contributing to the evolution of a bad engineering culture ruled by poor understandings?) It is very very difficult to know where to draw the line.
  - Izkata 2y ago
    Three axes in my mind to take into account: Likelihood, impact, and complexity. The article seems to mostly be about likelihood and complexity. I'd place more importance on impact and complexity. It's worth saving a headache later if it's simple to do right now, however unlikely it is, as long as it doesn't create a maintenance burden before that point.

lofaszvanitt 2y ago

This is why everyone uses microservices and React. We don't know how these work, but we are netfligs/farcebook level companiez, so we must.

mardifoufs 2y ago
What? How does react fit into that? It's just a web framework that works for any scale basically. If you are a react dev and familiar with react, it's as easy as anything else to use. Now sure if you're not familiar with it you'd have to learn it but that's not an issue with react.
- 123yawaworht456 2y ago
  ~90% of SPAs could be static HTML with a few kilobytes of vanilla JS sprinkled in for AJAX.
  - soerxpso 2y ago
    And they could also be react apps. What's your point? For a lot of people, a suitable react app is easier to make than the equivalent static HTML page with vanilla JS. The "it should be static HTML with a sprinkle of JS so the filesize is extra small!" guy is the one overengineering, here. The SPA's 20 customers are all using modern web browsers on modern first-world internet connections and don't care or notice that the page is not Stallman-approved.
    - Timber-6539 2y ago
      Stallman-approved pages will happily welcome any and all users visiting your site. Modern SPAs maybe not so much due to the overengineering and mostly unnecessary features.
    - walteweiss 2y ago
      [dead]
  - mardifoufs 2y ago

anal_reactor 2y ago

I'm stunned to realize that most developers just blindly follow whatever are the newest hottest "good practices" and completely ignore the actual goals that code should try to achieve. Literally yesterday I had an argument with a coworker because he wouldn't buy the argument "High-availability costs us $2000 per year extra, downtime costs us $100 per day, so if the database breaks once per year, there's no point having high-availability". Some time ago someone important from upper management announced that we need to pump out features faster to stay afloat as a business, even if they're a bit broken, and two hours later the most senior programmer in my team introduced us to new bureaucracy "because that's the best practice" and he wouldn't understand that what he's doing is directly at odds with the company's goals. Same guy also often accuses me of "not seeing the bigger picture". I gave up trying to reason with coworkers because it's just pain and changes nothing in the long run besides exhausting me.

mewpmewp2 2y ago
On the other side of the argument, does $2000 a year seem a lot for high-availability? Does downtime really cost only $100 per day? How was it calculated? How much does your business make? It would seem it should make more than 365 * $100 = $36,500 to be able to be in a position to hire people in the first place. Database downtime would potentially: 1) Break trust with customers. 2) Take away focus from your engs for a day or more which is also a cost. If an eng costs $200 per day for example, and you have 4 engs involved with, it's already a cost of $800, not to mention increase of odds of stress and burn out. And in US engs would cost of course much more than that, I was just picking a more favourable sum. 3) If it's a database that breaks, it's likely to cause further damages due to issues with data which might take indefinite time to fix. Overall in most businesses it would seem a no brainer to pay $2000 a year to have high availability and avoid any odds of a database breaking. It's very little compared to what you pay your engineers.
- anal_reactor 2y ago
  The whole thing is a side project with 5 customers total. If it dies it'll take a while before anyone notices.

SebFender 2y ago

In my last company that was always the golden rule - manage the present challenge with an eye on the future - but let's get things done.

JohnBooty 2y ago

Ugggggggggggh. My favorite part of the job is scaling/optimization, and my least favorite part of the job is... this. There are a LOT of "engineers" who understand that one thing might be faster than another thing, but lack the chops/wisdom to understand when it actually matters.

mewpmewp2 2y ago
Best example I've seen is wanting to optimize a frontend JS for loop that gets triggered maximally only once per page load, runs max 100 times in a loop with complicated data structures that will confuse any reader of that code why it was done like this and not just with a for loop.
- JohnBooty 2y ago
  It's such a Dunning-Kruger effect. You get legitimately talented engineers who are talented enough to do a legitimately hard thing: optimizing a complex piece of code. So, they don't realize that there's kind of another level of understanding beyond that (when to optimize vs. how to optimize) There's also the fact that some people just like optimizing shit for selfish or even slightly nefarious reasons. Some just like doing it because it's fun. Others like it because it gives them visible and easily quantified "wins." On the rarer and more nefarious side of things, they may enjoy now having "ownership" over a piece of complex code that only they can understand....
  - mewpmewp2 2y ago
    I don't know how to exactly classify this, but based on my experience it feels like a certain type of engs who have only been exposed to this type of way of doing things from when they started, like at uni. It's to do with going through certain type of education, after that certain type of enterprisey environment following very strict rules of design patterns and applying these patterns to absolutely everything they do. They haven't done many quicker projects or side projects to realize that simpler things can work absolutely well enough to realize there can be flexiblity. I think they are good people and outside of work I get along well with them, but during work I do feel frustrated and when I try to explain it to them, I get kind of a West World like response "Doesn't look like anything to me". Like they can't get it, because they only have the uni and enterprise experience, and not experience to build things on their own volition.

OutOfHere 2y ago

With SQL you really should support 10x your current user count, rounded up to the nearest power of 10. If you have 25 devices, this means you should support 250 rounded up to 1000. If you don't, it will come back to bite you really hard and really quickly. This is just something that you learn from experience. Obviously I am not talking about an emergency situation, but about planned design.