6 ms·

Keeping code simple: moving fast by avoiding over-engineering

44 points by mmphosis 3y ago

smokel 3y ago
I agree with the sentiment in the article, but I don't think there is much actionable advice to be taken from it. Understanding what "simple" and "clean" is, requires years of experience. A junior might get caught up in DRY or test-driven principles which do more harm than good. There are tons of blog posts out there that suggest that something is clean and simple, when it not so obviously is not. Take the entire frontend fiasco with HTML, CSS, and JavaScript. How can one go for the simplest architecture in that morass of historically accumulated complexity?
- leetrout 3y ago
  For folks that agree with the post I am replying to: Take a look at the writings of Christopher Alexander. What is missing, this ungraspable "thing" that we only gain through experience is written about as "quality without a name". Yes, there are parts of our profession that are cold, calculating, black and white computer science and mathematics but there is a equal or greater part of what we build on top of these abstractions that is what we create. And that creativity drives us and is more powerful than the black and white calculations and ultimately is what comes out as we show up day after day building a product or a service. https://ryanaferguson.com/2017/01/quality-without-name/ https://ryanaferguson.com/2017/01/quality-without-name/ https://en.wikipedia.org/wiki/The_Timeless_Way_of_Building https://en.wikipedia.org/wiki/The_Timeless_Way_of_Building
  - smokel 3y ago
    Thanks for the link. Might this be related to the concept of "quality" as brought up by Robert Pirsig in "Zen and the Art of Motorcycle Maintenance" (1974)? I don't think the "thing" is completely ungraspable. It is probably the same subject as that which philosophers have studied under the name of aesthetics for ages. Let's continue the rant from here: I tend to link everything back to the concept of evolution as a reason for existing, and aesthetics seems to me to be related to simplicity. If something is simple, it is most often better: i.e. a healthy body, instead of one full of pock marks, or a scientific explanation based on Occam's razor, instead of one based on the pattern of tea leaves. In the long run, simple things have proven to be extremely helpful to humans. Now, the problem with working on extremely complex things, such as societies, buildings, or software, is that these are not simple to begin with. So we have some humans finding extreme joy in trying to come up with ways to understand these in simple terms, even though the terms are still horribly complex to an uninitiated outsider. The building blocks do not always have, nor require, names. That makes them somewhat harder to grasp than your average object such as a table or a chair. But the entire process might be identified as quality, beauty, or cleanness. Or at least, that's what I think. Edit: To avoid common objections to the term "simple", consider that the reversal of something is only one hop away from the actual thing. That may still make a character with pock marks attractive, given that they have, say, an abundance of gold.
    - leetrout 3y ago
      Fascinating... I've not read Pirsig's book but have, of course, heard of it. A quick search led me to wikipedia and this bit: > "Quality," or "value," as described by Pirsig, cannot be defined because it empirically precedes any intellectual construction of it, namely due to the fact that quality (as Pirsig explicitly defines it) exists always as a perceptual experience before it is ever thought of descriptively or academically. https://en.wikipedia.org/wiki/Pirsig%27s_Metaphysics_of_Quality https://en.wikipedia.org/wiki/Pirsig%27s_Metaphysics_of_Qual...
leetrout 3y ago
This is like a "water is wet" post for me. Maybe it is because I have been doing this so long... maybe for another reason. But people need to hear it. The important part is buried at the bottom and I'll re-summarize in less words: Start with high code quality standards [with proficient, experienced devs] delivering the minimal amount of functionality to ship. Then rinse and repeat. With experience comes a developed taste for the general work of software development and a better discernment for where to draw the lines. How to decouple and how to recognize when the decoupling is wrong and refactor. And how to refactor appropriately without making up fear based excuses. If you are interested in more checkout "A Philosophy of Software Design" by Ousterhout, "Righting Software" by Löwy and "Mythical Man Month" by Brooks.
jurschreuder 3y ago
SQLite and Containerize is like 20 minutes to do. Saving as raw json seems more like over-engineering to me because you'll basically end up building a custom database from scratch if it grows.
- coolThingsFirst 3y ago
  Why do you need to containerize for the example he has given?
  - nerdponx 3y ago
    Hosting and continuous delivery become nearly trivial on a lot of platforms, dependency management gets a lot easier, and a consistent dev environment can be provided for the whole team with relatively little effort using multi-stage builds (set up runtime requirements in stage 1, add on the dev stuff in stage 2).
    - coolThingsFirst 3y ago
      ty
fumeux_fume 3y ago
I could be off here, but in my experience developing machine learning solutions, requirements come down from management and PMs. As a developer, I certainly love the idea of not over-engineering a solution, but if the boss wants a feature that requires complexity, there isn't a whole lot I can do. However, I read a lot of code for a living and it never ceases to amaze me how many developers immediately start building complex Python classes before understanding or creating a design for what they're doing.
chacham15 3y ago
The main problem that I see in my years of experience is: "how well does what I design hold up to ACTUAL product design changes?" Everything else comes second to that including this authors version of "build extensible systems around dependencies, this allows you to shift dependencies without shifting internal logic." That idea is good as long as you dont end up rebuilding an entire framework out of that extensible system, otherwise you shouldve used an existing one. If you have an extensible system, but dont use it, thats a lot of complexity that increases the maintenance burden. All of these tradeoffs make sense in their appropriate circumstances, but those circumstances largely rely on predicting the future of the product IMO.
nerdponx 3y ago
The article is confused and self-contradictory. It advises: 1. Punt on complexity 2. Plan for changing requirements Planning for changing requirements often introduces more complexity than not planning. That's the essence of the much-loved YAGNI and WET principles. The article seems to imply that using a frontend web framework, SQLite, and an ORM both introduce complexity and fail to accommodate changing requirements, and that this was the result of the programmer's inability to complete their project both within deadline and up to high standard. I'm not aware of any world in which this is true. Web frameworks, SQLite, and ORMs are flexible tools that encapsulate a lot of complexity. Their entire purpose is to make apps less complicated, by allowing you to reuse existing functionality, abstractions, and interfaces that the app dev does not need to develop, test, and maintain. > You start typing faster, but you also start skipping clean coding patterns. Your functions become more coupled, you directly call external libraries all over, and you skip writing any unit tests. Along the way you also cut and modify scope all over the place, hacking your initial spec to pieces. The article author then says: > I firmly believe writing clean code is just as fast as writing messy code - though it may take more experience and wisdom. Quality software is soft - easy to remold. Let’s take another look at our to-do list from earlier, but implement it a little more cleanly this time, and make it more malleable. But that's clearly not the case. Otherwise the developer in the story wouldn't have felt the need to cut corners on clean code in order to get a working product delivered on time! > initially implement the entire datastore as a JSON blob written to disk. That's clean? That's punting on complexity? Is that really the reason that the developer didn't meet their deadline? Have fun with all the DIY tooling and testing around that system. Yay, we anticipated one particular direction of growth. Now just as you're finishing migrating the entire storage backend to the JSON blob thing, the same PM shows up and tells you that the UX team is pissed off at how slow sorting and filtering got in the new version. Can you make it as fast as the old one? Oops, you blew it. Meanwhile if you had just stuck with the ORM, you could have migrated your storage backend from SQLite to MySQL and kept the rest of the app more or less identical. > You’ve just saved yourself a day of fiddling with a database and reading docs on an ORM someone else built. That's just nonsense. Don't pick new-to-you tools for short-turnaround projects. If you must do so, then you have either have unrealistic deadlines or are unqualified for your particular job. You could of course forego the ORM and write raw SQL strings. Does that solve anything here? No. Does it somehow make your code cleaner? No. Does it make the process of writing clean code faster? No. Does it make the author happy? Apparently, but that's not relevant at my job. The reason that writing clean code is (sometimes) slower is that writing clean code requires planning. People have this weird idea that planning is not agile and agile is not planning, so people don't plan out their code. A similar topic came up in a thread the other day about Dijkstra, but it bears repeating here. Your design process does not end once you've chosen your tools and drawn a couple of arrows on a cocktail napkin. Do you have at least a rough idea of what interface boundaries will be needed? What is your testing strategy? Do you even know what the important invariants of your system are, or which pieces of functionality might need to be tested? Do you need to implement any "algorithms" that can be implemented as pure functions? How do you intend to represent the data? Does that representation work well with your chosen storage backend, or will you need to do some kind of transformation between the database and the app? Etc. All that stuff is a whole lot more important than fussing over whether you've spent too much time reading library documentation. You can write just as much of a spaghetti platter mess with or without 3rd-party libraries. The article is not providing any useful advice for avoiding the spaghetti platter. It only asserts that a tangled linguine platter is better than a tangled spaghetti platter.
coolThingsFirst 3y ago
What does virtualizing the list if it gets too long refer to?
proc0 3y ago
> Over-engineering is an attempt to predict the future, while quality-engineering is maximizing extensibility. That's not really over-engineering, but rather building incorrect abstractions or building at the wrong abstraction level. In other words, it becomes over-engineering because a lot of effort is spent on an incorrect abstraction that is not general or flexible enough. That said, if you don't have time for a proper abstraction, then it is better to do as suggested and keep things as simple as possible, however that also has its downsides because it often leads to re-doing the work once you find out about more requirements. Good abstractions don't require predicting the future, they require "covering" the feature space in such a way that small changes to the "moving parts" will drastically change the behavior of the feature. So it's about creating a solution that is flexible and also powerful, with the trade-off being the transparency of the implementation, which might be difficult for junior engineers and/or managers to understand. I think when done right, you spend a little more time upfront designing but end up with a resilient, flexible, and powerful solution that can pivot quickly for the customer. Unfortunately, and also understandably, this is not a favorable approach in many companies because it is hard to manage and estimate properly.
mediumsmart 3y ago
A simple to-do app would be great even if it came with hundreds of thousands of lines of complicated code.
Zababa 3y ago
> Over-engineering is an attempt to predict the future, while quality-engineering is maximizing extensibility. The article is written as if extensibility is switching databases, but I've also encountered cases where extensibility is adding features, and all the indirection due to abstractions around databases, configuration, caching, view layer, etc makes it harder to extend. There's also the classic failure mode of "designing an interface but using only one implementation", which can lead to its friend "the read-and-write interface is leaky and we can't switch data stores!". And since we're on this, we may also talk about the "every service in this company uses MySQL and Spring Boot but for this small one I chose to use SQLite and Go and now we are in Hell".
- pdimitar 3y ago
  > "every service in this company uses MySQL and Spring Boot but for this small one I chose to use SQLite and Go and now we are in Hell". Not sure I understand why is that a bad thing?

44 points by mmphosis 3y ago

I agree with the sentiment in the article, but I don't think there is much actionable advice to be taken from it. Understanding what "simple" and "clean" is, requires years of experience. A junior might get caught up in DRY or test-driven principles which do more harm than good. There are tons of blog posts out there that suggest that something is clean and simple, when it not so obviously is not. Take the entire frontend fiasco with HTML, CSS, and JavaScript. How can one go for the simplest architecture in that morass of historically accumulated complexity?

leetrout 3y ago
For folks that agree with the post I am replying to: Take a look at the writings of Christopher Alexander. What is missing, this ungraspable "thing" that we only gain through experience is written about as "quality without a name". Yes, there are parts of our profession that are cold, calculating, black and white computer science and mathematics but there is a equal or greater part of what we build on top of these abstractions that is what we create. And that creativity drives us and is more powerful than the black and white calculations and ultimately is what comes out as we show up day after day building a product or a service. https://ryanaferguson.com/2017/01/quality-without-name/ https://ryanaferguson.com/2017/01/quality-without-name/ https://en.wikipedia.org/wiki/The_Timeless_Way_of_Building https://en.wikipedia.org/wiki/The_Timeless_Way_of_Building
- smokel 3y ago
  Thanks for the link. Might this be related to the concept of "quality" as brought up by Robert Pirsig in "Zen and the Art of Motorcycle Maintenance" (1974)? I don't think the "thing" is completely ungraspable. It is probably the same subject as that which philosophers have studied under the name of aesthetics for ages. Let's continue the rant from here: I tend to link everything back to the concept of evolution as a reason for existing, and aesthetics seems to me to be related to simplicity. If something is simple, it is most often better: i.e. a healthy body, instead of one full of pock marks, or a scientific explanation based on Occam's razor, instead of one based on the pattern of tea leaves. In the long run, simple things have proven to be extremely helpful to humans. Now, the problem with working on extremely complex things, such as societies, buildings, or software, is that these are not simple to begin with. So we have some humans finding extreme joy in trying to come up with ways to understand these in simple terms, even though the terms are still horribly complex to an uninitiated outsider. The building blocks do not always have, nor require, names. That makes them somewhat harder to grasp than your average object such as a table or a chair. But the entire process might be identified as quality, beauty, or cleanness. Or at least, that's what I think. Edit: To avoid common objections to the term "simple", consider that the reversal of something is only one hop away from the actual thing. That may still make a character with pock marks attractive, given that they have, say, an abundance of gold.
  - leetrout 3y ago
    Fascinating... I've not read Pirsig's book but have, of course, heard of it. A quick search led me to wikipedia and this bit: > "Quality," or "value," as described by Pirsig, cannot be defined because it empirically precedes any intellectual construction of it, namely due to the fact that quality (as Pirsig explicitly defines it) exists always as a perceptual experience before it is ever thought of descriptively or academically. https://en.wikipedia.org/wiki/Pirsig%27s_Metaphysics_of_Quality https://en.wikipedia.org/wiki/Pirsig%27s_Metaphysics_of_Qual...

leetrout 3y ago

This is like a "water is wet" post for me. Maybe it is because I have been doing this so long... maybe for another reason. But people need to hear it. The important part is buried at the bottom and I'll re-summarize in less words: Start with high code quality standards [with proficient, experienced devs] delivering the minimal amount of functionality to ship. Then rinse and repeat. With experience comes a developed taste for the general work of software development and a better discernment for where to draw the lines. How to decouple and how to recognize when the decoupling is wrong and refactor. And how to refactor appropriately without making up fear based excuses. If you are interested in more checkout "A Philosophy of Software Design" by Ousterhout, "Righting Software" by Löwy and "Mythical Man Month" by Brooks.

jurschreuder 3y ago

SQLite and Containerize is like 20 minutes to do. Saving as raw json seems more like over-engineering to me because you'll basically end up building a custom database from scratch if it grows.

coolThingsFirst 3y ago
Why do you need to containerize for the example he has given?
- nerdponx 3y ago
  Hosting and continuous delivery become nearly trivial on a lot of platforms, dependency management gets a lot easier, and a consistent dev environment can be provided for the whole team with relatively little effort using multi-stage builds (set up runtime requirements in stage 1, add on the dev stuff in stage 2).
  - coolThingsFirst 3y ago
    ty

fumeux_fume 3y ago

I could be off here, but in my experience developing machine learning solutions, requirements come down from management and PMs. As a developer, I certainly love the idea of not over-engineering a solution, but if the boss wants a feature that requires complexity, there isn't a whole lot I can do. However, I read a lot of code for a living and it never ceases to amaze me how many developers immediately start building complex Python classes before understanding or creating a design for what they're doing.

chacham15 3y ago

The main problem that I see in my years of experience is: "how well does what I design hold up to ACTUAL product design changes?" Everything else comes second to that including this authors version of "build extensible systems around dependencies, this allows you to shift dependencies without shifting internal logic." That idea is good as long as you dont end up rebuilding an entire framework out of that extensible system, otherwise you shouldve used an existing one. If you have an extensible system, but dont use it, thats a lot of complexity that increases the maintenance burden. All of these tradeoffs make sense in their appropriate circumstances, but those circumstances largely rely on predicting the future of the product IMO.

nerdponx 3y ago

The article is confused and self-contradictory. It advises: 1. Punt on complexity 2. Plan for changing requirements Planning for changing requirements often introduces more complexity than not planning. That's the essence of the much-loved YAGNI and WET principles. The article seems to imply that using a frontend web framework, SQLite, and an ORM both introduce complexity and fail to accommodate changing requirements, and that this was the result of the programmer's inability to complete their project both within deadline and up to high standard. I'm not aware of any world in which this is true. Web frameworks, SQLite, and ORMs are flexible tools that encapsulate a lot of complexity. Their entire purpose is to make apps less complicated, by allowing you to reuse existing functionality, abstractions, and interfaces that the app dev does not need to develop, test, and maintain. > You start typing faster, but you also start skipping clean coding patterns. Your functions become more coupled, you directly call external libraries all over, and you skip writing any unit tests. Along the way you also cut and modify scope all over the place, hacking your initial spec to pieces. The article author then says: > I firmly believe writing clean code is just as fast as writing messy code - though it may take more experience and wisdom. Quality software is soft - easy to remold. Let’s take another look at our to-do list from earlier, but implement it a little more cleanly this time, and make it more malleable. But that's clearly not the case. Otherwise the developer in the story wouldn't have felt the need to cut corners on clean code in order to get a working product delivered on time! > initially implement the entire datastore as a JSON blob written to disk. That's clean? That's punting on complexity? Is that really the reason that the developer didn't meet their deadline? Have fun with all the DIY tooling and testing around that system. Yay, we anticipated one particular direction of growth. Now just as you're finishing migrating the entire storage backend to the JSON blob thing, the same PM shows up and tells you that the UX team is pissed off at how slow sorting and filtering got in the new version. Can you make it as fast as the old one? Oops, you blew it. Meanwhile if you had just stuck with the ORM, you could have migrated your storage backend from SQLite to MySQL and kept the rest of the app more or less identical. > You’ve just saved yourself a day of fiddling with a database and reading docs on an ORM someone else built. That's just nonsense. Don't pick new-to-you tools for short-turnaround projects. If you must do so, then you have either have unrealistic deadlines or are unqualified for your particular job. You could of course forego the ORM and write raw SQL strings. Does that solve anything here? No. Does it somehow make your code cleaner? No. Does it make the process of writing clean code faster? No. Does it make the author happy? Apparently, but that's not relevant at my job. The reason that writing clean code is (sometimes) slower is that writing clean code requires planning. People have this weird idea that planning is not agile and agile is not planning, so people don't plan out their code. A similar topic came up in a thread the other day about Dijkstra, but it bears repeating here. Your design process does not end once you've chosen your tools and drawn a couple of arrows on a cocktail napkin. Do you have at least a rough idea of what interface boundaries will be needed? What is your testing strategy? Do you even know what the important invariants of your system are, or which pieces of functionality might need to be tested? Do you need to implement any "algorithms" that can be implemented as pure functions? How do you intend to represent the data? Does that representation work well with your chosen storage backend, or will you need to do some kind of transformation between the database and the app? Etc. All that stuff is a whole lot more important than fussing over whether you've spent too much time reading library documentation. You can write just as much of a spaghetti platter mess with or without 3rd-party libraries. The article is not providing any useful advice for avoiding the spaghetti platter. It only asserts that a tangled linguine platter is better than a tangled spaghetti platter.

coolThingsFirst 3y ago

What does virtualizing the list if it gets too long refer to?

proc0 3y ago

> Over-engineering is an attempt to predict the future, while quality-engineering is maximizing extensibility. That's not really over-engineering, but rather building incorrect abstractions or building at the wrong abstraction level. In other words, it becomes over-engineering because a lot of effort is spent on an incorrect abstraction that is not general or flexible enough. That said, if you don't have time for a proper abstraction, then it is better to do as suggested and keep things as simple as possible, however that also has its downsides because it often leads to re-doing the work once you find out about more requirements. Good abstractions don't require predicting the future, they require "covering" the feature space in such a way that small changes to the "moving parts" will drastically change the behavior of the feature. So it's about creating a solution that is flexible and also powerful, with the trade-off being the transparency of the implementation, which might be difficult for junior engineers and/or managers to understand. I think when done right, you spend a little more time upfront designing but end up with a resilient, flexible, and powerful solution that can pivot quickly for the customer. Unfortunately, and also understandably, this is not a favorable approach in many companies because it is hard to manage and estimate properly.

mediumsmart 3y ago

A simple to-do app would be great even if it came with hundreds of thousands of lines of complicated code.

Zababa 3y ago

> Over-engineering is an attempt to predict the future, while quality-engineering is maximizing extensibility. The article is written as if extensibility is switching databases, but I've also encountered cases where extensibility is adding features, and all the indirection due to abstractions around databases, configuration, caching, view layer, etc makes it harder to extend. There's also the classic failure mode of "designing an interface but using only one implementation", which can lead to its friend "the read-and-write interface is leaky and we can't switch data stores!". And since we're on this, we may also talk about the "every service in this company uses MySQL and Spring Boot but for this small one I chose to use SQLite and Go and now we are in Hell".

pdimitar 3y ago
> "every service in this company uses MySQL and Spring Boot but for this small one I chose to use SQLite and Go and now we are in Hell". Not sure I understand why is that a bad thing?