6 ms·

Software Development: Mistakes I Made

127 points by herodotus 6y ago

wmichelin 6y ago
This seems to directly contradict YAGNI. I find that when I "write for the future", I over-engineer things and they get messy and bloated.
- 0xFACEFEED 6y ago
  I find the opposite. This topic is fascinating to me actually. I'm a system oriented thinker. I don't know how I do it, but I'm able to design forward-thinking systems in my head... really well. Like large complex systems. I've since validated this with retrospectives (checking my past decisions, how they worked out, and how they compare to others). The systems turn out to be fairly elegant in their simplicity and ability to scale. So what's going on here? I think maybe the answer is pretty simple. Some people are good at building large abstract systems and some people aren't. In fact one thing I'm bad at is narrowly focusing on problems. My solutions are always good enough but not great there. Oversimplified example: if you were to ask me to build a highly performant sorting algorithm I'd probably just give you an okay solution. But if you were to ask me to design a distributed system that runs these sorting algorithms (where the algorithms itself is a black box; implemented already) I'd do really well!
- breischl 6y ago
  I took his meaning as "assume better hardware will be available in the future", which is not what you seem to mean, and may not be as true these days anyway. That said, when I've seen people over-engineer things (and maybe this isn't your problem, it's just what I've seen most) it's usually because instead of solving the problem in front of them, they created a flexible framework to solve a class of problems including the one in front of them. And it usually fails because they don't actually know what other problems they'll encounter - they guess and get it wrong. So their framework is flexible on Dimension X, but they actually need it to be flexible on Dimension Y. The best advice I read on this was to never build a framework until you have at least 3 examples of the problem. Once you've got 3 examples, you have a good feel for which ways it needs to be flexible. I would add that you should also make sure you'll eventually have more than 3, because if 3 is the total number of concrete examples you still don't need a framework. Also, it's entirely possible some smart guy/gal can give you those 3 examples before you even start building. A good product person can do that. If you have that, and they're pretty sure you'll actually get there, then go ahead and build it. Difficulty of changing it in the future also plays into this, but I've typed too much already. :)
kccqzy 6y ago
> However this time I didn’t rush to implement, thanks mainly to the influence of Ed Ashcroft. In retrospect Ed’s first impulse when he had an idea was to document, not implement. We produced a series of widely read papers and soon had other researchers and brilliant students of our own working on it. I'm not sure this actually applies to most people. The document vs implement struggle is something I see in modern software shops a lot. It certainly depends on the personality: some people just like thinking through everything and producing a coherent writeup before writing a single line of code, but for some writing code itself is an act that clarifies the still-murky concepts and helps to produce a good writeup. But in the end, most people aren't producing novel enough technologies that the benefit of a complete writeup is there. You don't write something brilliant and have other researchers work on it; you merely write something for the benefit of a future colleague understand your code quickly. It's a very different class of software development.
- jasonpeacock 6y ago
  Most modern software shops don't document at all. They would definitely benefit from writing down a problem statement and description of the solution so that everyone is on the same page. Documenting doesn't have to mean design-first development, it's a way to make sure you're working the right problems and building the desired solutions before you waste your time writing code.
  - mgkimsal 6y ago
    I've tried to force myself to write something in English before coding, and... it's not 100% followed, but it's often helped. Sometimes I'm not even prescribing the full 'solution', but it's a sketch of the pros/cons of a couple approaches. This can help me remind myself later of why a particular option was chosen, even if it turns out to be bad. I can't say if anyone else who looks at the code ever also reads what I've written - it seems that generally they don't (even though I announce it and it's in the commit activity stream and PRs).
- ansq 6y ago
  > "for some writing code itself is an act that clarifies the still-murky concepts and helps to produce a good writeup." This is my philosophy after ~2 years of working on distributed systems. I got sick of vague design discussions where everyone has a 5 minute memory. I am much happier to sit at my desk and figure it out by writing code, then produce a design doc once I have a solid prototype working. It feels more honest and real. At the same time, I wonder if I'm missing out on a different way of doing things.
akkartik 6y ago
Lovely stories hamstrung by the perceived need for a moral.
philipwhiuk 6y ago
Write for the future sounds great until you remember hindsight is 20-20 and predictions aren't.
- clarry 6y ago
  Yep. You can write for a future where desktops boast single-core speeds in the 10GHz range. That's how you get Crysis, which still has performance issues on the latest CPUs.
dullroar 6y ago
Back in the 1990s, when I was a software engineer at a database company in the Bay Area (rhymed with "PsyBass", if you pronounce "bass" as a musical instrument :), there was a joke among some of the architects along the lines of, "I DREW the boxes and arrows! If you can't implement it, that's you're fault!" Now, as a "software architect" in a really, really small shop, I basically just try and lead by example, and make sure my ideas work as code before inflicting them on anyone else. But that sorta sounds like keeping it all to myself until I am sure it works, per the article. I like to document (more than most), but have found over the years the only person who reads my documentation tends to be me. So writing docs or writing code, either way, tends to help me refine my ideas.
dacox 6y ago
Hey, Bill Wadge was a prof of mine! It was a topics in Artificial Intelligence course just before the Deep Learning boom. It was all about entailment and modal logics and frankly seemed really out of date. It was interesting, however.
frequentnapper 6y ago
I had the bad fortune of taking this guy's "computer science" class a long time ago as an undergrad. He spent the entire first week covering WW2 and other history. Then during the rest of the semester didn't teach anything. The icing on the cake was when we were handed evaluation forms he said to the class "It doesn't matter what you fill out because I have tenure."
- MetalGuru 6y ago
  I didn’t find any valuable insight in this. Get help and build for the future... ok? Also, I felt his anecdotes barely got whatever point he was trying to make across. And half the post consisted of name drops of people I’d never heard of.
  - herodotus 6y ago
    Butler Lampson: https://en.wikipedia.org/wiki/Butler_Lampson https://en.wikipedia.org/wiki/Butler_Lampson L. Peter Deutsch: https://en.wikipedia.org/wiki/L._Peter_Deutsch https://en.wikipedia.org/wiki/L._Peter_Deutsch The point is, these two people are incredible Computer Scientists.
- spicyramen 6y ago
  And?
- bade 6y ago
  What I meant was, say what you think, don't worry about getting me in trouble. As for the course , you''re saying I didn't teach for the future. Guilty.
  - frequentnapper 6y ago
    You didn't teach anything. Period. This was the opinion of all my classmates and others I knew who took your class. If you don't believe me, see for yourself: https://www.ratemyprofessors.com/ShowRatings.jsp?tid=48636 https://www.ratemyprofessors.com/ShowRatings.jsp?tid=48636
coldcode 6y ago
Everyone makes mistakes, the key is to find ways to avoid making them over and over.
some_eejit 6y ago
I was a CS student at Warwick (UK university) in the late 80s, and clearly remembering doing functional programming with Lucid. Never heard of it since.

127 points by herodotus 6y ago

This seems to directly contradict YAGNI. I find that when I "write for the future", I over-engineer things and they get messy and bloated.

0xFACEFEED 6y ago
I find the opposite. This topic is fascinating to me actually. I'm a system oriented thinker. I don't know how I do it, but I'm able to design forward-thinking systems in my head... really well. Like large complex systems. I've since validated this with retrospectives (checking my past decisions, how they worked out, and how they compare to others). The systems turn out to be fairly elegant in their simplicity and ability to scale. So what's going on here? I think maybe the answer is pretty simple. Some people are good at building large abstract systems and some people aren't. In fact one thing I'm bad at is narrowly focusing on problems. My solutions are always good enough but not great there. Oversimplified example: if you were to ask me to build a highly performant sorting algorithm I'd probably just give you an okay solution. But if you were to ask me to design a distributed system that runs these sorting algorithms (where the algorithms itself is a black box; implemented already) I'd do really well!
breischl 6y ago
I took his meaning as "assume better hardware will be available in the future", which is not what you seem to mean, and may not be as true these days anyway. That said, when I've seen people over-engineer things (and maybe this isn't your problem, it's just what I've seen most) it's usually because instead of solving the problem in front of them, they created a flexible framework to solve a class of problems including the one in front of them. And it usually fails because they don't actually know what other problems they'll encounter - they guess and get it wrong. So their framework is flexible on Dimension X, but they actually need it to be flexible on Dimension Y. The best advice I read on this was to never build a framework until you have at least 3 examples of the problem. Once you've got 3 examples, you have a good feel for which ways it needs to be flexible. I would add that you should also make sure you'll eventually have more than 3, because if 3 is the total number of concrete examples you still don't need a framework. Also, it's entirely possible some smart guy/gal can give you those 3 examples before you even start building. A good product person can do that. If you have that, and they're pretty sure you'll actually get there, then go ahead and build it. Difficulty of changing it in the future also plays into this, but I've typed too much already. :)

kccqzy 6y ago

> However this time I didn’t rush to implement, thanks mainly to the influence of Ed Ashcroft. In retrospect Ed’s first impulse when he had an idea was to document, not implement. We produced a series of widely read papers and soon had other researchers and brilliant students of our own working on it. I'm not sure this actually applies to most people. The document vs implement struggle is something I see in modern software shops a lot. It certainly depends on the personality: some people just like thinking through everything and producing a coherent writeup before writing a single line of code, but for some writing code itself is an act that clarifies the still-murky concepts and helps to produce a good writeup. But in the end, most people aren't producing novel enough technologies that the benefit of a complete writeup is there. You don't write something brilliant and have other researchers work on it; you merely write something for the benefit of a future colleague understand your code quickly. It's a very different class of software development.

jasonpeacock 6y ago
Most modern software shops don't document at all. They would definitely benefit from writing down a problem statement and description of the solution so that everyone is on the same page. Documenting doesn't have to mean design-first development, it's a way to make sure you're working the right problems and building the desired solutions before you waste your time writing code.
- mgkimsal 6y ago
  I've tried to force myself to write something in English before coding, and... it's not 100% followed, but it's often helped. Sometimes I'm not even prescribing the full 'solution', but it's a sketch of the pros/cons of a couple approaches. This can help me remind myself later of why a particular option was chosen, even if it turns out to be bad. I can't say if anyone else who looks at the code ever also reads what I've written - it seems that generally they don't (even though I announce it and it's in the commit activity stream and PRs).
ansq 6y ago
> "for some writing code itself is an act that clarifies the still-murky concepts and helps to produce a good writeup." This is my philosophy after ~2 years of working on distributed systems. I got sick of vague design discussions where everyone has a 5 minute memory. I am much happier to sit at my desk and figure it out by writing code, then produce a design doc once I have a solid prototype working. It feels more honest and real. At the same time, I wonder if I'm missing out on a different way of doing things.

akkartik 6y ago

Lovely stories hamstrung by the perceived need for a moral.

philipwhiuk 6y ago

Write for the future sounds great until you remember hindsight is 20-20 and predictions aren't.

clarry 6y ago
Yep. You can write for a future where desktops boast single-core speeds in the 10GHz range. That's how you get Crysis, which still has performance issues on the latest CPUs.

dullroar 6y ago

Back in the 1990s, when I was a software engineer at a database company in the Bay Area (rhymed with "PsyBass", if you pronounce "bass" as a musical instrument :), there was a joke among some of the architects along the lines of, "I DREW the boxes and arrows! If you can't implement it, that's you're fault!" Now, as a "software architect" in a really, really small shop, I basically just try and lead by example, and make sure my ideas work as code before inflicting them on anyone else. But that sorta sounds like keeping it all to myself until I am sure it works, per the article. I like to document (more than most), but have found over the years the only person who reads my documentation tends to be me. So writing docs or writing code, either way, tends to help me refine my ideas.

dacox 6y ago

Hey, Bill Wadge was a prof of mine! It was a topics in Artificial Intelligence course just before the Deep Learning boom. It was all about entailment and modal logics and frankly seemed really out of date. It was interesting, however.

frequentnapper 6y ago

I had the bad fortune of taking this guy's "computer science" class a long time ago as an undergrad. He spent the entire first week covering WW2 and other history. Then during the rest of the semester didn't teach anything. The icing on the cake was when we were handed evaluation forms he said to the class "It doesn't matter what you fill out because I have tenure."

MetalGuru 6y ago
I didn’t find any valuable insight in this. Get help and build for the future... ok? Also, I felt his anecdotes barely got whatever point he was trying to make across. And half the post consisted of name drops of people I’d never heard of.
- herodotus 6y ago
  Butler Lampson: https://en.wikipedia.org/wiki/Butler_Lampson https://en.wikipedia.org/wiki/Butler_Lampson L. Peter Deutsch: https://en.wikipedia.org/wiki/L._Peter_Deutsch https://en.wikipedia.org/wiki/L._Peter_Deutsch The point is, these two people are incredible Computer Scientists.
spicyramen 6y ago
And?
bade 6y ago
What I meant was, say what you think, don't worry about getting me in trouble. As for the course , you''re saying I didn't teach for the future. Guilty.
- frequentnapper 6y ago
  You didn't teach anything. Period. This was the opinion of all my classmates and others I knew who took your class. If you don't believe me, see for yourself: https://www.ratemyprofessors.com/ShowRatings.jsp?tid=48636 https://www.ratemyprofessors.com/ShowRatings.jsp?tid=48636

coldcode 6y ago

Everyone makes mistakes, the key is to find ways to avoid making them over and over.

some_eejit 6y ago

I was a CS student at Warwick (UK university) in the late 80s, and clearly remembering doing functional programming with Lucid. Never heard of it since.