10 ms·

> A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in. I thi

> A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in. I think this probably the most important concept for programmers to keep at the front of their minds when working day to day. So many of the problems I see come from developers (myself included) adding new business logic to a process, or fixing a bug and not thinking through the side effects of the change

HybridCurve 4y ago
I actually just started to refactor some of my code earlier this week to deal with this very problem. It was related to deserialized data being transposed from one complex digraph to another. The reason I did not catch it as being problematic sooner was precisely because the state management was loosely distributed within the transposing functions' call hierarchy. The code which was trying to manage the states had emerged from fixes for a series of different problems related to edge cases, and once the new problems started relating to fixes for the others, I had that "Ahhh, well sh*t.." moment you get when finding you were missing something important. > I think this probably the most important concept for programmers to keep at the front of their minds when working day to day. The lesson I had learned was to be more mindful when synthesizing a solution and there might be more state interdependence than we expect. Centralizing state management has helped me solve it in this case.
jmrobles 4y ago
The problem is even bigger when you are reviewing PRs, the lack of context is the big problem,imo.
- KMag 4y ago
  I agree. Good code design, regardless of paradigm used, encapsulates complexity in order to minimize the amount of context needed to understand any one bit of code. The better the encapsulation, the better we can scale our limited mental capacity to larger and more complex systems. If the complexity is better encapsulated, it's also faster and easier for the reviewer to acquire the necessary context. When I was first learning OOP, I wish someone had mentioned why classes are useful: they leverage our natural mental abstraction machinery for understanding the world. Walking down the road, we generally think of each car as a whole and ignore thinking about the thousands of parts that make it up. It would be overwhelming to simply cross a busy street if we were considering all of the millions of car components flowing down the streets or considering the various differences in cars all the time, rather than abstractly thinking of various differing collections of parts as just "car". All of this happens at a subconscious level, and we can consciously tap into some of this mental machinery to help us reason about code. Of course, we don't need OOP to leverage this mental machinery, but it's one technique for doing so.
mightybyte 4y ago
It took me awhile to realize but this is one of the big things that algebraic data types (ADTs) help you to do...design your data types so they have exactly the number of valid states that your domain has. To use another common FP way of saying it...make invalid states unrepresentable.
- ChrisMarshallNY 4y ago
  A classic way to do this, is enumerations.
  - the__alchemist 4y ago
    ADTs is a concept I've had a tough time understanding; the approximate definition that made the most impact with is considering them as a specific type of enumerations: Ones where each variant holds a value. Is this correct?
    - ChrisMarshallNY 4y ago
      That’s the way Swift enums work. Here’s an interesting little exploration: https://littlegreenviper.com/miscellany/swiftwater/writing-a-robust-little-parser-in-swift-epilogue/#peek https://littlegreenviper.com/miscellany/swiftwater/writing-a...
    - thesz 4y ago
      It is the other way around: enumerations are special cases of algebraic data types with no values attached. In algebraic data types we have a sum (union) of a labeled cartesian products, and each product may have arbitrary number of values: data MaybeInt = NoInt | AnInt Int data MaybeItsPair = NoIntsPair | AnIntPair Int Int The first constructor in both data types is a labeled (No...) product with empty number of value to apply cartesian product to. The second constructor in both data types is a cartesian product: in first case an Int and in case a pair of Ints. Of course you can have more involved data type: data Expr = Void | Const Int | Var String | Bin BinOp Expr Expr | Un UnOp Expr A label, two single value constructors and much more involved operations as well.
louthy 4y ago
> adding new business logic to a process, Adding logic is the smallest part of the problem: adding types, that aren't constrained to represent the smallest set of values needed for the problem at hand, that's the biggest problem. This 'looseness' doesn't show up as a compilation error and looks on the surface to be just fine. Then the real-world comes along and creates a value that should never exist and the code falls over. Constraining the data by properly defining types is the (primary) way to reduce the possible states an application can get in. This can be via sum-types (discriminated unions), or even making sure you don't use `int`, `string`, etc. and other types that are just one step away from being dynamic. When doing pure functional programming, along with properly constrained types, you tend to find the code writes itself. What I mean by this is that it becomes very much about composing functions via 'type lego'. Once the types fit, your code works [1]. This is a very profound feeling once you grok it, because it gives oneself a feeling of confidence in ones own code that doesn't happen in imperative land. I will sometimes go days or weeks without even running the code I'm writing because I just have confidence in it. [1] It's still possible to write logic errors, but they're the only major class of bugs you get really.
- japanman425 4y ago
  [dead]
- charcircuit 4y ago
  And how do you know you've constricted your data enough? You don't and you get bugs. You don't have a perfect specification you are building. In the micro scale in terms of variables this is not the biggest problem that exists with programs. Enums don't somehow become garbage because it's technically possible to overwrite them with garage. The macro state of the entire system adds much more complexity and bugs. There is too many things that you have to keep in mind and systems are too big for one single person to understand it all. Pure functional still has this macro state. The state is stored in the program counter and by carrying along the current state with it. If you break any programming task into fine enough steps it will feel like lego. It's nothing special. If I need to print out the numbers 1 through 10 there is a single canonical way my whole team could come up with independent from one another. >I will sometimes go days or weeks without even running the code I'm writing because I just have confidence in it. On a small scare this is possible but once you start working on a larger team or if you have to integrate your code with constantly changing systems you will want to be testing your code. >[1] It's still possible to write logic errors, but they're the only major class of bugs you get really. You can still make security bugs and performance bugs. Logic bugs is a giant category of bugs and are what most bugs belong to.
iExploder 4y ago
this is why I think tests are so vital. good luck trying to fix a bug in a complex project without doing some sort of test to make sure the change does not break one of the thousands of requirements implements so far. from my experience fixing a bug without validating the system has a very high chance to spawn multiple bugs found later on...
taberiand 4y ago
See also the Joe Armstrong talk "The Mess We're In" https://youtu.be/lKXe3HUG2l4 https://youtu.be/lKXe3HUG2l4
simplotek 4y ago
> So many of the problems I see come from developers (myself included) adding new business logic to a process, or fixing a bug and not thinking through the side effects of the change That's a great take, but I'd turn it upside down. Problems come from developers adding new features while not being mindful of the importance of helping out whoever is reading that code in the future to understand what are all the possible states. Anyone can hack together code that's all over the place, but forcing people in the future to do shotgun surgery is as bad as adding the bugs ourselves.
- dzdt 4y ago
  One thing I've seen that helps out immensely with this in a large complex code base is extremely liberal use of ASSERTs. Every time you have a function or clause that takes an object or data with many possible versions or states, but only will work correctly with particular assumptions, ASSERT that your assumptions are satisfied. Every time. This helps in several ways. At the time of writing the code, it makes you think about possible states that could be represented and the generality of what your new code actually handles, and so encourages you to write general solutions and avoid unhandled or mishandled cases. At the time of reading the code, it helps the reader see what cases the writer considered. It helps document the intended scope and use cases the code is designed for. At the time of extending surrounding code, it helps to catch cases where the extension introduced state combinations that had not been considered before. Its nice to make an extension, run a test case, and get an assert that tells you exactly what other function needs an extension. Without the assert you would make the extension and run your test case and get wrong behavior or a crash and have a big debugging mystery or even worse: apparently working but subtly wrong code. Others in the thread are talking about making invalid states unrepresentable, which is nice, but not always doable. In a big code base you don't have the luxury of redesigning things; usually you have to stick to more local changes. But you can always add ASSERTs -- that is a completely local way to mark invalid states as invalid.
  - lucidguppy 4y ago
    Haskell case expressions are nice. It seems like you can define which input values a function can process - and you can get a log of if/else logic removed.
makeitdouble 4y ago
I am always wondering how much are we expected to understand the code we’re working on. It is very easy to just say people should know everything, but in practice people don‘t review every single library, nor defend against every single possible case, and surely don’t keep everything in memory while working on any single problem. I really can’t imagine someone “fully understanding” how the linux kernel works for instance, or even a full graphic card driver, or the software on a POS works as working knowledge. So yeah, people cause bugs because they don’t know it all, but I’m not sure that’s saying much. PS: real progress is probably made starting from assuming people don’t understand, and having frameworks that somewhat keep the system manageable under that premise.
- boppo1 4y ago
  Total newb here, but according to Stroustrup, the whole point of programming is so you don't have to know everything. You're responsible for: A. Understanding relevant computing fundamentals B. Understanding the interfaces to code you use C. Writing reliable implementations and documenting them well D. Writing useful and appropriate interfaces for your implementations In a perfect world, this creates a nice chain where every programmer only needs to look after these four points and then everything will work seamlessly. Too bad we don't live there.
_448 4y ago
I think the best way to solve this problem is by writing small digestible message passing functions (like Erlang) and each function written using design by contrast (like Eiffel). That way the situation can be bought under some control when the software scales.

HybridCurve 4y ago

I actually just started to refactor some of my code earlier this week to deal with this very problem. It was related to deserialized data being transposed from one complex digraph to another. The reason I did not catch it as being problematic sooner was precisely because the state management was loosely distributed within the transposing functions' call hierarchy. The code which was trying to manage the states had emerged from fixes for a series of different problems related to edge cases, and once the new problems started relating to fixes for the others, I had that "Ahhh, well sh*t.." moment you get when finding you were missing something important. > I think this probably the most important concept for programmers to keep at the front of their minds when working day to day. The lesson I had learned was to be more mindful when synthesizing a solution and there might be more state interdependence than we expect. Centralizing state management has helped me solve it in this case.

jmrobles 4y ago

The problem is even bigger when you are reviewing PRs, the lack of context is the big problem,imo.

KMag 4y ago
I agree. Good code design, regardless of paradigm used, encapsulates complexity in order to minimize the amount of context needed to understand any one bit of code. The better the encapsulation, the better we can scale our limited mental capacity to larger and more complex systems. If the complexity is better encapsulated, it's also faster and easier for the reviewer to acquire the necessary context. When I was first learning OOP, I wish someone had mentioned why classes are useful: they leverage our natural mental abstraction machinery for understanding the world. Walking down the road, we generally think of each car as a whole and ignore thinking about the thousands of parts that make it up. It would be overwhelming to simply cross a busy street if we were considering all of the millions of car components flowing down the streets or considering the various differences in cars all the time, rather than abstractly thinking of various differing collections of parts as just "car". All of this happens at a subconscious level, and we can consciously tap into some of this mental machinery to help us reason about code. Of course, we don't need OOP to leverage this mental machinery, but it's one technique for doing so.

mightybyte 4y ago

It took me awhile to realize but this is one of the big things that algebraic data types (ADTs) help you to do...design your data types so they have exactly the number of valid states that your domain has. To use another common FP way of saying it...make invalid states unrepresentable.

ChrisMarshallNY 4y ago
A classic way to do this, is enumerations.
- the__alchemist 4y ago
  ADTs is a concept I've had a tough time understanding; the approximate definition that made the most impact with is considering them as a specific type of enumerations: Ones where each variant holds a value. Is this correct?
  - ChrisMarshallNY 4y ago
    That’s the way Swift enums work. Here’s an interesting little exploration: https://littlegreenviper.com/miscellany/swiftwater/writing-a-robust-little-parser-in-swift-epilogue/#peek https://littlegreenviper.com/miscellany/swiftwater/writing-a...
  - thesz 4y ago
    It is the other way around: enumerations are special cases of algebraic data types with no values attached. In algebraic data types we have a sum (union) of a labeled cartesian products, and each product may have arbitrary number of values: data MaybeInt = NoInt | AnInt Int data MaybeItsPair = NoIntsPair | AnIntPair Int Int The first constructor in both data types is a labeled (No...) product with empty number of value to apply cartesian product to. The second constructor in both data types is a cartesian product: in first case an Int and in case a pair of Ints. Of course you can have more involved data type: data Expr = Void | Const Int | Var String | Bin BinOp Expr Expr | Un UnOp Expr A label, two single value constructors and much more involved operations as well.

louthy 4y ago

> adding new business logic to a process, Adding logic is the smallest part of the problem: adding types, that aren't constrained to represent the smallest set of values needed for the problem at hand, that's the biggest problem. This 'looseness' doesn't show up as a compilation error and looks on the surface to be just fine. Then the real-world comes along and creates a value that should never exist and the code falls over. Constraining the data by properly defining types is the (primary) way to reduce the possible states an application can get in. This can be via sum-types (discriminated unions), or even making sure you don't use `int`, `string`, etc. and other types that are just one step away from being dynamic. When doing pure functional programming, along with properly constrained types, you tend to find the code writes itself. What I mean by this is that it becomes very much about composing functions via 'type lego'. Once the types fit, your code works [1]. This is a very profound feeling once you grok it, because it gives oneself a feeling of confidence in ones own code that doesn't happen in imperative land. I will sometimes go days or weeks without even running the code I'm writing because I just have confidence in it. [1] It's still possible to write logic errors, but they're the only major class of bugs you get really.

japanman425 4y ago
[dead]
charcircuit 4y ago
And how do you know you've constricted your data enough? You don't and you get bugs. You don't have a perfect specification you are building. In the micro scale in terms of variables this is not the biggest problem that exists with programs. Enums don't somehow become garbage because it's technically possible to overwrite them with garage. The macro state of the entire system adds much more complexity and bugs. There is too many things that you have to keep in mind and systems are too big for one single person to understand it all. Pure functional still has this macro state. The state is stored in the program counter and by carrying along the current state with it. If you break any programming task into fine enough steps it will feel like lego. It's nothing special. If I need to print out the numbers 1 through 10 there is a single canonical way my whole team could come up with independent from one another. >I will sometimes go days or weeks without even running the code I'm writing because I just have confidence in it. On a small scare this is possible but once you start working on a larger team or if you have to integrate your code with constantly changing systems you will want to be testing your code. >[1] It's still possible to write logic errors, but they're the only major class of bugs you get really. You can still make security bugs and performance bugs. Logic bugs is a giant category of bugs and are what most bugs belong to.

iExploder 4y ago

this is why I think tests are so vital. good luck trying to fix a bug in a complex project without doing some sort of test to make sure the change does not break one of the thousands of requirements implements so far. from my experience fixing a bug without validating the system has a very high chance to spawn multiple bugs found later on...

taberiand 4y ago

See also the Joe Armstrong talk "The Mess We're In" https://youtu.be/lKXe3HUG2l4 https://youtu.be/lKXe3HUG2l4

simplotek 4y ago

> So many of the problems I see come from developers (myself included) adding new business logic to a process, or fixing a bug and not thinking through the side effects of the change That's a great take, but I'd turn it upside down. Problems come from developers adding new features while not being mindful of the importance of helping out whoever is reading that code in the future to understand what are all the possible states. Anyone can hack together code that's all over the place, but forcing people in the future to do shotgun surgery is as bad as adding the bugs ourselves.

dzdt 4y ago
One thing I've seen that helps out immensely with this in a large complex code base is extremely liberal use of ASSERTs. Every time you have a function or clause that takes an object or data with many possible versions or states, but only will work correctly with particular assumptions, ASSERT that your assumptions are satisfied. Every time. This helps in several ways. At the time of writing the code, it makes you think about possible states that could be represented and the generality of what your new code actually handles, and so encourages you to write general solutions and avoid unhandled or mishandled cases. At the time of reading the code, it helps the reader see what cases the writer considered. It helps document the intended scope and use cases the code is designed for. At the time of extending surrounding code, it helps to catch cases where the extension introduced state combinations that had not been considered before. Its nice to make an extension, run a test case, and get an assert that tells you exactly what other function needs an extension. Without the assert you would make the extension and run your test case and get wrong behavior or a crash and have a big debugging mystery or even worse: apparently working but subtly wrong code. Others in the thread are talking about making invalid states unrepresentable, which is nice, but not always doable. In a big code base you don't have the luxury of redesigning things; usually you have to stick to more local changes. But you can always add ASSERTs -- that is a completely local way to mark invalid states as invalid.
- lucidguppy 4y ago
  Haskell case expressions are nice. It seems like you can define which input values a function can process - and you can get a log of if/else logic removed.

makeitdouble 4y ago

I am always wondering how much are we expected to understand the code we’re working on. It is very easy to just say people should know everything, but in practice people don‘t review every single library, nor defend against every single possible case, and surely don’t keep everything in memory while working on any single problem. I really can’t imagine someone “fully understanding” how the linux kernel works for instance, or even a full graphic card driver, or the software on a POS works as working knowledge. So yeah, people cause bugs because they don’t know it all, but I’m not sure that’s saying much. PS: real progress is probably made starting from assuming people don’t understand, and having frameworks that somewhat keep the system manageable under that premise.

boppo1 4y ago
Total newb here, but according to Stroustrup, the whole point of programming is so you don't have to know everything. You're responsible for: A. Understanding relevant computing fundamentals B. Understanding the interfaces to code you use C. Writing reliable implementations and documenting them well D. Writing useful and appropriate interfaces for your implementations In a perfect world, this creates a nice chain where every programmer only needs to look after these four points and then everything will work seamlessly. Too bad we don't live there.

_448 4y ago

I think the best way to solve this problem is by writing small digestible message passing functions (like Erlang) and each function written using design by contrast (like Eiffel). That way the situation can be bought under some control when the software scales.