4 ms·
Oh this, I've known for years. I learned it early, as a new OS engineer at Convergent Technologies. There was a nasty part of the kernel, where programmable tim
by JoeAltmaier 7y ago
Oh this, I've known for years. I learned it early, as a new OS engineer at Convergent Technologies. There was a nasty part of the kernel, where programmable timers lived, that was a large part of the change history of the OS. Looking at it, I saw folks had been thrashing for some time, to get all the issues resolved. But it was a '10 lbs of feathers in a 5 lb box' kind of thing. One change would cause another issue, or regress a bug.
So I took 2 days, made a chart of every path and pattern of events (restarting a timer from a timer callback; having another time interval expire while processing the previous; restarting while processing and another interval expires; restarted timer expires before completing previous expiration and on and on). Then writing exhaustive code to deal with every case. Then running every degenerate case in test code until it survived for hours.
It never had to be addressed again. But it did have to be addressed. So many folks are unwilling to face the music with complexity.
- heymijo 7y ago> One change would cause another issue, or regress a bug. > made a chart of every path and pattern of events > It never had to be addressed again Am I reading you correctly--every possible issue was knowable and preventable with enough prior thought?
- pjc50 7y agoState machines are a surprisingly powerful tool. They do require you to capture all the possible states of the system; the combinational explosion of "X while Y but not while Z" can get a bit nasty, but that can often force a redesign simply to reduce the state space. If all your logic is immutable apart from a small area around the state machine the state gets much easier to manage. You write everything as f(state, inputs) => next_state. You can at any point inspect or print the state of the state machine, and record its transitions so you have a clearer idea of how it got there if there was indeed a bug.
- heymijo 7y agoThank you, that's helpful. Is a Mealy state machine implied in your description?
- auggierose 7y agohaven't heard that term in ages :-) pretty much not since my first computer science lecture given by Wilfried Brauer :-D
- pjc50 7y agoI too have to look the Moore/Mealy distinction up every time, but yes. The real trick is finding all the state machines you've built in your program without noticing. Or are contained in dependencies. Parsers, TCP, SSL. Authentication is a state machine that usually has two states but may have more; places like Amazon have "we know who you are but you've not entered your password this session", for example. Anything involving message delivery. Anything transactional.
- JoeAltmaier 7y agoYes! This, so much this. At the hardware level there are countable events (interrupt, reprogram the device, change the timer list, callback, call from callback, etc) that can be listed and all combinations dealt with. I recall part of the solution was, scheduling callbacks. So there wasn't infinite recursion (call a timer expiration callback, have it re-start the timer, have it expire, call THAT callback and so on).
- vikiomega9 7y agoGreat advice on immutability, separation of concerns/domain logic, and side effects. I think modern languages or tooling should somehow enforce this. I think a lot of code will be better off for it. Specific patterns/zen (like another comment below) is what helps.
- lliamander 7y ago> but that can often force a redesign simply to reduce the state space I'm convinced that this idea, or something similar, is essentially what it means to write "algorithmic code". At a certain level, all code is just an algorithm (i.e. logic + control). But when we talk about Algorithms that one my find in a book on such things, inevitably what you have is code that was written to avoid unnecessary states in order to enforce certain guarantees with regard to performance, security, fault-tolerance, or some other desirable property.
- pjc50 7y agoTo me, there's an "algorithm" vs "business logic" continuum, where algorithms operate on cleanly specified mathematical operands and the difficulty is .. mathematical, while business logic isn't conceptually complicated but can be very wordy. A lot of the work in translating a business spec into algorithms is what JoeAltmaier did: enumerate all the cases. You end up with boxes labelled "on Thursdays when the user is in the US Virgin Islands and is using the compact UI", with conflicting assumptions from the postit notes taken from the meetings, and someone has to go and check what should happen in that box.
- lliamander 7y agoI agree that there is a continuum. I would say it's based on how easy it is to generalize over the state space. Stuff like sorting, cryptography, and consensus algorithms are often quite amenable to clean generalizations (which is often why abstract algebra can be useful in those domains[0]). There's also another axis which is the strength of the guarantees provided. Most interesting software, whether it is has many or few special cases, has strong guarantees that one is trying to make. Often in order to make stronger guarantees you have to add special case handling, but software which manages to have both strong guarantees and few special cases is the most interesting kind. But even stuff that is littered with special cases, whether it's query engines, control systems, workflow engines, resource schedulers, or chat applications, I think can benefit from the type of algorithmic thinking which strives to make stronger guarantees with fewer special cases. [0]https://news.ycombinator.com/item?id=22868840 https://news.ycombinator.com/item?id=22868840
- stagger87 7y agoRegarding being 'unwilling to face the music', is it possible that you were only able to perform this refactor due to all the 'thrashing' that came before you? Is it possible that the majority of commits leading up to your refactor were dealing with unforeseen complexities and addressing real bug fixes that you were able to conveniently take in and synthesize all at once? Refactoring too soon, before understanding the entire system (and all the possible changes/additions/issues that could arise) is probably worse (time-wise) than thrashing for a bit.
- JoeAltmaier 7y agoAs I recall (and it was a long time ago) the history gave some insight into the complexity. But nobody had exhaustively enumerated all the cases. I remember the chart had more conditions that had ever been addressed.
- hinkley 7y agoYou'd be surprised how few people look through commit history while bug fixing. There's a steady stream of people who don't understand why 'breaking' the commit history is a problem, and assume you have some sort of untreated OCD if you bother to even care.
- potta_coffee 7y agoCommit history has saved my ass more times than I can count. Especially useful on a codebase that has churned through a lot of developers that you've never met.
- hinkley 7y agoYeah, including other people's bugs. Worked a few places where people would just give up on bugs well before I did, and less than half of that was perseverance. The first time I recall doing this was when a QA person came to me to complain that in the last handful of releases we had done, were cycling back and forth between 2 bugs and they had just caught the same thing starting again in a pre-release build. After that it just kinda went into the toolbox to look at why code got a certain way instead of assuming the dev wasn't paying attention (or was in fact dim). If you know what someone is trying to do, you can find a solution that solves both problems (without necessarily using either solution).
- senderista 7y agoA great example of eliminating accidental complexity by embracing and clarifying the essential complexity. I have seen many such systems where a single exhaustive state machine could replace a mess of buggy and incomprehensible spaghetti code.
- hinkley 7y agoI think a critical skill for a developer is to be able to realize when you are frustrated, distance yourself from the problem and ask what is going on, 5 Why's style. Developers know about the Boiled Frog problem. But we sometimes don't know when we are the frog. We keep hill-climbing one solution because we are only trying to do one little thing so of course I should be able to do a small to medium thing to accomplish it. We also send out mixed messages on rewrites, and I'm guilty of it too. But there's gotta be some litmus test where we compare the number of interacting states in the requirements versus the number of interacting states in the code and just call Time of Death.
- blasterjeff151 7y agoThis is what RxJS was born for . . . :)