8 ms·

Reflections on Sudoku, or the Impossibility of Systematizing Thought

100 points by rjpower9000 1y ago

rjpower9000 1y ago
I was fascinated by the "Sudoku Affair", found myself speculating on the internal mindset of TDD advocates, and ended up with an unsatisfying conclusion that you can't systematize thought. Not my best writing but thought I'd share nonetheless.
- gnat 1y ago
  I think there’s an analogy in the dumb management dream that there’s a process by which you can replace these expensive skilled workers with cheap unskilled idiots and still get great results. Another variation: the magic process that lets you get great results from people who don’t care. Excellence doesn’t come solely from process. The benchmarks of great process with unskilled workers who don’t care is fast food, and this product is consistently mediocre. Consistency is good but consistent mediocrity is an unworthy ambition in many fields. NASA is massively process driven but they get people to space and back, and the people in their process are highly skilled and care deeply. Larry Wall, inventor of Perl, used to say (and probably still does) that complexity has to go somewhere. If you’re solving a complex problem then either it’s a simple program with complex tools, or a complex program with simple tools. That’s really stuck with me. The art of library, framework, language, API design is to provide a set of tools — if you make simple tools then complex problems become complex programs. And if you offer complex tools, complex problems can have simple solutions. And people will gripe at you for all the punctuation for decades. :) But the complexity exists and can’t be wished away. Which is kinda your point too. Thanks for writing!
  - NitpickLawyer 1y ago
    > there’s a process by which you can replace these expensive skilled workers with cheap unskilled idiots and still get great results. > Another variation: the magic process that lets you get great results from people who don’t care. Isn't that pretty much what the US military does? They take in "dumb" teens + docs + processes and get whatever it is they need out of this? It's also cheap (compared w/ industry). And by the end of the exercise the "dumb" teens become skilled in their own niche, some of them highly skilled (see nuclear specialists, ATCs, pilots, navigators, managers, procurement, etc.) The military processes are at the base of lots of organisational stuff we have in software dev and business as well (agile, scrum, ooda, etc)
    - gnat 1y ago
      I feel like you're saying that education (as practiced by the US military) is a somewhat reliable process for taking unskilled people and making them skilled. I agree. The US Military have admissions criteria. They bounce people out in Basic Training and in every other part of training. Not everyone gets to be an ATC, a pilot, or a nuclear specialist. Air Traffic Control turns out not to be a process you can put an unskilled person into. It requires training and careful integration, and once you have a skilled person with many hours invested in them, perhaps you can let them direct traffic. Training isn't a magic process that takes unskilled people who don't care and delivers great results every time. The equivalent in our world would be "I'll hire cheap people who don't know how to program, put them through a Bootcamp process, and then I'll have great programmers." That didn't work. I am trying to say that: every version of programming where there's been The Process You Just Have To Follow (from Jackson Structured Design) has failed significantly and hasn't been a substitute for hiring smart people who care. If someone came to me with a business plan that was "hire mediocre people who don't care, and we'll achieve great results because of My Process", I'd be veeeeery skeptical indeed.
    - WJW 1y ago
      Where do you get the idea that this is cheap? Militaries are somewhat famous for being incredible money sinks. They need incredibly specific skills that are not really taught in the civilian world. This means the military needs to train everyone at its own cost. So while the input might be "cheap, unskilled idiots", there is then a significant expense to turn them into expensive skilled non-idiots before they are ready for duty. As someone who worked inside the military for 14 years, it is also quite a stretch to say they consistently get "great" results tbh.
    - aleph_minus_one 1y ago
      > The military processes are at the base of lots of organisational stuff we have in software dev and business as well (agile, scrum [...]) The Agile Manifesto was exactly the counter-manifesto to this, and thus any methodology that calls itself agile (e.g. Scrum) is: Agile Manifesto > https://agilemanifesto.org/ https://agilemanifesto.org/ Principles behind the Agile Manifesto > https://agilemanifesto.org/principles.html https://agilemanifesto.org/principles.html
  - WJW 1y ago
    > Consistency is good but consistent mediocrity is an unworthy ambition in many fields. Sorry to take only a small part of an otherwise great comment but is this actually true? It seems to me that there is a great many fields in which consistency is more important than excellence, especially if the striving for excellence produces great misses as well sometimes. In a well-designed system with some allowed tolerance, as long as it's good enough you are fine. Take the electricity grid for example: There's no prizes for maintaining the frequency to within a nano-Hertz of the spec. There are very large fines for being outside the spec (+/-0.050 Hertz for the EU grid). Being consistently within spec is much more valuable than occasionally performing much better than the spec. It is only in extreme winner-takes-all fields like sports, spacefaring and entrepreneuring that being the absolute best is what you want. In most other fields being consistently decent beats out varying excellence. I definitely wouldn't want my dentist to take a risky moonshot in pursuit of excellence, for example.
    - gnat 1y ago
      "I definitely wouldn't want my dentist to take a risky moonshot in pursuit of excellence". Excellent comment. I was aware as I was writing it that consistent mediocrity is indeed a profitable target. Perhaps we're quibbling over "mediocrity"? Staying within the spec seems enough of the challenge for a grid, I'm not sure that I'd define excellence as narrower and narrower variation around the spec there. I was making a moral judgement in "unworthy". I like cars to come off the factory floor consistently good. I own a Tesla, this statement has high salience for me. That seems like a challenge. I wouldn't respect the Lada factory for consistently turning out cars that break down or fall apart, just as I don't respect McDonalds for consistently delivering underwhelming food. I acknowledge the consistency, I acknowledge it's profitable, but they're not achieving consistent greatness.
- PaulHoule 1y ago
  There's the principle "never let them see you sweat" and if you're trying to convince people of an idea like TDD you never want to be seen floundering. You don't publish anything about your sudoku solver until you've succeeded at it. Otherwise you're just proving "TDD sux" which, for all "X sux", TDD sux more than X.
- Yossarrian22 1y ago
  I'd be somewhat more charitable and say that it's an attempt to break down a big, complicated problem into chunks that are more easily digestible. The problem I think Jeffries ran into is that Sudoku and the search space is essentially atomic, where breaking it down further isn't helpful, while seeming like it's simple to break it down to row, column, and box and work from there.
schoen 1y ago
Isn't there a less-conceptual (but still conceptual) problem that correctness of software is commonly abrupt rather than continuous? You don't get a series of almost-right programs gradually approximating the right program, you have a correct program and variations on it may fail completely. Of course, whether this is literally true depends on what sort of algorithmic problem you're approaching. But there must be many real-world problems in which the very-nearly-correct program produces completely wrong behavior that doesn't resemble the correct behavior at all. In those circumstances, you couldn't expect to find the correct program empirically through iterative improvements. Edit: Maybe the incremental test-driven approach would work in cases where you have an externally given specification or design that already breaks the algorithmic part up into smaller, easier, and more verifiable pieces.
- AndrewDucker 1y ago
  Yes. The difference between "A program that does what you want" and "A program that crashes on startup" can be one character.
  - pixl97 1y ago
    Ya but the real fun ones are the "a program does what you want 99.993% of the time"
    - AndrewDucker 1y ago
      The annoying situation being that sometimes that 0.007% requires a major rearchitecture!
- aleph_minus_one 1y ago
  > Isn't there a less-conceptual (but still conceptual) problem that correctness of software is commonly abrupt rather than continuous? You don't get a series of almost-right programs gradually approximating the right program, you have a correct program and variations on it may fail completely. I consider it to be plausible that such a topology could exist (at least for many situations). The problem rather is that such a topology would likely behave very different from users' expectations.
IceDane 1y ago
Wow, the Ron Jeffries articles are sort of embarrassing, and he doesn't even realize. This why dogma never works.
- rjpower9000 1y ago
  I respect he's open about his work and struggles, and it's cool he's programming at 86, but it does seem like his approach makes it harder for him rather than easier. For example with the bowling score calculator, it's great to start with some tests, but I think he then marched towards a specific OOP formulation which obscured rather than clarified the problem.
JonChesterfield 1y ago
This was an interesting read but the source article is fascinating https://explaining.software/archive/the-sudoku-affair/ https://explaining.software/archive/the-sudoku-affair/ TDD as an incremental search from a simple start point towards a solutions. Put in those terms, it'll work when the path from start to solution is smoothly differentiable.
- PaulHoule 1y ago
  The smart way to write a Suduoku solver these days is to formulate the problem for an SMT solver, which is specifically intended to solve that kind of problem. The SMT solver already has tests, so you don't need to write tests. When I wrote my first Sudoku solver I started out with a solver that solved easy problems where, at each step, there was some square with only one possible solution. That didn't work for harder problems where you have to guess. Eventually I realized that you could just pick any square, try all the numbers, and recursively solve the remaining problem. This works no matter which square you pick but it is faster if you start with the square that has the minimum number of choices available. If you write a solver haphazardly like that you're very likely to wind up with two or more copies of the board because you're not sure what kind of queries you'll need to do. If you understand the problem, you won't. The role of the tests is complex here. In principle they could help you refactor your data structures, in practice the sheer bulk of them could reify the data structures you have and add to the burden of refactoring them. A code-golfed version of the solver could be smaller than the tests. Writing a chess program I found testing was a challenge. Move generators are easy to test, evaluation functions are easy to test. A decent alpha-beta search with transposition tables, killer heuristic and such is not so easy to test in terms of unit tests. Before I'd have the program play anyone I'd run a set of integration tests based on https://www.chessprogramming.org/Bratko-Kopec_Test https://www.chessprogramming.org/Bratko-Kopec_Test I think of a system I inherited that had a part that was prone to race conditions, part of vanquishing the race conditions was writing a "super hammer" test that would run a few 1000 threads for about 40 seconds. Maybe it still has race conditions, but they won't be triggered often. Long-running tests like that aren't really unit tests but for some problems they're the right tool. In general though long-running tests are a big problem because slow builds are a problem.
awanderingmind 1y ago
This is a rare type of article - a concrete analysis of different approaches to programming (that are arguably themselves reflections of different cognitive styles), that outlines the shortcomings of one approach in a specific domain, without generalising too much.
taeric 1y ago
You see similar in other arenas, too. And sometimes, as annoying as it is, popular techniques really do have better success than not. Even more annoying, unpopular techniques can often have better success rate than we care to acknowledge. The examples in my mind are: outlining, task breakdown, object modeling, and rote repetition.
mbb70 1y ago
I'm reminded of Rich Hickey's Hammock Driven Development talk, whose thesis for design is basically: 1. Think about the problem and write it all down 2. Research what other people have done to solve this problem 3. Think about it some more and write all that down too 4. Sleep on it I think the conclusion is the same: Good design does not naturally arise from good programming, and background/domain knowledge is essential
- DowsingSpoon 1y ago
  But Uncle Bob said that if I follow his TDD process by rote then my code will automatically attain good design! /s
  - earnestinger 1y ago
    My view of Uncle Bob is that his advise is attempt improve the average situation, even if taken literally as a gospell. It’s better that everybody do route tdd, than no tests at all. It’s better to have all methods exactly 3 lines of code, than all methods being 3 pages long. Uncle bob preaches better baseline, experience will teach all the nuance. Disclaimer: I dont do tdd. But it’s nice method of work to know.
    - vrighter 1y ago
      every time i heard a talk or read something of his I always come out with the impression that he has no idea about anything. Most of what he says sounds like bullshit to me.
      - tacitusarc 1y ago
        His advice needs to be contextualized within the 90s and 00s. He was correcting (or overcorrecting) for issues that were prevalent then: methods that were hundreds or thousands of lines, processes with hundreds of hours of design work without writing a single line of code, no testing (or maybe handing to a QA to test, if you’re lucky), and tangled messes of logic the author thought could be saved by adding a comment. Much of his advice seems silly or extreme now, but I believe that’s because he (and his cohort of likeminded programmers) won. Things have improved dramatically. The extreme advice worked. The problem, of course, is that you can’t just declare victory and move on. So now we have The Church of TDD and the like.
prmph 1y ago
And this is exactly thinking AI is going to make human thought and skill redundant is crazy. LLMs can never get to a point where you whip them up to solve any general intellectual challenge. Even creating CRUD apps (that are well-behaved, secure, performant, scalable, and maintainable) can't really be totally systematized.
fydorm 1y ago
Wow, I just read the original Sudoku post by Norvig yesterday for the first time, and now today this shows up here!
- layer8 1y ago
  It was bound to happen to someone.
groby_b 1y ago
It's making the same reasoning mistake that a lot of discussion of the Entscheidungsproblem makes - the problem talks about a generic algorithm to answer for all programs P if they can solve T, for all tasks T, the discussion assumes that you can't decide if P solves T for any P or T. With that in mind, let's look at the crux of the argument: "If we can't decide if a program P solves a task T, then we certainly can't solve the even harder problem of finding a program P that solves a given task." That's simply not true. We can decide if a program P solves a task T, for a specific program. Moreover, that means that for large classes of tasks, we actually can throw mud at the wall and see if it sticks - as long as it's decidable if the particular program P solves the particular task T. And for any problems you can exhaustively test, hey, you really can rely entirely on TDD and hill-climbing the problem space. Hence, bowling scores being easier than Sudoku solvers. As soon as you leave the (almost) exhaustively testable space, things become harder. And it's worth keeping in mind that TDD originates from a payroll system - something that's more amenable to exhaustive testing ("do all of our employees get the right amount, and did HR/finance stop yelling, and is our CFO not getting jailed for tax evasion") than a systematic approach. (Government plus corporate bureuacracy means that there are absolutely no deep structures to find. It's all about the special cases) You can still do "TDD" at a higher level. You just need to accept that TDD is simply a primitive form of formally establishing your constraints, and that some constraints are hard to establish purely through testing. But that there exist many formalism to allow you to express higher level constraints. This is at the core of the Haskell/Rust quip that "once the type checker/borrow checker is quiet, you can be confident the solution works". Maybe constraint-driven design would've been a better name.
- rjpower9000 1y ago
  I won't argue with the idea whether you can test a particular program produces a result (we aren't so interested in programs that run forever in any case). Your observation of the origins of TDD makes sense in this regard: there may be domains where the TDD approach is more useful, but it's not a generic panacea, and assuming it is can lead to sadness. My point was more about arbitrary programs and tasks. In general, I don't think there's any particular process you can follow that, in effect, reduces programming or math to a checklist. I'm sure there are techniques that can help structure many problems but they're necessarily fairly general e.g. "think about the problem, take a nap, think about it some more". What I was hoping to highlight was more the danger of trying to hill-climb without having the right set of skills, thus assuming you can solve a problem by following a particular pattern. To give an example, I remember trying to derive the quadratic equation when I was young, and no amount of algebraic repositioning was going to help me unless I know how to complete the square! But once that type of trick is in your toolbox, you can begin to solve all sorts of other problems. Even the bowling calculator can become hard if you don't know what your doing, not because TDD is bad or a wrong way to do things, but because you don't know how to reason about your program. Even bowling has 12^13 outcomes, with a bunch of silly special cases if you approach it the wrong way. Thus a truly naive approach might lead you down a weird direction.
gchamonlive 1y ago
> Wouldn't it be nice if you just had a general method you could apply to solve everything? Would it? Why? > But the reality is there's no single approach we can take. > I know, I know, this is all so deeply... unsatisfying. Why is it so unsatisfying? This is the only thing that I disagree with the post, that this need to have a general approach to solving problems is necessary for solving problems to be satisfying. What is so disconcerting about a problem that seems to be unsolvable? Why do we need to solve everything immediately? Maybe it's important not only to accept, but to embrace the fact that there is a sort of magic to the process of creativity that we can't capture in a bottle. But for that to happen we need to divorce ourselves from the necessity of productivity. This way we are more likely to be able to just sit with the problem without beating ourselves over not being able to solve it and without feeling like the problem owe us a resolution in return for our efforts. It doesn't prevent anyone however from pursuing better tools and more general approaches to systematizing thought and problem solving. We can do both -- tackling a problem creatively or systematically -- as long as we feel like we are able to choose doing it. I think if we feel like we must find these general methods, they will likely elude us indefinitely.
andrewlgood 1y ago
This sounds very similar to the challenge that companies had implementing Six Sigma (or other statistically-based quality programs). The tools significantly helped the average person improve the quality of their processes as determined by a variety of metrics. However, they were never going to tell you the solution or improvement to implement. They could give hints as to where to look for a solution, but someone still had to dream up the solution. For this reason, brain storming was always one of the most critical tools in the Six Sigma toolkit.

100 points by rjpower9000 1y ago

I was fascinated by the "Sudoku Affair", found myself speculating on the internal mindset of TDD advocates, and ended up with an unsatisfying conclusion that you can't systematize thought. Not my best writing but thought I'd share nonetheless.

gnat 1y ago
I think there’s an analogy in the dumb management dream that there’s a process by which you can replace these expensive skilled workers with cheap unskilled idiots and still get great results. Another variation: the magic process that lets you get great results from people who don’t care. Excellence doesn’t come solely from process. The benchmarks of great process with unskilled workers who don’t care is fast food, and this product is consistently mediocre. Consistency is good but consistent mediocrity is an unworthy ambition in many fields. NASA is massively process driven but they get people to space and back, and the people in their process are highly skilled and care deeply. Larry Wall, inventor of Perl, used to say (and probably still does) that complexity has to go somewhere. If you’re solving a complex problem then either it’s a simple program with complex tools, or a complex program with simple tools. That’s really stuck with me. The art of library, framework, language, API design is to provide a set of tools — if you make simple tools then complex problems become complex programs. And if you offer complex tools, complex problems can have simple solutions. And people will gripe at you for all the punctuation for decades. :) But the complexity exists and can’t be wished away. Which is kinda your point too. Thanks for writing!
- NitpickLawyer 1y ago
  > there’s a process by which you can replace these expensive skilled workers with cheap unskilled idiots and still get great results. > Another variation: the magic process that lets you get great results from people who don’t care. Isn't that pretty much what the US military does? They take in "dumb" teens + docs + processes and get whatever it is they need out of this? It's also cheap (compared w/ industry). And by the end of the exercise the "dumb" teens become skilled in their own niche, some of them highly skilled (see nuclear specialists, ATCs, pilots, navigators, managers, procurement, etc.) The military processes are at the base of lots of organisational stuff we have in software dev and business as well (agile, scrum, ooda, etc)
  - gnat 1y ago
    I feel like you're saying that education (as practiced by the US military) is a somewhat reliable process for taking unskilled people and making them skilled. I agree. The US Military have admissions criteria. They bounce people out in Basic Training and in every other part of training. Not everyone gets to be an ATC, a pilot, or a nuclear specialist. Air Traffic Control turns out not to be a process you can put an unskilled person into. It requires training and careful integration, and once you have a skilled person with many hours invested in them, perhaps you can let them direct traffic. Training isn't a magic process that takes unskilled people who don't care and delivers great results every time. The equivalent in our world would be "I'll hire cheap people who don't know how to program, put them through a Bootcamp process, and then I'll have great programmers." That didn't work. I am trying to say that: every version of programming where there's been The Process You Just Have To Follow (from Jackson Structured Design) has failed significantly and hasn't been a substitute for hiring smart people who care. If someone came to me with a business plan that was "hire mediocre people who don't care, and we'll achieve great results because of My Process", I'd be veeeeery skeptical indeed.
  - WJW 1y ago
    Where do you get the idea that this is cheap? Militaries are somewhat famous for being incredible money sinks. They need incredibly specific skills that are not really taught in the civilian world. This means the military needs to train everyone at its own cost. So while the input might be "cheap, unskilled idiots", there is then a significant expense to turn them into expensive skilled non-idiots before they are ready for duty. As someone who worked inside the military for 14 years, it is also quite a stretch to say they consistently get "great" results tbh.
  - aleph_minus_one 1y ago
    > The military processes are at the base of lots of organisational stuff we have in software dev and business as well (agile, scrum [...]) The Agile Manifesto was exactly the counter-manifesto to this, and thus any methodology that calls itself agile (e.g. Scrum) is: Agile Manifesto > https://agilemanifesto.org/ https://agilemanifesto.org/ Principles behind the Agile Manifesto > https://agilemanifesto.org/principles.html https://agilemanifesto.org/principles.html
- WJW 1y ago
  > Consistency is good but consistent mediocrity is an unworthy ambition in many fields. Sorry to take only a small part of an otherwise great comment but is this actually true? It seems to me that there is a great many fields in which consistency is more important than excellence, especially if the striving for excellence produces great misses as well sometimes. In a well-designed system with some allowed tolerance, as long as it's good enough you are fine. Take the electricity grid for example: There's no prizes for maintaining the frequency to within a nano-Hertz of the spec. There are very large fines for being outside the spec (+/-0.050 Hertz for the EU grid). Being consistently within spec is much more valuable than occasionally performing much better than the spec. It is only in extreme winner-takes-all fields like sports, spacefaring and entrepreneuring that being the absolute best is what you want. In most other fields being consistently decent beats out varying excellence. I definitely wouldn't want my dentist to take a risky moonshot in pursuit of excellence, for example.
  - gnat 1y ago
    "I definitely wouldn't want my dentist to take a risky moonshot in pursuit of excellence". Excellent comment. I was aware as I was writing it that consistent mediocrity is indeed a profitable target. Perhaps we're quibbling over "mediocrity"? Staying within the spec seems enough of the challenge for a grid, I'm not sure that I'd define excellence as narrower and narrower variation around the spec there. I was making a moral judgement in "unworthy". I like cars to come off the factory floor consistently good. I own a Tesla, this statement has high salience for me. That seems like a challenge. I wouldn't respect the Lada factory for consistently turning out cars that break down or fall apart, just as I don't respect McDonalds for consistently delivering underwhelming food. I acknowledge the consistency, I acknowledge it's profitable, but they're not achieving consistent greatness.
PaulHoule 1y ago
There's the principle "never let them see you sweat" and if you're trying to convince people of an idea like TDD you never want to be seen floundering. You don't publish anything about your sudoku solver until you've succeeded at it. Otherwise you're just proving "TDD sux" which, for all "X sux", TDD sux more than X.
Yossarrian22 1y ago
I'd be somewhat more charitable and say that it's an attempt to break down a big, complicated problem into chunks that are more easily digestible. The problem I think Jeffries ran into is that Sudoku and the search space is essentially atomic, where breaking it down further isn't helpful, while seeming like it's simple to break it down to row, column, and box and work from there.

schoen 1y ago

Isn't there a less-conceptual (but still conceptual) problem that correctness of software is commonly abrupt rather than continuous? You don't get a series of almost-right programs gradually approximating the right program, you have a correct program and variations on it may fail completely. Of course, whether this is literally true depends on what sort of algorithmic problem you're approaching. But there must be many real-world problems in which the very-nearly-correct program produces completely wrong behavior that doesn't resemble the correct behavior at all. In those circumstances, you couldn't expect to find the correct program empirically through iterative improvements. Edit: Maybe the incremental test-driven approach would work in cases where you have an externally given specification or design that already breaks the algorithmic part up into smaller, easier, and more verifiable pieces.

AndrewDucker 1y ago
Yes. The difference between "A program that does what you want" and "A program that crashes on startup" can be one character.
- pixl97 1y ago
  Ya but the real fun ones are the "a program does what you want 99.993% of the time"
  - AndrewDucker 1y ago
    The annoying situation being that sometimes that 0.007% requires a major rearchitecture!
aleph_minus_one 1y ago
> Isn't there a less-conceptual (but still conceptual) problem that correctness of software is commonly abrupt rather than continuous? You don't get a series of almost-right programs gradually approximating the right program, you have a correct program and variations on it may fail completely. I consider it to be plausible that such a topology could exist (at least for many situations). The problem rather is that such a topology would likely behave very different from users' expectations.

IceDane 1y ago

Wow, the Ron Jeffries articles are sort of embarrassing, and he doesn't even realize. This why dogma never works.

rjpower9000 1y ago
I respect he's open about his work and struggles, and it's cool he's programming at 86, but it does seem like his approach makes it harder for him rather than easier. For example with the bowling score calculator, it's great to start with some tests, but I think he then marched towards a specific OOP formulation which obscured rather than clarified the problem.

JonChesterfield 1y ago

This was an interesting read but the source article is fascinating https://explaining.software/archive/the-sudoku-affair/ https://explaining.software/archive/the-sudoku-affair/ TDD as an incremental search from a simple start point towards a solutions. Put in those terms, it'll work when the path from start to solution is smoothly differentiable.

PaulHoule 1y ago
The smart way to write a Suduoku solver these days is to formulate the problem for an SMT solver, which is specifically intended to solve that kind of problem. The SMT solver already has tests, so you don't need to write tests. When I wrote my first Sudoku solver I started out with a solver that solved easy problems where, at each step, there was some square with only one possible solution. That didn't work for harder problems where you have to guess. Eventually I realized that you could just pick any square, try all the numbers, and recursively solve the remaining problem. This works no matter which square you pick but it is faster if you start with the square that has the minimum number of choices available. If you write a solver haphazardly like that you're very likely to wind up with two or more copies of the board because you're not sure what kind of queries you'll need to do. If you understand the problem, you won't. The role of the tests is complex here. In principle they could help you refactor your data structures, in practice the sheer bulk of them could reify the data structures you have and add to the burden of refactoring them. A code-golfed version of the solver could be smaller than the tests. Writing a chess program I found testing was a challenge. Move generators are easy to test, evaluation functions are easy to test. A decent alpha-beta search with transposition tables, killer heuristic and such is not so easy to test in terms of unit tests. Before I'd have the program play anyone I'd run a set of integration tests based on https://www.chessprogramming.org/Bratko-Kopec_Test https://www.chessprogramming.org/Bratko-Kopec_Test I think of a system I inherited that had a part that was prone to race conditions, part of vanquishing the race conditions was writing a "super hammer" test that would run a few 1000 threads for about 40 seconds. Maybe it still has race conditions, but they won't be triggered often. Long-running tests like that aren't really unit tests but for some problems they're the right tool. In general though long-running tests are a big problem because slow builds are a problem.

awanderingmind 1y ago

This is a rare type of article - a concrete analysis of different approaches to programming (that are arguably themselves reflections of different cognitive styles), that outlines the shortcomings of one approach in a specific domain, without generalising too much.

taeric 1y ago

You see similar in other arenas, too. And sometimes, as annoying as it is, popular techniques really do have better success than not. Even more annoying, unpopular techniques can often have better success rate than we care to acknowledge. The examples in my mind are: outlining, task breakdown, object modeling, and rote repetition.

mbb70 1y ago

I'm reminded of Rich Hickey's Hammock Driven Development talk, whose thesis for design is basically: 1. Think about the problem and write it all down 2. Research what other people have done to solve this problem 3. Think about it some more and write all that down too 4. Sleep on it I think the conclusion is the same: Good design does not naturally arise from good programming, and background/domain knowledge is essential

DowsingSpoon 1y ago
But Uncle Bob said that if I follow his TDD process by rote then my code will automatically attain good design! /s
- earnestinger 1y ago
  My view of Uncle Bob is that his advise is attempt improve the average situation, even if taken literally as a gospell. It’s better that everybody do route tdd, than no tests at all. It’s better to have all methods exactly 3 lines of code, than all methods being 3 pages long. Uncle bob preaches better baseline, experience will teach all the nuance. Disclaimer: I dont do tdd. But it’s nice method of work to know.
  - vrighter 1y ago
    every time i heard a talk or read something of his I always come out with the impression that he has no idea about anything. Most of what he says sounds like bullshit to me.
    - tacitusarc 1y ago
      His advice needs to be contextualized within the 90s and 00s. He was correcting (or overcorrecting) for issues that were prevalent then: methods that were hundreds or thousands of lines, processes with hundreds of hours of design work without writing a single line of code, no testing (or maybe handing to a QA to test, if you’re lucky), and tangled messes of logic the author thought could be saved by adding a comment. Much of his advice seems silly or extreme now, but I believe that’s because he (and his cohort of likeminded programmers) won. Things have improved dramatically. The extreme advice worked. The problem, of course, is that you can’t just declare victory and move on. So now we have The Church of TDD and the like.

prmph 1y ago

And this is exactly thinking AI is going to make human thought and skill redundant is crazy. LLMs can never get to a point where you whip them up to solve any general intellectual challenge. Even creating CRUD apps (that are well-behaved, secure, performant, scalable, and maintainable) can't really be totally systematized.

fydorm 1y ago

Wow, I just read the original Sudoku post by Norvig yesterday for the first time, and now today this shows up here!

layer8 1y ago
It was bound to happen to someone.

groby_b 1y ago

It's making the same reasoning mistake that a lot of discussion of the Entscheidungsproblem makes - the problem talks about a generic algorithm to answer for all programs P if they can solve T, for all tasks T, the discussion assumes that you can't decide if P solves T for any P or T. With that in mind, let's look at the crux of the argument: "If we can't decide if a program P solves a task T, then we certainly can't solve the even harder problem of finding a program P that solves a given task." That's simply not true. We can decide if a program P solves a task T, for a specific program. Moreover, that means that for large classes of tasks, we actually can throw mud at the wall and see if it sticks - as long as it's decidable if the particular program P solves the particular task T. And for any problems you can exhaustively test, hey, you really can rely entirely on TDD and hill-climbing the problem space. Hence, bowling scores being easier than Sudoku solvers. As soon as you leave the (almost) exhaustively testable space, things become harder. And it's worth keeping in mind that TDD originates from a payroll system - something that's more amenable to exhaustive testing ("do all of our employees get the right amount, and did HR/finance stop yelling, and is our CFO not getting jailed for tax evasion") than a systematic approach. (Government plus corporate bureuacracy means that there are absolutely no deep structures to find. It's all about the special cases) You can still do "TDD" at a higher level. You just need to accept that TDD is simply a primitive form of formally establishing your constraints, and that some constraints are hard to establish purely through testing. But that there exist many formalism to allow you to express higher level constraints. This is at the core of the Haskell/Rust quip that "once the type checker/borrow checker is quiet, you can be confident the solution works". Maybe constraint-driven design would've been a better name.

rjpower9000 1y ago
I won't argue with the idea whether you can test a particular program produces a result (we aren't so interested in programs that run forever in any case). Your observation of the origins of TDD makes sense in this regard: there may be domains where the TDD approach is more useful, but it's not a generic panacea, and assuming it is can lead to sadness. My point was more about arbitrary programs and tasks. In general, I don't think there's any particular process you can follow that, in effect, reduces programming or math to a checklist. I'm sure there are techniques that can help structure many problems but they're necessarily fairly general e.g. "think about the problem, take a nap, think about it some more". What I was hoping to highlight was more the danger of trying to hill-climb without having the right set of skills, thus assuming you can solve a problem by following a particular pattern. To give an example, I remember trying to derive the quadratic equation when I was young, and no amount of algebraic repositioning was going to help me unless I know how to complete the square! But once that type of trick is in your toolbox, you can begin to solve all sorts of other problems. Even the bowling calculator can become hard if you don't know what your doing, not because TDD is bad or a wrong way to do things, but because you don't know how to reason about your program. Even bowling has 12^13 outcomes, with a bunch of silly special cases if you approach it the wrong way. Thus a truly naive approach might lead you down a weird direction.

gchamonlive 1y ago

> Wouldn't it be nice if you just had a general method you could apply to solve everything? Would it? Why? > But the reality is there's no single approach we can take. > I know, I know, this is all so deeply... unsatisfying. Why is it so unsatisfying? This is the only thing that I disagree with the post, that this need to have a general approach to solving problems is necessary for solving problems to be satisfying. What is so disconcerting about a problem that seems to be unsolvable? Why do we need to solve everything immediately? Maybe it's important not only to accept, but to embrace the fact that there is a sort of magic to the process of creativity that we can't capture in a bottle. But for that to happen we need to divorce ourselves from the necessity of productivity. This way we are more likely to be able to just sit with the problem without beating ourselves over not being able to solve it and without feeling like the problem owe us a resolution in return for our efforts. It doesn't prevent anyone however from pursuing better tools and more general approaches to systematizing thought and problem solving. We can do both -- tackling a problem creatively or systematically -- as long as we feel like we are able to choose doing it. I think if we feel like we must find these general methods, they will likely elude us indefinitely.

andrewlgood 1y ago

This sounds very similar to the challenge that companies had implementing Six Sigma (or other statistically-based quality programs). The tools significantly helped the average person improve the quality of their processes as determined by a variety of metrics. However, they were never going to tell you the solution or improvement to implement. They could give hints as to where to look for a solution, but someone still had to dream up the solution. For this reason, brain storming was always one of the most critical tools in the Six Sigma toolkit.