26 ms·

A Generation Lost in the Bazaar (2012)

260 points by rendx 9y ago

jasode 9y ago
If people want to discuss this again that's fine but I think it's important to point out previous discussions because the author (phkamp) participated in them: https://hn.algolia.com/?query=lost%20bazaar&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=lost%20bazaar&sort=byPopularit... Reading phkamp's comments in those threads adds more context to the ACM essay.
- bshanks 9y ago
  Specifically, the links to stories with phkamp's comments appear to be: Too many comments to list here in: https://news.ycombinator.com/item?id=4407188 https://news.ycombinator.com/item?id=4407188 Three comments in https://news.ycombinator.com/item?id=12251323 https://news.ycombinator.com/item?id=12251323 : https://news.ycombinator.com/item?id=12253059 https://news.ycombinator.com/item?id=12253059 https://news.ycombinator.com/item?id=12253175 https://news.ycombinator.com/item?id=12253175 https://news.ycombinator.com/item?id=12253027 https://news.ycombinator.com/item?id=12253027 Two comments in https://news.ycombinator.com/item?id=8812724 https://news.ycombinator.com/item?id=8812724 : https://news.ycombinator.com/item?id=8814319 https://news.ycombinator.com/item?id=8814319 https://news.ycombinator.com/item?id=8813888 https://news.ycombinator.com/item?id=8813888
jhiska 9y ago
Hyperbolic.
charlieflowers 9y ago
Similar to DNA that has evolved to create living organisms. Each evolutionary step was selected because it worked, but the result is in no way optimized to be understood by human brains. There's a hierarchy of systems: * non-working, random, chaotic * working, but not understood/understandable * working, and clearly understood by at least 1 human brain Clearly, each step in the hierarchy is less primitive than its predecessors. Also, since human brains are the greatest intelligence we know of, the 3rd bullet point seems like the pinnacle.
- shady-lady 9y ago
  > Each evolutionary step was selected because it worked not really. selected is the wrong word - survived is more appropriate. evolution isn't a set of preordained occurrences. it's a random mutation which happened to survive - this can be due to many reasons (only one of which is "it worked")
  - inimino 9y ago
    > selected is the wrong word It’s literally in the name “natural selection”. Survival is selection.
    - shady-lady 9y ago
      No.
    - SolaceQuantum 9y ago
      Selection in the evolutionary biology sense is not with the connotations of the common word. Selection is considered as a pressure on a population, and not a picking and choosing of individuals exactly, merely the picking off of individuals that for some reason or another cannot breed.
vidarh 9y ago
A big problem with his thesis is that I see just as much ugly crap in proprietary codebases where there is a clear structure of who is responsible for what. The problem is often that the economics does not favor the perfect. I'm working on a project for a client now where I know there are lots of ugly warts in the backend because we've been building abstractions in parallel with building the rest of the system and have not always had the time to go back and revise the old code. I would love to go back and sort that out. But to the client it is an extra expense now, both in terms of money, but more importantly in time spent now, vs. paying for it increased maintenance costs for people to - hopefully - clean it up over time later. But there's also often the other cost of learning and determining if a shared piece of code is appropriate. E.g. there's about half a dozen Ruby gems for handling ANSI escape codes, but for a recent project I ended up writing my own, because none of them seemed to cover what I needed, and it's not clear they should. A lot of code duplication happens because the cost of trying to avoid it often far outweighs the cost of maintaining code you know is doing exactly what you want it to. I do agree with his hate for the autotools family, though.
- charlieflowers 9y ago
  Interesting ... a system that starts out as understandable/understood can degrade into one that still works, but is not understood (obviously ... we see it every day). At that point, work (in the Physics sense) is required to lift it back to the outer energy level -- that of both working and being understood. In many cases, a reasonable option will be to leave it at the degraded level and continue to utilize it (after all, it still "works"). This all seems obvious to me for the first time, as I look at it from this perspective. But it has never dawned on me before, probably because every inch of my soul longs for the beauty of the highest level of the hierarchy -- "works and is understood".
  - k__ 9y ago
    Faster than you think. First they sell you a REST API, then it is more of a RPC, so you have to restructure. Later many endpoints return crap, even if they worked, so you start ignoring the responses. etc. pp.
- 9y ago
weego 9y ago
Aka how to extrapolate a few personal frustrations into sweeping criticism of an entire industry
- parasubvert 9y ago
  PHK is the creator and maintainer of Varnish, and has long been known for his diatribes.
marcus_holmes 9y ago
I love how it's all the fault of the dot-com generation, except the one definite example given was built in the 1980's...
hguhghuff 9y ago
I missed what the point of this was.... some sort of lament at how things aren't right for some reason?
- chris_wot 9y ago
  Do you understand m4?
  - tzahola 9y ago
    I don’t m8
    - DonHopkins 9y ago
      ESR would rather use M16.
ekidd 9y ago
From the OP: > ...there is no escaping that the entire dot-com era was a disaster for IT/CS in general and for software quality and Unix in particular. The dotcom era was almost 20 years ago. The Netscape IPO (1995) is now closer to the IBM PC (1981) than it is to 2017. Yes, a lot of terrible engineering happened during the dotcom boom, and most of it was proprietary code running on old-school Unix servers. If anything, Linux was a pretty solid alternative in those days. Sure, the GUI was a bad joke, and driver support was hit or miss. But if you chose your server hardware well, then Linux would give you years of uptime. Windows fell over several times a week. (Microsoft didn't get truly serious about quality and security until sometime around XP SP2, when they finally got sick of malware and buggy drivers.) So in 2017, what does a "bazaar"-style project look like? One modern example of a "bazaar" is Rust. (If you dislike Rust, you can insert plenty of other examples here.) All major design decisions are made through RFCs, after community feedback. There's no benevolent dictator, but rather a core team. Several key libraries like "serde" are actually maintained by separate teams. The compiler is released every 6 weeks. And yet, I have absolutely zero complaints about Rust's QA. There are plenty of test suites, and no code gets merged to master until those tests pass. To prevent regressions, somebody downloads all public Rust packages from crates.io, compiles them, runs their test suites, and checks for regressions. And so at work, we can upgrade our Rust compilers without fear. Now, to be fair, Rust is not for everybody. Not everybody wants "a better C++ minus the foot guns," or enjoys references and generic types. But if you do want those things, Rust is a good example of a community-driven project which delivers reliable, well-designed software. And I could point out 20 more modern open source projects with solid design and solid QA. We know how to do this now. And those inexperienced dotcom-era graduates now have houses in the suburbs. They're planning how to pay for their kids' college educations, and more than a few of them have some grey hair.
- erikbye 9y ago
  > And I could point out 20 more modern open source projects with solid design and solid QA. We know how to do this now. Then you should point them out, if they are of a standard you think people should learn from.
  - gurjeet 9y ago
scoot 9y ago
> "by more than 10,000 percent" If you're quoting numbers in percentage terms, and that term is more than 100%, there's proabably a better way to express it, in this case, "100 times" (or 100X if you prefer).
- wruza 9y ago
  I suspect that was said for those who wouldn't recognize the sound of “two orders of magnitude” even “if you hit them over the head with it”.
  - scoot 9y ago
    Yes, and if you don't understand two orders of magnitude, "10,000%" is going to be equally confusing, if not potentially misleading. "100 times" is clear and unambiguous (even if the claim itself is unsubstatiated).
pvg 9y ago
2012 and many previouslies: https://hn.algolia.com/?query=A%20Generation%20Lost%20in%20the%20Bazaar&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=A%20Generation%20Lost%20in%20t...
carlsborg 9y ago
The bazaar model gives you a meritocracy of ideas. You need a community process on top of that to ensure quality.
- AnIdiotOnTheNet 9y ago
  Linux Desktop does it horrifically badly. The community around it is terrible on a good day, so I guess that supports your assertion.
- notyourday 9y ago
  Bazaar is what happens when you design by a committee. Sometimes it just so happen that the project is small and the only committee members are rocket scientists and the project is about building a new rocket and painting it black. Having worked on a project like that, I can assure you it is an absolute beauty. The code that those guys shipped has been running in production on hundreds of millions of systems for nearly twenty years. It just works. In the other cases, however, it is a project to build a new car with new kind of square wheels because committee is a band of people who have some members that have never seen a car, some members that think it should go on tracks of Japan, some members that just like coloring spreadsheets, some members that wandered in directly from a bar after a happy hour. Oh and dont forget that guy who did something in a security field of the US Army, who keeps insisting that the wheels should be designed to withstand an IED. Most of bazaar projects are the 2nd, not the first.
jl6 9y ago
Can the author, or anyone, cite some modern examples of great quality software that we can look at and learn from?
- TheAceOfHearts 9y ago
  Check out SQLite [0]. Scroll down to the "How It All Fits Together" section, which points to several docs explaining its different parts, architecture, etc. libmpv has very well documented C interface [1], although I haven't had any occasion to use it. I haven't dug through the project source either, so I'm uncertain of its quality level. Fuchsia [2] looks very promising. Not everything is documented yet, but many of the parts which have docs are great. It's a bit of a rabbit hole though; when I first encountered it I spent a long time reading through their docs and poking around. [0] https://www.sqlite.org/src/doc/trunk/README.md https://www.sqlite.org/src/doc/trunk/README.md [1] https://github.com/mpv-player/mpv/blob/master/libmpv/client.h https://github.com/mpv-player/mpv/blob/master/libmpv/client.... [2] https://fuchsia.googlesource.com https://fuchsia.googlesource.com
randomsearch 9y ago
People may be interested in a great talk by Joe Armstrong (Erlang) along the same lines: https://vimeo.com/97408239 https://vimeo.com/97408239
hoosieree 9y ago
Part of this is the problem of "best practices" which should more rightly be called "usually best practices". Sometimes you're better off using a goto statement, or hard-coding a constant; but it's hard to justify unconventional choices to any team. It's particularly hard for programmers and engineers to know when to buck convention, because the combination of breadth and depth of domain knowledge makes it really hard to ever attain "expert" level in both.
upofadown 9y ago
The BSDs are known for having a more vertical management structure than other open source projects so I guess the author is contrasting FreeBSD ports to the project itself. It would of been a stronger argument if it had been made explicitly. OK, BSD ports have complex dependencies. That is optional, OpenBSD ports have relatively few dependencies. Is this a good thing or does it lead to more duplication? Packaging is a hard problem. What does the development model have to do with the result? The Bazaar idea is about developing the software in the first place, not what happens to it afterwards.
OliverJones 9y ago
Stone cathedrals (the ones still standing, anyhow) were built by generations of skilled craftspeople guided by a self-perpetuating hierarchy. The hierarchy had access to, and the craftspeople had the benefit of, ways of extracting wealth from the economy. The availability of unlimited resources allowed them to cover for plenty of mistakes. Sure, the cathedral structures look elegant and eternal. That's because time favors the stable ones. The other ones fell down long ago. It's the same reason that the orbits of planets are seemingly so well-ordered: everthing that wasn't in such an orbit has crashed into something else and disappeared. (Look at the surface of the moon for evidence of that.) Music and literature work like that. We think the age of Mozart was particularly wonderful, because we have Mozart's music. But there was plenty of other music in that era, most of it bellowed by bad musicians in taverns. That's not so different from today's music scene. Why shouldn't software work like that? Why isn't Dr. Kamp's lament morally the same as the lament of the guy with the cabin in the woods? I'm talking about the guy who finds himself with more and more neighbors, and grouses that "nobody respects the wilderness any more." We all respect elegant lasting structures (fork / exec / stdin / stdout is one such structure). a creative field with huge output will generate more elegant lasting structures. We just can't tell which ones will survive from where we sit in the middle of it all.
- matt4077 9y ago
  Your argument does make sense, but I'm now wondering if there was a significant number of cathedrals that collapsed? I don't remember ever hearing of such a thing happening without external factors like fire or war.
  - Rotten194 9y ago
    The Hagia Sophia was completed in just 5 years and "Earthquakes in August 553 and on 14 December 557 caused cracks in the main dome and eastern half-dome. The main dome collapsed completely during a subsequent earthquake on 7 May 558,[21] destroying the ambon, altar, and ciborium. The collapse was due mainly to the unfeasibly high bearing load and to the enormous shear load of the dome, which was too flat.[18] These caused the deformation of the piers which sustained the dome.[18] The emperor ordered an immediate restoration. He entrusted it to Isidorus the Younger, nephew of Isidore of Miletus, who used lighter materials and elevated the dome by "30 feet"[18] (about 6.25 meters or 20.5 feet)[clarification needed] – giving the building its current interior height of 55.6 meters (182 ft).[22] Moreover, Isidorus changed the dome type, erecting a ribbed dome with pendentives, whose diameter lay between 32.7 and 33.5 m.[18] Under Justinian's orders, eight Corinthian columns were disassembled from Baalbek, Lebanon, and shipped to Constantinople around 560.[23] This reconstruction, giving the church its present 6th-century form, was completed in 562."
k__ 9y ago
In the Bazaar I can at least fix bugs myself and look what the code says instead of fiddling around with incomplete documentations.
patkai 9y ago
When I read Cathedral and the bazaar I was genuinely surprised that it is not entitled "the cathedral _with_ a bazaar". Both have a role, e.g. the Linux kernel is a cathedral while GNU software is the bazaar - neither of them would make much sense without the other.
- chubot 9y ago
  Linux isn't accurately described as a cathedral. It really is the best example of a single project that's a bazaar. It has an incredible number of contributors, sometimes with differing agendas, and with local knowledge. And it shows. The API design of Linux is atrocious. It works well and supports a fantastic number of use cases, but it has no global coherence. cgroups is a good example: https://lwn.net/Articles/574317/ https://lwn.net/Articles/574317/ Because they had no precedent to work with for cgroups, the initial design was botched. Linux still relies on the "bones" of Unix, and whenever they have to do something new, they struggle with the API design.
  - patkai 9y ago
    You are probably right, but I couldn't find a better example of a benevolent dictator - like Linus - more or less _sourcing_ the building of the cathedral from the bazaar with a strong grip on the process. But I know that analogies shouldn't be stretched too far :)
- deleted 9y ago
  [deleted]
Jedd 9y ago
A metric I like to use with these kind of nostalgia-laden stories is how often the word 'free' appears, even in parenthesis. To save you some time, the answer is 14 times. And each of those 14 occurrences is exclusively as a prefix to 'BSD'. Complaints around how Eric failed to understand the the key element of free software, because he eschewed the word free, and when the subsequent reviewers missed this, should be easy to algorithmically flag.
trentnix 9y ago
Blame the “how” all you want but the primary blame is in the “who”. The industry has grown much larger and more quickly than the talent base. If the NBA were to suddenly add 200 teams, the average quality of each team would go down. How many “pretty good” programmers do you know now sitting in Director level or CTO positions? How many programmers with good aptitude find themselves being promoted prematurely as an organization struggles to keep that talent on staff? The quality of the average software team has been greatly diluted, the industry shows too little regard for experience, and hackers end up running the show on so many software teams. Naturally, the software looks like it was coded by hackers as a result.
wwarner 9y ago
Sure, m4 looks hoary now, but portability is the purpose and the legacy of the autoconf system, not opacity, sloppiness or laziness. The only alternative is an OS tightly bound to hardware.
freefrancisco 9y ago
Interesting that as people are questioning free markets in politics (both in the left and the right, not a jab to any particular side) we are also questioning free markets in design.
orblivion 9y ago
What the author failed to mention was why any of this was bad. Yes, it's a mess. And I don't like it either. And I like carefully planning things as well, and it's a personal source of frustration working for start up companies. But it's a mistake to assume that something well crafted and (in a sense centrally) planned actually interfaces well with the market. Maybe it does, but that argument needs actually to be made. Caveat: I haven't read Raymond's book myself.
fermigier 9y ago
Please someone add "(2012)" in the title.
- sctb 9y ago
  Done. Thanks!
VLM 9y ago
Its a strange article to read in the context of ageism. He also humorously quotes Brooks who's always been right and often quoted and basically never implemented outside maybe avionics and nuclear control rooms because "ship it now fix it later maybe" which adds to the irony that the article claims the cathedral ruined software because dogfooding and scratching ones own itch somehow miraculously results in worse software quality than "ship it now fix it later" mentality which surely pragmatically has resulted in low quality trash in the ecosystem. You have to keep in mind that its from more than half a decade ago which means its a generation (or two) in the industry which makes the "lost generation" talk even more weird and it makes it an interesting alternative reality alternative history from the past, a very interesting enjoyable sci fi read, while it being an alternate reality prediction that didn't pan out doesn't entirely ruin it. The Edwardian Era not having turned out like "War of the Worlds" postulated doesn't imply its bad story. Alternative histories are cool. I like it.
YeGoblynQueenne 9y ago
>> Today's Unix/Posix-like operating systems, even including IBM's z/OS mainframe version, as seen with 1980 eyes are identical; I just thought I'd clarify this for those among us fortunate enough to not have been acquainted with z/OS: the author does not mean that z/OS is a Unix. Rather, he is refering to the fact that there is a unix "mode" included in z/OS, Unix System Services, that can be started from inside the z/OS command line, TSO, to drop you to a Unix shell. It's a full Unix OS, too, complete with, well, for example- ed. So you can experience the supernatural horror of getting stuck insid ed, inside the z/OS command line, on a mainframe. Which is a bit like being in hell and having a nightmare about being in hell and having nightmares.
jhoechtl 9y ago
> and there is no escaping that the entire dot-com era was a disaster for IT/CS in general and for [...] Unix in particular. I honestly have no idea what ismeant by that. What background info do O need so that makes sense?
- syncsynchalt 9y ago
  Commercial Unix was destroyed around that time, but I don't think that's what PHK was referring to. The seeds of commercial Unix's destruction had been sewn in the decade before. I suspect he's referring to the change that happened in the late 90s as far as the level of quality required to exist in an OS distribution. Linux distributions had a way to bundling every hobbyist project on the planet, with practically no editorial oversight.
ccalvert 9y ago
One of the things I like about open source is its fecundity. When proprietary software ruled, we went to two or three companies to get most of our tools and libraries. Some were very good, some were terrible. We often had to wait years for features and bug fixes. Some key pieces seemed to never go away even though no one liked them. For instance, IE6. In the open source world, bug fixes often come much more quickly. If fixes don't appear, we at least have the source and can try to fix it -- though sometimes that's a high bar. If there is an obvious need for a tool or library, someone implements it and throws it up on GitHub. Sometimes the implementation is great, sometimes it is little more than a starting point for our own solutions. But at least we have the source to act as a starting point. Quality is important, and open source often encourages good quality, but as the writer points out it does not always do so. Sometimes it acts more like a neural net AI algorithm that keeps failing and failing until one day it succeeds. What open source does encourage is a rapid iteration as developers all over the world look for solutions to known problems. It's a messy solution, but it works surprisingly well. It also helps spread knowledge of how what the nuts and bolts of good software looks like.
EGreg 9y ago
That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it. It is hard to believe today, but under this embarrassing mess lies the ruins of the beautiful cathedral of Unix, deservedly famous for its simplicity of design, its economy of features, and its elegance of execution. (Sic transit gloria mundi, etc.) I can say that, for the last seven years, I ran our little company as a Cathedral and not as a Bazaar. We didn't import many packagess in PHP. We wrote our own code and made sure it integrated well with everything else. We didn't run package managers and didn't freeze versions. We wrote all our own stuff in the app layer. I can tell you, it's been... interesting. We know at the end of the day we are responsible for fixing stuff if it breaks. Not waiting for someone else to approve our pull request. Since Raymond's book, something major happened: DISTRIBUTED VERSION CONTROL SYSTEMS such as git became mainstream. So now you can build cathedrals IN THE BAZAAR! Anyone who wants to maintain package X can fork it and maintain it. Maintainers gossip their improvements in a sort of blockchain consensus protocol. If a maintainer falls behind, the package continues to be maintained by others. What I would like to see is MORE emphasis on content-addressable streams of data with history, instead of on the domains / urls from which they are fetched. In other words, more IPFS and less HTTP. That is the best of both worlds, and I describe it here: https://github.com/Qbix/architecture/wiki/Internet-2.0 https://github.com/Qbix/architecture/wiki/Internet-2.0
andrewfromx 9y ago
here's the best way to explain why the Bazaar is the right way: "nothing I ever do works correctly the first time but I'm on revision 8 before the programming by perfection committee gets revision 1 out the door".
DonHopkins 9y ago
Gnu Autoconf is the epitome of FUBAR Bazaar. >This is probably also why libtool's configure probes no fewer than 26 different names for the Fortran compiler my system does not have, and then spends another 26 tests to find out if each of these nonexistent Fortran compilers supports the -g option. >That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it. It is hard to believe today, but under this embarrassing mess lies the ruins of the beautiful cathedral of Unix, deservedly famous for its simplicity of design, its economy of features, and its elegance of execution. (Sic transit gloria mundi, etc.) https://en.wikipedia.org/wiki/Autoconf https://en.wikipedia.org/wiki/Autoconf Criticism There is some criticism that states that Autoconf uses dated technologies, has a lot of legacy restrictions, and complicates simple scenarios unnecessarily for the author of configure.ac scripts. In particular, often cited weak points of Autoconf are: General complexity of used architecture, most projects use multiple repetitions. Generated 'configure' is written in Bourne shell and thus Makefile generation is slow. Some people think that 'configure' scripts generated by autoconf provide only manual-driven command-line interface without any standardization. While it is true that some developers do not respect common conventions, such conventions do exist and are widely used. M4 is unusual and unknown to many developers. Developers will need to learn it to extend autoconf with non-standard checks. Weak backward and forward compatibility requires a wrapper script. Autoconf-generated scripts are usually large and rather complex. Although they produce extensive logging, debugging them can still be difficult. Due to these limitations, several projects that used GNU Build System switched to different build systems, such as CMake and SCons.
- foobarian 9y ago
  > the beautiful cathedral of Unix The biggest irony here is that in the Multics days the Unix design was considered an ugly heretical hack. It's all in the eye of the beholder.
everyone 9y ago
"Thirteen years ago also marks the apogee of the dot-com euphoria, where every teenager was a Web programmer and every college dropout had a Web startup" Yeah, this is hyperbole for dramatic effect. But a bit much no? It suggests the author has been living in one hell of a bubble.
erikb 9y ago
I also was such kind of engineer for years, who had the desire to be a cathedral architect instead of a dusty, worn-out bazaar trader. But the truth is, the bazaar is more successful. It needs less system, less skill to join (more skill to master though), less luck. Because there are 1200 libraries that basically do the same thing just with slight variation it means that if something is good in 500 of them a few of them will survive even a catastrophy like a bubble burst. And really funny is that it is also more scalable. Without architecture and oversight many more of the integrations between these little, inexperienced silos fail. But because there are so many more who could start, again a few will make it because they were lucky enough to do the right experiment at the right time. Surviving in such a jungle is of course harder, which is why we engineers, looking for a rather calm life, hate it so much. We want to focus on the code, not on survival. That's why we turn the facts and act like the Bazaar would be less successful where everything actually shows that it really is more successful. If you are like me, and really want to become successful, even if it means leaving your assumptions behind and accepting that you might've been wrong, then look out for people who survive quite well in this jungle, and learn from them. It's learning by doing though, since what you write into a book today might already be irrelevant tomorrow. And there are books with the more abstract concepts out there, but without feeling and doing it, you can't even understand them (but will think you do anyways). Real life feels more like learning Vim than learning Microsoft Word, but in a similar fashion it also offers rewards that are unexpected and come in sizes one wouldn't even dream about.
parasight 9y ago
"... endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it." Why is this generation of IT "professionals" clueless? Why can't they recognize a sound architecture? How to get a clue?
jstewartmobile 9y ago
This isn't a case of young whippersnappers not knowing what a cathedral looks like. This is a case of the system getting so large that the combinatorics are no longer surmountable. If I'm just starting on a program, and I realize that I zigged where I should have zagged, I just go through a handful of small files and make them zag. I believe Knuth did the same thing and re-wrote tex after his first attempt. Once the system gets larger, updating every boundary and interface to something sane falls somewhere between costly and insane. Don't get me wrong, I'm all for a re-do, it's just that nobody in the private sector is going to foot that bill until we're in crisis mode, and the wizards who remember what the mistakes were aren't getting any younger...

260 points by rendx 9y ago

If people want to discuss this again that's fine but I think it's important to point out previous discussions because the author (phkamp) participated in them: https://hn.algolia.com/?query=lost%20bazaar&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=lost%20bazaar&sort=byPopularit... Reading phkamp's comments in those threads adds more context to the ACM essay.

bshanks 9y ago
Specifically, the links to stories with phkamp's comments appear to be: Too many comments to list here in: https://news.ycombinator.com/item?id=4407188 https://news.ycombinator.com/item?id=4407188 Three comments in https://news.ycombinator.com/item?id=12251323 https://news.ycombinator.com/item?id=12251323 : https://news.ycombinator.com/item?id=12253059 https://news.ycombinator.com/item?id=12253059 https://news.ycombinator.com/item?id=12253175 https://news.ycombinator.com/item?id=12253175 https://news.ycombinator.com/item?id=12253027 https://news.ycombinator.com/item?id=12253027 Two comments in https://news.ycombinator.com/item?id=8812724 https://news.ycombinator.com/item?id=8812724 : https://news.ycombinator.com/item?id=8814319 https://news.ycombinator.com/item?id=8814319 https://news.ycombinator.com/item?id=8813888 https://news.ycombinator.com/item?id=8813888

jhiska 9y ago

Hyperbolic.

charlieflowers 9y ago

Similar to DNA that has evolved to create living organisms. Each evolutionary step was selected because it worked, but the result is in no way optimized to be understood by human brains. There's a hierarchy of systems: * non-working, random, chaotic * working, but not understood/understandable * working, and clearly understood by at least 1 human brain Clearly, each step in the hierarchy is less primitive than its predecessors. Also, since human brains are the greatest intelligence we know of, the 3rd bullet point seems like the pinnacle.

shady-lady 9y ago
> Each evolutionary step was selected because it worked not really. selected is the wrong word - survived is more appropriate. evolution isn't a set of preordained occurrences. it's a random mutation which happened to survive - this can be due to many reasons (only one of which is "it worked")
- inimino 9y ago
  > selected is the wrong word It’s literally in the name “natural selection”. Survival is selection.
  - shady-lady 9y ago
    No.
  - SolaceQuantum 9y ago
    Selection in the evolutionary biology sense is not with the connotations of the common word. Selection is considered as a pressure on a population, and not a picking and choosing of individuals exactly, merely the picking off of individuals that for some reason or another cannot breed.

vidarh 9y ago

A big problem with his thesis is that I see just as much ugly crap in proprietary codebases where there is a clear structure of who is responsible for what. The problem is often that the economics does not favor the perfect. I'm working on a project for a client now where I know there are lots of ugly warts in the backend because we've been building abstractions in parallel with building the rest of the system and have not always had the time to go back and revise the old code. I would love to go back and sort that out. But to the client it is an extra expense now, both in terms of money, but more importantly in time spent now, vs. paying for it increased maintenance costs for people to - hopefully - clean it up over time later. But there's also often the other cost of learning and determining if a shared piece of code is appropriate. E.g. there's about half a dozen Ruby gems for handling ANSI escape codes, but for a recent project I ended up writing my own, because none of them seemed to cover what I needed, and it's not clear they should. A lot of code duplication happens because the cost of trying to avoid it often far outweighs the cost of maintaining code you know is doing exactly what you want it to. I do agree with his hate for the autotools family, though.

charlieflowers 9y ago
Interesting ... a system that starts out as understandable/understood can degrade into one that still works, but is not understood (obviously ... we see it every day). At that point, work (in the Physics sense) is required to lift it back to the outer energy level -- that of both working and being understood. In many cases, a reasonable option will be to leave it at the degraded level and continue to utilize it (after all, it still "works"). This all seems obvious to me for the first time, as I look at it from this perspective. But it has never dawned on me before, probably because every inch of my soul longs for the beauty of the highest level of the hierarchy -- "works and is understood".
- k__ 9y ago
  Faster than you think. First they sell you a REST API, then it is more of a RPC, so you have to restructure. Later many endpoints return crap, even if they worked, so you start ignoring the responses. etc. pp.
9y ago

weego 9y ago

Aka how to extrapolate a few personal frustrations into sweeping criticism of an entire industry

parasubvert 9y ago
PHK is the creator and maintainer of Varnish, and has long been known for his diatribes.

marcus_holmes 9y ago

I love how it's all the fault of the dot-com generation, except the one definite example given was built in the 1980's...

hguhghuff 9y ago

I missed what the point of this was.... some sort of lament at how things aren't right for some reason?

chris_wot 9y ago
Do you understand m4?
- tzahola 9y ago
  I don’t m8
  - DonHopkins 9y ago
    ESR would rather use M16.

ekidd 9y ago

From the OP: > ...there is no escaping that the entire dot-com era was a disaster for IT/CS in general and for software quality and Unix in particular. The dotcom era was almost 20 years ago. The Netscape IPO (1995) is now closer to the IBM PC (1981) than it is to 2017. Yes, a lot of terrible engineering happened during the dotcom boom, and most of it was proprietary code running on old-school Unix servers. If anything, Linux was a pretty solid alternative in those days. Sure, the GUI was a bad joke, and driver support was hit or miss. But if you chose your server hardware well, then Linux would give you years of uptime. Windows fell over several times a week. (Microsoft didn't get truly serious about quality and security until sometime around XP SP2, when they finally got sick of malware and buggy drivers.) So in 2017, what does a "bazaar"-style project look like? One modern example of a "bazaar" is Rust. (If you dislike Rust, you can insert plenty of other examples here.) All major design decisions are made through RFCs, after community feedback. There's no benevolent dictator, but rather a core team. Several key libraries like "serde" are actually maintained by separate teams. The compiler is released every 6 weeks. And yet, I have absolutely zero complaints about Rust's QA. There are plenty of test suites, and no code gets merged to master until those tests pass. To prevent regressions, somebody downloads all public Rust packages from crates.io, compiles them, runs their test suites, and checks for regressions. And so at work, we can upgrade our Rust compilers without fear. Now, to be fair, Rust is not for everybody. Not everybody wants "a better C++ minus the foot guns," or enjoys references and generic types. But if you do want those things, Rust is a good example of a community-driven project which delivers reliable, well-designed software. And I could point out 20 more modern open source projects with solid design and solid QA. We know how to do this now. And those inexperienced dotcom-era graduates now have houses in the suburbs. They're planning how to pay for their kids' college educations, and more than a few of them have some grey hair.

erikbye 9y ago
> And I could point out 20 more modern open source projects with solid design and solid QA. We know how to do this now. Then you should point them out, if they are of a standard you think people should learn from.
- gurjeet 9y ago

scoot 9y ago

> "by more than 10,000 percent" If you're quoting numbers in percentage terms, and that term is more than 100%, there's proabably a better way to express it, in this case, "100 times" (or 100X if you prefer).

wruza 9y ago
I suspect that was said for those who wouldn't recognize the sound of “two orders of magnitude” even “if you hit them over the head with it”.
- scoot 9y ago
  Yes, and if you don't understand two orders of magnitude, "10,000%" is going to be equally confusing, if not potentially misleading. "100 times" is clear and unambiguous (even if the claim itself is unsubstatiated).

pvg 9y ago

2012 and many previouslies: https://hn.algolia.com/?query=A%20Generation%20Lost%20in%20the%20Bazaar&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=A%20Generation%20Lost%20in%20t...

carlsborg 9y ago

The bazaar model gives you a meritocracy of ideas. You need a community process on top of that to ensure quality.

AnIdiotOnTheNet 9y ago
Linux Desktop does it horrifically badly. The community around it is terrible on a good day, so I guess that supports your assertion.
notyourday 9y ago
Bazaar is what happens when you design by a committee. Sometimes it just so happen that the project is small and the only committee members are rocket scientists and the project is about building a new rocket and painting it black. Having worked on a project like that, I can assure you it is an absolute beauty. The code that those guys shipped has been running in production on hundreds of millions of systems for nearly twenty years. It just works. In the other cases, however, it is a project to build a new car with new kind of square wheels because committee is a band of people who have some members that have never seen a car, some members that think it should go on tracks of Japan, some members that just like coloring spreadsheets, some members that wandered in directly from a bar after a happy hour. Oh and dont forget that guy who did something in a security field of the US Army, who keeps insisting that the wheels should be designed to withstand an IED. Most of bazaar projects are the 2nd, not the first.

jl6 9y ago

Can the author, or anyone, cite some modern examples of great quality software that we can look at and learn from?

TheAceOfHearts 9y ago
Check out SQLite [0]. Scroll down to the "How It All Fits Together" section, which points to several docs explaining its different parts, architecture, etc. libmpv has very well documented C interface [1], although I haven't had any occasion to use it. I haven't dug through the project source either, so I'm uncertain of its quality level. Fuchsia [2] looks very promising. Not everything is documented yet, but many of the parts which have docs are great. It's a bit of a rabbit hole though; when I first encountered it I spent a long time reading through their docs and poking around. [0] https://www.sqlite.org/src/doc/trunk/README.md https://www.sqlite.org/src/doc/trunk/README.md [1] https://github.com/mpv-player/mpv/blob/master/libmpv/client.h https://github.com/mpv-player/mpv/blob/master/libmpv/client.... [2] https://fuchsia.googlesource.com https://fuchsia.googlesource.com

randomsearch 9y ago

People may be interested in a great talk by Joe Armstrong (Erlang) along the same lines: https://vimeo.com/97408239 https://vimeo.com/97408239

hoosieree 9y ago

Part of this is the problem of "best practices" which should more rightly be called "usually best practices". Sometimes you're better off using a goto statement, or hard-coding a constant; but it's hard to justify unconventional choices to any team. It's particularly hard for programmers and engineers to know when to buck convention, because the combination of breadth and depth of domain knowledge makes it really hard to ever attain "expert" level in both.

upofadown 9y ago

The BSDs are known for having a more vertical management structure than other open source projects so I guess the author is contrasting FreeBSD ports to the project itself. It would of been a stronger argument if it had been made explicitly. OK, BSD ports have complex dependencies. That is optional, OpenBSD ports have relatively few dependencies. Is this a good thing or does it lead to more duplication? Packaging is a hard problem. What does the development model have to do with the result? The Bazaar idea is about developing the software in the first place, not what happens to it afterwards.

OliverJones 9y ago

Stone cathedrals (the ones still standing, anyhow) were built by generations of skilled craftspeople guided by a self-perpetuating hierarchy. The hierarchy had access to, and the craftspeople had the benefit of, ways of extracting wealth from the economy. The availability of unlimited resources allowed them to cover for plenty of mistakes. Sure, the cathedral structures look elegant and eternal. That's because time favors the stable ones. The other ones fell down long ago. It's the same reason that the orbits of planets are seemingly so well-ordered: everthing that wasn't in such an orbit has crashed into something else and disappeared. (Look at the surface of the moon for evidence of that.) Music and literature work like that. We think the age of Mozart was particularly wonderful, because we have Mozart's music. But there was plenty of other music in that era, most of it bellowed by bad musicians in taverns. That's not so different from today's music scene. Why shouldn't software work like that? Why isn't Dr. Kamp's lament morally the same as the lament of the guy with the cabin in the woods? I'm talking about the guy who finds himself with more and more neighbors, and grouses that "nobody respects the wilderness any more." We all respect elegant lasting structures (fork / exec / stdin / stdout is one such structure). a creative field with huge output will generate more elegant lasting structures. We just can't tell which ones will survive from where we sit in the middle of it all.

matt4077 9y ago
Your argument does make sense, but I'm now wondering if there was a significant number of cathedrals that collapsed? I don't remember ever hearing of such a thing happening without external factors like fire or war.
- Rotten194 9y ago
  The Hagia Sophia was completed in just 5 years and "Earthquakes in August 553 and on 14 December 557 caused cracks in the main dome and eastern half-dome. The main dome collapsed completely during a subsequent earthquake on 7 May 558,[21] destroying the ambon, altar, and ciborium. The collapse was due mainly to the unfeasibly high bearing load and to the enormous shear load of the dome, which was too flat.[18] These caused the deformation of the piers which sustained the dome.[18] The emperor ordered an immediate restoration. He entrusted it to Isidorus the Younger, nephew of Isidore of Miletus, who used lighter materials and elevated the dome by "30 feet"[18] (about 6.25 meters or 20.5 feet)[clarification needed] – giving the building its current interior height of 55.6 meters (182 ft).[22] Moreover, Isidorus changed the dome type, erecting a ribbed dome with pendentives, whose diameter lay between 32.7 and 33.5 m.[18] Under Justinian's orders, eight Corinthian columns were disassembled from Baalbek, Lebanon, and shipped to Constantinople around 560.[23] This reconstruction, giving the church its present 6th-century form, was completed in 562."

k__ 9y ago

In the Bazaar I can at least fix bugs myself and look what the code says instead of fiddling around with incomplete documentations.

patkai 9y ago

When I read Cathedral and the bazaar I was genuinely surprised that it is not entitled "the cathedral _with_ a bazaar". Both have a role, e.g. the Linux kernel is a cathedral while GNU software is the bazaar - neither of them would make much sense without the other.

chubot 9y ago
Linux isn't accurately described as a cathedral. It really is the best example of a single project that's a bazaar. It has an incredible number of contributors, sometimes with differing agendas, and with local knowledge. And it shows. The API design of Linux is atrocious. It works well and supports a fantastic number of use cases, but it has no global coherence. cgroups is a good example: https://lwn.net/Articles/574317/ https://lwn.net/Articles/574317/ Because they had no precedent to work with for cgroups, the initial design was botched. Linux still relies on the "bones" of Unix, and whenever they have to do something new, they struggle with the API design.
- patkai 9y ago
  You are probably right, but I couldn't find a better example of a benevolent dictator - like Linus - more or less _sourcing_ the building of the cathedral from the bazaar with a strong grip on the process. But I know that analogies shouldn't be stretched too far :)
deleted 9y ago
[deleted]

Jedd 9y ago

A metric I like to use with these kind of nostalgia-laden stories is how often the word 'free' appears, even in parenthesis. To save you some time, the answer is 14 times. And each of those 14 occurrences is exclusively as a prefix to 'BSD'. Complaints around how Eric failed to understand the the key element of free software, because he eschewed the word free, and when the subsequent reviewers missed this, should be easy to algorithmically flag.

trentnix 9y ago

Blame the “how” all you want but the primary blame is in the “who”. The industry has grown much larger and more quickly than the talent base. If the NBA were to suddenly add 200 teams, the average quality of each team would go down. How many “pretty good” programmers do you know now sitting in Director level or CTO positions? How many programmers with good aptitude find themselves being promoted prematurely as an organization struggles to keep that talent on staff? The quality of the average software team has been greatly diluted, the industry shows too little regard for experience, and hackers end up running the show on so many software teams. Naturally, the software looks like it was coded by hackers as a result.

wwarner 9y ago

Sure, m4 looks hoary now, but portability is the purpose and the legacy of the autoconf system, not opacity, sloppiness or laziness. The only alternative is an OS tightly bound to hardware.

freefrancisco 9y ago

Interesting that as people are questioning free markets in politics (both in the left and the right, not a jab to any particular side) we are also questioning free markets in design.

orblivion 9y ago

What the author failed to mention was why any of this was bad. Yes, it's a mess. And I don't like it either. And I like carefully planning things as well, and it's a personal source of frustration working for start up companies. But it's a mistake to assume that something well crafted and (in a sense centrally) planned actually interfaces well with the market. Maybe it does, but that argument needs actually to be made. Caveat: I haven't read Raymond's book myself.

fermigier 9y ago

Please someone add "(2012)" in the title.

sctb 9y ago
Done. Thanks!

VLM 9y ago

Its a strange article to read in the context of ageism. He also humorously quotes Brooks who's always been right and often quoted and basically never implemented outside maybe avionics and nuclear control rooms because "ship it now fix it later maybe" which adds to the irony that the article claims the cathedral ruined software because dogfooding and scratching ones own itch somehow miraculously results in worse software quality than "ship it now fix it later" mentality which surely pragmatically has resulted in low quality trash in the ecosystem. You have to keep in mind that its from more than half a decade ago which means its a generation (or two) in the industry which makes the "lost generation" talk even more weird and it makes it an interesting alternative reality alternative history from the past, a very interesting enjoyable sci fi read, while it being an alternate reality prediction that didn't pan out doesn't entirely ruin it. The Edwardian Era not having turned out like "War of the Worlds" postulated doesn't imply its bad story. Alternative histories are cool. I like it.

YeGoblynQueenne 9y ago

>> Today's Unix/Posix-like operating systems, even including IBM's z/OS mainframe version, as seen with 1980 eyes are identical; I just thought I'd clarify this for those among us fortunate enough to not have been acquainted with z/OS: the author does not mean that z/OS is a Unix. Rather, he is refering to the fact that there is a unix "mode" included in z/OS, Unix System Services, that can be started from inside the z/OS command line, TSO, to drop you to a Unix shell. It's a full Unix OS, too, complete with, well, for example- ed. So you can experience the supernatural horror of getting stuck insid ed, inside the z/OS command line, on a mainframe. Which is a bit like being in hell and having a nightmare about being in hell and having nightmares.

jhoechtl 9y ago

> and there is no escaping that the entire dot-com era was a disaster for IT/CS in general and for [...] Unix in particular. I honestly have no idea what ismeant by that. What background info do O need so that makes sense?

syncsynchalt 9y ago
Commercial Unix was destroyed around that time, but I don't think that's what PHK was referring to. The seeds of commercial Unix's destruction had been sewn in the decade before. I suspect he's referring to the change that happened in the late 90s as far as the level of quality required to exist in an OS distribution. Linux distributions had a way to bundling every hobbyist project on the planet, with practically no editorial oversight.

ccalvert 9y ago

One of the things I like about open source is its fecundity. When proprietary software ruled, we went to two or three companies to get most of our tools and libraries. Some were very good, some were terrible. We often had to wait years for features and bug fixes. Some key pieces seemed to never go away even though no one liked them. For instance, IE6. In the open source world, bug fixes often come much more quickly. If fixes don't appear, we at least have the source and can try to fix it -- though sometimes that's a high bar. If there is an obvious need for a tool or library, someone implements it and throws it up on GitHub. Sometimes the implementation is great, sometimes it is little more than a starting point for our own solutions. But at least we have the source to act as a starting point. Quality is important, and open source often encourages good quality, but as the writer points out it does not always do so. Sometimes it acts more like a neural net AI algorithm that keeps failing and failing until one day it succeeds. What open source does encourage is a rapid iteration as developers all over the world look for solutions to known problems. It's a messy solution, but it works surprisingly well. It also helps spread knowledge of how what the nuts and bolts of good software looks like.

EGreg 9y ago

That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it. It is hard to believe today, but under this embarrassing mess lies the ruins of the beautiful cathedral of Unix, deservedly famous for its simplicity of design, its economy of features, and its elegance of execution. (Sic transit gloria mundi, etc.) I can say that, for the last seven years, I ran our little company as a Cathedral and not as a Bazaar. We didn't import many packagess in PHP. We wrote our own code and made sure it integrated well with everything else. We didn't run package managers and didn't freeze versions. We wrote all our own stuff in the app layer. I can tell you, it's been... interesting. We know at the end of the day we are responsible for fixing stuff if it breaks. Not waiting for someone else to approve our pull request. Since Raymond's book, something major happened: DISTRIBUTED VERSION CONTROL SYSTEMS such as git became mainstream. So now you can build cathedrals IN THE BAZAAR! Anyone who wants to maintain package X can fork it and maintain it. Maintainers gossip their improvements in a sort of blockchain consensus protocol. If a maintainer falls behind, the package continues to be maintained by others. What I would like to see is MORE emphasis on content-addressable streams of data with history, instead of on the domains / urls from which they are fetched. In other words, more IPFS and less HTTP. That is the best of both worlds, and I describe it here: https://github.com/Qbix/architecture/wiki/Internet-2.0 https://github.com/Qbix/architecture/wiki/Internet-2.0

andrewfromx 9y ago

here's the best way to explain why the Bazaar is the right way: "nothing I ever do works correctly the first time but I'm on revision 8 before the programming by perfection committee gets revision 1 out the door".

DonHopkins 9y ago

Gnu Autoconf is the epitome of FUBAR Bazaar. >This is probably also why libtool's configure probes no fewer than 26 different names for the Fortran compiler my system does not have, and then spends another 26 tests to find out if each of these nonexistent Fortran compilers supports the -g option. >That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it. It is hard to believe today, but under this embarrassing mess lies the ruins of the beautiful cathedral of Unix, deservedly famous for its simplicity of design, its economy of features, and its elegance of execution. (Sic transit gloria mundi, etc.) https://en.wikipedia.org/wiki/Autoconf https://en.wikipedia.org/wiki/Autoconf Criticism There is some criticism that states that Autoconf uses dated technologies, has a lot of legacy restrictions, and complicates simple scenarios unnecessarily for the author of configure.ac scripts. In particular, often cited weak points of Autoconf are: General complexity of used architecture, most projects use multiple repetitions. Generated 'configure' is written in Bourne shell and thus Makefile generation is slow. Some people think that 'configure' scripts generated by autoconf provide only manual-driven command-line interface without any standardization. While it is true that some developers do not respect common conventions, such conventions do exist and are widely used. M4 is unusual and unknown to many developers. Developers will need to learn it to extend autoconf with non-standard checks. Weak backward and forward compatibility requires a wrapper script. Autoconf-generated scripts are usually large and rather complex. Although they produce extensive logging, debugging them can still be difficult. Due to these limitations, several projects that used GNU Build System switched to different build systems, such as CMake and SCons.

foobarian 9y ago
> the beautiful cathedral of Unix The biggest irony here is that in the Multics days the Unix design was considered an ugly heretical hack. It's all in the eye of the beholder.

everyone 9y ago

"Thirteen years ago also marks the apogee of the dot-com euphoria, where every teenager was a Web programmer and every college dropout had a Web startup" Yeah, this is hyperbole for dramatic effect. But a bit much no? It suggests the author has been living in one hell of a bubble.

erikb 9y ago

I also was such kind of engineer for years, who had the desire to be a cathedral architect instead of a dusty, worn-out bazaar trader. But the truth is, the bazaar is more successful. It needs less system, less skill to join (more skill to master though), less luck. Because there are 1200 libraries that basically do the same thing just with slight variation it means that if something is good in 500 of them a few of them will survive even a catastrophy like a bubble burst. And really funny is that it is also more scalable. Without architecture and oversight many more of the integrations between these little, inexperienced silos fail. But because there are so many more who could start, again a few will make it because they were lucky enough to do the right experiment at the right time. Surviving in such a jungle is of course harder, which is why we engineers, looking for a rather calm life, hate it so much. We want to focus on the code, not on survival. That's why we turn the facts and act like the Bazaar would be less successful where everything actually shows that it really is more successful. If you are like me, and really want to become successful, even if it means leaving your assumptions behind and accepting that you might've been wrong, then look out for people who survive quite well in this jungle, and learn from them. It's learning by doing though, since what you write into a book today might already be irrelevant tomorrow. And there are books with the more abstract concepts out there, but without feeling and doing it, you can't even understand them (but will think you do anyways). Real life feels more like learning Vim than learning Microsoft Word, but in a similar fashion it also offers rewards that are unexpected and come in sizes one wouldn't even dream about.

parasight 9y ago

"... endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it." Why is this generation of IT "professionals" clueless? Why can't they recognize a sound architecture? How to get a clue?

jstewartmobile 9y ago

This isn't a case of young whippersnappers not knowing what a cathedral looks like. This is a case of the system getting so large that the combinatorics are no longer surmountable. If I'm just starting on a program, and I realize that I zigged where I should have zagged, I just go through a handful of small files and make them zag. I believe Knuth did the same thing and re-wrote tex after his first attempt. Once the system gets larger, updating every boundary and interface to something sane falls somewhere between costly and insane. Don't get me wrong, I'm all for a re-do, it's just that nobody in the private sector is going to foot that bill until we're in crisis mode, and the wizards who remember what the mistakes were aren't getting any younger...