5 ms·
Apache HTTPd Server Developers Considered Harmful
For the past 25 years, I have been the lead developer of the libapreq2 subproject within the Apache HTTPd Server Parent Project. The original idea of libapreq as a safe/performant HTML form and Cookie parsing library came out of a collaboration between Lincoln Stein and Doug MacEachern in the late 90s.
It was my vision back then to transform the library into a generic, non-Perl related C library that would support language bindings from other programming languages, which is why I pushed for the project to be homes under the HTTPd umbrella instead of the Apache-Perl project.
While this vision was wildly successful, with language bindings available for several languages like Perl, TCL, R, etc, ever since about 2010 its proven tragic for the existing user community consisting of all of them, not just Perl.
What happened? Philip Gollucci, a Perl/FreeBSD olleague of mine at the time, started agitating that we promote the project to be released from inside the HTTPd server itself. What Philip didn’t know very well back then was how utterly vapid and territorial that team had become, which would have meant having to collaborate with them directly on user-facing decisions about the code base.
In 2012, Philip got what he wanted and I stopped resisting, so he forked the existing project and copied the C library components into HTTPd core.
In 2016 I resigned from the Foundation en masse. You can guess the reasons.
In 2020 or so, Google’s Security Team took advantage of an alpha release of httpd 2.5 by fuzzing its 8 year old copy of apreq. It found a few hotspots that needed repair.
Instead of having the courtesy of reaching out to me, or anyone else involved in development of apreq, a junior engineer on the HTTPd team went about the business of “bug fixing” the vulnerabilities Google found. You can see a record of his trial and error work in every release since then.
But the coup de grace was the 2022 release of 2.17, wherein the rookie developer purposely introduced a fatal bug into the codebase, breaking a fifteen year old regression test.
If you are wondering how something with a broken regression test winds up on CPAN, you’ll have to look into how RELENG is done in the server project.
Long story short, they commented out the test and shipped it anyway, and called it a Security Release that fixed a vulnerability every prior release was susceptible to.
Why do I care now? Because I’m the sucker users reach out to for answers as a known subject matter expert.
This sucks, but I’m sorry to tell you that my days wearing the Superman cape at Apache ended 8 years ago.
- joesuf4 3y agoSee https://blogs.sunstarsys.com/joe/apache-considered-harmful https://blogs.sunstarsys.com/joe/apache-considered-harmful for the permalink.
- joesuf4 3y agoHere's a list of sites who announce mod_apreq2 in their ServerTokens. There are literally thousands of others who do not advertise the version number out of security concerns. https://webtechsurvey.com/technology/mod_apreq2 https://webtechsurvey.com/technology/mod_apreq2
- joesuf4 3y agoMost customers are running it behind a proxy server as part of their middle-tier webapp stack, so this isn't even a tiny fraction of the existing user base. And despite the decade-long warfare against its userbase by the community of glorified morticians at Apache HTTPd Server Project, it still has a growing customer-base!
- keikobadthebad 3y ago> In 2016 I resigned from the Foundation en masse. "En masse": All together and at the same time, in large numbers https://dictionary.cambridge.org/dictionary/english/en-masse https://dictionary.cambridge.org/dictionary/english/en-masse
- joesuf4 3y agoApologies if the meaning was unclear, but what it's meant to convey that I resigned ALL of my dozen-or-so positions from the org simultaneously.
- hu3 3y agoFor what is worth I understood your meaning immediately by applying the "most generous interpretation" principle.
- slau 3y agoMight I suggest phrasing it exactly like that then? “I resigned all 12 positions I held at Apache because X” is a lot clearer to me.
- fickle 3y agoor even further, naming some of the relevant positions (for example HTTP Server project management committee)
- joesuf4 3y agoHow peevish can you guys be?
- joesuf4 3y ago1/ because I don’t remember exactly how many, 2/ because the record keeping at Apache was so haphazard at the time, half of the places I resigned from left me on the roster anyway (like the HTTPd server project itself).
- kkielhofner 3y agoI’ll try to say this as respectfully and as kindly as possible. I’m genuinely trying to be helpful. Seeing these kinds of stories and interacting with the type of people who have them brings to mind an old saying: “If everyone you meet is an asshole, you’re the asshole” Note that I’m not saying that’s the case here or insulting you, that’s just the saying. This story is filled with what seems like (at best) “personality conflicts” and your very dim view of essentially everyone involved in it with the exception of yourself. It even ends with you essentially calling yourself Superman. I’m not sure what the point of this is (vent? warning?) but it doesn’t read like a sympathetic and even remotely balanced telling of events, that’s for sure. It may be worth reflecting on what you may have contributed to this if your goal is reducing the chance of repeats.
- foofie 3y agoIt reads like OP has been holding a grudge for a decade or so, and today decided to shit on a project he contributed way back in the past out of sheer entitlement. I'm not sure if that's the case, but that what's coming across. Small-minded pettiness.
- joesuf4 3y agoIt depends on whether you have any experience with F/OSS communities. If you don't , then 25 years is irrelevant, so yeah compressing it into 1 paragraph makes it seem petty.
- joesuf4 3y ago
- fickle 3y ago> For the past 25 years, I have been the lead developer of the libapreq2 subproject within the Apache HTTPd Server Parent Project. 25 years is a long time for an imaginary title on a vanity project. I guess it helps to work 12.5 years on, 12.5 years off (no commits, no releases). > In 2016 I resigned from the Foundation en masse. You can guess the reasons. NPD?
- joesuf4 3y agoAre you involved with the ASF at this point? You certainly are peevish, vapid, and territorial enough to be chair of some dead end project like HTTPd.
- joesuf4 3y agoFYI: https://projects.apache.org/committee.html?httpd#:~:text=Joe%20Schaefer https://projects.apache.org/committee.html?httpd#:~:text=Joe...