4 ms·

> ...I have never seen any evidence that the WMF has been following standard software engineering principles that were well-known when Mythical Man-Month was wa

> ...I have never seen any evidence that the WMF has been following standard software engineering principles that were well-known when Mythical Man-Month was was first published in 1975. If they had, we would be seeing things like requirements documents and schedules with measurable milestones. This part of the critique seems a little off, doesn't it? I don't know the state of WMF engineering, it very well may have problems, and a complete lack of documentation or planning is not a good sign, but the particular artifacts (requirements documents, schedules with milestones) mentioned here are more from the pre-Agile waterfall school of thought. Can anyone familiar with WMF engineering comment?

exizt88 9y ago
Agile approach doesn't mean that there are no requirements documents and no milestones. You're still supposed to write requirements in some form (e.g. user stories and test cases) and plan a few months ahead (while being ready to correct your course based on user feedback after every sprint).
smacktoward 9y ago
That's the point; he's arguing that WMF engineering practices are so disorganized that not only don't they qualify as agile, they don't even qualify as waterfall (which predates agile by several decades). Waterfall methodologies are deeply un-hip today, of course, but when they first coalesced they were a big improvement over what came before them, which was essentially nothing: an absence of any formal project management methodologies, with people cobbling together projects from bits and pieces of expertise learned in other disciplines. (Note that I have no idea how WMF's software engineering practices work, so I have no idea if this assertion is accurate or not. I'm just trying to clarify what I think Macon is arguing here.)
- dom0 9y ago
  "Waterfall methodology" is the ultimate straw man.
  - amscanne 9y ago
    It's hilarious. It has only existed as a thing to critcize, and the term itself actually originates in a paper describing why it's broken. No one has ever advocated for the "waterfall" approach. The ultimate straw man indeed :)
    - dragonwriter 9y ago
      > It has only existed as a thing to critcize No, this is false. > and the term itself actually originates in a paper describing why it's broken. So does the term "capitalism". Like capitalism, though, the waterfall method was a thing actually in wide use both before (the first paper describing it's use was about 20 years earlier than the critical one in which the term seems to have been first used) and after (it's been mandated by many institutions, particularly in government, even after that critical paper) being names in criticism. > No one has ever advocated for the "waterfall" approach. Actually, a number of large organizations, particularly governments, to this day mandate processes for software development projects, particularly large projects, that embody essentially the key features of the waterfall method, most critically that of doing full analysis across the whole scope before beginning development (often, in government, before getting approval for funding to open up contracting for the actual development work.) A lot of the contractors involved advertise that they use agile methods, but it ends often up being a kind of Scrum-within-waterfall monstrosity that managed to preserve the worst features of both.
      - amscanne 9y ago
        > No, this is false. Point me to someone espousing the benefits of the waterfall approach, please. > So does the term "capitalism". Like capitalism, though, the waterfall method was a thing actually in wide use both before (the first paper describing it's use was about 20 years earlier than the critical one in which the term seems to have been first used) and after (it's been mandated by many institutions, particularly in government, even after that critical paper) being names in criticism. I'm not saying that no one has ever tried building software this way. But the term and "methodology" are literally the collection of broken processes associated with early development. > Actually, a number of large organizations, particularly governments, to this day mandate processes for software development projects, particularly large projects, that embody essentially the key features of the waterfall method, most critically that of doing full analysis across the whole scope before beginning development (often, in government, before getting approval for funding to open up contracting for the actual development work.) How do you go to tender without reasonable complete requirements? The problem here isn't the development methodology, it's the fact that going to tender for development basically forces you into this position. Governments seem to be moving to in house development to solve this problem, but I'd hardly say that the original requirements gathering was the result of anyone advocating for the waterfall approach.
- elect_engineer 9y ago
  You are correct. I am arguing that WMF engineering practices are so disorganized that not only don't they qualify as agile, they don't even qualify as waterfall (which predates agile by several decades). In particular, I am part of the community of Wikipedia editors. Nobody asked us what flow or Knowledge Engine should look like. That's part of Waterfall AND Agile. Yes a dedicated person can go to the developer's separate Wiki and mailing lists, but on Wikipedia itself there is zero evidence that the principles I see at [ http://agilemanifesto.org/principles.html http://agilemanifesto.org/principles.html ] are in play. Again, the developers and their managers know this. I am convinced that they have been told in no uncertain terms that they will be fired if they interact with the Wikipedia community. (I am the author of the op-ed. A better version is at [ https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_Cancer https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C... ].)
- elect_engineer 9y ago
  You are correct. I am arguing that WMF engineering practices are so disorganized that not only don't they qualify as agile, they don't even qualify as waterfall (which predates agile by several decades). In particular, I am part of the community of Wikipedia editors. Nobody asked us what flow or Knowledge Engine should look like. That's part of Waterfall AND Agile. Yes a dedicated person can go to the developer's separate Wiki and mailing lists, but on Wikipedia itself there is zero evidence that the principles I see at [ http://agilemanifesto.org/principles.html http://agilemanifesto.org/principles.html ] are in play. Again, the developers and their managers know this. I am convinced that they have been told in no uncertain terms that they will be fired if they interact with the Wikipedia community. (I am the author of the op-ed. A better version is at [ https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_Cancer https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C... ].)
smsm42 9y ago
> and a complete lack of documentation or planning is not a good sign Neither it is true. There is documentation and planning. As pretty much every software project I've seen over my career, documentation could use some TLC (and unlike many other pieces of software, anybody can actually help it[1]), but it's not exactly described as "complete lack". There's a lot of documentation, though some areas are covered less than others - Mediawiki is a big piece of software, and a long-term organically grown project, and if you have any experience with those you know what it means. It is known and regular effort is taken to improve it. Same for planning - not exactly ideal, but "complete lack" is very, very far from the truth - moreover, unlike many other organizations, all the plans and all the internal workboards are public[2] (excluding security issues and sensitive information), so you can check for yourself. > Can anyone familiar with WMF engineering comment? Yes, I am familiar with it by virtue of being part of it (still not speaking for anyone by myself, off-the-clock, in completely personal capacity :) and I say this claim is completely false. Moreover, it is so obviously false and so easily disproven by public documents[3] that I wonder how one could publish that in a public media without bothering to do minimal due diligence. I mean, we all panic about "fake news" and stuff - shouldn't it make us to at least minimally try to check our claims with easy search or question on a mailing lists having dozens of people who could point out where the appropriate documents are? The author of the article seemingly believes it is unnecessary. I do respect his long-time contribution to Wikipedia (much more sizeable than mine) but that still does not entitle him to his own facts. Looks like he disagrees with some of the projects Wikimedia took on - like making user experience more friendly with Visual Editor and mobile support, both IMO excellent projects, but everybody is entitled to their own take on this. It is fine. What's not fine is claiming that not agreeing with him is equivalent to not having direction at all and wasting money and being cancer. That's way too far and completely untrue. [1] https://www.mediawiki.org/wiki/MediaWiki_Documentation_Day_2017 https://www.mediawiki.org/wiki/MediaWiki_Documentation_Day_2... [2] https://phabricator.wikimedia.org/ https://phabricator.wikimedia.org/ [3] https://www.mediawiki.org/wiki/Wikimedia_Engineering/2016-17_Goals https://www.mediawiki.org/wiki/Wikimedia_Engineering/2016-17...
swalling 9y ago
[Former product manager at the Wikimedia Foundation and longtime Wikipedia editor/admin here.] The author of the op-ed is a devoted editor but seems almost totally ignorant of how development is conducted. The Foundation has been doing transparent quarterly/yearly roadmap planning alongside their annual plan / budget cycle (which is shared all publicly). On a shorter timeframe, they are pretty serious about Agile/Scrum. You can see on https://wikimediafoundation.org/wiki/Staff_and_contractors https://wikimediafoundation.org/wiki/Staff_and_contractors that today they even have a team of half a dozen full time Scrum masters. If what he thinks is missing is serious, detailed planning, he's sorely wrong. The platform (MediaWiki) is still a FOSS community so you can find project requirements docs/roadmaps all over mediawiki.org, all the bugs on Phabricator, follow along on mailing lists, and even see commits on their Gerrit instance. Agile isn't my cup of tea personally and I could grok criticisms of the organization's software output (ignoring the fact that they're buried under 10+ years of technical debt...), but it takes minimal digging to find all their plans and timelines. I would venture that the author chose not to dig into this because he, like a lot of entrenched old school editors, viscerally hates some of the past attempts to make MediaWiki a modern collaboration platform, such as building a WYSIWYG editor and a threaded discussion system to replace wiki talk pages.
- saurik 9y ago
  Where can one find out what percentage of Wikimedia's developer staff resources (as opposed to open source contributors) are being allocated towards what projects? They are spending $31m this year on staff: what percentage of that is being spent on what kinds of staff (ex. software engineer vs. community manager), and what percentage of their developer staff is being used to build these aforementioned projects? If that number is extremely low then you can just discount that issue, but if that number is enourmous then more questions have to be asked (which would of course involve looking at the success metrics on those projects and what validation was done on them while they were designed). As it stands, Wikimedia is constantly asking for more money using the threat that Wikipedia will collapse, when for all we know most of their staff time is off building stuff like Wiktionary.
  - swalling 9y ago

exizt88 9y ago

Agile approach doesn't mean that there are no requirements documents and no milestones. You're still supposed to write requirements in some form (e.g. user stories and test cases) and plan a few months ahead (while being ready to correct your course based on user feedback after every sprint).

smacktoward 9y ago

That's the point; he's arguing that WMF engineering practices are so disorganized that not only don't they qualify as agile, they don't even qualify as waterfall (which predates agile by several decades). Waterfall methodologies are deeply un-hip today, of course, but when they first coalesced they were a big improvement over what came before them, which was essentially nothing: an absence of any formal project management methodologies, with people cobbling together projects from bits and pieces of expertise learned in other disciplines. (Note that I have no idea how WMF's software engineering practices work, so I have no idea if this assertion is accurate or not. I'm just trying to clarify what I think Macon is arguing here.)

dom0 9y ago
"Waterfall methodology" is the ultimate straw man.
- amscanne 9y ago
  It's hilarious. It has only existed as a thing to critcize, and the term itself actually originates in a paper describing why it's broken. No one has ever advocated for the "waterfall" approach. The ultimate straw man indeed :)
  - dragonwriter 9y ago
    > It has only existed as a thing to critcize No, this is false. > and the term itself actually originates in a paper describing why it's broken. So does the term "capitalism". Like capitalism, though, the waterfall method was a thing actually in wide use both before (the first paper describing it's use was about 20 years earlier than the critical one in which the term seems to have been first used) and after (it's been mandated by many institutions, particularly in government, even after that critical paper) being names in criticism. > No one has ever advocated for the "waterfall" approach. Actually, a number of large organizations, particularly governments, to this day mandate processes for software development projects, particularly large projects, that embody essentially the key features of the waterfall method, most critically that of doing full analysis across the whole scope before beginning development (often, in government, before getting approval for funding to open up contracting for the actual development work.) A lot of the contractors involved advertise that they use agile methods, but it ends often up being a kind of Scrum-within-waterfall monstrosity that managed to preserve the worst features of both.
    - amscanne 9y ago
      > No, this is false. Point me to someone espousing the benefits of the waterfall approach, please. > So does the term "capitalism". Like capitalism, though, the waterfall method was a thing actually in wide use both before (the first paper describing it's use was about 20 years earlier than the critical one in which the term seems to have been first used) and after (it's been mandated by many institutions, particularly in government, even after that critical paper) being names in criticism. I'm not saying that no one has ever tried building software this way. But the term and "methodology" are literally the collection of broken processes associated with early development. > Actually, a number of large organizations, particularly governments, to this day mandate processes for software development projects, particularly large projects, that embody essentially the key features of the waterfall method, most critically that of doing full analysis across the whole scope before beginning development (often, in government, before getting approval for funding to open up contracting for the actual development work.) How do you go to tender without reasonable complete requirements? The problem here isn't the development methodology, it's the fact that going to tender for development basically forces you into this position. Governments seem to be moving to in house development to solve this problem, but I'd hardly say that the original requirements gathering was the result of anyone advocating for the waterfall approach.
elect_engineer 9y ago
You are correct. I am arguing that WMF engineering practices are so disorganized that not only don't they qualify as agile, they don't even qualify as waterfall (which predates agile by several decades). In particular, I am part of the community of Wikipedia editors. Nobody asked us what flow or Knowledge Engine should look like. That's part of Waterfall AND Agile. Yes a dedicated person can go to the developer's separate Wiki and mailing lists, but on Wikipedia itself there is zero evidence that the principles I see at [ http://agilemanifesto.org/principles.html http://agilemanifesto.org/principles.html ] are in play. Again, the developers and their managers know this. I am convinced that they have been told in no uncertain terms that they will be fired if they interact with the Wikipedia community. (I am the author of the op-ed. A better version is at [ https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_Cancer https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C... ].)
elect_engineer 9y ago
You are correct. I am arguing that WMF engineering practices are so disorganized that not only don't they qualify as agile, they don't even qualify as waterfall (which predates agile by several decades). In particular, I am part of the community of Wikipedia editors. Nobody asked us what flow or Knowledge Engine should look like. That's part of Waterfall AND Agile. Yes a dedicated person can go to the developer's separate Wiki and mailing lists, but on Wikipedia itself there is zero evidence that the principles I see at [ http://agilemanifesto.org/principles.html http://agilemanifesto.org/principles.html ] are in play. Again, the developers and their managers know this. I am convinced that they have been told in no uncertain terms that they will be fired if they interact with the Wikipedia community. (I am the author of the op-ed. A better version is at [ https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_Cancer https://en.wikipedia.org/wiki/User:Guy_Macon/Wikipedia_has_C... ].)

smsm42 9y ago

> and a complete lack of documentation or planning is not a good sign Neither it is true. There is documentation and planning. As pretty much every software project I've seen over my career, documentation could use some TLC (and unlike many other pieces of software, anybody can actually help it[1]), but it's not exactly described as "complete lack". There's a lot of documentation, though some areas are covered less than others - Mediawiki is a big piece of software, and a long-term organically grown project, and if you have any experience with those you know what it means. It is known and regular effort is taken to improve it. Same for planning - not exactly ideal, but "complete lack" is very, very far from the truth - moreover, unlike many other organizations, all the plans and all the internal workboards are public[2] (excluding security issues and sensitive information), so you can check for yourself. > Can anyone familiar with WMF engineering comment? Yes, I am familiar with it by virtue of being part of it (still not speaking for anyone by myself, off-the-clock, in completely personal capacity :) and I say this claim is completely false. Moreover, it is so obviously false and so easily disproven by public documents[3] that I wonder how one could publish that in a public media without bothering to do minimal due diligence. I mean, we all panic about "fake news" and stuff - shouldn't it make us to at least minimally try to check our claims with easy search or question on a mailing lists having dozens of people who could point out where the appropriate documents are? The author of the article seemingly believes it is unnecessary. I do respect his long-time contribution to Wikipedia (much more sizeable than mine) but that still does not entitle him to his own facts. Looks like he disagrees with some of the projects Wikimedia took on - like making user experience more friendly with Visual Editor and mobile support, both IMO excellent projects, but everybody is entitled to their own take on this. It is fine. What's not fine is claiming that not agreeing with him is equivalent to not having direction at all and wasting money and being cancer. That's way too far and completely untrue. [1] https://www.mediawiki.org/wiki/MediaWiki_Documentation_Day_2017 https://www.mediawiki.org/wiki/MediaWiki_Documentation_Day_2... [2] https://phabricator.wikimedia.org/ https://phabricator.wikimedia.org/ [3] https://www.mediawiki.org/wiki/Wikimedia_Engineering/2016-17_Goals https://www.mediawiki.org/wiki/Wikimedia_Engineering/2016-17...

swalling 9y ago

[Former product manager at the Wikimedia Foundation and longtime Wikipedia editor/admin here.] The author of the op-ed is a devoted editor but seems almost totally ignorant of how development is conducted. The Foundation has been doing transparent quarterly/yearly roadmap planning alongside their annual plan / budget cycle (which is shared all publicly). On a shorter timeframe, they are pretty serious about Agile/Scrum. You can see on https://wikimediafoundation.org/wiki/Staff_and_contractors https://wikimediafoundation.org/wiki/Staff_and_contractors that today they even have a team of half a dozen full time Scrum masters. If what he thinks is missing is serious, detailed planning, he's sorely wrong. The platform (MediaWiki) is still a FOSS community so you can find project requirements docs/roadmaps all over mediawiki.org, all the bugs on Phabricator, follow along on mailing lists, and even see commits on their Gerrit instance. Agile isn't my cup of tea personally and I could grok criticisms of the organization's software output (ignoring the fact that they're buried under 10+ years of technical debt...), but it takes minimal digging to find all their plans and timelines. I would venture that the author chose not to dig into this because he, like a lot of entrenched old school editors, viscerally hates some of the past attempts to make MediaWiki a modern collaboration platform, such as building a WYSIWYG editor and a threaded discussion system to replace wiki talk pages.

saurik 9y ago
Where can one find out what percentage of Wikimedia's developer staff resources (as opposed to open source contributors) are being allocated towards what projects? They are spending $31m this year on staff: what percentage of that is being spent on what kinds of staff (ex. software engineer vs. community manager), and what percentage of their developer staff is being used to build these aforementioned projects? If that number is extremely low then you can just discount that issue, but if that number is enourmous then more questions have to be asked (which would of course involve looking at the success metrics on those projects and what validation was done on them while they were designed). As it stands, Wikimedia is constantly asking for more money using the threat that Wikipedia will collapse, when for all we know most of their staff time is off building stuff like Wiktionary.
- swalling 9y ago