6 ms·

"To replace out MediaWiki with a Java-based alternative (XWiki) ended up taking a total fo 24 dev-years over 4+ calendar years for the team, not counting the in

"To replace out MediaWiki with a Java-based alternative (XWiki) ended up taking a total fo 24 dev-years over 4+ calendar years for the team, not counting the interruptions to pretty much every other team at Amazon as their pages were getting constantly migrated and un-migrated" I would love to hear more about this. I'm guessing that's a cost of at least $4M? How was this approved? How did they allow it to continue for four years?

stu2010 6y ago
Amazon's total all-in cost per dev is almost certainly >$300k per year, so more like >$7 million if that's an accurate number. As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed.
- theli0nheart 6y ago
  > As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed. Maybe I'm being dense, but this still doesn't register for me. If 4+ calendar years of work for a single team continues with no noticeable progress, then there's an organizational issue, no matter the size of the company.
  - WJW 6y ago
    Of course there is, but that's not the point. Every large organisation has organisational issues and nobody in the history of humanity has found a way of preventing them yet. The only thing you can do when the group sizes get large is to try and keep the problem under control relative to how bad it is at other large companies.
    - theli0nheart 6y ago
      I'm lost; what specifically is not the point? As far as I could tell, the "reason" given by the OP for this waste was attributed to Amazon's scale. Could you explain how company size and broken corporate structure are correlated? Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure?
      - timerol 6y ago
        > I'm lost; what specifically is not the point? That there is an organizational issue. Of course there is an organizational issue. There will always be organizational issues. > Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure? Oh, it's definitely an organizational issue here, that's just not the main point. If you're willing to call this a "broken corporate structure," then you should revise your opinion of every company with more than 50 employees downward sharply. This is a normal corporate structure, not a broken one. > Could you explain how company size and broken corporate structure are correlated? When there are enough people, you will always end up with conflicting incentives somewhere. A larger organization will have more of these. In this case, it seems to be a clear case of competing priorities: IT wants to secure systems, and developers want to be maximally productive. I'm sure some folks in IT view Amazon paying ~$7M as a fair cost to remove known vulnerable tools from their infrastructure.
      - sbarre 6y ago
        It's unlikely that this went entirely unnoticed for 4 years, and I would guess that when the problem was understood, someone did the math on staying the course, versus starting over or reversing course. And after all the internal calculations were complete (which factor in way more than technical effort by a small dev team), they intentionally stayed the course. This is probably an extreme example because an internal wiki that is used company-wide will have an incredibly complex and hairy stakeholder matrix (both business and technical), which means consensus is very challenging. At some point the cost of the meetings required to work out a solution (both in time spent, and in time not spent on core work by the stakeholders) probably exceeds the cost of just letting a small team brute-force their way through a bad solution. Cost here isn't just money, it's focus, morale, disruption, conflict, risk, etc.. Keep in mind that you have the benefit of hindsight, and the comfort of looking at it from the outside, with a very incomplete picture of all the constraints and pressures at play.
  - mywittyname 6y ago
    It's pretty easy to have a project look like progress is being made in the moment, especially when presenting to the director+ level, where a manager is presenting on 4-5 different projects. It's even easier with projects that aren't important or highly-visible (as internal projects tend to be).
- nojito 6y ago
  >that's not enough money to really register at Amazon's scale. This viewpoint doesn't really match up with how enterprises operate. It's possible to function at amazon's scale without meticulous accounting and financial operations. The reason for this is that is extremely easy for wasteful spending to propagate throughout the enterprise if unchecked. Waste only appears after a project fails...it's easy to point out waste after the fact, but it's downright impossible to call it out as it's happening.
  - romanows 6y ago
    I guarantee there were some technical people saying this was a waste of time and money.
    - ThrowawayR2 6y ago
      You can find at least one technical person in a company who will say any particular project is a waste of time and money. It means nothing.
klohto 6y ago
I don't understand why are corporations obsessed with decommissioning wikis. When I was at Red Hat, they tried to decommission local country wiki in favor of their company-wide wiki (Mojo, absolute shit by the way). They tried for about 2 years, then finally somebody said "what the fuck are we doing" and the migration was stopped. I believe the banner "Decommissioning soon" still hangs there.
- chii 6y ago
  somebody (usually some kind of middle manager) needs to have projects to look after, otherwise it would show that they are dead weights.
- MattGaiser 6y ago
  Usually, someone saw a cool feature they wanted and everything moved to that. So often the main purpose of storing information gets lost to flashy new graphics.
- schnable 6y ago
  What happens is that different teams build out their own wikis , or come in via acquisition). The company then ends up with a dozen of them, and re-orgs occur and a single team now has two wikis, and people wonder why the information isn't centralized. Everyone agrees it will be great to have a singe company wide wiki and the project is born.
- breischl 6y ago
  Finding the right information is already difficult and a time-suck. Adding an extra place to look (and search interface, etc) for no extra value just makes it suck more. Plus there's all the TCO reasons - licensing costs, extra servers, extra maintenance, etc etc. You could also look at it as "Don't Repeat Yourself" writ large.
- novok 6y ago
  I've seen a engineering wiki migration done in 3 months at another bigco. Getting documentation off google docs on the other hand seems impossible from a behavior standpoint :(
pjc50 6y ago
Very few organisations really count the cost of things like this internally. And it would have been quoted as "relatively simple" in the beginning; just migrate a wiki from one piece of external software to another, how hard can it be? ("how hard can it be" should be on your Words That Prefigure Expensive Disasters list. It's the business equivalent of "hold my beer and watch this")
twic 6y ago
This week it's patio11's turn to tell us that big companies are big because they need to be big, not because they're inefficient: https://twitter.com/patio11/status/1255371954443505665 https://twitter.com/patio11/status/1255371954443505665 It's always satisfying to read reports from the trenches demonstrating the exact opposite.
- sbarre 6y ago
  Keep in mind stories like the wiki one are anecdotal (meaning it's not representative of all projects), and in fact this is covered in both Patrick's tweet thread and Dan Luu's linked essay ( https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/ ). So while they do their best to not have _every_ initiative go this way, large companies - even the mythical FAANGs - will inevitably have moments like this wiki debacle, and they need to be able to absorb them without material impact on the bottom line or having to fire a bunch of people for trying and failing.
  - twic 6y ago
    Dan and Patrick's accounts of bigness are precisely as anecdotal. I would love to have empirical data about this whole business. I have no idea how you would collect it.
    - EdJiang 6y ago
      Nobody makes fun of Coca-Cola for having 80k employees. "What do they have so many employees for? All they do are come up with combinations of sugar, water, and packaging. They don't even do the bottling!"
      - owenversteeg 6y ago
        If I look at Coca-Cola's jobs page, I see a bunch of jobs that actually do have to exist. A short list, quoting from their page: - Production operator: filling boxes, washing tanks, hook up and unload raw material delivery tankers - Maintenance Technician: Overhaul and install new equipment, maintenance on equipment. - CPS ID Laboratory Chemist: Ensure all raw and packaging materials and finished product comply with Company technical specifications including process water, ensure ingredients have all pertinent documentation before their final release. Meanwhile, you look at Uber's jobs page and you see this (descriptions are quoting from their page) - Business Development: We strengthen Uber’s position as the world’s leading mobility platform through creative and mutually beneficial commercial partnerships that unlock outsized value today and for many years to come. - Data Science & Analytics: We transform data into magical experiences. - Product: We create the vision for the future of urban mobility: deeply understanding our customers to solve their transportation needs with innovative technology. Sure, Coca-Cola probably has some fat that can be trimmed. But at the end of the day you can't fire the guy testing if the coke water is poisonous, you can't do without someone to load boxes, you can't do without the mechanic. Uber, however, can clearly use substantially fewer people creating "creative partnerships" or "transforming magical experiences" or "creating the future of urban mobility". Uber has around 30,000 employees to run a ridesharing business. (Didi, which does twice as many rides per day, has 10k.) Coca-Cola produces 110 billion bottles and 3,900 different beverages around the world. They own their anchor bottler in North America and they produce syrup for pretty much every country in the world! They're so big they used to own Columbia Pictures. I think Coca-Cola is a pretty bad example.
strgcmc 6y ago
There's a roughly ~12,000-word postmortem document that took an additional 2+ months to produce after the fact, which the OP has lightly quoted or summarized from in his post (including what you quoted). It really is an impressively magnificent failure at scale, but there's nothing really fundamentally earth-shattering about how this project went so far off the rails. Mistakes were made, but IMO nobody was outrageously negligent. The OP's assertion that it was primarily due to slavish devotion to InfoSec's insistence of getting off PHP-based MediaWiki, is incomplete and misleading. The team itself opted into the migration, and by their own admission in their own postmortem document, they wanted the opportunity to get off MediaWiki (killing two birds with one stone, both as a technology upgrade and as an InfoSec compliance thing), but they fundamentally vastly underappreciated the scale and complexity of the project (some top line #'s: ~4M total documents, 1.7M unique pages visited daily, 600K unique MAU, etc.). What probably doesn't come through in talking about this, is that the wiki is far from just a simple collection of text-based wiki pages in the standard sense; it serves more like something akin to an unholy abomination of a Wordpress-like platform with endless amounts of plugins and macros and templates, and gives you just enough programmability to be dangerous. The primary reason it took 4+ years, was essentially down to going through multiple rounds of failed migration attempts, including failed attempts at producing automatic translations of wiki's from one platform to another (which proved to be incredibly complex, due to the nature of how customizable MediaWiki was and how many teams had gone and done so many interesting/advanced things with it, which was great... except it made automatic translation/migration near-impossible).
- hamburga 6y ago
  OP here: take another look at the first sentence of that post-mortem doc. I don't think it's misleading to say that this was primarily driven by InfoSec. Though I don't disagree that the purported benefits of XWiki technology were considered as maybe a secondary factor.
  - strgcmc 6y ago
    Fair enough (though executive summaries are by nature, lossy contextually). I don't disagree that InfoSec was a driving force behind the project, and may have given it an unhelpful level of urgency (i.e. pushing for the team to make early decisions/plans too quickly, which anyways resulted in this banned PHP thing lasting 4+ years, seems counterproductive). But even if InfoSec was not a factor, the team knew that staying on MediaWiki was not sustainable, and some kind of migration would've happened eventually. So I guess what I'm saying is, IMO the perceived pressure from the InfoSec mandate was poorly managed by this team, and ended up exacerbating the problem, but fundamentally the choice to migrate to a different wiki platforms was something the team wanted to do (just maybe on a different schedule or with different plans/priorities, in an alternate universe). Migrating off MediaWiki was not done against their will.
afterburner 6y ago
> How was this approved? How did they allow it to continue for four years? I mean, it gave a lot of people something to do (and get paid to do) for 4 years. There's your answer, most of the time.
w0m 6y ago
Re: big company vs small -- Small companies fail all the time; projects like this pare more prevalent at big companies because it doesn't (always at least) kill them.
illumin8 6y ago
That's the famous w.amazon.com internal wiki that is basically the intranet portal for all of Amazon. I worked at AWS from 2016-2019 as a principal engineer. They were trying to deprecate w.amazon.com for the entire 3+ years I worked there, and were still trying when I left in early 2019. I can tell you that Amazon is pretty great in most ways, but they have a lot of really old school tools. For example, they still use majordomo to manage hundreds of thousands of email lists. They've customized and extended it in so many ways that it probably looks nothing like open source majordomo, but at it's core, it's still majordomo. They just wrapped 1990s majordomo CLI tool with a 1990s static HTML page that authenticates you with oauth and lets you create and manage email distribution lists. So many tools at Amazon are like that. But they actually function amazingly well once you adapt to their 1990s->early 2000s quirks. One comment I'll make is this: every environment will have preferred "paved" roads - use X language/toolchain and Y relational database, Z cloud/metal provider. Ultimately these details, other than X and a subset of Y, shouldn't matter if you build the right abstractions. And by "the right abstractions" I mean you should just be able to declare something like this: relational_database: mydb: size: 100GB readers: - auditservice readwriters: - application1 - application2

stu2010 6y ago

Amazon's total all-in cost per dev is almost certainly >$300k per year, so more like >$7 million if that's an accurate number. As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed.

theli0nheart 6y ago
> As far as how it gets approved, that's not enough money to really register at Amazon's scale. I've seen waste similar to that at companies much, much smaller than Amazon that goes entirely un-noticed. Maybe I'm being dense, but this still doesn't register for me. If 4+ calendar years of work for a single team continues with no noticeable progress, then there's an organizational issue, no matter the size of the company.
- WJW 6y ago
  Of course there is, but that's not the point. Every large organisation has organisational issues and nobody in the history of humanity has found a way of preventing them yet. The only thing you can do when the group sizes get large is to try and keep the problem under control relative to how bad it is at other large companies.
  - theli0nheart 6y ago
    I'm lost; what specifically is not the point? As far as I could tell, the "reason" given by the OP for this waste was attributed to Amazon's scale. Could you explain how company size and broken corporate structure are correlated? Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure?
    - timerol 6y ago
      > I'm lost; what specifically is not the point? That there is an organizational issue. Of course there is an organizational issue. There will always be organizational issues. > Or explain why 20+ person-years of engineering time wasted isn't a symptom of broken corporate structure? Oh, it's definitely an organizational issue here, that's just not the main point. If you're willing to call this a "broken corporate structure," then you should revise your opinion of every company with more than 50 employees downward sharply. This is a normal corporate structure, not a broken one. > Could you explain how company size and broken corporate structure are correlated? When there are enough people, you will always end up with conflicting incentives somewhere. A larger organization will have more of these. In this case, it seems to be a clear case of competing priorities: IT wants to secure systems, and developers want to be maximally productive. I'm sure some folks in IT view Amazon paying ~$7M as a fair cost to remove known vulnerable tools from their infrastructure.
    - sbarre 6y ago
      It's unlikely that this went entirely unnoticed for 4 years, and I would guess that when the problem was understood, someone did the math on staying the course, versus starting over or reversing course. And after all the internal calculations were complete (which factor in way more than technical effort by a small dev team), they intentionally stayed the course. This is probably an extreme example because an internal wiki that is used company-wide will have an incredibly complex and hairy stakeholder matrix (both business and technical), which means consensus is very challenging. At some point the cost of the meetings required to work out a solution (both in time spent, and in time not spent on core work by the stakeholders) probably exceeds the cost of just letting a small team brute-force their way through a bad solution. Cost here isn't just money, it's focus, morale, disruption, conflict, risk, etc.. Keep in mind that you have the benefit of hindsight, and the comfort of looking at it from the outside, with a very incomplete picture of all the constraints and pressures at play.
- mywittyname 6y ago
  It's pretty easy to have a project look like progress is being made in the moment, especially when presenting to the director+ level, where a manager is presenting on 4-5 different projects. It's even easier with projects that aren't important or highly-visible (as internal projects tend to be).
nojito 6y ago
>that's not enough money to really register at Amazon's scale. This viewpoint doesn't really match up with how enterprises operate. It's possible to function at amazon's scale without meticulous accounting and financial operations. The reason for this is that is extremely easy for wasteful spending to propagate throughout the enterprise if unchecked. Waste only appears after a project fails...it's easy to point out waste after the fact, but it's downright impossible to call it out as it's happening.
- romanows 6y ago
  I guarantee there were some technical people saying this was a waste of time and money.
  - ThrowawayR2 6y ago
    You can find at least one technical person in a company who will say any particular project is a waste of time and money. It means nothing.

klohto 6y ago

I don't understand why are corporations obsessed with decommissioning wikis. When I was at Red Hat, they tried to decommission local country wiki in favor of their company-wide wiki (Mojo, absolute shit by the way). They tried for about 2 years, then finally somebody said "what the fuck are we doing" and the migration was stopped. I believe the banner "Decommissioning soon" still hangs there.

chii 6y ago
somebody (usually some kind of middle manager) needs to have projects to look after, otherwise it would show that they are dead weights.
MattGaiser 6y ago
Usually, someone saw a cool feature they wanted and everything moved to that. So often the main purpose of storing information gets lost to flashy new graphics.
schnable 6y ago
What happens is that different teams build out their own wikis , or come in via acquisition). The company then ends up with a dozen of them, and re-orgs occur and a single team now has two wikis, and people wonder why the information isn't centralized. Everyone agrees it will be great to have a singe company wide wiki and the project is born.
breischl 6y ago
Finding the right information is already difficult and a time-suck. Adding an extra place to look (and search interface, etc) for no extra value just makes it suck more. Plus there's all the TCO reasons - licensing costs, extra servers, extra maintenance, etc etc. You could also look at it as "Don't Repeat Yourself" writ large.
novok 6y ago
I've seen a engineering wiki migration done in 3 months at another bigco. Getting documentation off google docs on the other hand seems impossible from a behavior standpoint :(

pjc50 6y ago

Very few organisations really count the cost of things like this internally. And it would have been quoted as "relatively simple" in the beginning; just migrate a wiki from one piece of external software to another, how hard can it be? ("how hard can it be" should be on your Words That Prefigure Expensive Disasters list. It's the business equivalent of "hold my beer and watch this")

twic 6y ago

This week it's patio11's turn to tell us that big companies are big because they need to be big, not because they're inefficient: https://twitter.com/patio11/status/1255371954443505665 https://twitter.com/patio11/status/1255371954443505665 It's always satisfying to read reports from the trenches demonstrating the exact opposite.

sbarre 6y ago
Keep in mind stories like the wiki one are anecdotal (meaning it's not representative of all projects), and in fact this is covered in both Patrick's tweet thread and Dan Luu's linked essay ( https://danluu.com/sounds-easy/ https://danluu.com/sounds-easy/ ). So while they do their best to not have _every_ initiative go this way, large companies - even the mythical FAANGs - will inevitably have moments like this wiki debacle, and they need to be able to absorb them without material impact on the bottom line or having to fire a bunch of people for trying and failing.
- twic 6y ago
  Dan and Patrick's accounts of bigness are precisely as anecdotal. I would love to have empirical data about this whole business. I have no idea how you would collect it.
  - EdJiang 6y ago
    Nobody makes fun of Coca-Cola for having 80k employees. "What do they have so many employees for? All they do are come up with combinations of sugar, water, and packaging. They don't even do the bottling!"
    - owenversteeg 6y ago
      If I look at Coca-Cola's jobs page, I see a bunch of jobs that actually do have to exist. A short list, quoting from their page: - Production operator: filling boxes, washing tanks, hook up and unload raw material delivery tankers - Maintenance Technician: Overhaul and install new equipment, maintenance on equipment. - CPS ID Laboratory Chemist: Ensure all raw and packaging materials and finished product comply with Company technical specifications including process water, ensure ingredients have all pertinent documentation before their final release. Meanwhile, you look at Uber's jobs page and you see this (descriptions are quoting from their page) - Business Development: We strengthen Uber’s position as the world’s leading mobility platform through creative and mutually beneficial commercial partnerships that unlock outsized value today and for many years to come. - Data Science & Analytics: We transform data into magical experiences. - Product: We create the vision for the future of urban mobility: deeply understanding our customers to solve their transportation needs with innovative technology. Sure, Coca-Cola probably has some fat that can be trimmed. But at the end of the day you can't fire the guy testing if the coke water is poisonous, you can't do without someone to load boxes, you can't do without the mechanic. Uber, however, can clearly use substantially fewer people creating "creative partnerships" or "transforming magical experiences" or "creating the future of urban mobility". Uber has around 30,000 employees to run a ridesharing business. (Didi, which does twice as many rides per day, has 10k.) Coca-Cola produces 110 billion bottles and 3,900 different beverages around the world. They own their anchor bottler in North America and they produce syrup for pretty much every country in the world! They're so big they used to own Columbia Pictures. I think Coca-Cola is a pretty bad example.

strgcmc 6y ago

There's a roughly ~12,000-word postmortem document that took an additional 2+ months to produce after the fact, which the OP has lightly quoted or summarized from in his post (including what you quoted). It really is an impressively magnificent failure at scale, but there's nothing really fundamentally earth-shattering about how this project went so far off the rails. Mistakes were made, but IMO nobody was outrageously negligent. The OP's assertion that it was primarily due to slavish devotion to InfoSec's insistence of getting off PHP-based MediaWiki, is incomplete and misleading. The team itself opted into the migration, and by their own admission in their own postmortem document, they wanted the opportunity to get off MediaWiki (killing two birds with one stone, both as a technology upgrade and as an InfoSec compliance thing), but they fundamentally vastly underappreciated the scale and complexity of the project (some top line #'s: ~4M total documents, 1.7M unique pages visited daily, 600K unique MAU, etc.). What probably doesn't come through in talking about this, is that the wiki is far from just a simple collection of text-based wiki pages in the standard sense; it serves more like something akin to an unholy abomination of a Wordpress-like platform with endless amounts of plugins and macros and templates, and gives you just enough programmability to be dangerous. The primary reason it took 4+ years, was essentially down to going through multiple rounds of failed migration attempts, including failed attempts at producing automatic translations of wiki's from one platform to another (which proved to be incredibly complex, due to the nature of how customizable MediaWiki was and how many teams had gone and done so many interesting/advanced things with it, which was great... except it made automatic translation/migration near-impossible).

hamburga 6y ago
OP here: take another look at the first sentence of that post-mortem doc. I don't think it's misleading to say that this was primarily driven by InfoSec. Though I don't disagree that the purported benefits of XWiki technology were considered as maybe a secondary factor.
- strgcmc 6y ago
  Fair enough (though executive summaries are by nature, lossy contextually). I don't disagree that InfoSec was a driving force behind the project, and may have given it an unhelpful level of urgency (i.e. pushing for the team to make early decisions/plans too quickly, which anyways resulted in this banned PHP thing lasting 4+ years, seems counterproductive). But even if InfoSec was not a factor, the team knew that staying on MediaWiki was not sustainable, and some kind of migration would've happened eventually. So I guess what I'm saying is, IMO the perceived pressure from the InfoSec mandate was poorly managed by this team, and ended up exacerbating the problem, but fundamentally the choice to migrate to a different wiki platforms was something the team wanted to do (just maybe on a different schedule or with different plans/priorities, in an alternate universe). Migrating off MediaWiki was not done against their will.

afterburner 6y ago

> How was this approved? How did they allow it to continue for four years? I mean, it gave a lot of people something to do (and get paid to do) for 4 years. There's your answer, most of the time.

w0m 6y ago

Re: big company vs small -- Small companies fail all the time; projects like this pare more prevalent at big companies because it doesn't (always at least) kill them.

illumin8 6y ago

That's the famous w.amazon.com internal wiki that is basically the intranet portal for all of Amazon. I worked at AWS from 2016-2019 as a principal engineer. They were trying to deprecate w.amazon.com for the entire 3+ years I worked there, and were still trying when I left in early 2019. I can tell you that Amazon is pretty great in most ways, but they have a lot of really old school tools. For example, they still use majordomo to manage hundreds of thousands of email lists. They've customized and extended it in so many ways that it probably looks nothing like open source majordomo, but at it's core, it's still majordomo. They just wrapped 1990s majordomo CLI tool with a 1990s static HTML page that authenticates you with oauth and lets you create and manage email distribution lists. So many tools at Amazon are like that. But they actually function amazingly well once you adapt to their 1990s->early 2000s quirks. One comment I'll make is this: every environment will have preferred "paved" roads - use X language/toolchain and Y relational database, Z cloud/metal provider. Ultimately these details, other than X and a subset of Y, shouldn't matter if you build the right abstractions. And by "the right abstractions" I mean you should just be able to declare something like this: relational_database: mydb: size: 100GB readers: - auditservice readwriters: - application1 - application2