4 ms·

I wonder if anybody's had similar experiences... I've worked at two companies with RFC processes. At both companies, the processes were -- and I hate to use su

I wonder if anybody's had similar experiences... I've worked at two companies with RFC processes. At both companies, the processes were -- and I hate to use such a strong word -- a total sham. I suppose the processes were crafted to look like some egalitarian meritocracy. In reality, it was just a test of who had the most political pull. If you had management on your side, your "RFC" was effectively law and dissenting voices were effectively career suicide. Which, you know... fine. I understand that choices are made based on cliques and political capital rather than anything else. Cool. That's okay! Sucks sometimes, but that's how the world works. Hopefully your org structure itself is at least something of a meritocracy, so that edicts from on high are of a generally high-enough quality. If they get it right-ish often enough, your org will be okay... probably. But what really rankled me was the fact that the RFC processes amounted to some kind of elaborate cosplay so that, I guess, folks could pretend that there was some sort of healthy collaborative process. If you're going to let a few "popular and management-blessed" engineers make all the decisions, fine, but don't add insult to injury by pretending otherwise. I've heard similar things from others, elsewhere. I hope there are at least a few companies dedicated to being something better.

eugenhotaj 4y ago
In my experience processes like these rarely work out as intended and usually add layers of bureaucracy for marginal benefit. It’s usually senior engineers or middle managers looking for “org wide impact” so they can get promoted.
- treis 4y ago
  At [current job] it's run by a group of architects that aren't responsible for delivering anything. Which works about as well as you might imagine. Had one legit tell me something like "that's for you to figure out" when I pointed out some implementation challenges. Of course now that the project is behind there's no architect in sight to help us out.
  - jugg1es 4y ago
    I recently joined a group of architects that aren't responsible for delivering anything and you are not totally wrong. But it isn't the architect role to involve themselves in weeds of the implementation - that's your job. They should be helping if the project is falling behind, though. Then again, if the project is falling behind, then that is probably more indicative of a poorly-run product team and/or a bad SDLC process.
    - hgsgm 4y ago
      If the architect is planting weeds, the architect should figure out how to kill the weeds. Ignorance of programming is omni excuse.
      - jugg1es 4y ago
        My point was that the architect shouldn't plant the weeds in the first place.
    - treis 4y ago
      I just hope one day I am good enough at programming to draw some boxes, tell someone else to figure it out, and then blame them when it doesn't work out
      - humanrebar 4y ago
        It's the job of all engineers involved to utterly understand one another. That goes for junior engineers cranking out unit tests as well as for architecture astronauts. If someone here is a junior and gets "figure it out" from some strategic level engineer, ask professional for clarification or a team-up until you understand. Involve management if the other engineer is uncooperative or fails to help. It's their job to explain slightly more than it's your job to understand. If all else fails, find an organization that expects the architecture astronauts to be explainers in chief as well.
    - JohnBooty 4y ago
      Then again, if the project is falling behind, then that is probably more indicative of a poorly-run product team and/or a bad SDLC process. That's a worrisomely glib take on the impact that architecture can have on the project especially since in practice "architecture" includes the choice of language, platform, and tooling for a project. Architecture choices directly or indirectly affect every line of code, every minute, every meeting, everything. As you said there are plenty of other places where things can go bad. But it's galling to hear an architect say, "Probably not the architecture, must be something else." a group of architects that aren't responsible for delivering anything And here's the root of the problem. The craft of delivering software is, well, delivering. The code itself is often somewhat trivial. Effective delivery is all about working about quirks and potholes and traps in the toolchain and architecture inefficiencies. As we all know software moves fast, and once you've divorced yourself from the reality of actually delivering anything at all, your knowledge quickly decays. You know abstract concepts and how to draw the rectangles on the whiteboard, but before very long you don't know how anything at all actually happens and soon you are unqualified to tell anybody else how to do it. There can be exceptions to this rule, like projects with an extraordinarily stable toolchain and environment. If you've been writing COBOL at a bank for 40 years you can probably step back into an advisory role and benefit the company by getting yourself out of the trenches and guiding others instead.
miketery 4y ago
I think the most important part of these processes is to document what’s being proposed and have a period for comments / inputs. I also think it’s ok for a monarchy as opposed to democracy ruling what gets approved, assuming merit based is not an option. In general merits (rationality?) can bubble up if it’s framed in correct way for upper levels (eg. risk, profit, time horizon), and can then sway the position. But yes on average what’s decide at upper levels is gospel. But that’s how corporations function, and that’s ok.
- JohnBooty 4y ago
  I also think it’s ok for a monarchy as opposed to democracy ruling what gets approved, assuming merit based is not an option. I completely agree. What I object to is the charade! In general merits (rationality?) can bubble up if it’s framed in correct way for upper levels Theoretically, yes, but it's trivially easy to fool upper levels with one-sided technical opinions. Choose the name of any technology or tech-related practice at random, pay me $20, and I'll write you a one-sided RFC about why our imaginary company should adopt it. I'll make it sound really good. Your boss will love it and if they don't, I promise their boss will.
  - Hermitian909 4y ago
    > Choose the name of any technology or tech-related practice at random, pay me $20, and I'll write you a one-sided RFC about why our imaginary company should adopt it. I'll make it sound really good. Your boss will love it and if they don't, I promise their boss will. Ultimately, business folks must always trust someone with technical expertise, trust is a necessary component of the system. Competent organizations ensure that someone technical has the relevant executive's ear, there are lots of ways to achieve this and it should be able to catch your one-sided RFC if you're proposing anything important. At the end of the day though, there's no process-based solution for an organization riddled with incompetence. Business must make judgment calls and if your company is staffed with people who consistently make bad calls process will not save you.
- t0mas88 4y ago
  > I also think it’s ok for a monarchy as opposed to democracy ruling what gets approved, assuming merit based is not an option. Just to share: I've been an outside advisor to a few companies where some developers may think "this is a politics based monarchy and we're not looking at proposals rationally". Where as an outsider it was clear to me that the higher up roles just had more context to make their decisions. Especially decisions coming from the CTO and VP Engineering kind of roles tended to be driven by input from the wider company strategy, due to daily interaction with the rest of the management team. In those cases there was a clear need for coaching the higher ups on how to explain and communicate their decisions, but what they were deciding were the right things. And everybody leaving before they improved communications would be on here posting about how bad their higher ups were and what a political mess it was. While that wasn't really the issue. I'm sure that's not always the case and big political dumpster fires exist (I've seen a few), but there are also many cases where it's only an issue with communicating more clearly from the (technical) management side of things.
Scubabear68 4y ago
Yep. Things like architectural review boards, RFC processes, corporate-wide approvals, etc are more about politics than anything else. Often - not always, but often - the people seeking to sit on these committees are the least likely to be able to actually build software. It is a function of these kinds of boards that will invariably draw those who aren’t so great at coding or architecture. A bit like being a film critic or book reviewer.
oscillonoscope 4y ago
That's the intended purpose and origination of these systems. In a traditional hardware firm, marketing is constantly coming up with ideas but you can't constantly change hardware without delaying the project. So you come up with an engineering change request system. This forces marketing to go through a bit of bureaucratic pain every time they want to add something to the project. The committee acts as a gatekeeper but mostly, that little bit of extra work makes marketing self-select the more important ideas to go through the process.
deleted 4y ago
[deleted]
raggi 4y ago
No collaboration process will remove people from the system. You should never expect it to, if you have problems with social power, that's a culture issue which isn't something you can control directly. You can discuss culture issues as a group and nudge toward changes. Process won't define culture - it might only nudge it, but it's very easy for established culture, especially that on the side of social power, to completely ignore the influences intended by process. What I read from Squares description in the article is a lot of power uncertainty, which is sadly all too common in the software industry. What comes through repeatedly in the problem description is any uncertainty defaulting to no communication. This problem won't be local to the RFC process, and ideally should be tackled more broadly while also looking to improve it in processes. Communication needs to be rewarded even if the technical content is inaccurate - more people need to speak up more, and with less concern, and receivers need to approach the commentary with a positive attitude to facilitate both not feeling pressure and to encourage ongoing communication. The approvers list is for example doing this, though for a privileged few. What will likely come to light later is that the approvers portion of the process is causing an asymmetry of inclusion, further feeding a lack of communication from other parts of the teams. An immediate concrete thing I would suggest that builds atop the existing described structure is to invite rotations for folks to join the "council" and review processes regularly but temporarily. This will increase inclusion and dilute exclusivity and prestige. Explicitly open space for participation and solicit input from visiting members, and again it is important to reward communication even if it did not have the desired content.
avgcorrection 4y ago
> egalitarian meritocracy Oxymoron.
- JohnBooty 4y ago
  Good catch, thanks. I was (incorrectly) using egalitarian as a synonym for "fair." Brain fart.
throwawaaarrgh 4y ago
> In reality, it was just a test of who had the most political pull. Every society works that way. When a group has to accept a decision, the most powerful person in the group decides. I don't understand why any educated person today would think meritocracy is a good thing. A child looking at Wikipedia for 10 minutes could see all the flaws in it, yet it keeps being touted as some idealized model of social contract. In reality it keeps white straight rich men in charge.
draw_down 4y ago
Amen. Really silly things happen too, like you go through all this process and then months later some hotshot asks “wait, why are we doing this?” when launch is imminent. Like we all agreed this was worth doing all along, right up to this very moment, and now we’re not sure all of a sudden. All the approvals, all the documents written and decisions we went through were apparently not compelling to this person. We could have had this discussion and perhaps killed the idea in the crib, now we’re having this discussion about something we’re about to launch. In other words the processes are a fig leaf, which as you say is fine, but then it doesn’t even do the job of the leaf.
dimal 4y ago
I worked at one company where (at least within our team of about 40) our process was legitimately good. There was a genuine back and forth discussion and good ideas tended to float to the top. Then we got acquired and things steadily changed. Slowly more and more decisions were made because some high level engineer or manager just wanted it that way. Of course, terrible ideas floated to the top and it lead to disastrous decisions. It is possible for the system to work, but it’s a fragile equilibrium and easily destroyed.
- alistairw 4y ago
  What made the process work so well? What about the culture made it so there was that back and forth? I'm wanting to build this into my teams and keen to hear what's worked for others
  - dimal 4y ago
    Empathy. Our team already had a great culture of empathy and mutual respect. People just tried to see things from other's POV, or if they couldn't, they'd at least be respectful. To behave differently would have just seemed weird. I think that our manager and the founding members of the team set the tone early, and as the team grew, it just became the norm. When we lost that manager, things declined. The worst part is that when I look back at my own behavior, I lost my empathy. I felt like a rat in a cage and I behaved accordingly. If you already have that kind of empathic culture, then I think it's just about starting a "good enough" process and refining it as you go. If you don't, then you have to build empathy into the team. I found the book Practical Empathy to be helpful. But basically, if you treat someone with empathy, they can't help but start to feel empathy for you. I think we're wired that way. If your team doesn't have a high level of empathy, be the first.
photonbeam 4y ago
You described my 2018-era experience in Uber’s infra org
potamic 4y ago
Not sure if you have read the post fully, but they seemed to have created this architecture review meeting precisely to overcome the problems you are talking about. The orgranization wide meeting is a way to make sure everyone sees the RFC, while the architecture meeting is an in-depth review with a small group of relevant people. Many people would agree the latter is needed in an organization beyond a certain scale, but I think the former is extremely important and often over looked. It provides a channel for people to be aware of what's happening across teams and build organizational context. It's also a forum for people to express themselves on anything they might have an opinion on and help establish a sort of the zeitgeist for the group. I've seen the alternative where you work within tight silos disconnected from the rest of the org, and it gets pretty dreary.
mirekrusin 4y ago
I find those prejudgement political games damaging as well to the point that I'm fantasizing about absurd ideas like anonymous version of slack/hn like chat where messages don't have identities to push it slightly towards merit based conversations. Upvote on messages/threads, not user accounts etc. I don't mean anonymous usernames, I mean no identity at all or at most random identities per message, sticky on reply chain (to know the original author answered your question at most).

eugenhotaj 4y ago

In my experience processes like these rarely work out as intended and usually add layers of bureaucracy for marginal benefit. It’s usually senior engineers or middle managers looking for “org wide impact” so they can get promoted.

treis 4y ago
At [current job] it's run by a group of architects that aren't responsible for delivering anything. Which works about as well as you might imagine. Had one legit tell me something like "that's for you to figure out" when I pointed out some implementation challenges. Of course now that the project is behind there's no architect in sight to help us out.
- jugg1es 4y ago
  I recently joined a group of architects that aren't responsible for delivering anything and you are not totally wrong. But it isn't the architect role to involve themselves in weeds of the implementation - that's your job. They should be helping if the project is falling behind, though. Then again, if the project is falling behind, then that is probably more indicative of a poorly-run product team and/or a bad SDLC process.
  - hgsgm 4y ago
    If the architect is planting weeds, the architect should figure out how to kill the weeds. Ignorance of programming is omni excuse.
    - jugg1es 4y ago
      My point was that the architect shouldn't plant the weeds in the first place.
  - treis 4y ago
    I just hope one day I am good enough at programming to draw some boxes, tell someone else to figure it out, and then blame them when it doesn't work out
    - humanrebar 4y ago
      It's the job of all engineers involved to utterly understand one another. That goes for junior engineers cranking out unit tests as well as for architecture astronauts. If someone here is a junior and gets "figure it out" from some strategic level engineer, ask professional for clarification or a team-up until you understand. Involve management if the other engineer is uncooperative or fails to help. It's their job to explain slightly more than it's your job to understand. If all else fails, find an organization that expects the architecture astronauts to be explainers in chief as well.
  - JohnBooty 4y ago
    Then again, if the project is falling behind, then that is probably more indicative of a poorly-run product team and/or a bad SDLC process. That's a worrisomely glib take on the impact that architecture can have on the project especially since in practice "architecture" includes the choice of language, platform, and tooling for a project. Architecture choices directly or indirectly affect every line of code, every minute, every meeting, everything. As you said there are plenty of other places where things can go bad. But it's galling to hear an architect say, "Probably not the architecture, must be something else." a group of architects that aren't responsible for delivering anything And here's the root of the problem. The craft of delivering software is, well, delivering. The code itself is often somewhat trivial. Effective delivery is all about working about quirks and potholes and traps in the toolchain and architecture inefficiencies. As we all know software moves fast, and once you've divorced yourself from the reality of actually delivering anything at all, your knowledge quickly decays. You know abstract concepts and how to draw the rectangles on the whiteboard, but before very long you don't know how anything at all actually happens and soon you are unqualified to tell anybody else how to do it. There can be exceptions to this rule, like projects with an extraordinarily stable toolchain and environment. If you've been writing COBOL at a bank for 40 years you can probably step back into an advisory role and benefit the company by getting yourself out of the trenches and guiding others instead.

miketery 4y ago

I think the most important part of these processes is to document what’s being proposed and have a period for comments / inputs. I also think it’s ok for a monarchy as opposed to democracy ruling what gets approved, assuming merit based is not an option. In general merits (rationality?) can bubble up if it’s framed in correct way for upper levels (eg. risk, profit, time horizon), and can then sway the position. But yes on average what’s decide at upper levels is gospel. But that’s how corporations function, and that’s ok.

JohnBooty 4y ago
I also think it’s ok for a monarchy as opposed to democracy ruling what gets approved, assuming merit based is not an option. I completely agree. What I object to is the charade! In general merits (rationality?) can bubble up if it’s framed in correct way for upper levels Theoretically, yes, but it's trivially easy to fool upper levels with one-sided technical opinions. Choose the name of any technology or tech-related practice at random, pay me $20, and I'll write you a one-sided RFC about why our imaginary company should adopt it. I'll make it sound really good. Your boss will love it and if they don't, I promise their boss will.
- Hermitian909 4y ago
  > Choose the name of any technology or tech-related practice at random, pay me $20, and I'll write you a one-sided RFC about why our imaginary company should adopt it. I'll make it sound really good. Your boss will love it and if they don't, I promise their boss will. Ultimately, business folks must always trust someone with technical expertise, trust is a necessary component of the system. Competent organizations ensure that someone technical has the relevant executive's ear, there are lots of ways to achieve this and it should be able to catch your one-sided RFC if you're proposing anything important. At the end of the day though, there's no process-based solution for an organization riddled with incompetence. Business must make judgment calls and if your company is staffed with people who consistently make bad calls process will not save you.
t0mas88 4y ago
> I also think it’s ok for a monarchy as opposed to democracy ruling what gets approved, assuming merit based is not an option. Just to share: I've been an outside advisor to a few companies where some developers may think "this is a politics based monarchy and we're not looking at proposals rationally". Where as an outsider it was clear to me that the higher up roles just had more context to make their decisions. Especially decisions coming from the CTO and VP Engineering kind of roles tended to be driven by input from the wider company strategy, due to daily interaction with the rest of the management team. In those cases there was a clear need for coaching the higher ups on how to explain and communicate their decisions, but what they were deciding were the right things. And everybody leaving before they improved communications would be on here posting about how bad their higher ups were and what a political mess it was. While that wasn't really the issue. I'm sure that's not always the case and big political dumpster fires exist (I've seen a few), but there are also many cases where it's only an issue with communicating more clearly from the (technical) management side of things.

Scubabear68 4y ago

Yep. Things like architectural review boards, RFC processes, corporate-wide approvals, etc are more about politics than anything else. Often - not always, but often - the people seeking to sit on these committees are the least likely to be able to actually build software. It is a function of these kinds of boards that will invariably draw those who aren’t so great at coding or architecture. A bit like being a film critic or book reviewer.

oscillonoscope 4y ago

That's the intended purpose and origination of these systems. In a traditional hardware firm, marketing is constantly coming up with ideas but you can't constantly change hardware without delaying the project. So you come up with an engineering change request system. This forces marketing to go through a bit of bureaucratic pain every time they want to add something to the project. The committee acts as a gatekeeper but mostly, that little bit of extra work makes marketing self-select the more important ideas to go through the process.

deleted 4y ago

[deleted]

raggi 4y ago

No collaboration process will remove people from the system. You should never expect it to, if you have problems with social power, that's a culture issue which isn't something you can control directly. You can discuss culture issues as a group and nudge toward changes. Process won't define culture - it might only nudge it, but it's very easy for established culture, especially that on the side of social power, to completely ignore the influences intended by process. What I read from Squares description in the article is a lot of power uncertainty, which is sadly all too common in the software industry. What comes through repeatedly in the problem description is any uncertainty defaulting to no communication. This problem won't be local to the RFC process, and ideally should be tackled more broadly while also looking to improve it in processes. Communication needs to be rewarded even if the technical content is inaccurate - more people need to speak up more, and with less concern, and receivers need to approach the commentary with a positive attitude to facilitate both not feeling pressure and to encourage ongoing communication. The approvers list is for example doing this, though for a privileged few. What will likely come to light later is that the approvers portion of the process is causing an asymmetry of inclusion, further feeding a lack of communication from other parts of the teams. An immediate concrete thing I would suggest that builds atop the existing described structure is to invite rotations for folks to join the "council" and review processes regularly but temporarily. This will increase inclusion and dilute exclusivity and prestige. Explicitly open space for participation and solicit input from visiting members, and again it is important to reward communication even if it did not have the desired content.

avgcorrection 4y ago

> egalitarian meritocracy Oxymoron.

JohnBooty 4y ago
Good catch, thanks. I was (incorrectly) using egalitarian as a synonym for "fair." Brain fart.

throwawaaarrgh 4y ago

> In reality, it was just a test of who had the most political pull. Every society works that way. When a group has to accept a decision, the most powerful person in the group decides. I don't understand why any educated person today would think meritocracy is a good thing. A child looking at Wikipedia for 10 minutes could see all the flaws in it, yet it keeps being touted as some idealized model of social contract. In reality it keeps white straight rich men in charge.

draw_down 4y ago

Amen. Really silly things happen too, like you go through all this process and then months later some hotshot asks “wait, why are we doing this?” when launch is imminent. Like we all agreed this was worth doing all along, right up to this very moment, and now we’re not sure all of a sudden. All the approvals, all the documents written and decisions we went through were apparently not compelling to this person. We could have had this discussion and perhaps killed the idea in the crib, now we’re having this discussion about something we’re about to launch. In other words the processes are a fig leaf, which as you say is fine, but then it doesn’t even do the job of the leaf.

dimal 4y ago

I worked at one company where (at least within our team of about 40) our process was legitimately good. There was a genuine back and forth discussion and good ideas tended to float to the top. Then we got acquired and things steadily changed. Slowly more and more decisions were made because some high level engineer or manager just wanted it that way. Of course, terrible ideas floated to the top and it lead to disastrous decisions. It is possible for the system to work, but it’s a fragile equilibrium and easily destroyed.

alistairw 4y ago
What made the process work so well? What about the culture made it so there was that back and forth? I'm wanting to build this into my teams and keen to hear what's worked for others
- dimal 4y ago
  Empathy. Our team already had a great culture of empathy and mutual respect. People just tried to see things from other's POV, or if they couldn't, they'd at least be respectful. To behave differently would have just seemed weird. I think that our manager and the founding members of the team set the tone early, and as the team grew, it just became the norm. When we lost that manager, things declined. The worst part is that when I look back at my own behavior, I lost my empathy. I felt like a rat in a cage and I behaved accordingly. If you already have that kind of empathic culture, then I think it's just about starting a "good enough" process and refining it as you go. If you don't, then you have to build empathy into the team. I found the book Practical Empathy to be helpful. But basically, if you treat someone with empathy, they can't help but start to feel empathy for you. I think we're wired that way. If your team doesn't have a high level of empathy, be the first.

photonbeam 4y ago

You described my 2018-era experience in Uber’s infra org

potamic 4y ago

Not sure if you have read the post fully, but they seemed to have created this architecture review meeting precisely to overcome the problems you are talking about. The orgranization wide meeting is a way to make sure everyone sees the RFC, while the architecture meeting is an in-depth review with a small group of relevant people. Many people would agree the latter is needed in an organization beyond a certain scale, but I think the former is extremely important and often over looked. It provides a channel for people to be aware of what's happening across teams and build organizational context. It's also a forum for people to express themselves on anything they might have an opinion on and help establish a sort of the zeitgeist for the group. I've seen the alternative where you work within tight silos disconnected from the rest of the org, and it gets pretty dreary.

mirekrusin 4y ago

I find those prejudgement political games damaging as well to the point that I'm fantasizing about absurd ideas like anonymous version of slack/hn like chat where messages don't have identities to push it slightly towards merit based conversations. Upvote on messages/threads, not user accounts etc. I don't mean anonymous usernames, I mean no identity at all or at most random identities per message, sticky on reply chain (to know the original author answered your question at most).