4 ms·

As someone who spent 8 years inside Red Hat, this is exactly why anecdotal "evidence" is garbage. Complaining that an engineering company is full of mostly whi

As someone who spent 8 years inside Red Hat, this is exactly why anecdotal "evidence" is garbage. Complaining that an engineering company is full of mostly white males (in an industry which is full of mostly white mails) is an example of nothing. Red Hat started programs to push LGBTQ+ people in 2014, women in leadership in 2017, neurodiversity acceptance in 2018, black leadership in 2018. You may think all of those things are late, but as driven company missions which they spend real money on (and real money on community involvement on), it's not insignificant. A "good-old-boy's club" at a company where only 25% of engineering was even in the US is a little mindblowing. The vast majority of RH's engineers when I left were in Pune, Brno, Tel Aviv, and scattered across Europe (eastern and western), with a reasonable showing from Brazil. Open source is, for better or worse, a place where a cult of personality matters. "This is a bad idea and we shouldn't do it" isn't personal. It's how the community interacts. It's an invitation to argue back and say "this is a good idea because XXX". This is basically how any upstream mailing list works also. While Red Hat did have some cultural problems, they were primarily around resistance to establishing an engineering presence in China (and it's hard to fault them for that -- my team was truly geographically distributed, and adding someone in APAC would have meant we literally never could have met at the same time), and regional cultural attitudes. I left last summer, along with a lot of other people, but your comment is apropos of nothing Red Hat was from 2013-2021.

wolverine876 5y ago
> garbage For sensitive issues, it's hard to imagine this will lead to a constructive, curious conversation. It would be interesting, for example, if you and the GP tried to examine why you have such different experiences? Maybe you were in different places, working for different people, and saw different things? Maybe different job descriptions? The world is always much bigger than what we imagine.
- evol262 5y ago
  This is not a "sensitive" issue, unless you consider GP's opinion "sensitive". I don't. I also don't think it would lead to a curious, constructive conversation. There is probably nothing OP could offer which would change what the experience of 8 years in RH engineering showed me. At best, it would lead to another long reply where I tried to explain what RH was actually like versus GP's impression. It's essentially GP posting on the Debian/OpenBSD/LK ML and saying "wow, these guys are arrogant". Well, they have strong opinions, and they need to justify them. If you don't have thick skin, you should probably avoid upstream discussions. It doesn't mean you should cast aspersions on the developers or their communities/companies. Linus is an incredibly nice person, but you wouldn't think it if all you saw was his mails blasting Intel for Spectre/Meltdown. You'd have a very different opinion of the man than who he is in 99% of interactions, and commenting about it on HN would be likely to get a reply similar to mine. GP only had "anecdotal" experience with Red Hat versus actually working inside the company. When I started, I was worried that it would be different "behind the curtain". It was not. The other who mentioned junior engineers speaking up if they disagreed was spot on. In 201x (2015? 2017? I dunno), RH changed their pay structure from being bimonthly to biweekly in the US to align with the rest of the world. There were hundreds of replies to the entire company debating whether it was better to pay us once a month and manage your money better, or once a week to be more consistent, or... That's what the company was. It was not by the time I left. Granted, when I started, RH HQ was still on a college campus. If all you ever read was upstream mailing lists, you'd get the feeling that everyone (Linux, Theo [not that he works for RH, obviously], Lennart, whomever) other than Dan Walsh was arrogant assholes. That's not representative of who they actually are. Painting Red Hat with a broad brush from incidental interactions versus the perspective of someone who spent nearly a decade there is dramatically different. Did it have problems? Sure. But none of the ones GP mentioned. The world is much bigger than what GP imagined from his anecdotal experience. RH was a unicorn, and it's gone. LinkedIn last summer was essentially rats fleeing a sinking ship. IBM spent $35B on a company with no actual assets (a few patents) other than smart, passionate people. They would have had to really try to screw it up. And they did. I don't honestly know if there will be another RH in the future, but I'm hoping there will be.
  - zxzax 5y ago
    I agree with the rest of your comment but I just want to respond to this part: >If you don't have thick skin, you should probably avoid upstream discussions I have a humble request, please discontinue this attitude on your own projects and please encourage other upstreams to also discontinue it. I'm baffled as to why so many open source maintainers seem to confuse needing to defend technical decisions with "having a thick skin," they are not the same. It's perfectly possible to be opinionated and strongly scrutinize a technical decision, while also not being harsh and rude towards the person presenting it. If you believe those mailing lists are unsalvageably hostile, then in my opinion, they should just be shut down. That's not the kind of place to have official project communication.
  - pabs3 5y ago
    Where are the people fleeing RH going to go? There are not many workplaces as large and as Free Software focussed as RH.
    - yourapostasy 5y ago
      Generally most M&A's like this turn into a talent smasher event (think atom smasher, aka particle accelerator), and the people scatter to the four winds in hundreds/thousands of different landing zones instead of coalescing into another convergent organization. Emergent consensus centered around such a large organization takes a lot of effort and time to build because it operates through social trust pathways (that predominantly operate in the time domain), and once fissioned, the byproducts do not transitively transfer that trust, it has to be built anew. This adds to my suspicions that the long-term IBM C-suite goal of the Red Hat acquisition is to eventually convert Red Hat to be like their captive mainframe revenue and profit stream. I would not at all be surprised to see a gradual deterioration in Red Hat quality in the coming 10 years. Watch the general level of support engineering skills in what will eventually be a Blue-washed Red Hat L2 Support. That is your canary. What you are used to today in that tier will be gradually relegated to the tech leads for L2. Don't look for tech leads by title; look for the leads who gates decisions to escalate to L3. Highly-skilled L2 organizations grant most if not all their support engineers autonomy to escalate to L3. Profit-squeezing L2 organizations (what I suspect may be happening with Red Hat in the coming years) will gate those escalations, and staff vacancies with relatively down-skilled engineers. Note that I also enjoy working with these relatively down-skilled engineers. No complaints about "bedside manners", and they can address common scenarios. Given enough time and exposure to more experienced staff, they would absolutely step up to highly-skilled L2 levels. The challenge is often these situations see those more experienced staff leave too soon. In my experience you need at least 7, ideally 14 years to season new staff to those skill levels, and the attrition rate is high over that time; troubleshooting well under pressure with grace is a very difficult combination of skills and characteristics to find. Two ways I've seen to effectively deal with this on the customer end is find an alternative solution (not really feasible in this case at this time, though I'm watching several possible developments), or up-skill your staff. You'll know you've reached the right skill level when your engineers can delve into the internals of the product sufficiently that they routinely convince the down-skill L2 support engineers to escalate their lead within a day or two of engaging. If my suspicions bear fruit (I really hope not, the vast majority of my clients hold massive people and capital investments in the Red Hat ecosystem), then this is extremely bullish for AWS. Their customer-obsession-to-a-fault culture would swiftly exploit IBM's strategy, and there would be no contest. I can see several simultaneous attacks AWS could mount to neutralize Red Hat's position if IBM's strategy at the end of the day really is what I currently suspect. I'd really like to hear from others that have found other workarounds for this eventuality.
notacoward 5y ago
Take that appeal to authority elsewhere, and the insults with it. As someone who was also at Red Hat for eight years (2009-2017) I'd say GP might have been off base but certainly not garbage. I heard more grossly sexist comments (or saw them on IRC) there than anywhere else. We're talking stuff that would lead to immediate mandatory diversity training the first time, or instant dismissal the second, at many companies (including IBM). "Who did she have to blow to get that job" particularly sticks in my mind. Most people didn't even bat an eye because that kind of stuff is totally allowable in the open-source communities these folks were part of. I also saw at least one person fired for refusal to work with people he considered "beneath" him socially. You might say it's good that the company took action, but it took over a year for that to happen and I don't think that would have been the case at a truly inclusive company. I suppose you can dismiss these (and plenty more I could relate) as mere anecdotes, but you haven't even provided those so that would be pretty hypocritical. The vast majority of people I knew at Red Hat ranged from industry-average to absolutely excellent, but overall I'd say even Facebook had a better internal culture in terms of diversity etc. Painting RH as anything more than a typical tech company in those areas is a bit absurd.

wolverine876 5y ago

> garbage For sensitive issues, it's hard to imagine this will lead to a constructive, curious conversation. It would be interesting, for example, if you and the GP tried to examine why you have such different experiences? Maybe you were in different places, working for different people, and saw different things? Maybe different job descriptions? The world is always much bigger than what we imagine.

evol262 5y ago
This is not a "sensitive" issue, unless you consider GP's opinion "sensitive". I don't. I also don't think it would lead to a curious, constructive conversation. There is probably nothing OP could offer which would change what the experience of 8 years in RH engineering showed me. At best, it would lead to another long reply where I tried to explain what RH was actually like versus GP's impression. It's essentially GP posting on the Debian/OpenBSD/LK ML and saying "wow, these guys are arrogant". Well, they have strong opinions, and they need to justify them. If you don't have thick skin, you should probably avoid upstream discussions. It doesn't mean you should cast aspersions on the developers or their communities/companies. Linus is an incredibly nice person, but you wouldn't think it if all you saw was his mails blasting Intel for Spectre/Meltdown. You'd have a very different opinion of the man than who he is in 99% of interactions, and commenting about it on HN would be likely to get a reply similar to mine. GP only had "anecdotal" experience with Red Hat versus actually working inside the company. When I started, I was worried that it would be different "behind the curtain". It was not. The other who mentioned junior engineers speaking up if they disagreed was spot on. In 201x (2015? 2017? I dunno), RH changed their pay structure from being bimonthly to biweekly in the US to align with the rest of the world. There were hundreds of replies to the entire company debating whether it was better to pay us once a month and manage your money better, or once a week to be more consistent, or... That's what the company was. It was not by the time I left. Granted, when I started, RH HQ was still on a college campus. If all you ever read was upstream mailing lists, you'd get the feeling that everyone (Linux, Theo [not that he works for RH, obviously], Lennart, whomever) other than Dan Walsh was arrogant assholes. That's not representative of who they actually are. Painting Red Hat with a broad brush from incidental interactions versus the perspective of someone who spent nearly a decade there is dramatically different. Did it have problems? Sure. But none of the ones GP mentioned. The world is much bigger than what GP imagined from his anecdotal experience. RH was a unicorn, and it's gone. LinkedIn last summer was essentially rats fleeing a sinking ship. IBM spent $35B on a company with no actual assets (a few patents) other than smart, passionate people. They would have had to really try to screw it up. And they did. I don't honestly know if there will be another RH in the future, but I'm hoping there will be.
- zxzax 5y ago
  I agree with the rest of your comment but I just want to respond to this part: >If you don't have thick skin, you should probably avoid upstream discussions I have a humble request, please discontinue this attitude on your own projects and please encourage other upstreams to also discontinue it. I'm baffled as to why so many open source maintainers seem to confuse needing to defend technical decisions with "having a thick skin," they are not the same. It's perfectly possible to be opinionated and strongly scrutinize a technical decision, while also not being harsh and rude towards the person presenting it. If you believe those mailing lists are unsalvageably hostile, then in my opinion, they should just be shut down. That's not the kind of place to have official project communication.
- pabs3 5y ago
  Where are the people fleeing RH going to go? There are not many workplaces as large and as Free Software focussed as RH.
  - yourapostasy 5y ago
    Generally most M&A's like this turn into a talent smasher event (think atom smasher, aka particle accelerator), and the people scatter to the four winds in hundreds/thousands of different landing zones instead of coalescing into another convergent organization. Emergent consensus centered around such a large organization takes a lot of effort and time to build because it operates through social trust pathways (that predominantly operate in the time domain), and once fissioned, the byproducts do not transitively transfer that trust, it has to be built anew. This adds to my suspicions that the long-term IBM C-suite goal of the Red Hat acquisition is to eventually convert Red Hat to be like their captive mainframe revenue and profit stream. I would not at all be surprised to see a gradual deterioration in Red Hat quality in the coming 10 years. Watch the general level of support engineering skills in what will eventually be a Blue-washed Red Hat L2 Support. That is your canary. What you are used to today in that tier will be gradually relegated to the tech leads for L2. Don't look for tech leads by title; look for the leads who gates decisions to escalate to L3. Highly-skilled L2 organizations grant most if not all their support engineers autonomy to escalate to L3. Profit-squeezing L2 organizations (what I suspect may be happening with Red Hat in the coming years) will gate those escalations, and staff vacancies with relatively down-skilled engineers. Note that I also enjoy working with these relatively down-skilled engineers. No complaints about "bedside manners", and they can address common scenarios. Given enough time and exposure to more experienced staff, they would absolutely step up to highly-skilled L2 levels. The challenge is often these situations see those more experienced staff leave too soon. In my experience you need at least 7, ideally 14 years to season new staff to those skill levels, and the attrition rate is high over that time; troubleshooting well under pressure with grace is a very difficult combination of skills and characteristics to find. Two ways I've seen to effectively deal with this on the customer end is find an alternative solution (not really feasible in this case at this time, though I'm watching several possible developments), or up-skill your staff. You'll know you've reached the right skill level when your engineers can delve into the internals of the product sufficiently that they routinely convince the down-skill L2 support engineers to escalate their lead within a day or two of engaging. If my suspicions bear fruit (I really hope not, the vast majority of my clients hold massive people and capital investments in the Red Hat ecosystem), then this is extremely bullish for AWS. Their customer-obsession-to-a-fault culture would swiftly exploit IBM's strategy, and there would be no contest. I can see several simultaneous attacks AWS could mount to neutralize Red Hat's position if IBM's strategy at the end of the day really is what I currently suspect. I'd really like to hear from others that have found other workarounds for this eventuality.

notacoward 5y ago

Take that appeal to authority elsewhere, and the insults with it. As someone who was also at Red Hat for eight years (2009-2017) I'd say GP might have been off base but certainly not garbage. I heard more grossly sexist comments (or saw them on IRC) there than anywhere else. We're talking stuff that would lead to immediate mandatory diversity training the first time, or instant dismissal the second, at many companies (including IBM). "Who did she have to blow to get that job" particularly sticks in my mind. Most people didn't even bat an eye because that kind of stuff is totally allowable in the open-source communities these folks were part of. I also saw at least one person fired for refusal to work with people he considered "beneath" him socially. You might say it's good that the company took action, but it took over a year for that to happen and I don't think that would have been the case at a truly inclusive company. I suppose you can dismiss these (and plenty more I could relate) as mere anecdotes, but you haven't even provided those so that would be pretty hypocritical. The vast majority of people I knew at Red Hat ranged from industry-average to absolutely excellent, but overall I'd say even Facebook had a better internal culture in terms of diversity etc. Painting RH as anything more than a typical tech company in those areas is a bit absurd.