21 ms·

One way a builder culture can fail

185 points by zdw 5y ago

jarofgreen 5y ago
People should check out Wardley mapping. It's a good tool to have conversations in a company/group about what you are doing that is commodity and what is custom and crucially if this is a good idea or not. ( Of course, management have to be open to these conversations and this alone won't save you from the terrible politics Rachel describes. )
MathMonkeyMan 5y ago
She seems mad by the end. > Just imagine what companies would look like if their hiring processes filtered out this kind of charlatan instead of looking for whether they could do some dumb coding card trick in a 45 minute interview block. I think they'd look very different, and a whole bunch of people would have to find another industry to prey upon. I would caution against hoping for a personality test that could weed out the phonies, though. "Behavioral interviews" are strange enough already, in my opinion, even if their goal might be admirable (e.g. identify the assholes). Anywhere there is money and prestige to be had, you can either keep things small or inevitably suffer the phonies.
- wmf 5y ago
  It seems like a leap from "person who wants to build X" to "charlatan".
  - markus_zhang 5y ago
    One probably spent too much time debugging someone's shit mountain code base, the original author got to enjoy bog bonuses and bragging about X in conferences and getting hired by similar companies to build another X.
  - Jtsummers 5y ago
    > I will now point out that whether X makes it to production can become disconnected from the notion of whether X actually belongs in production. In one outcome, it's brilliantly executed and things grow. But, this is about the downsides, and in that scenario, it's a stinking pile of garbage and needs to be thrown out, but it gets forced in, anyway. [emphasis added] She's left the positive case out for the charlatan comment, they wouldn't be charlatans (in this instance at least).
    - thaumasiotes 5y ago
      The guy building X isn't a charlatan in any case. The charlatan is the guy selling X, which is presumably the guy who hired the guy who builds it. If you tell me to build you a system you don't need, and I do, and you insist on using it against everyone else's better judgment, how did I become a charlatan?
      - lazyasciiart 5y ago
        Because if you're the expert on X, you can either stand up and agree with everyone that building X is not the right decision, or you can smile and nod and say yea, we should totally build our own X, it's a great idea. If you say no, it won't happen.
        em-bee 5y ago
        and they loose the job they were just hired for. because the more likely situation is that the newly hired expert on X is the only one realizing that X is wrong for this case.
        ed_elliott_asc 5y ago
        If you know you were being hired for one specific task shouldn’t you have talked about the fit before you start? I would say that is reasonable.
        dgb23 5y ago
        It's not always reasonable, because in the general case you don't know until later right? You discover the issues at some point when already working. Also you're implying that there is no economic dependence which is not always the case.
        lazyasciiart 5y ago
        The ones who lie to keep their job are the ones who get called 'charlatan'. Seems fair.
  - ThrowawayR2 5y ago
    We're not talking about some ordinary rank-and-file engineer here. The situation Rachel is talking about where the person is usually well-known among a significant fraction of their industry for their expertise in X. That level of seniority (principal level or beyond) is supposed to imply that they also know when -not- to use X and the responsibility to speak up, but sometimes either their belief in X is too strong ("When all you have is a hammer, everything looks like a nail.") or, in some cases I suspect, the monetary and career-enhancing incentives are high enough to overcome any unease they may have misapplying X. Then you get this situation where a mandate comes down for X is used where X is not really the appropriate thing to do.
    - jldugger 5y ago
      > The situation Rachel is talking about where the person is well-known among a significant fraction of their industry for their expertise in X. I don't know the specifics of Rachel's parable, if any. But by and large, the people who are most well known in industry for 'expertise in X' are the devrel / tech evangelists whose job it is to go on stage and be an expert, for one hour at a time (and perhaps a few extra hours on the show floor surrounding the conference). Well, the on-stage demos they execute only have to live as long as the lecture, and you don't have to fix bugs so much as rework the preso to avoid them. It's a long march from a proof of concept to production readiness, and I've seen one or two stories on HN about people who got burned hiring high profile devrels to lead an infra buildout. > sometimes either their belief in X is too strong ... or, ... the monetary and career-enhancing incentives are high enough to overcome any unease they may have One scenario I've seen play out: you hire a core maintainer for a product you depend on, but should probably be transitioning away from. The maintainer you typically don't have comparable expertise in anything else, so they might as well apply for unemployment before telling your boss you recommend against it. Which I suppose is covered in 'monetary incentives' but with tinge of regret about the whole thing, especially if hired away from some a more stable role.
      - pjmlp 5y ago
        You also have those whose consulting career is fully based on being 'experts in X', being on stage doing talks on X, is the mean to sell themselves as 'experts in X', alongside writting books about how cool X happens to be no matter what.
  - fuzzfactor 5y ago
    It doesn't have to be a malicious charlatan to pounce on value that has already been built, without them having actual consideration for future value that should be more rewarding and more sensibly worth working toward instead. The corporate-climber-who-can't-build can often do this and nothing else, and they can sometimes even subconsciously exhibit some of the behaviors that end up with the same outcome, and it often can actually be unintentional.
- deanCommie 5y ago
  Honestly, I don't think it's that hard. I assume we're talking about fairly senior candidates here. "Tell me about the most complicated projects you contributed a significant part of the design, implementation to, or ideally both." Start with the problem they aimed to solve. Probe for how they chose the solution, and what alternatives they considered. Get into the details of specifically what artifacts they produced, and if part of a team what role they played. Get them to describe the solution in technical detail. Probe into trade-offs they had to make or compromises. Ask what aspects they found novel or innovative. Once they finish describing the system, ask if it was successful, and how they measured it. If they were around after launch, how did they operate it, monitor it, what unexpected challenges they encountered that they didn't foresee. By the way, all along it doesn't matter if the domain or technology they are describing to you is totally different from your own experience. A senior+ engineer should be able to explain their domain to someone equally technical in a different domain with some competency. Finally, since you now understand the problem space and the solution yourself, ask them "How would you scale this out to 10-100X" (by some metric). Altogether, this is a dense 40-45 minutes extremely well spent. And it weeds out the phonies and charlatans. Some yellow flags that individually aren't a problem but when you start seeing multiple of them turn into a red flag: * Someone who didn't question the requirements and just accepted them from upstream. * Someone that wrote a lot of code on the project, but didn't make any of the tough decisions. * Someone that avoids talking about what they did, and focuses too much on "we" * Someone who did all the initial decisions but did none of the actual implementation work. * Someone who describes how X solved the problem, but you realize that's the only option they really considered * Someone who left before the project finished, or immediately after. * Someone who isn't sure if the project was a success or didn't see it their responsibility to find out.
  - cbsks 5y ago
    As someone who just went through a dozen interviews, this line of questioning is by far my most favorite one to receive.
  - pstuart 5y ago
    This cannot be upvoted enough.
- pm90 5y ago
  Filtering out the assholes is an accepted policy today but it didn’t just happen. Assholes were accepted and that sort of attitude wasn’t discouraged, until people started speaking up about it and others realized they probably had seen that dynamic. I see this article as a step in extending the anti-asshole policy to people who engage in this sort of behavior. Can I tell you a perfect personality test that will always work and never fail? Absolutely not. Even the anti asshole processes don’t always work. But it seems a good thing that this sort of behavior is at least being described and reported on.
  - taurath 5y ago
    Every company that I’ve been in that had an anti asshole policy has meant that they don’t tolerate assholes at the line engineer level, since that’s reserved for management, especially director and up. I guess it’s better, but the proof is in what the management team looks like.
- JeremyNT 5y ago
  > She seems mad by the end. You'll see this sort of things a lot from Rachel's writing, but I tend to write it off as a stylistic choice. She kind of channels the BOFH [0] sometimes, I think :) [0] https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell
markus_zhang 5y ago
Regarding builders who jump between solutions to build X, I don't think the issue is ONLY the hiring process.
- bombcar 5y ago
  It’s often just a symptom of cargo culting at a managerial or even Board level. Google does X so we need to do X.
  - markus_zhang 5y ago
    Yeah I think the same. A boat leaking from top.
  - andi999 5y ago
    Can you tell where I can find the Xs Google does? My boss wants to know. :)
  - crate_barre 5y ago
    I wanted to smack someone at work for doing this (being facetious). It was truly demoralizing to watch that person go through with it too. I think I even heard one of them verbally utter ‘that’s not how real big boys do it, that’s not how the pros do it’. They said words like that out loud, absolutely shameless. This hilarious part about it was, they never actually worked at a big company.
theamk 5y ago
Wait, there is some disconnect here, the last paragraph does not follow at all. To see it more easily, here is her post but with some specific, made up example instead of "X": A chocolate factory wants to build a new dark chocolate line. They hire a very experienced industrial engineer, who just recently finished building car engine line for the Ford factory. However, once the new engineer started to work on chocolate line, they found it extremely boring and devoid of challenges. So they somehow convinced management that the chocolate factory needs to start producing car engines for their delivery trucks. This is obviously a bad idea, and many people point that, but the engineer has secured support of one of VPs, so the project goes ahead. Eventually the line is ready, new engines are produced and installed in all the new trucks. They proceed to break down frequently, the deliveries are late, customers are unhappy, and everyone is having a bad time. Everyone is blaming engineer now and calling them "charlatan", but the engineer does not understand why: They really worked their ass off building the new engine line, working 80 hours a week to realize their dream. It was not easy -- the factory was only doing chocolate before, so switching to heavy machinery was quite a challenge! And the reason customers are unhappy is because of the management -- this project should have gotten way more resources! Then we could test more and iron out all the bugs instead of having to go to production too early. Rachel's final paragraph puts the blame squarely on the engineer, with wishes that "hiring processes filtered out this kind of charlatan" and "whole bunch of people would have to find another industry to prey upon", but that engineer is not a charlatan -- they have a legitimate technical talent, and they used it to build real stuff. No hiring process can filter them out -- they will pass a technical interview and take-home test, they have great communication skills, they have a github page with open source code, and they can talk passionately about their past successful projects. Instead, I'd put most of the blame on the management. In a large company, it is inevitable that someone will work on stuff that is useless or could harm the main product. Maybe because engineers lack perspective, or they overestimate themselves, or they just don't care about company's goals and want to do what they find fun. The company should have some mechanisms to prevent this -- maybe management should listen to feedback from engineers, or there should be technical committee, or at least people should not be rewarded when they ship a broken product. If the company has no such mechanisms, it will have problems as described in the blog.
- 5y ago
jackblemming 5y ago
I love the call out at the end. I believe much of the post is a slight at Google (and much of big tech), where building complicated crap is one of the fastest ways of promotion. Rachel is completely right, "big tech" has a problem of attracting politician types, who are focused on abusing systems for their own gain. You can hardly blame them though, the company as a whole is doing the same thing. The incentives are misaligned at pretty much every level.
- shadowgovt 5y ago
  Given when she was with the company, the X in the room is Google+.
  - wmf 5y ago
    Google+ actually seemed kind of good to me? Like maybe the problem with Google+ was that not that it existed but that they didn't invest enough in it.
    - int0x2e 5y ago
      I'd argue that they invested way too much into Google+ actually. They tried forcing it on everyone, almost killing the YouTube community in the process, and people started fighting back. Facebook didn't gain its users overnight and without a good amount of manipulation and tricks on their part, but they were smart enough to not make it feel as aggressive or directed as Google did with Google+
      - deleted 5y ago
        [deleted]
      - pornel 5y ago
        G+ had a Real Name Policy, apparently for no reason other than copying Facebook. And then because every Google product had to have G+, Google effectively banned anonymity/pseudonymity everywhere. It was so dumb and heavy-handed. For me that was the end of using Google products.
qPM9l3XJrF 5y ago
>I've told some stories about what happens when you end up at a company that builds nothing and instead rents everything from some vendor. This sounded interesting but I searched for her blog and I couldn't find it. Does anyone know which post this refers to?
- minism 5y ago
  Not sure but possibly one of her books? https://rachelbythebay.com/store/ https://rachelbythebay.com/store/
- greenyouse 5y ago
  Here are a couple for sure: - https://rachelbythebay.com/w/2020/02/08/miss/ - https://rachelbythebay.com/w/2019/10/13/firewall/ maybe these too: - https://rachelbythebay.com/w/2020/08/17/potato/ - https://rachelbythebay.com/w/2019/11/10/scale/
  - qPM9l3XJrF 5y ago
    Thanks!
onion2k 5y ago
These two remarks don't line up... They're a builder, all right, as long as you want them to build X. Perhaps they have been building X at some other company, and now your company has hired them to come and do that same job over here instead. Just imagine what companies would look like if their hiring processes filtered out this kind of charlatan The company specifically filtered them in to build the X product they're renowned for building at other companies. The hiring process is working perfectly if the company decides they want X, hires a famous X builder, and that person then builds X in their role. I think the point Rachel is alluding to is that the person should have realised X is inappropriate and said so, but if your world is X then it can be hard to see the broader picture. We've all tried to leverage the tools we know best to do jobs they're not perfect for rather than start from zero with something more appropriate, especially when there's a looming deadline or a lot of pressure from higher ups. To call someone a charlatan for doing that seems grossly unfair.
- yjftsjthsd-h 5y ago
  > the person should have realised X is inappropriate and said so, but if your world is X then it can be hard to see the broader picture. Also, for political reasons, nobody else can or will question it, so there's no safety mechanism.
- turbinerneiter 5y ago
  I think she is not specifically talking about the exact same person here. She specifies people putting themselves over the success of the team later in the text, right in front of the charlatan statement. The x builder can have this affect, on the one end of the spectrum what you describes happens, on the other end of the spectrum charlatanerie happens.
- Moru 5y ago
  An engineer is expected to speak up when told to construct X out of Y when Y is not good enough for it. He is personally responsible for the lives at stake. A software engineer is not expected to speak up when told to build X out of Y but to push through no matter what.
kfk 5y ago
This is quite normal in big companies and it doesn’t just apply to building stuff, buying stuff has the same problem. Managers will propose a certain product vendor and after roll out will continue to support it, no matter the consequences. We had to use a terrible “data virtualizzation” tool for 2 years because IT “leaders” had bought into it and now would not admit their mistake.
- cloverich 5y ago
  Unfortunately this is somewhat ingrained in the sales cycle. I finally understood it at a prior company when they bought Saleforce. By the time they realized SF wasn't going to be a good fit, they were 50k in and had already signed a multi year contract. Charge obscene amounts, and offer steep discounts the earlier and longer the company is willing to contract for. It becomes a sunk cost and bonus points if the company starts developing competencies in it (negating that many people are sufficiently detached that they don't even realize its a bad fit). Its a relatively advanced skill knowing how to navigate such a scenario at the various stages.
downWidOutaFite 5y ago
I'm in such a situation right now. We got this new person that is known for having built a great a/b testing solution that scaled at a large top-tier company so they seemed like a great hire. Problem is we already have an ab test solution that is integrated into everything. Maybe it's not the best solution ever but it does everything we need, nobody has been complaining about it. But since we got this hire we're now having to do a large rewrite that's not really buying us anything and nobody can really question it because people are invested in that we got this great hire.
- int0x2e 5y ago
  I would argue that this person is not a great hire for you then. If they cannot contribute to something you actually need, how could them bring awesome at something else help? Imagine a different situation - there's a fellow engineer who's also an olympic gold medalist in some sport. They may be awesome to have around. They may have cool stories. They may have connections. But unless they can harness that towards sometging that helps your mission - it really won't make them any better as an engineer.
  - pjmlp 5y ago
    I have been there, I have been part of teams that were created to do X, when it wasn't really required, just some architects thought so. So when everything is in place, from management point of view it is easier to charge ahead than acknowledge it was a bad decision. In the end when everything falls appart, they will either no longer be there, or find other reasons why it failed.
brunooliv 5y ago
In my limited experience, this is rarely so simple as to being able to push back on a technical decision when the reality will feel like pushing a square through a circle peg. I mean, as rightfully pointed out, if the reasons are more political than technical, pushing back can have nefarious consequences for the people involved. Doing our jobs as engineers is about raising the alarms when needed, but, if all you get is resistance back... Assuming it's not your product, not your company and your ties to the thing are mostly technical... Then if advocate for the "not your problem" attitude and you can simply be content to let it crash and burn and show up the next day with a mop to fix it. Dealing with incompetent management is never nice or easy, but I'd advocate that as software developers, we're quite privileged because we get paid pretty well, work is versioned in source control and most messes are physically unseen, and cleaning up is "usually" not a big deal.
tjpnz 5y ago
Where the hilarity really ensues is when X gets reflected in the hiring process. If you're working in a sufficiently small industry you can sometimes tell if the builder has paid a visit without checking their LinkedIn. How would you integrate NIH X into architecture Y? Sometimes you can even tell which iteration of X the company ended up with. "Fucker's settin' up franchises!"
- lpapez 5y ago
  I work at one such company. The amount of resources thrown into X is disturbing, and any attempts to discuss benchmarking changes so that the useleness of X can be revealed are shut down aggresively.
0xFACEFEED 5y ago
> I will now point out that whether X makes it to production can become disconnected from the notion of whether X actually belongs in production. ... is the crux of her point. So, um, who decides what belongs in production? I generally like her essays but this has a weird senseless negative tone to it. Look... I get it. We've all come across bozos. A lot of these bozos aren't the derpy clowns we (hope? wish?) they would be. No - they're usually very confident, very aggressive, and very very goal oriented. Yes it's absolutely infuriating watching them somehow gain the confidence of the C-level folks, obtain a lot of resources, and burn everything to the ground all while claiming plausible deniability. Wishing that they'd get filtered out by the hiring process is <insert idiom of improbability>. The real culprit is your director of engineering. That person is making terrible decisions but is so good at avoiding accountability that you place your blame on the bozo. Hot take: if your company keeps hiring bozos then your executives are doo-doo. Leave the company. Oh, you can't because you're making 300k base + 2 years away from 800k in RSUs? Congrats. You're part of the problem (it's not your fault). The same reason why you won't leave your company is the same reason Mr. Bozo gets a crack at "innovating" at that same company. People like Rachael (who obviously love this work at their very core) are doomed to end up a jaded curmudgeon unless they 1) start their own company 2) join a tiny company. Big tech is hell. Make your money and GTFO.
- fuzzfactor 5y ago
  end up a jaded curmudgeon? Success is not guaranteed, but that can be a well-earned element that at least greatly reduces the odds of failure when they start their own company.
- rvba 5y ago
  > So, um, who decides what belongs in production? Product Owner, or just management? On a side note: in "mature" markets a bricklayer nor an architect, nor constructor can really build anything they want without anyone looking.
  - TheOtherHobbes 5y ago
    I'm confused why anyone would think this is about optics and not about performance. If something Does Not Work Reliably - when it has no reasonable reason for not doing so - then it Does Not Belong In Production. As for mature markets - useless and/or broken buildings happen all the time. Sometimes this is because a starchitect is good at making buildings that look incredible but bad at making buildings that don't have problems. (Calatrava, Frank Lloyd Wright, many more.) Sometimes it's because the project is extractive and the goal is money at the expense of quality - or even usability. (Many housing estates and not a few McMansions.) Both of these are different to a tech co hiring the equivalent of a minor starchitect to Make A Thing Happen, but getting something non-performant and/or impossible to maintain. The difference is that unless it's a greenfield startup Thing is likely to be easy to define and its performance should be easy to measure objectively with simple metrics like uptime, throughput, and so on.
ed_elliott_asc 5y ago
This seems oddly specific ha ha
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
Marazan 5y ago
For those who are confused about this post it is clearly a long form sub-tweet. It is directed at a specific organisation and person but anonymised. But..... it is totally generalisable. I have seen this happen myself. Rank-and-file team rail loaded into using Senior Person's X because Senior Person has made X rather than it being the best fir for the job.
tester34 5y ago
Probably Charlatan here, AMA Our $company lacked of automation of some process and after months of wasting hours doing it manually I decided to automate that. I had no exp with real ci/cd tools and thought our process was already crazy enough (using annoying source control tool, heavy dependency between compilation and external soft's specific versions) that we need to build our tool to manage it. I built MVP as CLI and it worked. As time passed everybody wanted to add tons of stuff to it - like Web GUI, history, logs, managing software versions when creating binaries and even performing code analysis (because we were heavily dependent on 3rd software we needed to detect some colissions) Overall, it saved us from doing hundreds manual, sometimes error-prone tasks per year. It allowed us to add customizations like that code analysis using compiler's API But the other hand maybe we could just use some Jenkins or some shit and achieve similar stuff without having to invest time into it and maintain - it's combo of Visual Studio + MS Build + Team Foundation Version Control + .NET Framework, so stuff tends to make problems from time to time I feel that despite all good things that this project brought, then we could just use existing tool and maybe achieve similar results, but I don't know for sure. I feel like sometimes I used this project to escape boring CRUDing and used it as a learning opportunity - maybe I abused it too much? Or maybe should I treat it as "Google's 20%"?
- Ensorceled 5y ago
  This is a completely different scenario and, also, a not so subtle humble brag. You're duct taping process onto an existing product cycle. I've experienced the 'X' factor the author is talking about and it's something like "we're moving everything to Linux because that's what I do" and then you go 18 months without a release and the release you do make is a bugfix release to the old infrastructure, your customers leave and you get bought for debt.
  - tester34 5y ago
    >a not so subtle humble brag. I'm not sure how could I rewrite it to avoid that feel, yet still include enough details to let people kinda evaluate whether it was reasonable or not
davewritescode 5y ago
I will say that avoiding "builder culture" probably saved my job. I joined a company that had migrated from an open source identity management solution to a homegrown one and it had ran perfectly well in production for many years but was getting creaky and didn't offer the features that were needed for a SaaS offering. VPs gave me the option of determining whether we migrated to an identity vendor or built our own. I've contributed code to popular OIDC implementations on GitHub, I've written code in widely used open source JWT libraries and I felt extremely comfortable working in that space so my inclination was towards building. The problem was to staff a quality engineering team was going to cost 5x the yearly price of the most expensive service and hiring other people in that space was proving to be very difficult and we didn't have time to take a year to build it and then start migration In the end I chose to go with a vendor and a smaller team to support a migration. We migrated in less than 6 months and a year later we ended up getting contacted by law enforcement about some activity that was happening our system. The auditing system on the vendors side allowed us to quickly diagnose what exactly had happened and let us give data to folks who needed it. Had that been our homegrown system, it would've been 100% an unmitigated disaster that took days to unravel. Had we decided to build it ourselves, there's no way we could've hit production as quickly as we did.
- jimktrains2 5y ago
  It really depends on how well the offering matches the need. My personal experience is that outsourcing core bits where vendors don't match exactly the needs ends up causing more overhead that if we had written the service ourselves.
- cloverich 5y ago
  One unwritten trade off that's in addition, not a knock on your stance just an observation, is that teams that build one quality product can often build another. To some extent when you choose to build you are also choosing to build a culture that knows how to build. So you have to think about not just the next thing you need, but the five things after that. I've worked at companies that bought literally everything, and what I found is the engineers there knew a lot about how to plugin to these various systems, but when they needed to build something complex together they weren't able to do so very well, or at all. Which might be fine, but you should head in that direction with intention.
bertr4nd 5y ago
I don’t know. I’ve been in the position of being an “X-builder” (I’m a dyed-in-the-wool compiler engineer), and I’ve been through something like what she describes, and nothing about the decisions involved seemed as obvious (or as malicious/egotistical) as she makes it sound. Like, maybe I don’t quite want to build an X, but other engineers and managers have their own reasons why we should, and after we discuss/argue for a little while, I decide that, OK, fine, I don’t entirely agree but I’m willing to concede that maybe I’m wrong and give it a shot. The thing we end up building is OK, but not stellar. We have an outage or two but they’re minor and not an ongoing headache. We have a few wins but it’s not the runaway success we hoped. Maybe we should have gone with Y instead of building our own X? Would it have been better that way? Am I a charlatan? I don’t know.
TheDesolate0 5y ago
well...I can't unread that waste of time, space, and typing...
seahawks78 5y ago
> "This is just one of the ways an unprincipled "builder culture" can backfire, particularly if you have people running the show who have no problem putting their own gains ahead of the company. They'll push all kinds of garbage if it makes them hit their goals and get a bigger bonus, a promotion, or some kind of shiny new position." In lots of places I have worked there is a particular term that is often used to precisely describe this situation, it is called POA aka "Promotion Oriented Architecture". In POA, you deliberately add a lot of accidental complexity to come up with a fiendishly complicated solution to a relatively simple problem. In short, the complexity stems from the solution space rather than the problem space.

185 points by zdw 5y ago

People should check out Wardley mapping. It's a good tool to have conversations in a company/group about what you are doing that is commodity and what is custom and crucially if this is a good idea or not. ( Of course, management have to be open to these conversations and this alone won't save you from the terrible politics Rachel describes. )

MathMonkeyMan 5y ago

She seems mad by the end. > Just imagine what companies would look like if their hiring processes filtered out this kind of charlatan instead of looking for whether they could do some dumb coding card trick in a 45 minute interview block. I think they'd look very different, and a whole bunch of people would have to find another industry to prey upon. I would caution against hoping for a personality test that could weed out the phonies, though. "Behavioral interviews" are strange enough already, in my opinion, even if their goal might be admirable (e.g. identify the assholes). Anywhere there is money and prestige to be had, you can either keep things small or inevitably suffer the phonies.

wmf 5y ago
It seems like a leap from "person who wants to build X" to "charlatan".
- markus_zhang 5y ago
  One probably spent too much time debugging someone's shit mountain code base, the original author got to enjoy bog bonuses and bragging about X in conferences and getting hired by similar companies to build another X.
- Jtsummers 5y ago
  > I will now point out that whether X makes it to production can become disconnected from the notion of whether X actually belongs in production. In one outcome, it's brilliantly executed and things grow. But, this is about the downsides, and in that scenario, it's a stinking pile of garbage and needs to be thrown out, but it gets forced in, anyway. [emphasis added] She's left the positive case out for the charlatan comment, they wouldn't be charlatans (in this instance at least).
  - thaumasiotes 5y ago
    The guy building X isn't a charlatan in any case. The charlatan is the guy selling X, which is presumably the guy who hired the guy who builds it. If you tell me to build you a system you don't need, and I do, and you insist on using it against everyone else's better judgment, how did I become a charlatan?
    - lazyasciiart 5y ago
      Because if you're the expert on X, you can either stand up and agree with everyone that building X is not the right decision, or you can smile and nod and say yea, we should totally build our own X, it's a great idea. If you say no, it won't happen.
      - em-bee 5y ago
        and they loose the job they were just hired for. because the more likely situation is that the newly hired expert on X is the only one realizing that X is wrong for this case.
        ed_elliott_asc 5y ago
        If you know you were being hired for one specific task shouldn’t you have talked about the fit before you start? I would say that is reasonable.
        dgb23 5y ago
        It's not always reasonable, because in the general case you don't know until later right? You discover the issues at some point when already working. Also you're implying that there is no economic dependence which is not always the case.
        lazyasciiart 5y ago
        The ones who lie to keep their job are the ones who get called 'charlatan'. Seems fair.
- ThrowawayR2 5y ago
  We're not talking about some ordinary rank-and-file engineer here. The situation Rachel is talking about where the person is usually well-known among a significant fraction of their industry for their expertise in X. That level of seniority (principal level or beyond) is supposed to imply that they also know when -not- to use X and the responsibility to speak up, but sometimes either their belief in X is too strong ("When all you have is a hammer, everything looks like a nail.") or, in some cases I suspect, the monetary and career-enhancing incentives are high enough to overcome any unease they may have misapplying X. Then you get this situation where a mandate comes down for X is used where X is not really the appropriate thing to do.
  - jldugger 5y ago
    > The situation Rachel is talking about where the person is well-known among a significant fraction of their industry for their expertise in X. I don't know the specifics of Rachel's parable, if any. But by and large, the people who are most well known in industry for 'expertise in X' are the devrel / tech evangelists whose job it is to go on stage and be an expert, for one hour at a time (and perhaps a few extra hours on the show floor surrounding the conference). Well, the on-stage demos they execute only have to live as long as the lecture, and you don't have to fix bugs so much as rework the preso to avoid them. It's a long march from a proof of concept to production readiness, and I've seen one or two stories on HN about people who got burned hiring high profile devrels to lead an infra buildout. > sometimes either their belief in X is too strong ... or, ... the monetary and career-enhancing incentives are high enough to overcome any unease they may have One scenario I've seen play out: you hire a core maintainer for a product you depend on, but should probably be transitioning away from. The maintainer you typically don't have comparable expertise in anything else, so they might as well apply for unemployment before telling your boss you recommend against it. Which I suppose is covered in 'monetary incentives' but with tinge of regret about the whole thing, especially if hired away from some a more stable role.
    - pjmlp 5y ago
      You also have those whose consulting career is fully based on being 'experts in X', being on stage doing talks on X, is the mean to sell themselves as 'experts in X', alongside writting books about how cool X happens to be no matter what.
- fuzzfactor 5y ago
  It doesn't have to be a malicious charlatan to pounce on value that has already been built, without them having actual consideration for future value that should be more rewarding and more sensibly worth working toward instead. The corporate-climber-who-can't-build can often do this and nothing else, and they can sometimes even subconsciously exhibit some of the behaviors that end up with the same outcome, and it often can actually be unintentional.
deanCommie 5y ago
Honestly, I don't think it's that hard. I assume we're talking about fairly senior candidates here. "Tell me about the most complicated projects you contributed a significant part of the design, implementation to, or ideally both." Start with the problem they aimed to solve. Probe for how they chose the solution, and what alternatives they considered. Get into the details of specifically what artifacts they produced, and if part of a team what role they played. Get them to describe the solution in technical detail. Probe into trade-offs they had to make or compromises. Ask what aspects they found novel or innovative. Once they finish describing the system, ask if it was successful, and how they measured it. If they were around after launch, how did they operate it, monitor it, what unexpected challenges they encountered that they didn't foresee. By the way, all along it doesn't matter if the domain or technology they are describing to you is totally different from your own experience. A senior+ engineer should be able to explain their domain to someone equally technical in a different domain with some competency. Finally, since you now understand the problem space and the solution yourself, ask them "How would you scale this out to 10-100X" (by some metric). Altogether, this is a dense 40-45 minutes extremely well spent. And it weeds out the phonies and charlatans. Some yellow flags that individually aren't a problem but when you start seeing multiple of them turn into a red flag: * Someone who didn't question the requirements and just accepted them from upstream. * Someone that wrote a lot of code on the project, but didn't make any of the tough decisions. * Someone that avoids talking about what they did, and focuses too much on "we" * Someone who did all the initial decisions but did none of the actual implementation work. * Someone who describes how X solved the problem, but you realize that's the only option they really considered * Someone who left before the project finished, or immediately after. * Someone who isn't sure if the project was a success or didn't see it their responsibility to find out.
- cbsks 5y ago
  As someone who just went through a dozen interviews, this line of questioning is by far my most favorite one to receive.
- pstuart 5y ago
  This cannot be upvoted enough.
pm90 5y ago
Filtering out the assholes is an accepted policy today but it didn’t just happen. Assholes were accepted and that sort of attitude wasn’t discouraged, until people started speaking up about it and others realized they probably had seen that dynamic. I see this article as a step in extending the anti-asshole policy to people who engage in this sort of behavior. Can I tell you a perfect personality test that will always work and never fail? Absolutely not. Even the anti asshole processes don’t always work. But it seems a good thing that this sort of behavior is at least being described and reported on.
- taurath 5y ago
  Every company that I’ve been in that had an anti asshole policy has meant that they don’t tolerate assholes at the line engineer level, since that’s reserved for management, especially director and up. I guess it’s better, but the proof is in what the management team looks like.
JeremyNT 5y ago
> She seems mad by the end. You'll see this sort of things a lot from Rachel's writing, but I tend to write it off as a stylistic choice. She kind of channels the BOFH [0] sometimes, I think :) [0] https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell

markus_zhang 5y ago

Regarding builders who jump between solutions to build X, I don't think the issue is ONLY the hiring process.

bombcar 5y ago
It’s often just a symptom of cargo culting at a managerial or even Board level. Google does X so we need to do X.
- markus_zhang 5y ago
  Yeah I think the same. A boat leaking from top.
- andi999 5y ago
  Can you tell where I can find the Xs Google does? My boss wants to know. :)
- crate_barre 5y ago
  I wanted to smack someone at work for doing this (being facetious). It was truly demoralizing to watch that person go through with it too. I think I even heard one of them verbally utter ‘that’s not how real big boys do it, that’s not how the pros do it’. They said words like that out loud, absolutely shameless. This hilarious part about it was, they never actually worked at a big company.

theamk 5y ago

Wait, there is some disconnect here, the last paragraph does not follow at all. To see it more easily, here is her post but with some specific, made up example instead of "X": A chocolate factory wants to build a new dark chocolate line. They hire a very experienced industrial engineer, who just recently finished building car engine line for the Ford factory. However, once the new engineer started to work on chocolate line, they found it extremely boring and devoid of challenges. So they somehow convinced management that the chocolate factory needs to start producing car engines for their delivery trucks. This is obviously a bad idea, and many people point that, but the engineer has secured support of one of VPs, so the project goes ahead. Eventually the line is ready, new engines are produced and installed in all the new trucks. They proceed to break down frequently, the deliveries are late, customers are unhappy, and everyone is having a bad time. Everyone is blaming engineer now and calling them "charlatan", but the engineer does not understand why: They really worked their ass off building the new engine line, working 80 hours a week to realize their dream. It was not easy -- the factory was only doing chocolate before, so switching to heavy machinery was quite a challenge! And the reason customers are unhappy is because of the management -- this project should have gotten way more resources! Then we could test more and iron out all the bugs instead of having to go to production too early. Rachel's final paragraph puts the blame squarely on the engineer, with wishes that "hiring processes filtered out this kind of charlatan" and "whole bunch of people would have to find another industry to prey upon", but that engineer is not a charlatan -- they have a legitimate technical talent, and they used it to build real stuff. No hiring process can filter them out -- they will pass a technical interview and take-home test, they have great communication skills, they have a github page with open source code, and they can talk passionately about their past successful projects. Instead, I'd put most of the blame on the management. In a large company, it is inevitable that someone will work on stuff that is useless or could harm the main product. Maybe because engineers lack perspective, or they overestimate themselves, or they just don't care about company's goals and want to do what they find fun. The company should have some mechanisms to prevent this -- maybe management should listen to feedback from engineers, or there should be technical committee, or at least people should not be rewarded when they ship a broken product. If the company has no such mechanisms, it will have problems as described in the blog.

5y ago

jackblemming 5y ago

I love the call out at the end. I believe much of the post is a slight at Google (and much of big tech), where building complicated crap is one of the fastest ways of promotion. Rachel is completely right, "big tech" has a problem of attracting politician types, who are focused on abusing systems for their own gain. You can hardly blame them though, the company as a whole is doing the same thing. The incentives are misaligned at pretty much every level.

shadowgovt 5y ago
Given when she was with the company, the X in the room is Google+.
- wmf 5y ago
  Google+ actually seemed kind of good to me? Like maybe the problem with Google+ was that not that it existed but that they didn't invest enough in it.
  - int0x2e 5y ago
    I'd argue that they invested way too much into Google+ actually. They tried forcing it on everyone, almost killing the YouTube community in the process, and people started fighting back. Facebook didn't gain its users overnight and without a good amount of manipulation and tricks on their part, but they were smart enough to not make it feel as aggressive or directed as Google did with Google+
    - deleted 5y ago
      [deleted]
    - pornel 5y ago
      G+ had a Real Name Policy, apparently for no reason other than copying Facebook. And then because every Google product had to have G+, Google effectively banned anonymity/pseudonymity everywhere. It was so dumb and heavy-handed. For me that was the end of using Google products.

qPM9l3XJrF 5y ago

>I've told some stories about what happens when you end up at a company that builds nothing and instead rents everything from some vendor. This sounded interesting but I searched for her blog and I couldn't find it. Does anyone know which post this refers to?

minism 5y ago
Not sure but possibly one of her books? https://rachelbythebay.com/store/ https://rachelbythebay.com/store/
greenyouse 5y ago
Here are a couple for sure: - https://rachelbythebay.com/w/2020/02/08/miss/ - https://rachelbythebay.com/w/2019/10/13/firewall/ maybe these too: - https://rachelbythebay.com/w/2020/08/17/potato/ - https://rachelbythebay.com/w/2019/11/10/scale/
- qPM9l3XJrF 5y ago
  Thanks!

onion2k 5y ago

These two remarks don't line up... They're a builder, all right, as long as you want them to build X. Perhaps they have been building X at some other company, and now your company has hired them to come and do that same job over here instead. Just imagine what companies would look like if their hiring processes filtered out this kind of charlatan The company specifically filtered them in to build the X product they're renowned for building at other companies. The hiring process is working perfectly if the company decides they want X, hires a famous X builder, and that person then builds X in their role. I think the point Rachel is alluding to is that the person should have realised X is inappropriate and said so, but if your world is X then it can be hard to see the broader picture. We've all tried to leverage the tools we know best to do jobs they're not perfect for rather than start from zero with something more appropriate, especially when there's a looming deadline or a lot of pressure from higher ups. To call someone a charlatan for doing that seems grossly unfair.

yjftsjthsd-h 5y ago
> the person should have realised X is inappropriate and said so, but if your world is X then it can be hard to see the broader picture. Also, for political reasons, nobody else can or will question it, so there's no safety mechanism.
turbinerneiter 5y ago
I think she is not specifically talking about the exact same person here. She specifies people putting themselves over the success of the team later in the text, right in front of the charlatan statement. The x builder can have this affect, on the one end of the spectrum what you describes happens, on the other end of the spectrum charlatanerie happens.
Moru 5y ago
An engineer is expected to speak up when told to construct X out of Y when Y is not good enough for it. He is personally responsible for the lives at stake. A software engineer is not expected to speak up when told to build X out of Y but to push through no matter what.

kfk 5y ago

This is quite normal in big companies and it doesn’t just apply to building stuff, buying stuff has the same problem. Managers will propose a certain product vendor and after roll out will continue to support it, no matter the consequences. We had to use a terrible “data virtualizzation” tool for 2 years because IT “leaders” had bought into it and now would not admit their mistake.

cloverich 5y ago
Unfortunately this is somewhat ingrained in the sales cycle. I finally understood it at a prior company when they bought Saleforce. By the time they realized SF wasn't going to be a good fit, they were 50k in and had already signed a multi year contract. Charge obscene amounts, and offer steep discounts the earlier and longer the company is willing to contract for. It becomes a sunk cost and bonus points if the company starts developing competencies in it (negating that many people are sufficiently detached that they don't even realize its a bad fit). Its a relatively advanced skill knowing how to navigate such a scenario at the various stages.

downWidOutaFite 5y ago

I'm in such a situation right now. We got this new person that is known for having built a great a/b testing solution that scaled at a large top-tier company so they seemed like a great hire. Problem is we already have an ab test solution that is integrated into everything. Maybe it's not the best solution ever but it does everything we need, nobody has been complaining about it. But since we got this hire we're now having to do a large rewrite that's not really buying us anything and nobody can really question it because people are invested in that we got this great hire.

int0x2e 5y ago
I would argue that this person is not a great hire for you then. If they cannot contribute to something you actually need, how could them bring awesome at something else help? Imagine a different situation - there's a fellow engineer who's also an olympic gold medalist in some sport. They may be awesome to have around. They may have cool stories. They may have connections. But unless they can harness that towards sometging that helps your mission - it really won't make them any better as an engineer.
- pjmlp 5y ago
  I have been there, I have been part of teams that were created to do X, when it wasn't really required, just some architects thought so. So when everything is in place, from management point of view it is easier to charge ahead than acknowledge it was a bad decision. In the end when everything falls appart, they will either no longer be there, or find other reasons why it failed.

brunooliv 5y ago

In my limited experience, this is rarely so simple as to being able to push back on a technical decision when the reality will feel like pushing a square through a circle peg. I mean, as rightfully pointed out, if the reasons are more political than technical, pushing back can have nefarious consequences for the people involved. Doing our jobs as engineers is about raising the alarms when needed, but, if all you get is resistance back... Assuming it's not your product, not your company and your ties to the thing are mostly technical... Then if advocate for the "not your problem" attitude and you can simply be content to let it crash and burn and show up the next day with a mop to fix it. Dealing with incompetent management is never nice or easy, but I'd advocate that as software developers, we're quite privileged because we get paid pretty well, work is versioned in source control and most messes are physically unseen, and cleaning up is "usually" not a big deal.

tjpnz 5y ago

Where the hilarity really ensues is when X gets reflected in the hiring process. If you're working in a sufficiently small industry you can sometimes tell if the builder has paid a visit without checking their LinkedIn. How would you integrate NIH X into architecture Y? Sometimes you can even tell which iteration of X the company ended up with. "Fucker's settin' up franchises!"

lpapez 5y ago
I work at one such company. The amount of resources thrown into X is disturbing, and any attempts to discuss benchmarking changes so that the useleness of X can be revealed are shut down aggresively.

0xFACEFEED 5y ago

> I will now point out that whether X makes it to production can become disconnected from the notion of whether X actually belongs in production. ... is the crux of her point. So, um, who decides what belongs in production? I generally like her essays but this has a weird senseless negative tone to it. Look... I get it. We've all come across bozos. A lot of these bozos aren't the derpy clowns we (hope? wish?) they would be. No - they're usually very confident, very aggressive, and very very goal oriented. Yes it's absolutely infuriating watching them somehow gain the confidence of the C-level folks, obtain a lot of resources, and burn everything to the ground all while claiming plausible deniability. Wishing that they'd get filtered out by the hiring process is <insert idiom of improbability>. The real culprit is your director of engineering. That person is making terrible decisions but is so good at avoiding accountability that you place your blame on the bozo. Hot take: if your company keeps hiring bozos then your executives are doo-doo. Leave the company. Oh, you can't because you're making 300k base + 2 years away from 800k in RSUs? Congrats. You're part of the problem (it's not your fault). The same reason why you won't leave your company is the same reason Mr. Bozo gets a crack at "innovating" at that same company. People like Rachael (who obviously love this work at their very core) are doomed to end up a jaded curmudgeon unless they 1) start their own company 2) join a tiny company. Big tech is hell. Make your money and GTFO.

fuzzfactor 5y ago
end up a jaded curmudgeon? Success is not guaranteed, but that can be a well-earned element that at least greatly reduces the odds of failure when they start their own company.
rvba 5y ago
> So, um, who decides what belongs in production? Product Owner, or just management? On a side note: in "mature" markets a bricklayer nor an architect, nor constructor can really build anything they want without anyone looking.
- TheOtherHobbes 5y ago
  I'm confused why anyone would think this is about optics and not about performance. If something Does Not Work Reliably - when it has no reasonable reason for not doing so - then it Does Not Belong In Production. As for mature markets - useless and/or broken buildings happen all the time. Sometimes this is because a starchitect is good at making buildings that look incredible but bad at making buildings that don't have problems. (Calatrava, Frank Lloyd Wright, many more.) Sometimes it's because the project is extractive and the goal is money at the expense of quality - or even usability. (Many housing estates and not a few McMansions.) Both of these are different to a tech co hiring the equivalent of a minor starchitect to Make A Thing Happen, but getting something non-performant and/or impossible to maintain. The difference is that unless it's a greenfield startup Thing is likely to be easy to define and its performance should be easy to measure objectively with simple metrics like uptime, throughput, and so on.

ed_elliott_asc 5y ago

This seems oddly specific ha ha

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

Marazan 5y ago

For those who are confused about this post it is clearly a long form sub-tweet. It is directed at a specific organisation and person but anonymised. But..... it is totally generalisable. I have seen this happen myself. Rank-and-file team rail loaded into using Senior Person's X because Senior Person has made X rather than it being the best fir for the job.

tester34 5y ago

Probably Charlatan here, AMA Our $company lacked of automation of some process and after months of wasting hours doing it manually I decided to automate that. I had no exp with real ci/cd tools and thought our process was already crazy enough (using annoying source control tool, heavy dependency between compilation and external soft's specific versions) that we need to build our tool to manage it. I built MVP as CLI and it worked. As time passed everybody wanted to add tons of stuff to it - like Web GUI, history, logs, managing software versions when creating binaries and even performing code analysis (because we were heavily dependent on 3rd software we needed to detect some colissions) Overall, it saved us from doing hundreds manual, sometimes error-prone tasks per year. It allowed us to add customizations like that code analysis using compiler's API But the other hand maybe we could just use some Jenkins or some shit and achieve similar stuff without having to invest time into it and maintain - it's combo of Visual Studio + MS Build + Team Foundation Version Control + .NET Framework, so stuff tends to make problems from time to time I feel that despite all good things that this project brought, then we could just use existing tool and maybe achieve similar results, but I don't know for sure. I feel like sometimes I used this project to escape boring CRUDing and used it as a learning opportunity - maybe I abused it too much? Or maybe should I treat it as "Google's 20%"?

Ensorceled 5y ago
This is a completely different scenario and, also, a not so subtle humble brag. You're duct taping process onto an existing product cycle. I've experienced the 'X' factor the author is talking about and it's something like "we're moving everything to Linux because that's what I do" and then you go 18 months without a release and the release you do make is a bugfix release to the old infrastructure, your customers leave and you get bought for debt.
- tester34 5y ago
  >a not so subtle humble brag. I'm not sure how could I rewrite it to avoid that feel, yet still include enough details to let people kinda evaluate whether it was reasonable or not

davewritescode 5y ago

I will say that avoiding "builder culture" probably saved my job. I joined a company that had migrated from an open source identity management solution to a homegrown one and it had ran perfectly well in production for many years but was getting creaky and didn't offer the features that were needed for a SaaS offering. VPs gave me the option of determining whether we migrated to an identity vendor or built our own. I've contributed code to popular OIDC implementations on GitHub, I've written code in widely used open source JWT libraries and I felt extremely comfortable working in that space so my inclination was towards building. The problem was to staff a quality engineering team was going to cost 5x the yearly price of the most expensive service and hiring other people in that space was proving to be very difficult and we didn't have time to take a year to build it and then start migration In the end I chose to go with a vendor and a smaller team to support a migration. We migrated in less than 6 months and a year later we ended up getting contacted by law enforcement about some activity that was happening our system. The auditing system on the vendors side allowed us to quickly diagnose what exactly had happened and let us give data to folks who needed it. Had that been our homegrown system, it would've been 100% an unmitigated disaster that took days to unravel. Had we decided to build it ourselves, there's no way we could've hit production as quickly as we did.

jimktrains2 5y ago
It really depends on how well the offering matches the need. My personal experience is that outsourcing core bits where vendors don't match exactly the needs ends up causing more overhead that if we had written the service ourselves.
cloverich 5y ago
One unwritten trade off that's in addition, not a knock on your stance just an observation, is that teams that build one quality product can often build another. To some extent when you choose to build you are also choosing to build a culture that knows how to build. So you have to think about not just the next thing you need, but the five things after that. I've worked at companies that bought literally everything, and what I found is the engineers there knew a lot about how to plugin to these various systems, but when they needed to build something complex together they weren't able to do so very well, or at all. Which might be fine, but you should head in that direction with intention.

bertr4nd 5y ago

I don’t know. I’ve been in the position of being an “X-builder” (I’m a dyed-in-the-wool compiler engineer), and I’ve been through something like what she describes, and nothing about the decisions involved seemed as obvious (or as malicious/egotistical) as she makes it sound. Like, maybe I don’t quite want to build an X, but other engineers and managers have their own reasons why we should, and after we discuss/argue for a little while, I decide that, OK, fine, I don’t entirely agree but I’m willing to concede that maybe I’m wrong and give it a shot. The thing we end up building is OK, but not stellar. We have an outage or two but they’re minor and not an ongoing headache. We have a few wins but it’s not the runaway success we hoped. Maybe we should have gone with Y instead of building our own X? Would it have been better that way? Am I a charlatan? I don’t know.

TheDesolate0 5y ago

well...I can't unread that waste of time, space, and typing...

seahawks78 5y ago

> "This is just one of the ways an unprincipled "builder culture" can backfire, particularly if you have people running the show who have no problem putting their own gains ahead of the company. They'll push all kinds of garbage if it makes them hit their goals and get a bigger bonus, a promotion, or some kind of shiny new position." In lots of places I have worked there is a particular term that is often used to precisely describe this situation, it is called POA aka "Promotion Oriented Architecture". In POA, you deliberately add a lot of accidental complexity to come up with a fiendishly complicated solution to a relatively simple problem. In short, the complexity stems from the solution space rather than the problem space.