5 ms·
If you succeed you will fail
- anotherhue 3y agoI've seen versions of this, anytime someone builds on lambda and then points it at something not built on lambda. The impedance mismatch is usually too great and speaks to some serious deficiency at the architecture stage.
- doctor_eval 3y agoThey said the system could be brought down by a RPI over DSL. My guess is that the Auth system, which became the bottleneck, authenticated everywhere at once, which would be super slow. Not sure what lambda had to do with this. The whole SAP thing reads like a legacy nightmare.
- teaearlgraycold 3y agoI don’t think lambda is the problem. But my guess is that using a highly scalable system that forces you to make convenience tradeoffs for that scale isn’t the best solution here. A monolithic app could handle the scale required with better devex. But I also don’t know much at all about what’s going on in that system. If the serverless tradeoffs has no real downsides for their specific use cases then all I said is moot.
- doctor_eval 3y agoYeah - I’m not personally a fan of the specific tech discussed in the article - 100 proxy requests/second could be handled by a RPi, but I suspect it was more complex than a simple proxy - but it was pretty clear that this wasn’t really the issue. The point was that the launch was predictably going to fail, it was predicted, and then it failed. And this was due to the kind of arrogant, disconnected management decisions that are sadly very familiar to many of us here at HN.
- thirdplace_ 3y agoreminds me of the similar sentiment that a failure may be required in order to get all parties to accept that a change/improvement is needed.
- 2600guy 3y agoCause one person with the support of c level makes a decision, we can all stand back and watch 70+ years of success crash and burn in flames.
- vsareto 3y ago>I also learnt to only become as involved and care as much as the customer. If they aren’t willing to go that extra mile then why should I? There was probably a good portion of that army of ops and developers that felt the same way. And even if you can move mountains to point fingers and clean out all of the bad people, train the average ones, and keep the good ones around, you will likely get little compensation out of it, so no one does that. Even then, it's hard to blame incompetent people getting jobs and holding on to them. There's a lack of standards for skill and training so naturally people slip through the variety of bespoke vetting processes that companies come up with. There's also some economic reasons why people hold on to jobs for far longer than they should, even if they know they're bad at it. Large amounts of bad complexity slowly erodes morale, but bad decisions like this one can set it on fire. Accruing bad complexity may not have any solid failures to trigger reassessment, which is why it often becomes a problem and stays a problem. It's an obvious problem for new people with fresh perspectives, while incumbents might have gotten used to it.
- ilaksh 3y agoPoor management from executives seems to be the norm. - They failed to get the internal developer to buy in to the project or really work with the other team. - They insisted on launching even though it was explained that the system wasn't ready. - Insisted on having a launch party when failure was expected.
- JackFr 3y agoAs a newly minted bad manager, it rankles her that there is somehow a moral judgement of bad managers. Developers, QA, product can all be incompetent, but when a manager is incompetent, he’s “evil”. In all seriousness, incompetence should never be rewarded and bad behavior should be called out, but in nearly every case we’re all really trying to do our best.
- whoknowsidont 3y agoThe difference is that it's managements job to identify, correct, or quarantine all those bad reports. There's also easy and cheap ways that devs, QA, and product are labelled as "incompetent." Proving that management is incompetent is a fairly large task in and of itself, and the people that can hold them accountable are all peers who generally wouldn't dare to "rock the boat" so to speak. I've termed this whole set of phenomenon as a "managerial crisis", and it's been written about extensively since the last 60's.
- YZF 3y agoAny good references you can recommend?
- heisenbit 3y ago> nearly every case we’re all really trying to do our best. If you sitting in the same boat this is a reasonable approach most of the time. Even then you may have to deal with an occasional irrationally acting person. However if you are in another boat e.g. providing services or products there are a lot fewer barriers for misbehavior and some may be even rewarded. Win-win and win-loose are different games and deluding yourself in what game you find yourself in a quick way to fail big.
- ChazDazzle 3y ago> I learnt a few things from this. The first is, some companies or organisations within them need failure in order to progress. Encountered a similar situation a few years back while working for a government client. My advice to the team was that we needed to “let the train wreck happen.”
- gretch 3y agoI agree with this, but I think you also have to ready to document the course of events and (in the most diplomatically possible way) say “I told you so”. Otherwise the people who are accountable for the failure will say “oh who could have seen this coming” and refuse to learn anything for the next instance.
- pbronez 3y agoYeah this stood out to me as well. I think the reason for this is that observed production failure is _certain_. Hard fails are undeniable. They obviously need to be addressed and obviously deserve resources. The amount of deserved resources can be clearly calculated by projecting the concrete, observed costs of the failure forward in time. Before prod goes down, there is much more uncertainty: - even an expert engineering assessment has some level of uncertainty - engineering may not have a full appreciation for the business context of the work, and might over-weight technical issues relative to other concerns - if the engineers are contractors, or otherwise organizationally distant from the experience owners, that inserts a trust gap which further increases uncertainty - the business owner’s projections are themselves uncertain. Is the expected launch volume really that high, or is it aspirational? - the costs of failure are uncertain too… if the system goes down, how hard will it go down? What will that actually cost in lost revenue? Fuzzier stuff like brand reputation is even harder to quantify. Meanwhile the costs of paying the contracted development team another 2 months on the same project are quite concrete. The team already spent significant political capital to force a change on an incumbent team. Now they’re saying they want more money because it still doesn’t work?? The big open question is - what was the cost of the failed launch? How long did it take to get the system back up and running at scale? What did it cost in terms of user loyalty? How does that compare to the concrete cost of holding launch until the auth system was upgraded? Different people will answer those questions in different ways. What matters is how the customer answers those questions, whether their bosses believe that answer, and their bosses judgment of the overall situation.
- ttr2021 3y ago"Fall back on blaming the process when failure comes around and avoid ever pointing fingers or owning it." Some organisations can do this, but I've see plenty that might outwardly try to avoid pointing fingers, but you can tell that despite warnings given, they do blame. I told several leaders that one of their systems was literally on the brink and we were fighting fires on it every other day, and the processes in place were horribly broken. 12 months later shit hit the fan, my feedback was that I wasn't proactive enough, and they basically threw me and my team under the bus. This despite budget under cutting, limits in hiring, enormous optimisation, education of other teams, research. Just getting 'No' or no real outcomes all the time on any escalation. The leaders did this serially across the business, and lacked ownership on this aspect to even give people the resources and autonomy to do anything about it, yet would come looking when it came time, to throw other teams under the bus like this
- aledalgrande 3y agoReally sad to see people like these, in most companies I worked at, in leadership positions.
- isaacremuant 3y agoThey set themselves up for success no matter what (take credit for hard arbitrary deadline being successful or blame whoever can't meet it), which clearly leads them to be in those leadership positions. It's very common.
- jimmaswell 3y agoLesson: Get everything in writing, save it, and forward it to the person, their boss, and whoever else you can CC when someone lies about it.
- laserlight 3y agoI don't think not having evidence was the problem that GP had. Upper management asked GP to be more proactive. So, they expected GP to be more assertive or take measures regardless of management's approval, which would be doomed still in other ways.
- smitty1e 3y agoI tell people that we do great work for its own sake; without parental affection; knowing that, brilliant or bogus, the underside of the bus beckons.
- adamc 3y agoWhen you see that people are going to get thrown under the bus, it is prudent to not be in the vicinity. You can't stop managers from doing it, but you can stop volunteering to be one of them.
- don-code 3y agoAs an engineer, I've never personally understood the desire to release anyway, even when a system has known critical deficiencies like this one. Sure, perfect is the enemy of good, but I'd also argue that inoperable is the enemy of good - it only serves to erode trust in your product, if your team chose to release an unusable product. Some years ago, I worked on a system not quite as bad as what the author described, but close. We released a new product with a known, quite bad security vulnerability (I'd made sure our product team was _extremely_ aware of this), as well as no monitoring to speak of. The deadline had been communicated for around one year, but nobody had ever really discussed the significance of the date, other than it was what we were all death-marching to, and we needed to deliver. What did that date turn out to be? The head of product management's birthday, which was revealed to the rest of the company on the highly-celebrated the launch date. People were just kissing ass. I left several months later. It feels unconscionable to me that a company could have launched an incomplete, insecure, customer-facing product just to give a birthday gift to a leader, but I suspect this sort of thing is common.
- nicbou 3y ago> I've never personally understood the desire to release anyway I work alone with no deadlines. Sometimes I have to release something or I'll never finish. Perfect is the enemy of the good, as the saying goes. I often realise that half the "blockers" are actually nice-to-haves that I'm willing to complete much later. However I don't consider the thing finished, only released. In a dev team, the post-release bug-fixing and refactoring never takes place. Managers immediately move on.
- Tomte 3y agoI used to work in a company where important dates (inauguration of new development building, important product release etc.) were set to the birthday of the owner/CEO's late husband. Come what may. If your product was to be released somewhere in that half of the year, this date it was.
- geosh 3y agoIt's like a communist party opens a dam or a factory earlier just because it's the party's anniversary.
- nine_zeros 3y ago> I also learnt to only become as involved and care as much as the customer. If they aren’t willing to go that extra mile then why should I? This is what I use at work to keep myself sane. If the highly paid VPs, directors, managers don't care about something, why should I? This leads to a lot of friction with manager where they want to put blame on individuals, months after they ignored the warnings. But realistically, it is their fault and they will get fired soon enough if they try to make everything a problem of engineers.
- GianFabien 3y agoIn my experience the engineers get blamed and often fired if they push back. The managers seem to acquire immunity if they can pass the blame and play politics.
- nine_zeros 3y agoYou are right. Engineers are the ultimate scapegoat. Which is why engineers hate management in this industry. Nobody likes "leaders" who will throw reports under the bus when the time comes. The only way out is for engineers to spend more time recognizing a manager falling off the cliff and switching teams/companies. Doing actual work is not enough to protect one from politics.
- taneq 3y agoYou keep going until something stops you. So if you do everything right and solve every problem thrown at you, that just means you're going to push harder. Eventually something's going to break. This isn't 'failure', it's just how development, business, really everything works.
- GianFabien 3y agoI have worked with well over a dozen organizations with ERP systems from SAP, Oracle and IBM. When organizations are large enough to need such systems, the management has grown to the level of being driven by politics and hubris. Any warnings by technical staff are rejected because they contradict the management's belief that they know better.
- weekendvampire 3y agoNot to be that guy but that should be a comma and not a semicolon.
- rgmerk 3y agoWorth considering: why did the managers ignore the writer’s warning of the apocalypse when the system went live? One possibility that comes to mind is that they’ve heard dire warnings before from other engineers which have turned out to be exaggerated. Engineers do have Chicken Little tendencies sometimes, and the long term results can be similar.
- nine_zeros 3y ago> One possibility that comes to mind is that they’ve heard dire warnings before from other engineers which have turned out to be exaggerated. Tells me that management is incompetent. What exactly are they managing if they don't understand engineering estimation and consequences?
- jaggederest 3y agoWhen warnings are taken seriously, and no disaster happens: Clearly, the warnings were unnecessary. When warnings are not given and a disaster happens: clearly the team was incompetent and should have seen it coming. When warnings are not given or ignored and no disaster happens: they were overblown and pointless, why do we even have a disaster recovery team. When warnings are taken seriously and a disaster still happens: it's the worst of both worlds. You can't win in this situation. No combination of warning/not warning and disaster/no disaster results in success. If you can't trust your managers to cogently weigh the risks and costs, there's nothing you can do.
- rgmerk 3y agoMy point is that engineers sometimes view anything that goes wrong with their tiny corner of the system as a disaster, which is not always the case. My (engineering) boss is more risk-tolerant than I am, but after 30 years in the business I respect his judgement that a particular risk is worth running.
- b800h 3y agoWhere was architecture in scoping?
- Sparkyte 3y agoJust commenting on the "If you succeed you will fail.". Failure is envitable and to succeed at something you have had to have had experience encountering a problem. If you succeeded first time encountering a problem it is likely you'll never know the solution to the problem and thus it is envitable failure will eventually become of it. It is enough to say success is built on a mountain of failures. Only difference between something being a failure and it becoming a success is that attempts made to produce a desired result.
- ajani 3y agoCompanies usually don't have all their parts aligned or even functioning properly. And you cannot change that. Your work isn't just the product, it is navigating an imperfect structure to achieve what you need to, despite the fact that the imperfect structure is the one contracting the product/service, and is (mostly unwittingly) standing in the way of that. It's not fair, it's not fun. It is what it is.
- calderknight 3y agoThe sooner that you find success the sooner that you'll fail So don't, don't, don't Don't play the chords of fame
- riazrizvi 3y agoA strangely, poorly performing authentication system with a good one a few weeks behind the launch. An oddly configured circular fail over strategy. The thing I’ve noticed about entrentched employees who don’t perform is that while they are not so good at creating/building things, they are really good at pulling strings to make themselves look good at the expense of others. I don’t think it was a great look for them that you were brought in to build this, and once you start pointing out flaws in other people’s systems you become a walking target. I’ve made this mistake too many times in the past, determined not to do it in the future.
- pdimitar 3y agoOkay, you will not do it but the underlying problem remains: you'll take the blame for other people's mistakes. How do you deal with that?
- riazrizvi 3y agoThe situation is you are brought on board to do work, and you have in-house collaborators and you'll take the blame for their mistakes? Step back and find an outcome where your work doesn't harm their reputation/security. So how do you make them into allies? Or how do you not matter to them? Figure that out...
- pdimitar 3y ago> The situation is you are brought on board to do work, and you have in-house collaborators and you'll take the blame for their mistakes? People really are like that, yes: Monday the boss says "X is now responsible for Y" and lo and behold, Tuesday EVERYONE already says "Y is not working well, X, do something, it's your fault!". > Step back and find an outcome where your work doesn't harm their reputation/security. So how do you make them into allies? Or how do you not matter to them? Figure that out... I have figured out part of that and I was genuinely asking if you have additional insights.
- riazrizvi 3y ago
- duxup 3y agoIt’s interesting how such things impact an employe. There are times where things have to fail in order for change to happen, but employees also generally want to make things that are meaningful, and work. It makes the process to inevitable failure really frustrating / demoralizing for the employees.
- datadrivenangel 3y agoI love watching fake deadlines go past. Good, cheap, fast, pick at most two. Maybe one. Sometimes none. Occasionally negative.