25 ms·
Suits ignored IT's warnings, so the tech team went for the neck
- fennecbutt 3y agoBut they never learn. And this experience didn't teach em. What they get is destroying a business due to random/bad decisions, which is basically what most business majors seem to operate on. Then they get their golden handshake and leave to become a C level at another place where one of their buddies from their fancy business school is already parasiting. Most of them seem to barely get by by operating on the surface level strategy of "do whatever x creates y money more". When they succeed, it's their good job, when they fail, it's the bad job of everyone under them. They only look at surface level money, they'll take surface level "we can make 35% more if we do x" rather than the deeper and more considered "we can make 25% more if we do x but limited by y for z reasons because x has a negative impact on customers but mitigated by y, costing 10% but guaranteeing the 25 via happy customers and stable systems".
- devhead 3y agoreminds me of the convo i had with the ceo randomly after an episode of Silicon Valley and he asked about security because they hacked a fridge...i said yep, super easy; we had a bigger budget the next week :D
- bomewish 3y agoHilarious and all too believable. What on earth is wrong with organisations??
- politelemon 3y agoBean counters, among other departments, are distant from the technology powering their business. They only see tech as a cost sink and don't understand or care about the implications of their decisions. Actually you can scale that out to more than just the finance departments. Another example is in large orgs, tech teams don't meet customers and don't understand real world usage and subsequently don't care or put thoughts into the implications of their actions. Inefficiencies of scale? Would that be the term for it
- lucidguppy 3y agoGive a team a budget - if you achieve your goals and come in under budget you get half the under-spend as a bonus. Don't let the bean counters force budgets - incentivize them.
- tonyedgecombe 3y ago>What on earth is wrong with organisations?? They get too big.
- xvilka 3y ago> What on earth is wrong with organisations?? They are made of people.
- Spiwux 3y agoMaybe I'm too spiteful, but with leadership like that I would've just looked for a new job and let the entire ship sink.
- highwaylights 3y agoYeah but you’d be blamed for the problems in your absence and the warnings you issued beforehand would be magically forgotten.
- saiya-jin 3y ago... and it wouldn't matter a nano-bit, life is too short to care what others say about you in some office you will never see again 10km away
- highwaylights 3y agoKinda, but it could harm your future prospects though if you’re unaware you’re getting badmouthed in your absence and word gets round. One degree of separation is all it would take before people who’ve never met or worked with you have been persuaded you’re a problem.
- saiya-jin 3y ago... still my advice is that life is too short to care about such things. In my experience people (guys too) that indulge a bit too much in rumors are massively insecure folks with various other issues, and it shows. Have confidence in your skills and the rest are details, especially if we talk about IT where job mobility is generally way above average, you really don't have to kiss assess or be worried what others think/say about you to be successful.
- ejb999 3y agoface it - no matter where you leave from, or why you leave, your successors will almost certainly blame you for anything goes wrong in the first few months after you leave - you can't worry about it, because there is nothing you can do about it.
- littlestymaar 3y ago> Once usage ticked past 50 percent, the IT team again made a pitch for an ISDN order. And were again rebuffed – with instructions not to ask again until utilization was nudging 100 percent. This reminds me a lot about Covid perception by authorities: it's as if people in charge are all unable to make any kind of anticipation based on existing trend and have to see the disaster in front of them before reacting.
- xvilka 3y agoWhat's worse, fast forward N years into the future until the next pandemic, and all lessons will be dutifully forgotten, triggering exactly the same sluggish and misjudged response.
- lakpan 3y agoDon’t worry we won’t have to wait that long. However that won’t matter, people will protest any measure before politicians are able to react.
- littlestymaar 3y ago> people will protest Having a vocal minority protesting against something usually don't slow down politicians when they have clear goals[1], but they become a convenient excuse not to act when they don't want to. [1] I've marched in vain for many causes, most of which had much, much more intense protests than what Covid restrictions triggered.
- lakpan 3y agoLet's say you marched for "in favor of choice", but the politicians chose to ignore you because they're backed by just-as-large numbers of people who feel strongly the other way. With COVID, people who felt strongly about staying at home, were probably the minority, especially as time went on. If a new pandemic starts in 10 years, people will still be burnt out from the last one and resist closure in greater numbers, immediately.
- praptak 3y agoIt might be seen as underhanded but in fact it was the diligent and proper thing to do. Creating calamity in a controlled way is a tried and true way of being sure one is prepared for the time it strikes uninvited, see DiRT exercises.
- janandonly 3y agoPeople may claim this is an unlikely story, or entirely made up. But I don’t think it is. In fact, if you want to get anything done in a big organisation, you have to make management feel your pain. This is not being cynical, it’s simply how the world works.
- genmud 3y agoI don’t believe that this type of thing hasn’t happened, but the article talks about ISDN and even in the early 90s that wouldn’t be very common for a non branch office. Then they talk about traffic shaping/QOS and that’s when I call BS. I don’t think the Cisco 2500/2600 routers (which basically everyone used) even supported those features until many, many years later. Maybe they were talking about T1? I’m getting very r/thathappened vibes.
- mdp2021 3y ago> how the world works How communication is sometimes bound to work. Those who expect agents to perform all mental steps on their own to understand situations are in queue for harsh surprises.
- atq2119 3y agoThis makes a lot of sense - but how? By definition, management does a different job from individual contributors. How do you make a manager feel the pain of a messy code base, for example?
- therealdrag0 3y agoIf your edge manager doesn’t understand the value in clean code then you’re screwed. However that doesn’t mean it will be invested in. Clean Vs messy are binary descriptors that devs love to throw around. A lot of time messy code is not worth fixing. Managers have to make judgement calls. But what they should always let you do is marginally improve the code each time you’re in it, and especially let you invest in test coverage.
- ipaddr 3y agoMaking changes take longer. This will take 6 months today or 2 months in clean code base
- gostsamo 3y agoI like the stories on the register, but keep in mind that those stories are always told from only one of the sides.
- eastbound 3y agoAnd now they have double ISDN while the first one is used at 55%, so 26% capacity usage in total.
- threeseed 3y agoOr maybe the CEO knew something that the IT people didn't. Like for example the business was struggling and they weren't expecting the same growth i.e. their capacity planning calculations were incorrect and there was no need for an additional link.
- throw0101b 3y ago> Once usage ticked past 50 percent, the IT team again made a pitch for an ISDN order. And were again rebuffed – with instructions not to ask again until utilization was nudging 100 percent. If you know that installation of the new line will take N months, and traffic is creeping up X% each month, simple arithmetic will tell you how far ahead you should order more capacity. The instructions of waiting until 100% shows a lack of understanding of logistical capacity planning (perhaps through a bad explanation/communication of the problem).
- forinti 3y agoI once worked with this awful third-party software that caused tons of trouble for our clients. We had an in-house solution on the works, but some people wanted to renew the contract and keep using the same buggy software. Sometimes I would spend the whole day just fixing stuff on this system. I made sure all the tickets opened by our clients went to the people who wanted to keep the old solution. It worked, we moved to our system and we rarely have issues with it.
- throwaway154 3y agoI once worked with new awful in-house software that (thankfully) depended on 3rd party software. We had the legacy in-house system parallel running, which was even more awful. The legacy's awful came from it being unable to do anything it didn't already do without extensive re-wiring, and a team who believed if they don't share their knowledge of it they had a job for life. The replacement's awful came from it being inefficiently programmed with large DB latency due to the team's belief at throwing more CPU behind queries instead of re-writing queries. The teams sat next to each other but didn't speak. The legacy team called in a bomb threat to a package left under the IT director's desk. It was taken very seriously. While their cause was already lost, that didn't help them personally. The new system team eventually called Oracle, who came and re-wrote their queries.
- jl6 3y agoWhen you say called in a bomb threat, do you mean they reported a package as suspicious?
- Aeolun 3y agoI think it means they call the police (from a payphone if they are smart), and tell them there’s a bomb under the IT directors desk that’ll explode in x minutes/hours.
- throwaway154 3y agoThe call-in was an anonymous, apparently, call to the company's switchboard stating a bomb package had been left in the vicinity of the desk. We were evacuated to a separate part of the building for a few hours while it was dealt with.
- jl6 3y agoBeing able to clearly articulate risk, and the business consequences of technical consequences, is a key skill for IT folk. It’s possible that management at this bank were profoundly thick in the head, but it’s also possible the tech team didn’t explain it very well.
- cromulent 3y agoPeople also find it hard to understand or recognise exponential growth.
- Jabbles 3y agoPerhaps using a log-scale y-axis would help!
- jl6 3y agoThat’s part of the communication challenge. If the audience doesn’t understand exponential growth (kind of surprising for a bank exec) then you still have to explain the problem in a way they will understand. “The current growth trajectory means we’ll be out of capacity in a month, and it takes 3 weeks to install more capacity, so you’ve got a week to find the budget. If we don’t do this, customers will experience a slower service which may lead to complaints and reputational damage.” It’s also possible that the execs understood perfectly and were happy to accept the risk of the service being slower for a short period of time. Maybe the Internet-using customer base was tiny, or the service in question wasn’t critical, or the execs were handling general financial pressures leading them to want to sweat every asset until it was impossible to ignore, or maybe they just understood that “the internet being slow” was normal for the 90s and it would be unlikely to hit their reputation in the long term.
- komali2 3y agoOr the execs were sweating assets as you say, but also understood that they could lay the blame entirely on IT when things started breaking.
- Jeff_Brown 3y ago
- mrkeen 3y agoIt's too much to hope for, but I wish the right people could do this for the nonsense around e2e encryption these past few years. Imagine if politicians' https connections could be identified and automatically downgraded to http.
- krisoft 3y agoWhat would happen then? How would that lead to the change you hope for?
- mrkeen 3y agoI think they would stop pushing so hard to break encryption if the people sitting next to them in cafes started live-tweeting the politicians' internet traffic.
- krisoft 3y agoBut that misses the point. The politicians are not arguing for a world where nobody can encrypt. They want strong encryption which can only be opened by two groups: the intended recipient, and the authorities. We know that such a system will inevitably lead to other groups also gaining the ability. Rich criminals, security services all around the globe (friendlies and the enemies too). Either by bribing, or impersonating authority figures or by stealing the backdoor keys. What is at dispute with the politicians is the probability of this happening. How would a service voluntarily leaking the private information of politicians convince them that we cannot build a backdoor into encryption without that backdoor proliferating widely?
- bartkappenburg 3y agoMy first reaction was: ha, funny and a nice way to leverage your power as an IT person. Kind of revenge of the nerds towards the ‘suits’. But then I realized that IT abused their power and lied to the suits which is outright wrong. At the end of the day you’re in IT and the suits are in charge. Don’t like that? Find another job or grab a position that gets you this power. The problem is: you don’t always oversee the consequences of your roque actions. In this case the stakes were low, but where do you draw the line? If I were a suit and found out that you deliberately throttled the connection to force us in a decision I would be furious. That is simply not your call.
- te_chris 3y agoThat sort of bootlicking has a history of leading to ‘problems’.
- threeseed 3y agoYour job working for a company is to respect the decisions of your manager. Choosing not to undermine them is not bootlicking. It's called being a professional.
- hibernator149 3y agoYour responsibility as a professional is towards the customer and company. Not towards incompetent individuals. Don't call yourself a professional when you are reday to do unprofesssional work at the first sign of resistance.
- threeseed 3y agoYour customer and company doesn't approve your paycheck. Your line manager does. Going behind their back because you don't agree with their decisions or actions is not professional.
- A_non_e-moose 3y ago
- i386 3y agoAustralia? Banking? I assume the early 2000s? Absolutely plausible.
- threeseed 3y agoWilfully sabotaging executive's network connection in order to get more money for another link because your boss couldn't justify it properly is inexcusable. These people deserve to be fired and the behaviour is disgraceful. I can't imagine working with anyone like this.
- dsr_ 3y agoSabotaging? Clearly not. They applied proactive network management to fairly allocate an expensive resource according to the anticipated needs of the company.
- dickersnoodle 3y ago[flagged]
- lakpan 3y agoI’d agree with you if the problem wasn’t real, but we all know that utilization at 50% will cause trouble. I say these people were skilled in saving the company’s ass. They could have just as well waited for the “I told you so” moment, which is what I learned to do.
- al_borland 3y agoThe problem with waiting for the "I told you so" moment, is all the people you warned conveniently forget about all the warnings they were given. Then if you bring it up and say, "I told you so," or provide proof, you get labeled as petty or focusing on the past instead of fixing the problem in front of you. I mostly keep the "I told you sos" in my head for these reasons.
- lakpan 3y agoYeah, at that point I either kindly "add context" with an issue from last year or quietly bask in schadenfreude.
- NoboruWataya 3y agoPlot twist: This was 1999, the brave techies convinced the dastardly "suits" to double down on the internet revolution, the dotcom bubble burst and the bank went bust.
- devhead 3y agoat least they survived Y2K :D
- dreamcompiler 3y agoLooks like BOFH is writing under a pseudonym. https://www.theregister.com/offbeat/bofh/ https://www.theregister.com/offbeat/bofh/
- deleted 3y ago[deleted]
- brauhaus 3y agoSometimes it's hard to communicate technical problems to non-technical people. They article calls that as going for the neck, but I think of it as effective communication.
- freetanga 3y agoI get this is probably a reflection of corporate culture in the 90s (a bit cartoonish, I think) but I think this attitude does little to make Tech taken more seriously and “at the table”. Instead we self-reinforce the image of childish kids doing mischief and pranks when the things don’t go our way. The CTO was at fault in this story, for failing to convey the gravity and for being used as a doormat if things went bad.
- throw101010 3y agoBold of you to assume there was a CTO back in the days of ISDN... even in a bank.
- devhead 3y agoi suspect they meant the CIO; i agree, it was their role to relay the real need here not just push a request over without already securing approval from the rest of their c suite peers.
- lr4444lr 3y agoAh yes, "managing upward"...
- hamdouni 3y agoReading some comments, like 'this is wrong because your job is to respect hierarchy decisions', reminds me how corporate still are military minded...
- dijit 3y agowhich is ironic, because a large part of current/western military leadership is "coming up with the plan together"- because it builds buy-in from the lower ranks.
- Spooky23 3y agoLike LARPers everywhere, corporate dudes emulating the army tend to project what they feel, which isn’t very informed.
- sofixa 3y ago> military minded There are different military philisophies, not all are top->down strict hierarchies. E.g. in the Prussian army of old, missions were communicated through objectives (I want to achieve X, your part is to do Y, while these other guys will do Z), and execution is up to to the local officers who actually see the things as they are locally and have all the knowledge about what they need to do, how it fits in the wider plans, and are best placed to decide how to do it. Furthermore, if an office or NCO disagreed with their direct commander's orders, they could go above their head to their commander's commander to protest. Add in the general staff system where staff officers had to service in command as well as part of their training, and could counteract commander's orders, it made for a very robust and flexible system capable of adapting to evolving situations and where officers didn't hesitate to argue with their superiors, go above their heads, and even disobey orders if they really thought it made sense.
- unsupp0rted 3y agoWhat happened to the relationship if one went above an officer’s head? Would they be on their direct superior’s “hurt this person as much as possible from now on” list?
- lloydatkinson 3y agoA tragically believable story.
- quickthrower2 3y agoWhat I don’t get is “50% utilization” as if you have a faucet on half way. Surely with a single pipe and a spikey load you would get occasional performance degradation anyway? I would imagine that “shit experience” time percentage driving up sharply (depending on the distribution) so that even 60% would be intolerable. For comparison imagine a bar decides to have 1 server based on average demand. Then Friday night comes.
- regularfry 3y agoYes, latency will depend on the variability of the load. That's Kingman's Formula for you.
- worewood 3y agoHe didn't go for the neck. The corporate view was that the executives detected the problem, allocated resources to it, and were applauded when it was "solved". Heck, some may even have got a promotion. Going for the neck would mean documenting everything and letting the bomb tick, and when it exploded make it clear to even higher-ups what was happening. You'd probably go as well but that's not a good place to work anyway.
- em-bee 3y agoread the dailywtf for more and better stories like this
- deleted 3y ago[deleted]
- Finnucane 3y agoThis works for getting attention from IT too. I’ve had issues that IT said they couldn’t fix, and I just put tickets on the system for every instance the issue came up. Then, oddly fixes appeared.
- bob1029 3y agoI think the real problem being illustrated here is that the IT side of the shop needs autonomy to do what it knows to be correct somewhat independently of the business side of the shop. Ultimately, trust is what governs how much oversight you should be expected to suffer in an organization. In my shop, I don't have to ask for permission to do anything unless it touches our customers in some way. If I see a machine that is struggling or a SaaS plan that is reaching quota, I simply upgrade. Every now and then someone will ask me about a new bill or increase, but I don't have to subject myself to multiple depositions to move the ball forward. The suits should consider the downsides of creating an environment where one has to plead for permission to conduct the most trivial tech work. How much innovation is going straight into a burning trash can over convoluted change policies that were established in some ivory tower a decade+ ago? Can we somehow reframe the organization in a way where the business is modeled as a customer of the IT team? That seems to me like a much simpler way to think of things, because at the end of the day if the business fails, the IT shop is has no raison d'etre.
- vladd 3y agoIf the business is the customer, upgrading a machine increases the customer price hence it requires communication as it changes the benefits/cost value proposition.
- Aeolun 3y agoNot upgrading a machine affects quality of service. If they don’t trust that that’ll happen until they can feel the problem themselves, what do they have an IT team for?
- bob1029 3y agoThe trick is to wrap your technology with a service-style abstraction so you don't have to communicate all of these painful little details. Then, all of those details get rolled up into something like a quarterly report. The terms of this relationship (contract) should be approximately along lines of "you provide us a service with a certain level of quality and in return we grant you an annual budget of $X". At a certain point, having detailed analysis of every last expense will cost you way more money than it will save.
- cvccvroomvroom 3y agoI had a Cisco 1604 ISDN router setup at work to autodial to save money rather than being connected all of the time. Unfortunately, I discovered IBM AIX when the web browser package was installed contained periodic telemetry dial-home to Big Blue every hour or so that prevented the lab network from idling happily until I added a firewall rule to the router to block that sneaky shit. Even in the late 90's, Microsoft, Sun, and Novell weren't as brazen about telemetry as IBM.
- ramses0 3y agoEarly in my career, I'd cobbled together an internal server which effectively did monitoring (RRDTool) against the internal BugZilla(!) server. The end result was speedily cached charts and graphs of bug counts by team, by project, by release, etc. Over time, the system had grown to be basically _THE_ source of truth for the release management process for a Top-500 website of the era. Go/No-go decisions were based on looking at the trend-lines for bug graphs. Projects were merged to "trunk" only if their P1 counts were low enough. When I decided to leave I wanted to make sure things were left in a sustainable place, and asked around for who to transition the server to (it was a random beige box in the QA Lab, circa 2005). "It's Debian running MySQL, it has an auto-update script which will bug you if you need to update packages, here's how to restart the server, here's the root password, here's how to update/deploy it, docs are in this wiki." I'd given 4 weeks notice, and week 2.5 rolls around and even though I'd kept asking around about who to transition this pet project to, nobody was swinging by my desk (so to speak) to pick up the responsibility. So I made a little script: `@hourly: if (( $RANDOM % 2 )) ; then ifdown eth0 else ifup eth0; fi` Suddenly people started showing up at my desk... "What's going on, can you fix it, what's happening?" Sorry, nope, can't fix it, the network on it is down... see, I can't ssh into it, by the way, this is what could very easily happen once I've left. Just wait an hour or two and it should come back online. If you send someone over who you want to be taking care of this machine I can walk over to the keyboard with them and show them how to fix it. Once they committed to sending over someone to transition to, I turned down the BOFH to a 25% chance of 15 minute outages...and all was right with the world. Later on that week, a really good guy, one of the main end-users of the software (sorry Mike!) came over and explained the relative immaturity of my actions, but I couldn't help but point out their effectiveness. In retrospect, it was an internal "API brownout" but that didn't have a name yet. The concept of "controlled unreliability" (eg: chaos-monkey) being a tool to increase business resiliency is something to take to heart, if even only unofficially.
- Aeolun 3y agoSo did you eventually transition it to someone? Wouldn’t be the first time that I’ve seen someone commit to something and not do it without notice. Or learn to live with random 15m outages because it’s apparently less frustrating than doing something.
- spandextwins 3y agoI just love awkward English.
- crazygringo 3y agoHonestly, I don't believe this story for a second -- and believe me, I've witnessed my fair share of corporate incompetence, so it's not that. But it's that if there's anything executives are good at, it's looking at graphs, extrapolated trend lines, and setting deadlines before disaster strikes. This is basically one of the core, day-to-day competencies of the executive suite, because there are so many things that can go wrong regarding finances, infrastructure, legal, etc. There are plenty of things executives can be bad at managing, e.g. when it comes to tech debt or user experience or company morale -- especially anything subjective or hard-to-measure like these. But capacity planning for any measurable resource, be it an internet pipe or otherwise? This is MBA 101 stuff. So this particular story just reads like a fantasy of "tech people smart, business people dumb". If an executive team couldn't handle something this simple, there's no way they could have been running a functioning, established bank in the first place. It just doesn't add up.
- croquemonsieur 3y agoI've worked for several large corporations now directly supporting c levels, and their primary skillset is self confidence - a belief that they deserve their position.
- underyx 3y agoI agree but I think the more likely disconnect here is that the tech team couldn’t be bothered to properly present their case. They might’ve just passed on a note with a vague “we will run out of resources soon” or something.
- darreninthenet 3y agoGiven when this was (early 90s going by the story?) I could easily believe it... nowadays there would be an executive level senior manager ultimately responsible for "run" who would actually have a clue on forward planning for capacity but back then running capacity to its limit, being told several times "no" along the way and then being blamed when it all fell over were very much the norm, especially in banks, in my experience.
- asylteltine 3y agoBoomer executives especially at banks are like children. They only understand what they see with their own eyes. They can’t think in abstract terms.
- wazoox 3y agoAh yes. In my first job the CFO repeatedly rejected IT manager's requests for a proper backup system of the AS/400 that held all of the business (ERP, CRM, accounting) because backing up the work of 300 people on 8" floppies simply didn't work. One day, a major disk crash occurred, and the whole company went frozen for a couple of weeks (couldn't fulfil orders; couldn't manage support requests; couldn't make any proper commercial proposal; couldn't find a customer's number or address from the database; etc). IIRC some disks were even sent to Kroll Ontrack. After that disaster, of course, a proper backup unit was bought...
- BizarroLand 3y agoHappened to me too. Worked for a remote 2-year college not too long ago that had all of its data on an ancient as/400 system that went down every couple of weeks for a day or two. No backups, the tape drive was busted. No replacement parts for things like the bleeding edge 10mbps NIC. I kept pressing that we need to replace it or migrate off of it to a cloud server, kept getting denied. "It's not in the budget" "It would cost too much". I mean, we're talking a few hundred dollars a month for the minuscule usage we had, compared to the massive risk that if the system went down it would cause the entire college to collapse and possibly be permanently shuttered. Finally the ball dropped and it went down. For days you could access it at the main terminal but it would not speak to the network at all. People are starting to panic. Talks of bringing in an AS/400 repair specialist at over $100/hr are being floated. Finally, I took a last ditch effort to fix the nic and managed to pull it off. I let them know that I had managed to fix it and they all breathed a massive sigh, but before they could finish I let them know that this might be the last time it comes back up and that we need to back it up somewhere before that happens. 6 weeks later we had a nice shiny cloud based AS/400 and I held the funerary rites as I shut down that old lumbering beast for the last time. Final clock, nearly 25 years of service.
- cendyne 3y ago"We need a test environment!" "We don't have the budget for it." Months later, a student developer runs a delete query in production with the where clause commented out. Oops, an entire table got wiped. Well at least there are backups... except a background job noticed and quickly created 4 years of invoices and emailed every past customer to pay again. A test environment arrived soon after.
- ris 3y agoThinking, or even knowing you're right isn't sufficient grounds for concocting fraudulent ways of proving it. This approach runs the risk of blowing up in your face if discovered.
- Modified3019 3y ago“Regomise” kinda threw me. Here’s what it’s supposed to mean: https://english.stackexchange.com/a/546162 https://english.stackexchange.com/a/546162 >The Register itself uses the term Regomiser as a user name generator for its registration. If this contributor is a "reader Regomized as 'Felix,'" it would mean a reader registered and was reassigned that random name.
- voiceblue 3y agoIs it regomiser or regomizer? Or has the term remomize been regomised as regomiser?