23 ms·

“In roughly two hours, 1647 devices are about to be wiped”

603 points by terom 4y ago

0xbadcafebee 4y ago
It's like a dog owner whose dog has been shitting on the sidewalk for so long that the owner eventually can't walk on the sidewalk without stepping in their own dog's shit. But they will absolutely blame the dog.
smcg 4y ago
"Hey, there's a rake up ahead. I'd recommend not stepping on it." "Ow! Why did this happen?"
- hprotagonist 4y ago
  doctor, doctor, it hurts when i do $this well don’t do $that then
  - Oxidation 4y ago
    Private healthcare: there is only one dollar sign and it changed hands between those sentences.
- gtsteve 4y ago
  Yes I remember this from my consulting days well. Me: If we do A, B will happen Client: Do A anyway, we'll deal with B later. (time passes) Client: OH MY GOD B HAS HAPPENED
  - aidenn0 4y ago
    A friend of mine started writing his predictions and putting them in sealed, dated envelopes. He said the 3rd time he pulled one out Carnac[1] style, management actually started listening to him. Nobody really got "I predicted this 18 months ago" but the theatrics apparently drove the point home. 1: https://en.wikipedia.org/wiki/Carnac_the_Magnificent https://en.wikipedia.org/wiki/Carnac_the_Magnificent
    - ahazred8ta 4y ago
      A dirty trick, but an effective one. May the fleas of a thousand camels infest you, and may your arms be to short too scratch.
charles_f 4y ago
We need a retrospective, how could you let this happen?
- tenebrisalietum 4y ago
  1. You told us to do it. 2. We did it. 3. It was done.
  - Oxidation 4y ago
    4. Bonuses all round?
pessimizer 4y ago
My reaction to this tweet was surprisingly intense. It's like the plot to a horror movie, or the 5 minute opening credit montage of a post-apocalyptic film.
dijksterhuis 4y ago
> Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent. I see that life as enterprise service desk hasn’t changed much. “Nobody tells me nuthin!” Shout out to the ever under appreciated service desk folks out there.
- ilyt 4y ago
  "We need that machine tomorrow?" "Why didn't you tell us you hired something new?" "They work here for 2 weeks now"
  - dylan604 4y ago
    are new hires really being referred to somethings now?
    - smiley1437 4y ago
      That's a pronoun I haven't seen yet
      - mrguyorama 4y ago
        Honestly it's a way more accurate reflection of how managers see employees
        flerchin 4y ago
        :feels:
        dylan604 4y ago
        ?? Isn't feeling the employees an HR violation?
        kevin_thibedeau 4y ago
kerblang 4y ago
I would go with an incremental progression of something like 100 random devices a week for 17 weeks, so that people see the tidal wave eating others and suddenly "get it". Less overwhelming for the service/support desk folk too.
- Oxidation 4y ago
  Or start with the "most critical" devices: those belonging to the highest-ranking users. They're the high-value hacking targets so it's only reasonable. Might unstick a few wallets better than hitting mostly rank-and-file for something they have no control over.
  - gumby 4y ago
    Or incrementally degrade service (lock out web browser, then email...)
  - jdironman 4y ago
    The problem with that is, it's most likely those devices which are already compliant / up to date.
  - basch 4y ago
    If you are this close to a deadline, it doesnt really matter anymore. There is no avoiding the iceberg. You can fix your organization for next time, which is what the org believe it has initiated.
    - Oxidation 4y ago
      Well, yes you don't do this two hours out, you'd need to do it some time in the 13 months they knew about it for it to work. But at the end of the day, they did the job they were paid to do and were clear about the looming impact. It's not their job to also wipe their clients' metaphorical bottoms when they were ignored.
- kneebonian 4y ago
  Based on the thread I think it was Legal and Compliance forcing it. It may have been regulatory required. Also they say they are managing on behalf of their client, so their position as mercenaries is just to follow orders.
danjc 4y ago
As much as I hate the concept, sometimes brownouts are the only way to force people to make a change and avoid complete disaster on D day
- danpalmer 4y ago
  Why do you hate the concept? I've seen it done a few times in different areas and I think it's a neat way of notifying people in large organisations/ecosystems. As long as the brownout is done after the support period, i.e. it's "contractually" ok to do so, it seems like a good idea.
  - jaywalk 4y ago
    Brownouts can be very useful in finding impacted systems that may have been overlooked as well. Last month, I had to do some updates because a customer's API had moved to a new URL on a new server. My team and I identified (what we thought was) everything using the API and did the updates. A week later, the customer notified us that they were still seeing some traffic on the old URL, but all they could give us was the IP address it was coming from. Unfortunately this IP address belonged to a server that hosts a lot of our smaller applications, so it didn't really help locate the offender. So I just added a firewall rule to block access to the IP address of the customer's old server, and sure enough I heard the scream 15 minutes later. Removed the rule to get that application back up and running, got it updated to the new URL, and all was good.
- nunez 4y ago
  Scream testing is underrated for sure.
twawaaay 4y ago
The CEO will have a convo with two of his managers that will resemble me when I talk to my two sons when they get into a fight and start pulling out all of their excuses.
jmholla 4y ago
Mostly unrelated, but I hate websites like this that think they're way of handling arrow keys for scrolling should be implemented over how every other web page does it. I lost my place so many times when I mindlessly tried to scroll again with the arrow keys.
- gambiting 4y ago
  And even though this website only contains his posts and nothing else, individual posts are minimised and have to be clicked on to unfold, which scrolls the entire page for me on mobile chrome and I have to find it again. It's a usability nightmare.
  - fragmede 4y ago
    This website is a well known infosec Mastodon host. The linked site is to a specific person's feed but the site local feed, with many other individuals' posts is at https://infosec.exchange/public/local https://infosec.exchange/public/local
  - LordDragonfang 4y ago
    Top right of the content bar has an eyeball icon with "show more for all", which expands them all at once, but agreed, this isn't great UI or UX (still better than twitter though!)
  - Krisjohn 4y ago
    At a particular point the author specifically indicates that for the sake of reducing stuff that many readers might find uninteresting or too frequent, they will be condensing larger posts. > Alright, things are moving faster now, so I might condense-toot to avoid pollution. But I guess the chronically online need something to complain about.
    - gambiting 4y ago
      It's not the condensation of content I'm complaining about, it's the fact that when you try to unfold a post the entire website scrolls and puts you somewhere else. That's just bad UI.
AtlasBarfed 4y ago
I'm guessing upper management prioritized the update of these devices with downstream management rather than overburden them with other stuff. So in the end, this is just one piss poor managed division abusing another piss poor managed division. Who gets the heat? Probably the lowest level people. Why "wipe" them? That seems unnecessarily punative. You can see the "don't give a shit I work with a predatory organization" oozing from everywhere. The security guy is trying to claim that they've sent out many many notices, but really this is just an excuse to abuse other people in a machiavellian abusive organization. "Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent." Whoops, missed an email and a meeting in there bucko. And it's the SECOND company where this was "implemented" or "specced"? This sounds like someone checked a box or compliance or ass-covering upper management slid this under the table, but all the people it ACTUALLY AFFECTS didn't get any input or opinion on the matter. And when push came to shove over funding it that person had probably moved on to bigger and better things. So since you get to do it, you seem to be gleefully doing it. Great job.
- ilyt 4y ago
  > Why "wipe" them? That seems unnecessarily punative. There was something in the thread about the devices coming out of support by manufacturer, which was already extended by a year.
- Blackthorn 4y ago
  What exactly are you suggesting this person does? The policy to wipe clearly came from the company's compliance department. They warned them over and over what was about to happen, and went above and beyond doing it with multiple meetings and phone calls.
  - whstl 4y ago
    Exactly. In this situation all you must do is warn people of the risks and document, document, document. Which the article writer seems to have done.
- iso1631 4y ago
furyofantares 4y ago
At a company run like that, I doubt these 1647 employees, or however many are using these devices, are really doing much anyway. * Seems I've misunderstood; I was corrected downthread.
- rvba 4y ago
  I kind of disagree. Probably 80% of users are not exactly employee of the month but they probably do something. 20% can probably handle some legacy stuff that requires an old computer. Or a migration. In addition some can be management, so they wont be making decisions for some time.
  - furyofantares 4y ago
    Perhaps I've misunderstood, but if you're warned repeatedly that you'll lose access to your device(s) and haven't taken any action, I have to think you don't find it very important.
    - db48x 4y ago
      That’s not how it works. Your manager’s boss’s boss was warned a year ago that a dozen laptops used by people in his department were going to go end–of–life at the end of this month. Nobody warned you about it at all; you were just plugging away at whatever tasks were assigned to you by your manager. Your manager might have known about it but was probably only told that you would be getting a new laptop “soon”. Someone was supposed to be taking care of it, but nobody really knew who, or when, etc. So when your laptop didn’t work right this morning, you called the tech support department, who ironically were the only department who didn’t know this problem was coming.
      - furyofantares 4y ago
        Ah, yeah ok. Thanks for explaining.
      - flerchin 4y ago
        I tend to think that was a misplay on the part of the original author. If they had notified the 1647 users that their machines would be wiped a month in advance, then a bottom up pressure to get it resolved would have occurred. Few folks are as invested in their daily work as the people who will be blocked.
mysterydip 4y ago
> "So for a whole year, they knew this was coming. But nobody wants all that additional spend, so close to year end. Departments bickering over who’s responsibility it was, who’s budget it came out of, and so on. So everyone dug their heels in, and we continued to shout “iceberg!” from the sidelines." I've seen this play out multiple times over the years. What's the solution?
- protastus 4y ago
  The solution is escalation.
  - kulahan 4y ago
    Yep. The whole point of escalation is so someone can tell everyone to shut up and do X. Solves all these problems.
  - madaxe_again 4y ago
    Correct answer. You do not win by playing a defensive game, and it’s essential to realise that the organisation’s goals are not your goals. The correct strategy here is to go on the offensive. Make friends with your manager’s manager’s manager. Just go full on social bribery mode. Invite them for Christmas dinner. Even if they decline, it will make them remember you, and next time, ask them to a barbecue or a picnic instead, and they’ll say yes. If you can’t throw that high, shoot for your manager’s manager. Once you’re in, get the dagger in your manager’s back, but only once you’ve made them a pariah, take their role, and repeat until you retire wealthy. Remember to take every opportunity to accrue political capital. Eventually you will be in a position to fire people who make poor decisions, but you won’t, because the salary is good, and retirement is only a few years off. You literally cannot prevent this behaviour in any human-operated organisation of any scale beyond a fistful of people - you can only co-opt it.
  - tedunangst 4y ago
    Who should escalate to who and what should that person have done in this scenario?
  - 4y ago
kabdib 4y ago
I never let a company have access to my devices. They can buy me a phone if they want to (the last place I worked did).
- mousetree 4y ago
  In my experience most companies would provide the device if they require MDM. Would definitely be a hard no if they asked me to MDM a personal device
  - aidenn0 4y ago
    Pretty much every MDM advertises BYOD support, so presumably somebody is doing that?
    - Macha 4y ago
      I know our company lets you BYOD if you don't want to deal with also carrying around the company supplied devices. They have the opposite problem, as the company lets you pick between the top end iPhone and Samsung with a 2 year replacement window, so people are tempted to use their company supplied device as their personal device also.
  - whstl 4y ago
    I once worked at a company that paid "device rental" money to employees that used their own devices. It needed MDM, but you could rescind it at any time. Some co-workers would finance laptops and use the "rental" money to pay the installments. I did the math and it wasn't worth it for me, though.
- Aissen 4y ago
  That's the thing, it's not your device, it's your work device.
- rejectfinite 4y ago
  Yes. But MDM/Intune can also be set to company devices.
RedShift1 4y ago
How were the users informed? Did they even understand what's going on? If I receive an email saying my device is out of compliance, I'd ask, out of compliance with what? How do I check? How do I get in compliance? The way this is communicated to the users and what actions they had available to them makes all the difference here.
- jdironman 4y ago
  I would also have thought there should be alerts for devices going out of compliance soon. I'd set that for months back to account for lead times and deal with it as it comes. CC finance / procurement on the alerts if necessary.
- jodrellblank 4y ago
  Skimming the thread it appears to be middle management being informed, not users. The company devolved IT purchasing out to individual non-IT departments. Many of the purchased devices were past end of support life. Legal and Compliance set a hard cutoff when they could not be connected anymore and would not budge. This was known at least 12 months ago as the company bought extended support for some of the devices. IT told the department managers these devices needed updating/replacing over hundreds of emails and dozens of calls and meetings. Department managers took no action. Somehow the CTO was unaware.
- jaywalk 4y ago
  Somewhere in the thread the author mentions that they were explicitly not allowed to inform users.
MichaelZuo 4y ago
"For anyone wondering why we don’t just lift the compliance restrictions, we don’t specify it. Their Compliance department does, and as it’s a large company and the affected users are less than 25% of overall workforce… no exception will be made. One side of the org is going b-a-n-a-n-a-s and the other is taking a very parental “well you should have thought about that” tone. You kinda have to admire their commitment to the cause." I want to know what their org chart looks like.
- baq 4y ago
  Probably something along those lines https://images.app.goo.gl/9bRVF4EeZW4SJbqC9 https://images.app.goo.gl/9bRVF4EeZW4SJbqC9
  - Aachen 4y ago
    That link does not open for me, can someone post whatever that's supposed to redirect to?
    - bombela 4y ago
      An organization chart with three main groups. Pointing guns at each others.
    - shoo 4y ago
      see also https://goomics.net/62/ https://goomics.net/62/
    - chrisandchris 4y ago
      It's a picture of several org charts, each within a balloon by themselfes but connect together by a line. However, they are all pointing guns at each other.
    - teknofobi 4y ago
      It’s the Microsoft org chart with guns pointing between divisions, e.g here: https://www.businessinsider.com/big-tech-org-charts-2011-6 https://www.businessinsider.com/big-tech-org-charts-2011-6
deleted 4y ago
[deleted]
manv1 4y ago
The cost should be billed to the department with the users that were affected. The laptops are assigned to those employees, so normally any kind of compliance/spend should be associated with them as well. It doesn't matter if those costs are mandated by compliance, it's up to each department to keep up. FYI for those non-corporate readers, if there's an actual compliance department that means that the cost of non-compliance is really, really high. That either means financial or government/DoD.
- pas 4y ago
  it might also be healthcare, or nuclear/aviation, no? > The cost should be billed to the department with the users that were affected. It doesn't really matter. These are arbitrary slicings and dicings of one big monolithic blob anyway, internally connected by interdependent causal links. It's a "simple" management decision how compliant the whole operation gets to be, also weighing the cost of compliance versus consequences of non-compliance (from simple things like "have to massage the scope during the next audit" to "might impact a potential lawsuit" to regulator fines us to personal criminal responsibility for the CEO), similarly it's a management decision how much spending each org/department can do on getting compliant without requesting budget for it, etc, etc. In this case the priorities are clear and it just happens that there is this big discontinuity (crisis!) that will probably look like a bad overall trade off, something that should have been prevented. Will this even result in some change to the whole decision-making hierarchy? Unlikely. If this hierarchy was able to completely impervious to all the input from below (emails, calls, meetings, 1 year grace period, etc) then it likely does not matter that these devices will get wiped.
smiley1437 4y ago
Maybe I'm naive but shouldn't there have been an associated budgeted line item for this compliance requirement? Might have made things go smoother
- p_l 4y ago
  There should have been, but according to the thread no department deigned to put it in their budgets, despite having a year-long extension.
  - Krisjohn 4y ago
    Surely no senior management would increase the financial burden on a department or branch without also increasing its budget, right? Right?
    - p_l 4y ago
      Given that the yearly extension was supposedly long enough that there should have been more than one budgeting cycle where the department could request money to get the budget increase... All the management there fucked up. Possibly with the exception of Compliance/Legal and IT.
mnw21cam 4y ago
"To use the Mastodon web application, please enable JavaScript." I don't want to "use the Mastodon web application", whatever that means. I just want to read a web page like a normal person. Sigh Does anyone have the article text handy?
- weberer 4y ago
  Don't know why your comment is grey. Forced Javascript is a blight on the internet. https://archive.is/fxqui https://archive.is/fxqui
  - 0xbadcafebee 4y ago
    Blight on the web, you mean. The internet is what you pass packets over. The web is what you pass memes over.
    - gpderetta 4y ago
      I'm sure there must be an IP-Over-Meme implementation somewhere.
    - EvanAnderson 4y ago
      The Internet is a thing is used to pass HTTPS which we then tunnel all new protocols over.
  - pc86 4y ago
    I actually upvoted the comment because it's true (and it's positive now anyway), but it was probably negative because people complaining about the platform on which a given link is posted is pretty boring, and happens pretty regularly. Surprisingly regularly especially when you consider that approximately 0% of regular web users have JS disabled so there's not exactly a strong incentive to build for that crowd.
  - arcanemachiner 4y ago
    JavaScript is fine. It's just a tool. Like a hammer. A hammer can be an important part of building breathtaking architecture. It can also be used to gouge somebody's eyes out. Just like JavaScript. EDIT: That "web app" was very nice to use.
rcoveson 4y ago
Popular opinion of the decade: Social media is vile. Why are we live-tooting (ugh) our client's private disasters? The indiscretion is staggering. This account casts its author in as bad a light as it does their client. I wouldn't want to work with either.
- macspoofing 4y ago
  >Why are we live-tooting (ugh) our client's private disasters? Indeed. I'm glad they are, because it is fascinating but very odd. I'm sure it's not hard to identify the customer either.
  - guhidalg 4y ago
    I'm glad they are too. Let's take another example: what if the client was a healthcare provider and instead of us merely chuckling at the inconvenience of losses we witnessed deaths from management's incompetence. Would you still want the event to stay confidential if someone you knew died? I'm glad someone is discreetly share details of the situation to signal to the world "Hey if you fuck up compliance people can die, please don't do it like this" instead of keeping it confidential.
  - LordDragonfang 4y ago
    >I'm sure it's not hard to identify the customer either. OP seems rather sure of the opposite: >But I felt I needed to address one particular concern that has been repeatedly raised. That of the identity of the company in question. >I’m a professional, and I’ve been doing this a long ol’ time. There is no way I’m going to risk the identity of the company, or my reputation, or the potential legal consequences for some interaction on social media. >So to clarify, enough details of the incident and those involved have been changed to protect their identity and everyone else involved. I am confident that you could work at the company involved and not even be aware this happened, even after reading this Partly due to scale and partly due to managerial secrecy.
- cschep 4y ago
  I found the writer to be downright empathetic. People make these decisions on purpose to cause their employees this much pain and you're worried about defending their feelings from a writer who is actually deploying empathy? I am so confused by this position. If the writing was nasty and exposing specifics, sure, but it very much is not.
jenadine 4y ago
I don't understand what's going on? What kind of devices are we talking about?
- rejectfinite 4y ago
  Laptops. This is using Microsoft Endpoint manager/Intune when they talk about “compliance”. IT admins can set a policy like "must have 6 number PIN for login", if it does not then it is out of compliance. This can mean nothing at all but if the company wants to act on it, it can.
  - wstuartcl 4y ago
    and it sounds like in this case, anything that was out of compliance (in any regard) was acted on by wiping the device and deregistering it on the deadline day -- read this as 1700 laptops or desktops getting wiped in one day.
- db48x 4y ago
  Probably laptops.
- GartzenDeHaes 4y ago
  OP mentioned Microsoft Intune, which is used to manage phones (System Center is used for laptops).
  - ollien 4y ago
    I've since left my gig with MS systems, but I seem to recall seeing some sort of InTune client on my laptop. Is my memory failing me or is the client just weirdly named?
    - jabroni_salad 4y ago
      intune is the cloud replacement for Configuration Manager. It's been renamed a bunch of times over the years. I'm pretty sure they call it Endpoint Manager right now.
ramshanker 4y ago
Ohhh the classic enterprise PROCEDURES ;) I concluded one of the last year deals in 8 months total from the first Mail I sent. Fortunately, 1 day before deadline (31st Dec) there were 4 different departments heads (each at least 2 level above my rank but still below C-level) involved with extended working hours on 30th December…… Ha ha. So when the next renewal comes up, I am gonna kick-start the procedure 12 months in advance. :D For my mental peace.
- qup 4y ago
  I think the real lesson is not to start the procedure sooner, it's to set the deadline earlier.
harel 4y ago
I like his avatar image. I've just finished restoring and pumping some steroids on an Amiga 500. It's still open next to me and it's nice to have that logo pop up on an unrelated context.
paultopia 4y ago
I could easily see any big university perpetrating this.
gwbas1c 4y ago
Does anyone have any context? What company is this? How did they get into this situation?
- CoastalCoder 4y ago
  I'm curious too. I'm not sure I could tell if this was truth or fiction. The depiction is so close to IT / compliance-office revenge fantasies, so fiction seems plausible.
- GartzenDeHaes 4y ago
  This sounds a lot like federal government contractor / FISMA compliance. I was in a similar situation with VPN remote access device non-compliance, but we ended up ignoring the compliance requirements since Important People were using the VPN.
  - jabroni_salad 4y ago
    That happened when XP was decommissioned. The project's due date was set by a congressional order. The calendar ticks over whether you are ready for it or not!
xorcist 4y ago
There's actually a whitehat ransomware-as-a-service? Now you're telling me!
- rejectfinite 4y ago
  That's kind of what all RMMs like n-able, ninja, connectwise, kaseya vsa, intune is. It is a "backdoor" into corporate computers so that IT can install programs, reboot, install/force updates, run commands, wipe devices etc...
jliptzin 4y ago
I found a really bad vulnerability in a dating app once, allowed anyone to see all other user’s exact locations…contacted the CEO to let him know to fix it. He acknowledged. Thought that was it. A few months go by, I decide to check again. Still hasn’t been fixed, emailed again, acknowledged again. On and on and on. About a year went by for them to finally implement this fix which should take all of 10 minutes, I mean at the very least all you have to do is introduce some entropy into the gps coordinates of the user. Hopefully I am the only one that found it. It’s pretty astonishing how much people just don’t care even the C suite.
- mtsr 4y ago
  That's what responsible disclosure is for. Having a set deadline before an issue becomes public at least puts some pressure on the company to fix it. Not out of spite, or anything, but because it's the only way to protect the users, instead of just the owners.
- kgeist 4y ago
  Sounds familiar. In July 2022 I found a vulnerability in one of our systems (easy to exploit and basically allows anyone to authenticate as anyone, full access to LDAP accounts), I reported it and they made a fix which they supposedly deployed. The infosec department was notified everything was OK now. I decided to recheck it a few months later (I took it personally because someone could pose as me) and found out they somehow forgot to actually deploy it even though the original ticket was marked as fixed/closed. I notified the original team and they promised to deploy it "very soon" which didn't happen again. Basically every week I had to post "still not fixed" to their chat for a few months. Every time the project manager would promise it would be deployed soon but then would forget about it. Countless emails to the infosec department about the situation. It was finally deployed in January 2023, a fix which had been ready (coded and tested) for half a year by that time! Deploying it took literally 15 minutes. In fact, I could (and was ready to) deploy it myself because I have the required privileges but I was part of a different team by then and it felt wrong to mess with their release cycles on my own.
  - t0astbread 4y ago
WaitWaitWha 4y ago
As others have asked, how did it get to this point? Were the meetings, emails, phone calls had the right people in them? Was the escalation up the org chain? Unclear from the tweets.
tomxor 4y ago
How is it that not a single person "in the know" (of which apparently there were a great many) had the sense to simply take this directly to the CTO, seeing as how clear middle management was failing a massive, critical and time sensitive task. It doesn't matter if you are the Janitor, it's obvious they are going to want to clear all the red tape out of the way as soon as they find out. What is it some kind of "not my problem"ism? Madness.
- Spivak 4y ago
  From the thread it seems like the company specifically took procurement away from the CTO and pushed it down to the individual departments and so there were a whole group of "final desks" that needed to agree on a collective purchase but didn't.
EVa5I7bHFq9mnYK 4y ago
So the author sold his work ethic for Twitter likes. He knew the disaster will happen and haven't done enough to prevent it. I'm 100% sure the disaster could be prevented by taking up a phone and finding the people capable of solving the problem.
- ibejoeb 4y ago
  This is addressed in the thread.
- tomxor 4y ago
  > and haven't done enough to prevent it Nuh uh, go read the thread.
  - EVa5I7bHFq9mnYK 4y ago
    I have. It's all about "I've sent all the formally required emails. Now preparing the popcorn and going to polish that bombastic blog post". If he didn't find the correct person and correct words, he hasn't done his job.
    - Dylan16807 4y ago
      It sounds to me like they did a lot more than required in trying to convince people this was a bad idea. But they also didn't go out of the loop to find the right person.
    - ruune 4y ago
      28 phone calls in 14 days too. And with everyone seemingly aware, another phone call probably wouldn't have helped
    - tedunangst 4y ago
      What is their job? Their job is to turn off the devices. They're not in charge of new device procurement.
ploum 4y ago
In "Work without email", Cal Newport explain a case where a whole financial institution was margin-called for the exact same reason. They knew it was coming. They were willing to fix it. They spent weeks exchanging emails on how to setup a meeting to solve the problem. The problem eventually solved itself. Had a pretty similar experience with management early in my career that was wide-opening on how incompetent every single manager was. Became a manager myself with the intention of avoiding that. I could not. Changed career path.
- momojo 4y ago
  > Became a manager myself with the intention of avoiding that. I could not. I appreciate the honesty. What was that experience like?
  - ploum 4y ago
    As a manager, you simply don’t have the time to dig into technical issues. You can’t take uninterrupted 4 hours to enter into the code and debug something. When you are a new manager with a deep experience of the technology, you don’t see it immediately. But the longer you manage, the more your experience become irrelevant (for example: my team switched from Angular to React. I never did any React and there was no way for a manager to dig into it at the same rate as the team). It took me two jobs as a manager to realise that, at least in software development, a manager’s job is to pretend. To make uninformed decisions and lead the team without understanding anything of what is happening. You also spend your time negotiating with upper levels that want everything without even thinking about the implications (I’m not talking about costs or time, I really had meeting with really high levels managers who asked me, straight in the eyes, to make "a solution with all the advantages and without the disadvantages" and they were very proud of their line). I learned that very high level management meeting are dumb and boring, that those people don’t even have the slightest clue what they are talking about and spend hours discussing micromanagement discussion (I attended a very high-level meeting where I replaced my n+1 and they litteraly spent one hour discussing who should send an email to X to ask him to send an email to Y. I took notes of that one because I feared nobody would believe me). But I also reached the conclusion that managers are necessary. I even had a very good one who told me after one week: "I’m a manager, I have no idea how you are doing your thing. My job is to set a goal with you then your job is to ask me every time I could help with your job. Also, I’m here to insulate you from the administrative shit". I tried to become a manager like that. I also lived by the credo: "If anything fails in my team, it’s my fault, I will not put the fault to individuals in my team". I learned that this work only with very good teams and independant individuals. Some people need to be taken by the hand and a good manager will offer psychological help. But this only work if the layer above is also working that way. I ended fighting with my N+1 because they absolutely wanted to fire someone from my team. Needless to say, a CEO and friend told me I was not a good manager. I would never become one if I didn’t change the way I was looking at things. So, in conclusion : there are good managers. But they do not last long. They either quit or becomes bad managers which is the only way to climb in the hierarchy : lick upper levels asses and tell them that any problem is because of the individuals in your team. If you do that properly, you will never stay long enough in a team to have any impact anyway. Don’t try to deliver. Pretend you do it by saying it in a powerpoint. And tell your developers that everything is due yesterday.
racl101 4y ago
I hear Clock Town Day 3 music playing.
tflinton 4y ago
I've had a situation at a previous employer where a contract lost its ownership due to a reorg after downsizing. The new org was completely unaware of the contract lapse until services were turned off. The existing contract lapse had also lapsed the vendor review requirements and finances standing and thus getting a new contract in place, signed and paid took compliance, legal, finance and IT to all get together with the C-level staff to get services turned back on. Longest outage i've ever seen in my life.
AndrewKemendo 4y ago
Yup - and guess what, the only people hurt by this are the lowest level people. No manager or exec will feel any pain from this incompetence. Someone who isn't responsible but couldn't do their job effectively as a result is having a worse life now. This is one of the primary reasons why I am totally done with Tech The distance between users and builders is so excessively far apart now and the levels of abstraction for actually building things that are robust is just not even a consideration in software-centric design. Literally everything you have built after IDK 2000(?) will have exactly this issue. Just hope you're not at a scale that crushes people. Software is the language of alienation and increasingly becoming unethical, as these systems are becoming increasingly impactful on the most vulnerable with no buttresses or supports preventing this kind of malfeasance. It's not trivial. These are people's livelihoods at stake.
- opamp 4y ago
  > No manager or exec will feel any pain from this incompetence. Someone who isn't responsible but couldn't do their job effectively as a result is having a worse life now. > This is one of the primary reasons why I am totally done with Tech Is it that much better in other fields?
  - sillyquiet 4y ago
    no, its not. Not in any white collar professions, not in any blue collar jobs, not in the military, not in academia. You are only safe when you don't have bosses and all responsibility and power rests on yourself.
    - AndrewKemendo 4y ago
      Almost...you find coworkers and share responsibility mutually. You don't need to put it all on your own back.
  - AndrewKemendo 4y ago
    Yes. For example German workers have better than average protection from exploitation This is from 2018 and shows where you can actually have some rights as a worker. It's not hopeless: https://www.ituc-csi.org/IMG/pdf/ituc-global-rights-index-2018-en-final-2.pdf https://www.ituc-csi.org/IMG/pdf/ituc-global-rights-index-20...
ivraatiems 4y ago
Fascinating to read, but couldn't the author get in trouble for posting like this about one of their employer's customers? Where I work, which is a much lower-stakes environment, talking about our customers' issues or choices in public like this is a huge no-no. I'd get fired if someone found me out. Especially since if the customer is large, and the decisions have anything to do with my company's revenue, it could be considered MNPI.
somecompanyguy 4y ago
apparently nobody quantified the projected losses, because that would have caused this to not be ignored. i blame everyone involved including the service provider. everybody did something wrong
mdip 4y ago
Oh, do I remember these days. I spent 17 years at a global multi-national telecom through two mergers and a bankruptcy (and a half). We were a large organization (between 5,500 and 22,000 depending on the year). During much of that time, "who paid for what" was a big issue. The thread alludes to the issue: IT says "you need to buy new hardware every X years", department already has less than no budget, has no budget for new PCs and perceives no need for new PCs for workers that could get away with much less than they're using, now. It was a funny little game that was played because IT would get dinged in their compliance metrics if staff was out of date (and staff hated old hardware/blamed IT), but management would get dinged for spending too much and have little incentive to buy new hardware until the last second. Meanwhile, C-Level executives on both sides get to say "your problem". The difference, here, is that someone gave IT a pretty large sledge-hammer and permission to use it in order to force departments to push for more budget. In our case (and I'm sure others), a bit (a lot?) of non-compliance was normal. Personally, I think the take that "IT should own the budget" isn't as great as it sounds. It solves one problem: distributing the payment among budgets creates a "shared responsibility" that ultimately becomes "pass the buck". It also happens low enough from C-Levels that "they don't have to think about it." Having IT own the budget solves this because at least one C-Level is going to have to account for a large enough expense that it's likely to be a little better planned for. It won't always be better planned for ... depending on the company or manager, it won't often be better planned for. Unfortunately, the consequence of this poor planning only extends to the IT budget. Since compliance is non-negotiable, the largest line-item on the budget -- IT's staff -- is the next hit. In the former model, "making the budget deficit up" is naturally spread throughout the company, in this model, it all hits IT.
- pas 4y ago
  All in all this is business as usual. You can't work because your work device is unfit to do the work? It's not really your problem as an employee. It's the employers' responsibility to provide the tools, and it's up to each management how they solve it, with what trade offs. This one picked "bugdet first, compliance second, worker/client satisfaction and business continuity last".
cm2187 4y ago
Having worked all my life in large organisations, this sounds very familiar. A lot of people would rather the company to go bust than to challenge an internal policy written by a group of people largely above their level of competence, and completely unaware and unconcerned of the implications of their policies. One of the things you realise when you get closer to management is that those policies shouldn't be taken too seriously if they contradict common sense.
- throwaway892238 4y ago
  OTOH, people who risk their career to challenge an internal policy written by a group of people largely above their pay grade and not answerable to them, at best become pariahs who are ignored, and at worst are fired for "not being a team player". Most people are only concerned with their own little corporate corner and doing the least effort that keeps them in paychecks. Trying to follow the spirit of a rule rather than the letter, or pushing for change to improve things overall, is never appreciated.
  - cm2187 4y ago
    By middle management maybe. By senior management, what gets you promoted is the ability to fight back, challenge things that don't make sense and to get things done.
    - Volundr 4y ago
      If your middle management is full of yes men (yes people?) it's because that's who senior leadership chose to promote. The idea that there exists some level where C-level execs in that organization are going to suddenly appreciate dissent is wishful thinking. These kinds of cultures begin at the top.
- bonestamp2 4y ago
  We call those people/policies the "Business Prevention Department"... In other words, they're the department that makes it difficult for everyone else to generate revenue. Sometimes they're right, but often they're too rigid to operate in reality and instead of protecting the company they actually hurt it.
deepsun 4y ago
Kudos to Compliance team to keep the high bar on their work and not let anyone go like "oh this superuser password is just for testing and telemetry, we certainly won't forget to remove it before release".
Marco30 4y ago
[dead]
Marco30 4y ago
https://support.google.com/maps/answer/6258979?hl=sk&dark=1 https://support.google.com/maps/answer/6258979?hl=sk&dark=1
sacnoradhq 4y ago
I work for a MAANG in a related area at very large scale. 1. BYOD is a terrible idea. Work and home devices should be separate, even at the cost of multiple devices. 2. More than emails, they need push notifications and background wallpaper changes on the machine with specific instructions before doing anything terrible. Soft power configuration changes can do much more without potentially destroying value. 3. It's much easier to lock out users from their account with a message about mandatory remediation steps to regain access. 4. Issue newer machines on a lifecycle with an easy return program. When people get new machines, they'll almost always cough up the old ones. This solves the issue and maintains fleet health and reliability. 5. Consider replacing laptops with VDI and Chromebooks where possible.
- Chris2048 4y ago
  > 1. BYOD is a terrible idea. Work and home devices should be separate, even at the cost of multiple devices. You could have your own work device at home, separate from "personal use" items. The difference is that you can upgrade & reclaim hardware without having to bring it in to IT.
  - sacnoradhq 4y ago
    The IT department must horrible (or perhaps the employee doesn't understand money) if they prefer to subsidize their employer's CapEx and take a pay cut.

603 points by terom 4y ago

It's like a dog owner whose dog has been shitting on the sidewalk for so long that the owner eventually can't walk on the sidewalk without stepping in their own dog's shit. But they will absolutely blame the dog.

smcg 4y ago

"Hey, there's a rake up ahead. I'd recommend not stepping on it." "Ow! Why did this happen?"

hprotagonist 4y ago
doctor, doctor, it hurts when i do $this well don’t do $that then
- Oxidation 4y ago
  Private healthcare: there is only one dollar sign and it changed hands between those sentences.
gtsteve 4y ago
Yes I remember this from my consulting days well. Me: If we do A, B will happen Client: Do A anyway, we'll deal with B later. (time passes) Client: OH MY GOD B HAS HAPPENED
- aidenn0 4y ago
  A friend of mine started writing his predictions and putting them in sealed, dated envelopes. He said the 3rd time he pulled one out Carnac[1] style, management actually started listening to him. Nobody really got "I predicted this 18 months ago" but the theatrics apparently drove the point home. 1: https://en.wikipedia.org/wiki/Carnac_the_Magnificent https://en.wikipedia.org/wiki/Carnac_the_Magnificent
  - ahazred8ta 4y ago
    A dirty trick, but an effective one. May the fleas of a thousand camels infest you, and may your arms be to short too scratch.

charles_f 4y ago

We need a retrospective, how could you let this happen?

tenebrisalietum 4y ago
1. You told us to do it. 2. We did it. 3. It was done.
- Oxidation 4y ago
  4. Bonuses all round?

pessimizer 4y ago

My reaction to this tweet was surprisingly intense. It's like the plot to a horror movie, or the 5 minute opening credit montage of a post-apocalyptic film.

dijksterhuis 4y ago

> Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent. I see that life as enterprise service desk hasn’t changed much. “Nobody tells me nuthin!” Shout out to the ever under appreciated service desk folks out there.

ilyt 4y ago
"We need that machine tomorrow?" "Why didn't you tell us you hired something new?" "They work here for 2 weeks now"
- dylan604 4y ago
  are new hires really being referred to somethings now?
  - smiley1437 4y ago
    That's a pronoun I haven't seen yet
    - mrguyorama 4y ago
      Honestly it's a way more accurate reflection of how managers see employees
      - flerchin 4y ago
        :feels:
        dylan604 4y ago
        ?? Isn't feeling the employees an HR violation?
        kevin_thibedeau 4y ago

kerblang 4y ago

I would go with an incremental progression of something like 100 random devices a week for 17 weeks, so that people see the tidal wave eating others and suddenly "get it". Less overwhelming for the service/support desk folk too.

Oxidation 4y ago
Or start with the "most critical" devices: those belonging to the highest-ranking users. They're the high-value hacking targets so it's only reasonable. Might unstick a few wallets better than hitting mostly rank-and-file for something they have no control over.
- gumby 4y ago
  Or incrementally degrade service (lock out web browser, then email...)
- jdironman 4y ago
  The problem with that is, it's most likely those devices which are already compliant / up to date.
- basch 4y ago
  If you are this close to a deadline, it doesnt really matter anymore. There is no avoiding the iceberg. You can fix your organization for next time, which is what the org believe it has initiated.
  - Oxidation 4y ago
    Well, yes you don't do this two hours out, you'd need to do it some time in the 13 months they knew about it for it to work. But at the end of the day, they did the job they were paid to do and were clear about the looming impact. It's not their job to also wipe their clients' metaphorical bottoms when they were ignored.
kneebonian 4y ago
Based on the thread I think it was Legal and Compliance forcing it. It may have been regulatory required. Also they say they are managing on behalf of their client, so their position as mercenaries is just to follow orders.

danjc 4y ago

As much as I hate the concept, sometimes brownouts are the only way to force people to make a change and avoid complete disaster on D day

danpalmer 4y ago
Why do you hate the concept? I've seen it done a few times in different areas and I think it's a neat way of notifying people in large organisations/ecosystems. As long as the brownout is done after the support period, i.e. it's "contractually" ok to do so, it seems like a good idea.
- jaywalk 4y ago
  Brownouts can be very useful in finding impacted systems that may have been overlooked as well. Last month, I had to do some updates because a customer's API had moved to a new URL on a new server. My team and I identified (what we thought was) everything using the API and did the updates. A week later, the customer notified us that they were still seeing some traffic on the old URL, but all they could give us was the IP address it was coming from. Unfortunately this IP address belonged to a server that hosts a lot of our smaller applications, so it didn't really help locate the offender. So I just added a firewall rule to block access to the IP address of the customer's old server, and sure enough I heard the scream 15 minutes later. Removed the rule to get that application back up and running, got it updated to the new URL, and all was good.
nunez 4y ago
Scream testing is underrated for sure.

twawaaay 4y ago

The CEO will have a convo with two of his managers that will resemble me when I talk to my two sons when they get into a fight and start pulling out all of their excuses.

jmholla 4y ago

Mostly unrelated, but I hate websites like this that think they're way of handling arrow keys for scrolling should be implemented over how every other web page does it. I lost my place so many times when I mindlessly tried to scroll again with the arrow keys.

gambiting 4y ago
And even though this website only contains his posts and nothing else, individual posts are minimised and have to be clicked on to unfold, which scrolls the entire page for me on mobile chrome and I have to find it again. It's a usability nightmare.
- fragmede 4y ago
  This website is a well known infosec Mastodon host. The linked site is to a specific person's feed but the site local feed, with many other individuals' posts is at https://infosec.exchange/public/local https://infosec.exchange/public/local
- LordDragonfang 4y ago
  Top right of the content bar has an eyeball icon with "show more for all", which expands them all at once, but agreed, this isn't great UI or UX (still better than twitter though!)
- Krisjohn 4y ago
  At a particular point the author specifically indicates that for the sake of reducing stuff that many readers might find uninteresting or too frequent, they will be condensing larger posts. > Alright, things are moving faster now, so I might condense-toot to avoid pollution. But I guess the chronically online need something to complain about.
  - gambiting 4y ago
    It's not the condensation of content I'm complaining about, it's the fact that when you try to unfold a post the entire website scrolls and puts you somewhere else. That's just bad UI.

AtlasBarfed 4y ago

I'm guessing upper management prioritized the update of these devices with downstream management rather than overburden them with other stuff. So in the end, this is just one piss poor managed division abusing another piss poor managed division. Who gets the heat? Probably the lowest level people. Why "wipe" them? That seems unnecessarily punative. You can see the "don't give a shit I work with a predatory organization" oozing from everywhere. The security guy is trying to claim that they've sent out many many notices, but really this is just an excuse to abuse other people in a machiavellian abusive organization. "Service Desk is now aware that everyone else except them was aware, and now IT is absolutely incandescent." Whoops, missed an email and a meeting in there bucko. And it's the SECOND company where this was "implemented" or "specced"? This sounds like someone checked a box or compliance or ass-covering upper management slid this under the table, but all the people it ACTUALLY AFFECTS didn't get any input or opinion on the matter. And when push came to shove over funding it that person had probably moved on to bigger and better things. So since you get to do it, you seem to be gleefully doing it. Great job.

ilyt 4y ago
> Why "wipe" them? That seems unnecessarily punative. There was something in the thread about the devices coming out of support by manufacturer, which was already extended by a year.
Blackthorn 4y ago
What exactly are you suggesting this person does? The policy to wipe clearly came from the company's compliance department. They warned them over and over what was about to happen, and went above and beyond doing it with multiple meetings and phone calls.
- whstl 4y ago
  Exactly. In this situation all you must do is warn people of the risks and document, document, document. Which the article writer seems to have done.
iso1631 4y ago

furyofantares 4y ago

At a company run like that, I doubt these 1647 employees, or however many are using these devices, are really doing much anyway. * Seems I've misunderstood; I was corrected downthread.

rvba 4y ago
I kind of disagree. Probably 80% of users are not exactly employee of the month but they probably do something. 20% can probably handle some legacy stuff that requires an old computer. Or a migration. In addition some can be management, so they wont be making decisions for some time.
- furyofantares 4y ago
  Perhaps I've misunderstood, but if you're warned repeatedly that you'll lose access to your device(s) and haven't taken any action, I have to think you don't find it very important.
  - db48x 4y ago
    That’s not how it works. Your manager’s boss’s boss was warned a year ago that a dozen laptops used by people in his department were going to go end–of–life at the end of this month. Nobody warned you about it at all; you were just plugging away at whatever tasks were assigned to you by your manager. Your manager might have known about it but was probably only told that you would be getting a new laptop “soon”. Someone was supposed to be taking care of it, but nobody really knew who, or when, etc. So when your laptop didn’t work right this morning, you called the tech support department, who ironically were the only department who didn’t know this problem was coming.
    - furyofantares 4y ago
      Ah, yeah ok. Thanks for explaining.
    - flerchin 4y ago
      I tend to think that was a misplay on the part of the original author. If they had notified the 1647 users that their machines would be wiped a month in advance, then a bottom up pressure to get it resolved would have occurred. Few folks are as invested in their daily work as the people who will be blocked.

mysterydip 4y ago

> "So for a whole year, they knew this was coming. But nobody wants all that additional spend, so close to year end. Departments bickering over who’s responsibility it was, who’s budget it came out of, and so on. So everyone dug their heels in, and we continued to shout “iceberg!” from the sidelines." I've seen this play out multiple times over the years. What's the solution?

protastus 4y ago
The solution is escalation.
- kulahan 4y ago
  Yep. The whole point of escalation is so someone can tell everyone to shut up and do X. Solves all these problems.
- madaxe_again 4y ago
  Correct answer. You do not win by playing a defensive game, and it’s essential to realise that the organisation’s goals are not your goals. The correct strategy here is to go on the offensive. Make friends with your manager’s manager’s manager. Just go full on social bribery mode. Invite them for Christmas dinner. Even if they decline, it will make them remember you, and next time, ask them to a barbecue or a picnic instead, and they’ll say yes. If you can’t throw that high, shoot for your manager’s manager. Once you’re in, get the dagger in your manager’s back, but only once you’ve made them a pariah, take their role, and repeat until you retire wealthy. Remember to take every opportunity to accrue political capital. Eventually you will be in a position to fire people who make poor decisions, but you won’t, because the salary is good, and retirement is only a few years off. You literally cannot prevent this behaviour in any human-operated organisation of any scale beyond a fistful of people - you can only co-opt it.
- tedunangst 4y ago
  Who should escalate to who and what should that person have done in this scenario?
- 4y ago

kabdib 4y ago

I never let a company have access to my devices. They can buy me a phone if they want to (the last place I worked did).

mousetree 4y ago
In my experience most companies would provide the device if they require MDM. Would definitely be a hard no if they asked me to MDM a personal device
- aidenn0 4y ago
  Pretty much every MDM advertises BYOD support, so presumably somebody is doing that?
  - Macha 4y ago
    I know our company lets you BYOD if you don't want to deal with also carrying around the company supplied devices. They have the opposite problem, as the company lets you pick between the top end iPhone and Samsung with a 2 year replacement window, so people are tempted to use their company supplied device as their personal device also.
- whstl 4y ago
  I once worked at a company that paid "device rental" money to employees that used their own devices. It needed MDM, but you could rescind it at any time. Some co-workers would finance laptops and use the "rental" money to pay the installments. I did the math and it wasn't worth it for me, though.
Aissen 4y ago
That's the thing, it's not your device, it's your work device.
rejectfinite 4y ago
Yes. But MDM/Intune can also be set to company devices.

RedShift1 4y ago

How were the users informed? Did they even understand what's going on? If I receive an email saying my device is out of compliance, I'd ask, out of compliance with what? How do I check? How do I get in compliance? The way this is communicated to the users and what actions they had available to them makes all the difference here.

jdironman 4y ago
I would also have thought there should be alerts for devices going out of compliance soon. I'd set that for months back to account for lead times and deal with it as it comes. CC finance / procurement on the alerts if necessary.
jodrellblank 4y ago
Skimming the thread it appears to be middle management being informed, not users. The company devolved IT purchasing out to individual non-IT departments. Many of the purchased devices were past end of support life. Legal and Compliance set a hard cutoff when they could not be connected anymore and would not budge. This was known at least 12 months ago as the company bought extended support for some of the devices. IT told the department managers these devices needed updating/replacing over hundreds of emails and dozens of calls and meetings. Department managers took no action. Somehow the CTO was unaware.
jaywalk 4y ago
Somewhere in the thread the author mentions that they were explicitly not allowed to inform users.

MichaelZuo 4y ago

"For anyone wondering why we don’t just lift the compliance restrictions, we don’t specify it. Their Compliance department does, and as it’s a large company and the affected users are less than 25% of overall workforce… no exception will be made. One side of the org is going b-a-n-a-n-a-s and the other is taking a very parental “well you should have thought about that” tone. You kinda have to admire their commitment to the cause." I want to know what their org chart looks like.

baq 4y ago
Probably something along those lines https://images.app.goo.gl/9bRVF4EeZW4SJbqC9 https://images.app.goo.gl/9bRVF4EeZW4SJbqC9
- Aachen 4y ago
  That link does not open for me, can someone post whatever that's supposed to redirect to?
  - bombela 4y ago
    An organization chart with three main groups. Pointing guns at each others.
  - shoo 4y ago
    see also https://goomics.net/62/ https://goomics.net/62/
  - chrisandchris 4y ago
    It's a picture of several org charts, each within a balloon by themselfes but connect together by a line. However, they are all pointing guns at each other.
  - teknofobi 4y ago
    It’s the Microsoft org chart with guns pointing between divisions, e.g here: https://www.businessinsider.com/big-tech-org-charts-2011-6 https://www.businessinsider.com/big-tech-org-charts-2011-6

deleted 4y ago

[deleted]

manv1 4y ago

The cost should be billed to the department with the users that were affected. The laptops are assigned to those employees, so normally any kind of compliance/spend should be associated with them as well. It doesn't matter if those costs are mandated by compliance, it's up to each department to keep up. FYI for those non-corporate readers, if there's an actual compliance department that means that the cost of non-compliance is really, really high. That either means financial or government/DoD.

pas 4y ago
it might also be healthcare, or nuclear/aviation, no? > The cost should be billed to the department with the users that were affected. It doesn't really matter. These are arbitrary slicings and dicings of one big monolithic blob anyway, internally connected by interdependent causal links. It's a "simple" management decision how compliant the whole operation gets to be, also weighing the cost of compliance versus consequences of non-compliance (from simple things like "have to massage the scope during the next audit" to "might impact a potential lawsuit" to regulator fines us to personal criminal responsibility for the CEO), similarly it's a management decision how much spending each org/department can do on getting compliant without requesting budget for it, etc, etc. In this case the priorities are clear and it just happens that there is this big discontinuity (crisis!) that will probably look like a bad overall trade off, something that should have been prevented. Will this even result in some change to the whole decision-making hierarchy? Unlikely. If this hierarchy was able to completely impervious to all the input from below (emails, calls, meetings, 1 year grace period, etc) then it likely does not matter that these devices will get wiped.

smiley1437 4y ago

Maybe I'm naive but shouldn't there have been an associated budgeted line item for this compliance requirement? Might have made things go smoother

p_l 4y ago
There should have been, but according to the thread no department deigned to put it in their budgets, despite having a year-long extension.
- Krisjohn 4y ago
  Surely no senior management would increase the financial burden on a department or branch without also increasing its budget, right? Right?
  - p_l 4y ago
    Given that the yearly extension was supposedly long enough that there should have been more than one budgeting cycle where the department could request money to get the budget increase... All the management there fucked up. Possibly with the exception of Compliance/Legal and IT.

mnw21cam 4y ago

"To use the Mastodon web application, please enable JavaScript." I don't want to "use the Mastodon web application", whatever that means. I just want to read a web page like a normal person. Sigh Does anyone have the article text handy?

weberer 4y ago
Don't know why your comment is grey. Forced Javascript is a blight on the internet. https://archive.is/fxqui https://archive.is/fxqui
- 0xbadcafebee 4y ago
  Blight on the web, you mean. The internet is what you pass packets over. The web is what you pass memes over.
  - gpderetta 4y ago
    I'm sure there must be an IP-Over-Meme implementation somewhere.
  - EvanAnderson 4y ago
    The Internet is a thing is used to pass HTTPS which we then tunnel all new protocols over.
- pc86 4y ago
  I actually upvoted the comment because it's true (and it's positive now anyway), but it was probably negative because people complaining about the platform on which a given link is posted is pretty boring, and happens pretty regularly. Surprisingly regularly especially when you consider that approximately 0% of regular web users have JS disabled so there's not exactly a strong incentive to build for that crowd.
- arcanemachiner 4y ago
  JavaScript is fine. It's just a tool. Like a hammer. A hammer can be an important part of building breathtaking architecture. It can also be used to gouge somebody's eyes out. Just like JavaScript. EDIT: That "web app" was very nice to use.

rcoveson 4y ago

Popular opinion of the decade: Social media is vile. Why are we live-tooting (ugh) our client's private disasters? The indiscretion is staggering. This account casts its author in as bad a light as it does their client. I wouldn't want to work with either.

macspoofing 4y ago
>Why are we live-tooting (ugh) our client's private disasters? Indeed. I'm glad they are, because it is fascinating but very odd. I'm sure it's not hard to identify the customer either.
- guhidalg 4y ago
  I'm glad they are too. Let's take another example: what if the client was a healthcare provider and instead of us merely chuckling at the inconvenience of losses we witnessed deaths from management's incompetence. Would you still want the event to stay confidential if someone you knew died? I'm glad someone is discreetly share details of the situation to signal to the world "Hey if you fuck up compliance people can die, please don't do it like this" instead of keeping it confidential.
- LordDragonfang 4y ago
  >I'm sure it's not hard to identify the customer either. OP seems rather sure of the opposite: >But I felt I needed to address one particular concern that has been repeatedly raised. That of the identity of the company in question. >I’m a professional, and I’ve been doing this a long ol’ time. There is no way I’m going to risk the identity of the company, or my reputation, or the potential legal consequences for some interaction on social media. >So to clarify, enough details of the incident and those involved have been changed to protect their identity and everyone else involved. I am confident that you could work at the company involved and not even be aware this happened, even after reading this Partly due to scale and partly due to managerial secrecy.
cschep 4y ago
I found the writer to be downright empathetic. People make these decisions on purpose to cause their employees this much pain and you're worried about defending their feelings from a writer who is actually deploying empathy? I am so confused by this position. If the writing was nasty and exposing specifics, sure, but it very much is not.

jenadine 4y ago

I don't understand what's going on? What kind of devices are we talking about?

rejectfinite 4y ago
Laptops. This is using Microsoft Endpoint manager/Intune when they talk about “compliance”. IT admins can set a policy like "must have 6 number PIN for login", if it does not then it is out of compliance. This can mean nothing at all but if the company wants to act on it, it can.
- wstuartcl 4y ago
  and it sounds like in this case, anything that was out of compliance (in any regard) was acted on by wiping the device and deregistering it on the deadline day -- read this as 1700 laptops or desktops getting wiped in one day.
db48x 4y ago
Probably laptops.
GartzenDeHaes 4y ago
OP mentioned Microsoft Intune, which is used to manage phones (System Center is used for laptops).
- ollien 4y ago
  I've since left my gig with MS systems, but I seem to recall seeing some sort of InTune client on my laptop. Is my memory failing me or is the client just weirdly named?
  - jabroni_salad 4y ago
    intune is the cloud replacement for Configuration Manager. It's been renamed a bunch of times over the years. I'm pretty sure they call it Endpoint Manager right now.

ramshanker 4y ago

Ohhh the classic enterprise PROCEDURES ;) I concluded one of the last year deals in 8 months total from the first Mail I sent. Fortunately, 1 day before deadline (31st Dec) there were 4 different departments heads (each at least 2 level above my rank but still below C-level) involved with extended working hours on 30th December…… Ha ha. So when the next renewal comes up, I am gonna kick-start the procedure 12 months in advance. :D For my mental peace.

qup 4y ago
I think the real lesson is not to start the procedure sooner, it's to set the deadline earlier.

harel 4y ago

I like his avatar image. I've just finished restoring and pumping some steroids on an Amiga 500. It's still open next to me and it's nice to have that logo pop up on an unrelated context.

paultopia 4y ago

I could easily see any big university perpetrating this.

gwbas1c 4y ago

Does anyone have any context? What company is this? How did they get into this situation?

CoastalCoder 4y ago
I'm curious too. I'm not sure I could tell if this was truth or fiction. The depiction is so close to IT / compliance-office revenge fantasies, so fiction seems plausible.
GartzenDeHaes 4y ago
This sounds a lot like federal government contractor / FISMA compliance. I was in a similar situation with VPN remote access device non-compliance, but we ended up ignoring the compliance requirements since Important People were using the VPN.
- jabroni_salad 4y ago
  That happened when XP was decommissioned. The project's due date was set by a congressional order. The calendar ticks over whether you are ready for it or not!

xorcist 4y ago

There's actually a whitehat ransomware-as-a-service? Now you're telling me!

rejectfinite 4y ago
That's kind of what all RMMs like n-able, ninja, connectwise, kaseya vsa, intune is. It is a "backdoor" into corporate computers so that IT can install programs, reboot, install/force updates, run commands, wipe devices etc...

jliptzin 4y ago

I found a really bad vulnerability in a dating app once, allowed anyone to see all other user’s exact locations…contacted the CEO to let him know to fix it. He acknowledged. Thought that was it. A few months go by, I decide to check again. Still hasn’t been fixed, emailed again, acknowledged again. On and on and on. About a year went by for them to finally implement this fix which should take all of 10 minutes, I mean at the very least all you have to do is introduce some entropy into the gps coordinates of the user. Hopefully I am the only one that found it. It’s pretty astonishing how much people just don’t care even the C suite.

mtsr 4y ago
That's what responsible disclosure is for. Having a set deadline before an issue becomes public at least puts some pressure on the company to fix it. Not out of spite, or anything, but because it's the only way to protect the users, instead of just the owners.
kgeist 4y ago
Sounds familiar. In July 2022 I found a vulnerability in one of our systems (easy to exploit and basically allows anyone to authenticate as anyone, full access to LDAP accounts), I reported it and they made a fix which they supposedly deployed. The infosec department was notified everything was OK now. I decided to recheck it a few months later (I took it personally because someone could pose as me) and found out they somehow forgot to actually deploy it even though the original ticket was marked as fixed/closed. I notified the original team and they promised to deploy it "very soon" which didn't happen again. Basically every week I had to post "still not fixed" to their chat for a few months. Every time the project manager would promise it would be deployed soon but then would forget about it. Countless emails to the infosec department about the situation. It was finally deployed in January 2023, a fix which had been ready (coded and tested) for half a year by that time! Deploying it took literally 15 minutes. In fact, I could (and was ready to) deploy it myself because I have the required privileges but I was part of a different team by then and it felt wrong to mess with their release cycles on my own.
- t0astbread 4y ago

WaitWaitWha 4y ago

As others have asked, how did it get to this point? Were the meetings, emails, phone calls had the right people in them? Was the escalation up the org chain? Unclear from the tweets.

tomxor 4y ago

How is it that not a single person "in the know" (of which apparently there were a great many) had the sense to simply take this directly to the CTO, seeing as how clear middle management was failing a massive, critical and time sensitive task. It doesn't matter if you are the Janitor, it's obvious they are going to want to clear all the red tape out of the way as soon as they find out. What is it some kind of "not my problem"ism? Madness.

Spivak 4y ago
From the thread it seems like the company specifically took procurement away from the CTO and pushed it down to the individual departments and so there were a whole group of "final desks" that needed to agree on a collective purchase but didn't.

EVa5I7bHFq9mnYK 4y ago

So the author sold his work ethic for Twitter likes. He knew the disaster will happen and haven't done enough to prevent it. I'm 100% sure the disaster could be prevented by taking up a phone and finding the people capable of solving the problem.

ibejoeb 4y ago
This is addressed in the thread.
tomxor 4y ago
> and haven't done enough to prevent it Nuh uh, go read the thread.
- EVa5I7bHFq9mnYK 4y ago
  I have. It's all about "I've sent all the formally required emails. Now preparing the popcorn and going to polish that bombastic blog post". If he didn't find the correct person and correct words, he hasn't done his job.
  - Dylan16807 4y ago
    It sounds to me like they did a lot more than required in trying to convince people this was a bad idea. But they also didn't go out of the loop to find the right person.
  - ruune 4y ago
    28 phone calls in 14 days too. And with everyone seemingly aware, another phone call probably wouldn't have helped
  - tedunangst 4y ago
    What is their job? Their job is to turn off the devices. They're not in charge of new device procurement.

ploum 4y ago

In "Work without email", Cal Newport explain a case where a whole financial institution was margin-called for the exact same reason. They knew it was coming. They were willing to fix it. They spent weeks exchanging emails on how to setup a meeting to solve the problem. The problem eventually solved itself. Had a pretty similar experience with management early in my career that was wide-opening on how incompetent every single manager was. Became a manager myself with the intention of avoiding that. I could not. Changed career path.

momojo 4y ago
> Became a manager myself with the intention of avoiding that. I could not. I appreciate the honesty. What was that experience like?
- ploum 4y ago
  As a manager, you simply don’t have the time to dig into technical issues. You can’t take uninterrupted 4 hours to enter into the code and debug something. When you are a new manager with a deep experience of the technology, you don’t see it immediately. But the longer you manage, the more your experience become irrelevant (for example: my team switched from Angular to React. I never did any React and there was no way for a manager to dig into it at the same rate as the team). It took me two jobs as a manager to realise that, at least in software development, a manager’s job is to pretend. To make uninformed decisions and lead the team without understanding anything of what is happening. You also spend your time negotiating with upper levels that want everything without even thinking about the implications (I’m not talking about costs or time, I really had meeting with really high levels managers who asked me, straight in the eyes, to make "a solution with all the advantages and without the disadvantages" and they were very proud of their line). I learned that very high level management meeting are dumb and boring, that those people don’t even have the slightest clue what they are talking about and spend hours discussing micromanagement discussion (I attended a very high-level meeting where I replaced my n+1 and they litteraly spent one hour discussing who should send an email to X to ask him to send an email to Y. I took notes of that one because I feared nobody would believe me). But I also reached the conclusion that managers are necessary. I even had a very good one who told me after one week: "I’m a manager, I have no idea how you are doing your thing. My job is to set a goal with you then your job is to ask me every time I could help with your job. Also, I’m here to insulate you from the administrative shit". I tried to become a manager like that. I also lived by the credo: "If anything fails in my team, it’s my fault, I will not put the fault to individuals in my team". I learned that this work only with very good teams and independant individuals. Some people need to be taken by the hand and a good manager will offer psychological help. But this only work if the layer above is also working that way. I ended fighting with my N+1 because they absolutely wanted to fire someone from my team. Needless to say, a CEO and friend told me I was not a good manager. I would never become one if I didn’t change the way I was looking at things. So, in conclusion : there are good managers. But they do not last long. They either quit or becomes bad managers which is the only way to climb in the hierarchy : lick upper levels asses and tell them that any problem is because of the individuals in your team. If you do that properly, you will never stay long enough in a team to have any impact anyway. Don’t try to deliver. Pretend you do it by saying it in a powerpoint. And tell your developers that everything is due yesterday.

racl101 4y ago

I hear Clock Town Day 3 music playing.

tflinton 4y ago

I've had a situation at a previous employer where a contract lost its ownership due to a reorg after downsizing. The new org was completely unaware of the contract lapse until services were turned off. The existing contract lapse had also lapsed the vendor review requirements and finances standing and thus getting a new contract in place, signed and paid took compliance, legal, finance and IT to all get together with the C-level staff to get services turned back on. Longest outage i've ever seen in my life.

AndrewKemendo 4y ago

Yup - and guess what, the only people hurt by this are the lowest level people. No manager or exec will feel any pain from this incompetence. Someone who isn't responsible but couldn't do their job effectively as a result is having a worse life now. This is one of the primary reasons why I am totally done with Tech The distance between users and builders is so excessively far apart now and the levels of abstraction for actually building things that are robust is just not even a consideration in software-centric design. Literally everything you have built after IDK 2000(?) will have exactly this issue. Just hope you're not at a scale that crushes people. Software is the language of alienation and increasingly becoming unethical, as these systems are becoming increasingly impactful on the most vulnerable with no buttresses or supports preventing this kind of malfeasance. It's not trivial. These are people's livelihoods at stake.

opamp 4y ago
> No manager or exec will feel any pain from this incompetence. Someone who isn't responsible but couldn't do their job effectively as a result is having a worse life now. > This is one of the primary reasons why I am totally done with Tech Is it that much better in other fields?
- sillyquiet 4y ago
  no, its not. Not in any white collar professions, not in any blue collar jobs, not in the military, not in academia. You are only safe when you don't have bosses and all responsibility and power rests on yourself.
  - AndrewKemendo 4y ago
    Almost...you find coworkers and share responsibility mutually. You don't need to put it all on your own back.
- AndrewKemendo 4y ago
  Yes. For example German workers have better than average protection from exploitation This is from 2018 and shows where you can actually have some rights as a worker. It's not hopeless: https://www.ituc-csi.org/IMG/pdf/ituc-global-rights-index-2018-en-final-2.pdf https://www.ituc-csi.org/IMG/pdf/ituc-global-rights-index-20...

ivraatiems 4y ago

Fascinating to read, but couldn't the author get in trouble for posting like this about one of their employer's customers? Where I work, which is a much lower-stakes environment, talking about our customers' issues or choices in public like this is a huge no-no. I'd get fired if someone found me out. Especially since if the customer is large, and the decisions have anything to do with my company's revenue, it could be considered MNPI.

somecompanyguy 4y ago

apparently nobody quantified the projected losses, because that would have caused this to not be ignored. i blame everyone involved including the service provider. everybody did something wrong

mdip 4y ago

Oh, do I remember these days. I spent 17 years at a global multi-national telecom through two mergers and a bankruptcy (and a half). We were a large organization (between 5,500 and 22,000 depending on the year). During much of that time, "who paid for what" was a big issue. The thread alludes to the issue: IT says "you need to buy new hardware every X years", department already has less than no budget, has no budget for new PCs and perceives no need for new PCs for workers that could get away with much less than they're using, now. It was a funny little game that was played because IT would get dinged in their compliance metrics if staff was out of date (and staff hated old hardware/blamed IT), but management would get dinged for spending too much and have little incentive to buy new hardware until the last second. Meanwhile, C-Level executives on both sides get to say "your problem". The difference, here, is that someone gave IT a pretty large sledge-hammer and permission to use it in order to force departments to push for more budget. In our case (and I'm sure others), a bit (a lot?) of non-compliance was normal. Personally, I think the take that "IT should own the budget" isn't as great as it sounds. It solves one problem: distributing the payment among budgets creates a "shared responsibility" that ultimately becomes "pass the buck". It also happens low enough from C-Levels that "they don't have to think about it." Having IT own the budget solves this because at least one C-Level is going to have to account for a large enough expense that it's likely to be a little better planned for. It won't always be better planned for ... depending on the company or manager, it won't often be better planned for. Unfortunately, the consequence of this poor planning only extends to the IT budget. Since compliance is non-negotiable, the largest line-item on the budget -- IT's staff -- is the next hit. In the former model, "making the budget deficit up" is naturally spread throughout the company, in this model, it all hits IT.

pas 4y ago
All in all this is business as usual. You can't work because your work device is unfit to do the work? It's not really your problem as an employee. It's the employers' responsibility to provide the tools, and it's up to each management how they solve it, with what trade offs. This one picked "bugdet first, compliance second, worker/client satisfaction and business continuity last".

cm2187 4y ago

Having worked all my life in large organisations, this sounds very familiar. A lot of people would rather the company to go bust than to challenge an internal policy written by a group of people largely above their level of competence, and completely unaware and unconcerned of the implications of their policies. One of the things you realise when you get closer to management is that those policies shouldn't be taken too seriously if they contradict common sense.

throwaway892238 4y ago
OTOH, people who risk their career to challenge an internal policy written by a group of people largely above their pay grade and not answerable to them, at best become pariahs who are ignored, and at worst are fired for "not being a team player". Most people are only concerned with their own little corporate corner and doing the least effort that keeps them in paychecks. Trying to follow the spirit of a rule rather than the letter, or pushing for change to improve things overall, is never appreciated.
- cm2187 4y ago
  By middle management maybe. By senior management, what gets you promoted is the ability to fight back, challenge things that don't make sense and to get things done.
  - Volundr 4y ago
    If your middle management is full of yes men (yes people?) it's because that's who senior leadership chose to promote. The idea that there exists some level where C-level execs in that organization are going to suddenly appreciate dissent is wishful thinking. These kinds of cultures begin at the top.
bonestamp2 4y ago
We call those people/policies the "Business Prevention Department"... In other words, they're the department that makes it difficult for everyone else to generate revenue. Sometimes they're right, but often they're too rigid to operate in reality and instead of protecting the company they actually hurt it.

deepsun 4y ago

Kudos to Compliance team to keep the high bar on their work and not let anyone go like "oh this superuser password is just for testing and telemetry, we certainly won't forget to remove it before release".

Marco30 4y ago

[dead]

Marco30 4y ago

https://support.google.com/maps/answer/6258979?hl=sk&dark=1 https://support.google.com/maps/answer/6258979?hl=sk&dark=1

sacnoradhq 4y ago

I work for a MAANG in a related area at very large scale. 1. BYOD is a terrible idea. Work and home devices should be separate, even at the cost of multiple devices. 2. More than emails, they need push notifications and background wallpaper changes on the machine with specific instructions before doing anything terrible. Soft power configuration changes can do much more without potentially destroying value. 3. It's much easier to lock out users from their account with a message about mandatory remediation steps to regain access. 4. Issue newer machines on a lifecycle with an easy return program. When people get new machines, they'll almost always cough up the old ones. This solves the issue and maintains fleet health and reliability. 5. Consider replacing laptops with VDI and Chromebooks where possible.

Chris2048 4y ago
> 1. BYOD is a terrible idea. Work and home devices should be separate, even at the cost of multiple devices. You could have your own work device at home, separate from "personal use" items. The difference is that you can upgrade & reclaim hardware without having to bring it in to IT.
- sacnoradhq 4y ago
  The IT department must horrible (or perhaps the employee doesn't understand money) if they prefer to subsidize their employer's CapEx and take a pay cut.