27 ms·

When Employees Use Software That IT Hasn’t Approved

247 points by r0n0j0y 7y ago

tyingq 7y ago
"Soon enough the CIO sniffed out the project and called her in to a disciplinary council." Somebody has apparently lost touch with who the customer for IT is.
- chrismatheson 7y ago
  I've seen this over & over, i always think of it as a sort of a god complex from "admin" rights.
  - burfog 7y ago
    It's fear of copyright lawsuits and even having the Business Software Alliance convince federal marshals raid the business. It's fear of malware. It's fear of data being locked up in the format used by an employee's personal tool. Above all, it's fear of being held responsible for any of the above. It's also a fear of being not needed.
    - raverbashing 7y ago
      And the best way to avoid all first 3 is doing your job and giving them the correct tools the employee needs (even for the things they don't "strictly" need for work but might be useful)
      - SerLava 7y ago
        Yeah, it seems like the VP's choices were A) risk IT security or B) guarantee total personal career failure Having the resources to install a CRM for someone this year is a fundamental part of any properly equipped IT department. Can you call your IT security strong if you are dangling irresistible incentives to break the security?
- spydum 7y ago
  No, CIO role often carries responsibility for security. VP violates policy is like skirting regulation - yes it cost less money, but for all you know they are not compliant with policy and aren’t doing the whole job. However it does often seem like IT doesn’t consider SaaS solutions - they always want to build something their selves without doing cost analysis.
  - tyingq 7y ago
    Sure, you need security. I would, though, expect to be summarily fired if I proposed something like a "disciplinary council" for when I had a disagreement with my customers. If you need rules to force the business to engage with you, you've failed.
    - lightbritefight 7y ago
      If a large part of your job is security, and your "customers" had opted to start stealing product off the floor because it was "easier than waiting in a line", you would be fired for not bringing it up. Thats the situation the CIO had to respond to. Just because its not part if your role to consider security implications of these SaaS services doesnt mean he's out of line for doing so.
      - tyingq 7y ago
        To true up the analogy, they are waiting in a line that's 3 years long.
  - thisisnico 7y ago
    IT Does consider SaaS solutions. When the business executives see the cost of the solutions, the business leaders decide to roll your own. SaaS isn't the end-all be all for everything. It's all about value add and achieving a goal at the end of the day. Trust me IT would much rather roll a SaaS solution, far, far less of a headache and less overhead for the department.
  - burfog 7y ago
    I have to use SaaS solutions for work, and the security situation terrifies me. I have to put my corporate password, with access to all sorts of important stuff, into a sketchy 3rd-party web site. This looks mighty bad.
- thisisnico 7y ago
  To be honest, I find it odd when you treat it as if everyone else that you work with is a customer. I don't believe in this philosophy. The business is my customer. The business is what IT is trying to protect. If you have individuals that are not following policies, they would be disciplined like HR would discipline for not following policies. It's all in place to protect the business and what's best for the business. Sure you'd like admin rights to your own machine, that will help you individually, but will it help the business as a whole if we get hit with cryptowall again?
  - mc32 7y ago
    Agreed. Everyone is beholden to the company and its principles, not the CEO or manager, though there should be alignment, but when there isn’t, then it’s the company.
  - kyriee 7y ago
    Did you miss the line where it was written "before we started the program, we were losing revenue. Now we increased it by 1MM$ / month"? It seems that the whole point of the article is that while IT thinks that it is serving the customer (the business), it actually lost track of what it was needed to: helping it succeed.
  - bob33212 7y ago
    It goes both ways. Is the business user taking inappropriate risks to get his own work done quicker? Or is IT denying access to vendors to minimize his personal responsibility?
  - scarface74 7y ago
    I find most “IT security policies” that hamper developers to be mostly security theatre. No matter how many policies they put in place, since they aren’t developers, one junior developer can write: var sql = “select * from Customer where firstname = ‘“ + firstname + “‘“; And thwart all of your security “best practices.” I was the lead dev at a medium size non tech company, and the hoops I had to go through to get anything done dealing with the “security team” was ridiculous and of course I didn’t have access to production to troubleshoot for awhile. I had ultimate control of all the code that did go through the process. If I were to do something stupid or purposefully malicious, while I didn’t have access to the environment - my code did. As far as someone mistakingly installing a “crypto wall”, if a user can download a program that doesn’t require admin access, that program has access to the user’s files. The system can be restored much easier than the user’s data.
- mc32 7y ago
  It depends. If the company has instituted SSO and MFA and someone goes out and uses a solution that is outside of that, they could be exposing the company to liability.
  - tyingq 7y ago
    It's the fact that something like a "disciplinary council" exists, and that it was the first tactic the CIO went with that bothers me. There's a reason they went around the CIO. I suspect if the CIO had met with this person they could have learned and helped.
    - mc32 7y ago
      The problem is bureaucracy and unwillingness of IT to be agile and responsive. Their weapon is procedure which quashes initiatives. At the same time though, going outside like that can have major effects on compliance if they are subject to audits, aside from the security considerations.
    - PeterisP 7y ago
      In the situation described in the article, the manager had gone to the CIO but CIO refused to help them. "The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP." The manager had a valid and valuable reason ("increased revenue $1M per month") to require a service that CIO and their organization was unable to provide in a reasonable timeframe, but other companies on the market were.
- imtringued 7y ago
  The "customer" doesn't care about IT and wants to do things behind the back of the CIO. The "customer" goes to extreme lengths to hide the fact that he is violating company policy by purchasing SaaS with his own credit card. If that employee leaves one day, all the data inside the SaaS is gone because nobody else knew about it. A malicious employee could also use it to extort the company.
adwww 7y ago
how about trust your staff.
- geekamongus 7y ago
  What could possibly go wrong?
- pjmlp 7y ago
  Unfortunely many prove that they aren't worthy of such trust and open the company to liability issues, data loss, lawsuits and other security issues.
- jcims 7y ago
  Domain Admin for everyone! Trust is always contextualized. I trust my mom to watch my kids but not to remove my appendix. The purpose of technical controls and security policy is to wall off areas where employee capabilities or motivations are too uneven or complex to safely expose them without a risk of loss that's incompatible with the appetite of the business.
- justinclift 7y ago
  Maybe, "empower your staff as they each prove competence", instead?
burfog 7y ago
I've been amused by VMWare being on the strictly-enforced official software list, and the VM being considered data. Nothing in the VM counts as software! It's not even being sneaky. Official policy is that the VM is data.
geekamongus 7y ago
It sounds like the author recommends embracing the Agile philosophy of letting your teams choose their tools, then working with IT/Sec to make sure implementation is sound. I like that philosophy.
- FooHentai 7y ago
  Great until you have five teams, each having chosen a different tool, and now you're wondering why the IT support costs are out of control. Still possible to support but requires a different model e.g. one where IT delivers a new, unconfigured workstation to your new team member and it's up to them to build it. If it breaks, their loss of productivity is their problem and not ITs. Authority for something (e.g. software selection) must go in conjunction for responsibly for consequences arising. Those things must always move in lockstep to avoid perverse outcomes.
  - closeparen 7y ago
    Our IT does not support development environments. Just network, backups, printing, client certs. Dev tools support is all essentially peer to peer and ad hoc, with escalation to the internal owner of the tool (another engineer) sometimes possible. If you mess up in an unrecoverable way, IT will give you a loaner to work on while they reimage your machine. It works fine.
S_A_P 7y ago
I see this a lot in consulting. When a new CIO (or CEO or other C level) arrives, they want to make their mark with a digital transformation intiative. This usually just means that the new C level employee is coming into a medium to large business and would like to add a bullet point to their resume and get that new shiny object everyone is talking about. Tableau, Salesforce, Data lakes, blockchain, ERP, Identity Management and "cloud" projects are often the result. It seems to also stem from the new C level employee having a close relationship with a sales rep/partner/C level employee at the vendor. Left a project a couple years ago that had Hadoop interfaces from every system. The user count of all this data? exactly 0. One somewhat disturbing trend I've seen at some of the largest corporations- cut/outsource IT support staff to near egregiously low levels to "save money". At the same time kick off 7-9 figure ERP/consulting projects that at best provide fractional value to the organization. Of course there are counterpoints to this. One of Houston's major pipeline operators pulled off a digital transformation and actually ended up with well designed, highly integrated and easily maintained systems. It took about 5-7 years and had a few reboots, but it eventually landed. That brings me to my final point. These projects often have a timeline that is divorced from reality. Whatever time frame you think a major IT project will take. Double it. twice, then add 50% and you are close. It also seems that C level folks are hesitant to hire boutique/small shops that have industry experience and years of experience in favor of big consulting. Nobody every gets fired for hiring Accenture/Deloitte/PwC. What usually happens in the non trivial niches is that these big shops sleeve the boutiques through them to get things done...
- maxxxxx 7y ago
  All you are writing sounds exactly like my company.
  - ct520 7y ago
    Ditto. Working in the enterprise industry I can relate to a lot of what SAP is talking about.
- lzol 7y ago
  This resonates so much and seems to be a major trend in non-traditional tech companies. I've mostly worked in the financial industry and the executives' knowledge of technology is almost always horrible. As you said, a couple buzz words and very set opinions on the ways to do things. It's like they get pet projects in their head from reading an article in a magazine and get locked into it. I don't really have an issue with CXXs being ignorant about a subject. No one knows everything. What I do have an issue with is when they act like an expert ignoring all the people who are actually experts in a particular area. It'd be like me going into a room full of IT people and saying EBITDA a bunch of times claiming to be an accounting wizard ready to lead a major initiative. It's frustrating but I've learned all I can really do is smile and watch the show.
maxxxxx 7y ago
This is exactly extremely common. In my company there is this constant battle about the devs having admin rights on their machines. We need admin rights to do our job. We have had dozens of meetings explaining the situation but IT can’t come up with a solution so the devs go around security because they have no alternative if they want to finish their work . Same with Dropbox. They block it but we have suppliers who use Dropbox. So the result is that people download confidential files from Dropbox on their home computers or phones and transfer them to their work machines. In my view security shouldn’t be isolated at corporate headquarters but they should be close to the end users so they see what users need to do and help them to balance security with getting things done. They can’t just block stuff without providing alternatives or they will either hurt the business or they will be circumvented.
- 7748394773639 7y ago
  I have my own PC strapped under my desk connected to public Wi-Fi. I use it to do all my work. My company pc is left turned on but disconnected, sitting on top of my desk to dissuade suspicion.
  - maxxxxx 7y ago
    I have a WiFi router hidden so we can do some specific network testing. We also have a consumer DSL line so we can test on a network where IT doesn’t block random stuff. It’s not even much of a secret. Everybody local knows this stuff and every six months a guy at IT headquarters throws a fit, doesn’t provide an alternative and gets ignored. I have thought about sending them recordings of previous meetings so we don’t have to repeat the series of same meetings every six to twelve months.
    - miahi 7y ago
      We had the same kind of "hidden" router at my previous employer, as we had a team of ~10 developers who needed a lot of different access to different servers and between eachother. We put the name of another tenant in the office building as the SSID :). After about a year the network through that router started to slow down. When we checked why we realized that we had more than 40 wifi clients, as other (non-dev) teams learned about the network through the grapevine and actually the new employees from sales (the neighboring office) were told to connect to that network directly - as the process to connect to the "normal" network was too much of a hassle.
brixon 7y ago
Let's ignore the SaaS security issues for a second. When IT says "No" it's not like the area asking is going to go away and not try to solve their problem. Organizations are going to find ways to solve their issues and IT can either help from the beginning or help clean up the mess later. I try to take the stance of offering the right solution and a lot of the times a now solution at the same time. There is no saying "No" in the long term, either help them now or get stuck with the shadow solution the magic macro guy cobbled together that became a critical business function.
- rosege 7y ago
  In my last job I tried to handle this in a similar way. The issue we ran into though was that often these managers would not properly evaluate the software. They would get wowed by the sales guys and sign up for huge contracts without, sometimes, even checking with IT or testing other vendors.
  - tvanantwerp 7y ago
    Whenever I'm wearing my IT hat at work, a big part of it is being a detective looking for clues that somebody, somewhere, is about to do this. Then I can insert myself into that process. It would be preferable to be included in the beginning, but one must work within the reality of the situation.
- x38iq84n 7y ago
  I have seen IT being unaware of and unwilling to meet requirements of highly specialized technical teams, such as network engineering. You cannot have a TELNET client because the use of TELNET is prohibited by corporate policy, test TCP connections another way. You don't need vim when you have vi. You can't have admin rights but we don't support drivers for RS232 dongle so nope. Sometimes it's quite a challenge to get some work done.
  - user5994461 7y ago
    Use netcat. Telnet is pretty hard to procure since it's not included by default since Windows 7.
drevil-v2 7y ago
I still shudder thinking about my time working as a developer on corporate IT locked down IBM leased laptops. Every time I did npm install I needed to request admin access to Windows which took 2-3 hours to action by IBM team sitting on the other side of the world in India. One day a grey beard took pity on me and installed a Linux VM where I was admin, copied the security certs from the Windows host and I could access all corporate resources at my leisure. Never logged a single IT Helpdesk ticket after that.
- abledon 7y ago
  wow. that sounds horrible. did they ask you fizzbuzz or mergesort in the interview?
  - maxxxxx 7y ago
    Probably not. The interviewer didn’t know how to do fizz buzz himself.
    - mxuribe 7y ago
      When has the lack of info/knowledge ever stopped interviewers from asking such questions? ;-)
    - deleted 7y ago
      [deleted]
- deleted 7y ago
  [deleted]
- flurdy 7y ago
  Yup, back then my way around corporate locked down Windows machines was to only get permission to install VMware Workstation or Virtualbox. Then fullscreen Ubuntu in a VM from then on. Slower but no restrictions. Even better when Workstation supported multiple screens.
  - Macha 7y ago
raxxorrax 7y ago
> If you don’t think this is happening in your organization, think again That story probably never happened anyway. But the essence of the article is very true. I never have been in a corp where IT enforces 100% conformity anyway (apart from medical industry). Sure, there are actual successful attacks, but that is mostly not the fault of unsanctioned programs. But there are systems where people should not just start to use any system, because information gets lost on the way. That would include CRM and ERP in my opinion. That a company can exist without a CRM is questionable to begin with and solutions are plentiful. If they did not have anything like that... If the story were true, it would not be the fault of Chief Input/Output.
- cannonedhamster 7y ago
  I've been in corporate IT where this happened. All company apps were built internally. None were able to run on anything past Windows XP. On top of my regular help desk, asset management, software project, and lease refresh program I was also somehow supposed to make the software work with Windows 7 as they had let the developers go. This is the same company that refused my sane security requirements and ignored just about everything until too late. I hear they have since outsourced IT and networking and it's failing dramatically, but they are saving money right?
deleted 7y ago
[deleted]
philipodonnell 7y ago
> The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?” Shut it down right now and ask the VP to tender their resignation. Any company doing a 3-year SAP implementation is a very large company. That $1M in additional revenue pales in comparison to the risk introduced by sharing company or personal customer data with a vendor who has not passed the required security auditing. Data is no longer a thing to be thrown around in search of additional revenue and "but I made money" or "I had to because IT is slow" is not a post hoc rationalization for the behavior. Regardless of the merits of large enterprises acting this way, this is a VP who clearly cannot function within the enhanced risk-controlled environment of one and should find a position with a smaller company where they have more freedom to pursue personal initiatives at the VP-level. Those companies exist. Go find one.
- enraged_camel 7y ago
  >>Any company doing a 3-year SAP implementation is a very large company. Not at all. I have a client with ~100 employees who are past year 2 of their Salesforce implementation because the director of technology keeps changing priorities and project requirements.
  - philipodonnell 7y ago
    That's fair, but just using the time-frames in the article, 6 months had passed before the VP got caught and still had 3 years left on the implementation. I think a director of technology who take 3.5+ years to do a CRM implementation at a 100-person company... isn't doing a very good job. :-)
- arkades 7y ago
ConfusedDog 7y ago
This is my current situation. Being a SWE, IT and security are always putting out fires with networks or upper echelon cybersecurity violation complaints (mostly people downloading software without authorization). They have very little time, almost none for investigating new software, and all software must be installed by them. End of the day, nothing gets done on our work computers. I once waited two months for them to say no for a piece of solution we as the team approved. It's absolutely frustrating.
noonespecial 7y ago
Its the same dilemma companies face with their legal team. The "safest" thing to is nothing at all so sometimes the overabundance of caution hamstrings business growth. That's what the CEO is there to figure out.
exabrial 7y ago
IT is a service to the rest of the company. If you don't approach it with a servant's heart, people will go find their own solutions without you and you'll be part of the cleanup crew.
jmkd 7y ago
Joining Google was an eye-opener for me on this. Was the first time I encountered an IT department (TechStop) that didn't act like a police force and instead had your back, helping you get where you needed to be. Was always the first thing I would show guests on a tour of the campus.
- repolfx 7y ago
  But you're probably a developer? TechStop is/was great. But Windows users had locked down workstations where IT whitelisted binaries. I assume the approval process sucked about as much as normal. Many Google employees use desktop Linux which is basically unheard of outside the tech world. That by itself simplifies things quite a bit. Not many people writing viruses posing as screensavers for Google's in house Linux strain. Anyone who cracks that is probably an APT attacker and those require different approaches.
- alexhutcheson 7y ago
  CorpEng also does an amazing job building in-house apps & tools to let people get their job done.
tvanantwerp 7y ago
The ideal IT team is one that proactively learns the needs of others in the business and works with them to solve problems. It's no wonder that so many companies end up with shadow IT when so many IT teams are just people who tell you "no" whenever you ask for something. Doing it right is harder in the near-term, but much easier in the long-term as you're not putting out so many fires or going to "disciplinary council" meetings.
deleted 7y ago
[deleted]
eithed 7y ago
To me it reads like this - VP didn't care about the consequences of utilizing their solution and didn't care about IT; they simply wanted their stuff done, without acknowedgling prioratization of tasks. The proper way this could have been resolved is by VP utilizing people's skills they've hired. Does this solution look good and will accomplish the task that was prioritized? Excellent! Pass it to IT to evaluate. If the task has specification - excellent, have somebody in IT look for a product that ticks all the boxes and let's choose it together.
fphhotchips 7y ago
How many years since The Phoenix Project and this conversation has barely moved an inch? CIO probably wins this battle and gets the VP fired, but will be mystified when they're reporting to the CFO or a Chief Digital Officer when it happens 3 more times by the end of the year.
- scarejunba 7y ago
  It is wonderful. Each crack in a behemoth’s process is a point of leverage for a startup. We will outcompete by being better.
- finnthehuman 7y ago
  >How many years since The Phoenix Project and this conversation has barely moved an inch? People are still fighting the lessons in The Mythical Man Month; it's gonna be a while.
hartator 7y ago
Isn't IT something from the past? I would expect people knowing how to use a computer and what they need to do their job.
- kleborp 7y ago
  As a lowly help desk technician perusing this thread, you couldn't be any further from the truth. The software devs never open tickets, everyone else does and its for the most banal problems
  - rhinoceraptor 7y ago
    I've opened plenty of tickets as a developer, mostly the tickets are IT's garbage software or horrible network setups preventing me from doing my work.
  - silverfox17 7y ago
    A lot of people who post here don't seem to realize that a lot of companies still have IT departments.. a lot of people here are also developers who don't realize they can easily do things to compromise data, even if they think they know better.
- Macha 7y ago
  When MacOS borks itself in an update because the endpoint management software does something funky with partitions, I'd rather delegate that to a team that's handled three cases of that this week and get on with my job description than diagnose the exact steps to resolve a problem that'll often require access I don't have anyway.
- jillesvangurp 7y ago
  My thoughts exactly. It's been a while since I last saw a company with an actual IT department and even longer where they had an actual clue. The reality is that IT in most SMEs simply sucks and no longer requires a college education. Working in IT for an SME is not a career plan. You're at constant risk of being outsourced and essentially all you do is done better by a gazillion companies as a service that probably cost less than a few months of your salary. Frankly, most SMEs would be better off doing exactly that. Most startups I work with do this from day one for obvious reasons. Once you hit a certain scale different dynamics may kick in but even then, outsourcing is an option. As a freelancer these days, I bring my own laptop and am granted access to stuff for the duration of the project. It's understood and expected of me that I do such things as encrypt disks, use 2FA, and don't use "secret" as the password. Most stuff I access for these projects is SAAS based. I'd probably walk away from projects where that wasn't the case.
protomyth 7y ago
Its the user who downloaded a program they "needed" which had malware which sent out a lot of spam email because this was a user that did announcements which basically got an e-mail server listed on blacklists that creates these IT policies. You want to treat people like responsible adults, but they aren't the ones who have to deal with the fallout. Developers know the score for the most part, so full privileges are expected with the caveat, if it all goes bad, we are wiping the machine[1], not doing a recovery. IT dreads the moment we are called to account for something some user decided they needed to do. 1) most developers understand backup tools and code control - those that don't, well...... with great power comes great responsibility
- dx87 7y ago
  Yep, a company I worked at hired a tech writer that downloaded some cracked version of software that included ransomware on their first day of work because they said they didn't want to wait for the company to get them a legitimate copy.
  - toyg 7y ago
    Well, that used to be common practice... in the '90s. Thank $deity for the rise of opensource.
    - protomyth 7y ago
      OpenSource still isn't a force in that particular area. Its Microsoft or Adobe there.
      - toyg 7y ago
        Yeah, what I meant is that, these days, the culture is such that one assumes there will be an OSS tool somewhere, before one even considers a sketchy binary. Maybe the OSS option will be inferior, but it's almost guaranteed that it will get some stuff done and not nuke your machine. That's a significant improvement (of course we know that having a github repo is no guarantee and blablabla, but it correlates well enough for most purposes).
thomasjudge 7y ago
I work in an IT organization & I see (in the sense of witness) both sides of this. We are over-tasked and under-resourced and new projects/ideas/initiatives that come in the door go into a backlog of requests. So I see business/end users signing up on their own for SAAS solutions to solve their problems.
- argd678 7y ago
  Right, the CIO is also being held to a lower standard than a P&L. The CIO could have planed out an interim solution to meet the business needs quicker. If the tables were turned, say the CIO needed to deliver a service and didn’t have a big enough budget, then what?
sgt101 7y ago
Well, that cleared that up then! Gosh I had no idea that the solution would be so simple. It does shock me that the people who've had their whole infrastructure compromised and held to ransom by viruses and the people who've been held over a barrel by suppliers or had vast amounts of money burned by being locked into a dozen vendor contracts for the same service are so silly and hysterical about it when the solution is as simple as "identify when you need to be best in class and stay small everywhere else".
thisisit 7y ago
As a born and bread corporate (mostly banks) corporate IT guy, I used to frown upon this behavior. Then I got one of my bigger career breaks because the finance team went behind IT, bought a software and installed it in a machine which kept under their desk. They further hired people from an IT service company to configure the machine. The configuration was so bad that it exposed the company's network to whole wide world. Google contacted the company and after searching high and low IT security managed to track the pc down and take it away. Finance team promised to hire someone with skillset required to run the software in a closed environment. And that's how I ended up getting my job.
- lapnitnelav 7y ago
  Really interesting story, especially if you put it in the context of being corporate world and a finance team, both of which aren't the type to rock the boat. It says a lot about the struggles of enabling change in structures with strong silos that (what are probably among the most risk averse type of) people would go to such lengths.
vinay_ys 7y ago
Does your IT team use key loggers or other employee monitoring software at your company? I hear some big trillion dollar companies do this. Is that true?
jrjarrett 7y ago
This thread hits home. I switched jobs a few years ago because the IT policies on workstations were being ratcheted down to make my job as a developer difficult to impossible. Now, the company I work for, ostensibly a _software_ company, got its ISO certifications, which meant policies and procedures that make developing hard or impossible again. How does a software business _successfully_ implement stringent access controls while still allowing for efficient software development? I'd like to see/hear what works.
- CrossWired 7y ago
  I'm heading down this path right now. How do I obtain my certs while also allowing enough freedom for the dev teams to operate. We have to deal with the fallout when they screw something up, there has to be a happy medium somewhere.
  - user5994461 7y ago
    What's the issue specifically? Developers don't need admin rights for much of anything in this decade. No need to bother with that. Common software has to be made available in self-service, so developers can install development tools like notepad++ or visual studio. Deployment is usually the challenge because you have to store binaries somewhere, copy it to some random servers and finally execute it, each step causing numerous security headaches, so there has to be some approved tooling to handle that.
    - stefan_ 7y ago
      You can't get past "docker run" without admin rights. No, the challenge is access to production data without auditing.
    - jrjarrett 7y ago
      Developers don't need admin rights for much of anything in this decade. No need to bother with that. Please defend this position. My experience, mostly with Linux-like tools, is that those tools are built with the assumption they are being used by someone who knows what they are doing, and that they have the appropriate level of control of the machine -- they are tools for professionals to build tools. If you don't have rights to install or execute them, you're done. You can't make any forward progress.
davvolun 7y ago
> The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?” Maybe not for this particular project, but another interpretation of that is "who cares about security if we're making money" which is a very dangerous argument as well.
- yebyen 7y ago
  When a person says "we need this infrastructure project" and a project is commissioned, acknowledging the need, it is in my experience that unfortunately that person's job function is rarely placed on hold until the appropriate infrastructure has been made available. "Who cares about your pie-in-the-sky infrastructure project, my boss continues to measure our real performance with basic accounting, and is expecting to be able to report on growth each quarter, which I can't help without tools" seems to be a bit closer to the argument posed here, IMHO.
  - davvolun 7y ago
    Of course that's the real question with the actual story in the actual article -- could they have implemented their security on the temporary infrastructure, or "good-enough" security, if the CIO knew about it? In that story, is the blame on the VP for going ahead instead of getting dialogue started between CEO, VP and CIO? Is it on the CIO for just saying "no" instead of recognizing the need and the value? Is it on the CEO for failing to empower the VP and CIO to get that conversation started themselves? And then, it's all well and good to worry about the bottom line first, until you're sitting in Equifax's shoes right.
    - yebyen 7y ago
rsuelzer 7y ago
Our IT security department was incentivized to deny everything from new tools to new internal applications. We had an outside firm making security decisions and if there were any security issues it would end up being on them. So as long as they did not allow us to release any products and or install any software they could not be held responsible. I made friends with a lower level contractor who told me off the record to use my judgement on what to install to get the job done, because the security department would never approve anything new unless directly instructed to by the CEO. Fast forward three months, there was a major security flaw on our website (also built with outsourced labor) which allowed anyone to access private data without a login. A few of us had reported to the security department that the code running the website was so poorly written that the odds of being insecure were close to 100 percent. We suggested upgrading the website and rewriting the code, and management was on board with this but security department refused to allow us to use any new frameworks since they were not approved. Of course in a matter of a few months the site was hacked and millions were spent as a result. I quit this job after we were unable to release several products after a year even though we jumped through every hoop we needed to. That department killed all innovation.
- closetohome 7y ago
  I think it really depends on the company. If you're something like a nontechnical non-profit, sure, turn that decision making over to IT. In that case IT is performing a vital, skilled function. But in most software shops, the workers are probably more qualified than the IT department to be making decisions about what applications to use, and what kind of security they need. IT is just there to make things run and fix them when they break. They don't really need to offer guidance.
la_barba 7y ago
Hehe, if you think this is nuts, come to pharma. We can't do jack shit with our machines. If you so much as change the time on your machine, that is a 'data integrity breach', and if your actions are determined to be malicious it can result in a firing.
- GuB-42 7y ago
  To be honest, changing the time on a machine is a very serious concern. Accurate timekeeping is crucial in security, that's how you connect events together.
  - la_barba 7y ago
    Well, all the rigid policies like no dropbox or no FTP or no whatever, also arise from serious concerns. I just wanted to point out another seemingly innocuous one. Most of our equipment is not internet connected, and we need to manually change the time for daylight savings or other corrections. We have a company policy and procedure to do that periodically so that our audit trails are accurate. Sometimes folks get busy and the shop floor guys take matters into their own hands.
alkonaut 7y ago
Security training focuses way to much on email phishing and not enough on this kind of stuff. Actually getting your work done, managing your own computer. Of course people can't be trusted if they havem't been trained. How to handle USB drives. What and from where you can download and run programs. What actually IS a program and what isn't. Many of us learned this the hard way by playing lots of cracked games in the 90s. But not everyone did that. Try explaining to a non-technical person how how a desktop background image isn't a program so it's basically safe to grab from anywhere, while a screen saver is definitely a program and usually unsafe to get from most places, and a word document is some times a program that might eat your computer. Training could involve things like "which of these 5 webbpages would you consider it safe to download and run executable from"? Having too cumbersome rules around security just means it's ignored or circumvented, increasing risks.
- JakeTheAndroid 7y ago
  It's tough, I give security awareness trainings myself and I completely agree with what you're saying. However, that's a lot of information to give to a group of new employees that can span any department and technical understanding. I actually was talking today with a customer during a logical assessment about if I talked about downloading malware in the training. I dedicate an entire section to downloading documents, but I don't really give people the information you're talking about. I tell them how to avoid ever having to download anything, and if they must do it, how to try and do it properly. All of this is ended with the process on how to report incidents because eventually something bad will happen. As a company you kind of expect this to be solved at a number of layers. Endpoint management should hopefully help resolve this issue. Restricting web access where it makes sense can help. Sec Awareness Training helps keep people aware. Etc, etc, etc. You hope your controls are what save you from incidents, because there is no way you can effectively train your entire company on security topics to a degree that they can make good, security conscious decisions. That said, many of these SAT's are really just checking compliance requirements, because thats the real need. I put my own training together starting with what I know needs to be covered for compliance (pii handling, passwords, acceptable use policy, common threats, security incident response reporting, etc). Anything else that makes it in is purely because I have extra time and I know it to be important.
noja 7y ago
There's a difference between using software that IT has approved and shipping customer data outside of the company.
TheRealDunkirk 7y ago
I just witnessed a very similar situation, on a smaller scale, but there are many of these in my company, and they add up. Boss: "We need access to the database of our primary application that you wrote for us so that we can pull the data into this new tool to track progress." IT: "No. Not only can we not give you access to YOUR data in YOUR application that we wrote on YOUR dime, we will not allow you to have this new application written by someone who isn't in our group. If you wanted something like this, why didn't you just ask us? We would have written this for you." Boss: "We had a meeting about this over a year and a half ago, and you told me that you didn't even have the time to discuss it further." IT: "... Well, we're still not going to let you do this." IT is effectively holding the rest of the company hostage, and the corporate technical debt is becoming epic. So skunkworks solutions will continue to be developed.
dingo_bat 7y ago
In our company people just started using free slack en mass, boycotting the horrible IT approved Skype for business. When it was discovered that thousands of employees were using slack, the CTO had to step in and tell IT to fuck off, and started paying for the full version.
jmspring 7y ago
This brings back a memory. The only time I was fired "for cause". Summer after my freshman year at college, I was temping and got an assignment doing real estate purchase comps with a company in the East Bay. At the time, there were laser printers, but often printing sucked up CPU time and let's just say multitasking was still not a widespread thing. I found myself tired of sitting around. I found a TSR / print spooler that would use RAM and offload the process of printing. This allowed me to keep working. My productivity (as a temp) was higher than many others including the person I was "reporting to" at the company. They found the print spooler, labeled it "unapproved software", and I was walked out the door. The funny thing is, a friend at the time (and I didn't realize it) was higher up in the management. He reached out to me on a multi-line BBS that was popular in the area and offered me a full time job a few days later. I was in school and obviously declined. Working the rest of the summer for a Chemical Engineer in Martinez/Benecia ended up being incredibly more interesting. So it was a net win.
jccalhoun 7y ago
I just wish I had a decent computer at work that didn't have 3rd party antivirus that would just slow the software to a crawl.
analog31 7y ago
Something that's crossed my mind is John Gall's observation that complex systems operate in failure mode 100% of the time. I understand "failure mode" to mean that built-in guards have been bypassed in order to enable the system to do anything at all. Germane to this thread, the "guards" are IT approvals. I suspect that if a business is complex enough to have IT policy, that policy is always being bypassed in some way, at any given time. Somebody is using unofficial software, or using official software in an unofficial way.

247 points by r0n0j0y 7y ago

"Soon enough the CIO sniffed out the project and called her in to a disciplinary council." Somebody has apparently lost touch with who the customer for IT is.

chrismatheson 7y ago
I've seen this over & over, i always think of it as a sort of a god complex from "admin" rights.
- burfog 7y ago
  It's fear of copyright lawsuits and even having the Business Software Alliance convince federal marshals raid the business. It's fear of malware. It's fear of data being locked up in the format used by an employee's personal tool. Above all, it's fear of being held responsible for any of the above. It's also a fear of being not needed.
  - raverbashing 7y ago
    And the best way to avoid all first 3 is doing your job and giving them the correct tools the employee needs (even for the things they don't "strictly" need for work but might be useful)
    - SerLava 7y ago
      Yeah, it seems like the VP's choices were A) risk IT security or B) guarantee total personal career failure Having the resources to install a CRM for someone this year is a fundamental part of any properly equipped IT department. Can you call your IT security strong if you are dangling irresistible incentives to break the security?
spydum 7y ago
No, CIO role often carries responsibility for security. VP violates policy is like skirting regulation - yes it cost less money, but for all you know they are not compliant with policy and aren’t doing the whole job. However it does often seem like IT doesn’t consider SaaS solutions - they always want to build something their selves without doing cost analysis.
- tyingq 7y ago
  Sure, you need security. I would, though, expect to be summarily fired if I proposed something like a "disciplinary council" for when I had a disagreement with my customers. If you need rules to force the business to engage with you, you've failed.
  - lightbritefight 7y ago
    If a large part of your job is security, and your "customers" had opted to start stealing product off the floor because it was "easier than waiting in a line", you would be fired for not bringing it up. Thats the situation the CIO had to respond to. Just because its not part if your role to consider security implications of these SaaS services doesnt mean he's out of line for doing so.
    - tyingq 7y ago
      To true up the analogy, they are waiting in a line that's 3 years long.
- thisisnico 7y ago
  IT Does consider SaaS solutions. When the business executives see the cost of the solutions, the business leaders decide to roll your own. SaaS isn't the end-all be all for everything. It's all about value add and achieving a goal at the end of the day. Trust me IT would much rather roll a SaaS solution, far, far less of a headache and less overhead for the department.
- burfog 7y ago
  I have to use SaaS solutions for work, and the security situation terrifies me. I have to put my corporate password, with access to all sorts of important stuff, into a sketchy 3rd-party web site. This looks mighty bad.
thisisnico 7y ago
To be honest, I find it odd when you treat it as if everyone else that you work with is a customer. I don't believe in this philosophy. The business is my customer. The business is what IT is trying to protect. If you have individuals that are not following policies, they would be disciplined like HR would discipline for not following policies. It's all in place to protect the business and what's best for the business. Sure you'd like admin rights to your own machine, that will help you individually, but will it help the business as a whole if we get hit with cryptowall again?
- mc32 7y ago
  Agreed. Everyone is beholden to the company and its principles, not the CEO or manager, though there should be alignment, but when there isn’t, then it’s the company.
- kyriee 7y ago
  Did you miss the line where it was written "before we started the program, we were losing revenue. Now we increased it by 1MM$ / month"? It seems that the whole point of the article is that while IT thinks that it is serving the customer (the business), it actually lost track of what it was needed to: helping it succeed.
- bob33212 7y ago
  It goes both ways. Is the business user taking inappropriate risks to get his own work done quicker? Or is IT denying access to vendors to minimize his personal responsibility?
- scarface74 7y ago
  I find most “IT security policies” that hamper developers to be mostly security theatre. No matter how many policies they put in place, since they aren’t developers, one junior developer can write: var sql = “select * from Customer where firstname = ‘“ + firstname + “‘“; And thwart all of your security “best practices.” I was the lead dev at a medium size non tech company, and the hoops I had to go through to get anything done dealing with the “security team” was ridiculous and of course I didn’t have access to production to troubleshoot for awhile. I had ultimate control of all the code that did go through the process. If I were to do something stupid or purposefully malicious, while I didn’t have access to the environment - my code did. As far as someone mistakingly installing a “crypto wall”, if a user can download a program that doesn’t require admin access, that program has access to the user’s files. The system can be restored much easier than the user’s data.
mc32 7y ago
It depends. If the company has instituted SSO and MFA and someone goes out and uses a solution that is outside of that, they could be exposing the company to liability.
- tyingq 7y ago
  It's the fact that something like a "disciplinary council" exists, and that it was the first tactic the CIO went with that bothers me. There's a reason they went around the CIO. I suspect if the CIO had met with this person they could have learned and helped.
  - mc32 7y ago
    The problem is bureaucracy and unwillingness of IT to be agile and responsive. Their weapon is procedure which quashes initiatives. At the same time though, going outside like that can have major effects on compliance if they are subject to audits, aside from the security considerations.
  - PeterisP 7y ago
    In the situation described in the article, the manager had gone to the CIO but CIO refused to help them. "The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP." The manager had a valid and valuable reason ("increased revenue $1M per month") to require a service that CIO and their organization was unable to provide in a reasonable timeframe, but other companies on the market were.
imtringued 7y ago
The "customer" doesn't care about IT and wants to do things behind the back of the CIO. The "customer" goes to extreme lengths to hide the fact that he is violating company policy by purchasing SaaS with his own credit card. If that employee leaves one day, all the data inside the SaaS is gone because nobody else knew about it. A malicious employee could also use it to extort the company.

adwww 7y ago

how about trust your staff.

geekamongus 7y ago
What could possibly go wrong?
pjmlp 7y ago
Unfortunely many prove that they aren't worthy of such trust and open the company to liability issues, data loss, lawsuits and other security issues.
jcims 7y ago
Domain Admin for everyone! Trust is always contextualized. I trust my mom to watch my kids but not to remove my appendix. The purpose of technical controls and security policy is to wall off areas where employee capabilities or motivations are too uneven or complex to safely expose them without a risk of loss that's incompatible with the appetite of the business.
justinclift 7y ago
Maybe, "empower your staff as they each prove competence", instead?

burfog 7y ago

I've been amused by VMWare being on the strictly-enforced official software list, and the VM being considered data. Nothing in the VM counts as software! It's not even being sneaky. Official policy is that the VM is data.

geekamongus 7y ago

It sounds like the author recommends embracing the Agile philosophy of letting your teams choose their tools, then working with IT/Sec to make sure implementation is sound. I like that philosophy.

FooHentai 7y ago
Great until you have five teams, each having chosen a different tool, and now you're wondering why the IT support costs are out of control. Still possible to support but requires a different model e.g. one where IT delivers a new, unconfigured workstation to your new team member and it's up to them to build it. If it breaks, their loss of productivity is their problem and not ITs. Authority for something (e.g. software selection) must go in conjunction for responsibly for consequences arising. Those things must always move in lockstep to avoid perverse outcomes.
- closeparen 7y ago
  Our IT does not support development environments. Just network, backups, printing, client certs. Dev tools support is all essentially peer to peer and ad hoc, with escalation to the internal owner of the tool (another engineer) sometimes possible. If you mess up in an unrecoverable way, IT will give you a loaner to work on while they reimage your machine. It works fine.

S_A_P 7y ago

I see this a lot in consulting. When a new CIO (or CEO or other C level) arrives, they want to make their mark with a digital transformation intiative. This usually just means that the new C level employee is coming into a medium to large business and would like to add a bullet point to their resume and get that new shiny object everyone is talking about. Tableau, Salesforce, Data lakes, blockchain, ERP, Identity Management and "cloud" projects are often the result. It seems to also stem from the new C level employee having a close relationship with a sales rep/partner/C level employee at the vendor. Left a project a couple years ago that had Hadoop interfaces from every system. The user count of all this data? exactly 0. One somewhat disturbing trend I've seen at some of the largest corporations- cut/outsource IT support staff to near egregiously low levels to "save money". At the same time kick off 7-9 figure ERP/consulting projects that at best provide fractional value to the organization. Of course there are counterpoints to this. One of Houston's major pipeline operators pulled off a digital transformation and actually ended up with well designed, highly integrated and easily maintained systems. It took about 5-7 years and had a few reboots, but it eventually landed. That brings me to my final point. These projects often have a timeline that is divorced from reality. Whatever time frame you think a major IT project will take. Double it. twice, then add 50% and you are close. It also seems that C level folks are hesitant to hire boutique/small shops that have industry experience and years of experience in favor of big consulting. Nobody every gets fired for hiring Accenture/Deloitte/PwC. What usually happens in the non trivial niches is that these big shops sleeve the boutiques through them to get things done...

maxxxxx 7y ago
All you are writing sounds exactly like my company.
- ct520 7y ago
  Ditto. Working in the enterprise industry I can relate to a lot of what SAP is talking about.
lzol 7y ago
This resonates so much and seems to be a major trend in non-traditional tech companies. I've mostly worked in the financial industry and the executives' knowledge of technology is almost always horrible. As you said, a couple buzz words and very set opinions on the ways to do things. It's like they get pet projects in their head from reading an article in a magazine and get locked into it. I don't really have an issue with CXXs being ignorant about a subject. No one knows everything. What I do have an issue with is when they act like an expert ignoring all the people who are actually experts in a particular area. It'd be like me going into a room full of IT people and saying EBITDA a bunch of times claiming to be an accounting wizard ready to lead a major initiative. It's frustrating but I've learned all I can really do is smile and watch the show.

maxxxxx 7y ago

This is exactly extremely common. In my company there is this constant battle about the devs having admin rights on their machines. We need admin rights to do our job. We have had dozens of meetings explaining the situation but IT can’t come up with a solution so the devs go around security because they have no alternative if they want to finish their work . Same with Dropbox. They block it but we have suppliers who use Dropbox. So the result is that people download confidential files from Dropbox on their home computers or phones and transfer them to their work machines. In my view security shouldn’t be isolated at corporate headquarters but they should be close to the end users so they see what users need to do and help them to balance security with getting things done. They can’t just block stuff without providing alternatives or they will either hurt the business or they will be circumvented.

7748394773639 7y ago
I have my own PC strapped under my desk connected to public Wi-Fi. I use it to do all my work. My company pc is left turned on but disconnected, sitting on top of my desk to dissuade suspicion.
- maxxxxx 7y ago
  I have a WiFi router hidden so we can do some specific network testing. We also have a consumer DSL line so we can test on a network where IT doesn’t block random stuff. It’s not even much of a secret. Everybody local knows this stuff and every six months a guy at IT headquarters throws a fit, doesn’t provide an alternative and gets ignored. I have thought about sending them recordings of previous meetings so we don’t have to repeat the series of same meetings every six to twelve months.
  - miahi 7y ago
    We had the same kind of "hidden" router at my previous employer, as we had a team of ~10 developers who needed a lot of different access to different servers and between eachother. We put the name of another tenant in the office building as the SSID :). After about a year the network through that router started to slow down. When we checked why we realized that we had more than 40 wifi clients, as other (non-dev) teams learned about the network through the grapevine and actually the new employees from sales (the neighboring office) were told to connect to that network directly - as the process to connect to the "normal" network was too much of a hassle.

brixon 7y ago

Let's ignore the SaaS security issues for a second. When IT says "No" it's not like the area asking is going to go away and not try to solve their problem. Organizations are going to find ways to solve their issues and IT can either help from the beginning or help clean up the mess later. I try to take the stance of offering the right solution and a lot of the times a now solution at the same time. There is no saying "No" in the long term, either help them now or get stuck with the shadow solution the magic macro guy cobbled together that became a critical business function.

rosege 7y ago
In my last job I tried to handle this in a similar way. The issue we ran into though was that often these managers would not properly evaluate the software. They would get wowed by the sales guys and sign up for huge contracts without, sometimes, even checking with IT or testing other vendors.
- tvanantwerp 7y ago
  Whenever I'm wearing my IT hat at work, a big part of it is being a detective looking for clues that somebody, somewhere, is about to do this. Then I can insert myself into that process. It would be preferable to be included in the beginning, but one must work within the reality of the situation.
x38iq84n 7y ago
I have seen IT being unaware of and unwilling to meet requirements of highly specialized technical teams, such as network engineering. You cannot have a TELNET client because the use of TELNET is prohibited by corporate policy, test TCP connections another way. You don't need vim when you have vi. You can't have admin rights but we don't support drivers for RS232 dongle so nope. Sometimes it's quite a challenge to get some work done.
- user5994461 7y ago
  Use netcat. Telnet is pretty hard to procure since it's not included by default since Windows 7.

drevil-v2 7y ago

I still shudder thinking about my time working as a developer on corporate IT locked down IBM leased laptops. Every time I did npm install I needed to request admin access to Windows which took 2-3 hours to action by IBM team sitting on the other side of the world in India. One day a grey beard took pity on me and installed a Linux VM where I was admin, copied the security certs from the Windows host and I could access all corporate resources at my leisure. Never logged a single IT Helpdesk ticket after that.

abledon 7y ago
wow. that sounds horrible. did they ask you fizzbuzz or mergesort in the interview?
- maxxxxx 7y ago
  Probably not. The interviewer didn’t know how to do fizz buzz himself.
  - mxuribe 7y ago
    When has the lack of info/knowledge ever stopped interviewers from asking such questions? ;-)
  - deleted 7y ago
    [deleted]
deleted 7y ago
[deleted]
flurdy 7y ago
Yup, back then my way around corporate locked down Windows machines was to only get permission to install VMware Workstation or Virtualbox. Then fullscreen Ubuntu in a VM from then on. Slower but no restrictions. Even better when Workstation supported multiple screens.
- Macha 7y ago

raxxorrax 7y ago

> If you don’t think this is happening in your organization, think again That story probably never happened anyway. But the essence of the article is very true. I never have been in a corp where IT enforces 100% conformity anyway (apart from medical industry). Sure, there are actual successful attacks, but that is mostly not the fault of unsanctioned programs. But there are systems where people should not just start to use any system, because information gets lost on the way. That would include CRM and ERP in my opinion. That a company can exist without a CRM is questionable to begin with and solutions are plentiful. If they did not have anything like that... If the story were true, it would not be the fault of Chief Input/Output.

cannonedhamster 7y ago
I've been in corporate IT where this happened. All company apps were built internally. None were able to run on anything past Windows XP. On top of my regular help desk, asset management, software project, and lease refresh program I was also somehow supposed to make the software work with Windows 7 as they had let the developers go. This is the same company that refused my sane security requirements and ignored just about everything until too late. I hear they have since outsourced IT and networking and it's failing dramatically, but they are saving money right?

deleted 7y ago

[deleted]

philipodonnell 7y ago

> The CIO admitted that he had been approached and explained that he had informed the VP that IT already had a project with SAP to deliver what the VP needed. “Yes, but that won’t be ready for me to use for three years, and I need something today,” retorted the VP. The CIO was silent. Then the CEO asked the VP, “I’ve known you for ten years. You don’t seem like someone who would do something to harm the company. Why did you do this?” The VP hit right back: “Since I started this digital customer acquisition program, we’ve increased revenue $1M per month. Before we were losing revenue. If you want, I can shut it down right now. What do you want me to do?” Shut it down right now and ask the VP to tender their resignation. Any company doing a 3-year SAP implementation is a very large company. That $1M in additional revenue pales in comparison to the risk introduced by sharing company or personal customer data with a vendor who has not passed the required security auditing. Data is no longer a thing to be thrown around in search of additional revenue and "but I made money" or "I had to because IT is slow" is not a post hoc rationalization for the behavior. Regardless of the merits of large enterprises acting this way, this is a VP who clearly cannot function within the enhanced risk-controlled environment of one and should find a position with a smaller company where they have more freedom to pursue personal initiatives at the VP-level. Those companies exist. Go find one.

enraged_camel 7y ago
>>Any company doing a 3-year SAP implementation is a very large company. Not at all. I have a client with ~100 employees who are past year 2 of their Salesforce implementation because the director of technology keeps changing priorities and project requirements.
- philipodonnell 7y ago
  That's fair, but just using the time-frames in the article, 6 months had passed before the VP got caught and still had 3 years left on the implementation. I think a director of technology who take 3.5+ years to do a CRM implementation at a 100-person company... isn't doing a very good job. :-)
arkades 7y ago

ConfusedDog 7y ago

This is my current situation. Being a SWE, IT and security are always putting out fires with networks or upper echelon cybersecurity violation complaints (mostly people downloading software without authorization). They have very little time, almost none for investigating new software, and all software must be installed by them. End of the day, nothing gets done on our work computers. I once waited two months for them to say no for a piece of solution we as the team approved. It's absolutely frustrating.

noonespecial 7y ago

Its the same dilemma companies face with their legal team. The "safest" thing to is nothing at all so sometimes the overabundance of caution hamstrings business growth. That's what the CEO is there to figure out.

exabrial 7y ago

IT is a service to the rest of the company. If you don't approach it with a servant's heart, people will go find their own solutions without you and you'll be part of the cleanup crew.

jmkd 7y ago

Joining Google was an eye-opener for me on this. Was the first time I encountered an IT department (TechStop) that didn't act like a police force and instead had your back, helping you get where you needed to be. Was always the first thing I would show guests on a tour of the campus.

repolfx 7y ago
But you're probably a developer? TechStop is/was great. But Windows users had locked down workstations where IT whitelisted binaries. I assume the approval process sucked about as much as normal. Many Google employees use desktop Linux which is basically unheard of outside the tech world. That by itself simplifies things quite a bit. Not many people writing viruses posing as screensavers for Google's in house Linux strain. Anyone who cracks that is probably an APT attacker and those require different approaches.
alexhutcheson 7y ago
CorpEng also does an amazing job building in-house apps & tools to let people get their job done.

tvanantwerp 7y ago

The ideal IT team is one that proactively learns the needs of others in the business and works with them to solve problems. It's no wonder that so many companies end up with shadow IT when so many IT teams are just people who tell you "no" whenever you ask for something. Doing it right is harder in the near-term, but much easier in the long-term as you're not putting out so many fires or going to "disciplinary council" meetings.

deleted 7y ago

[deleted]

eithed 7y ago

To me it reads like this - VP didn't care about the consequences of utilizing their solution and didn't care about IT; they simply wanted their stuff done, without acknowedgling prioratization of tasks. The proper way this could have been resolved is by VP utilizing people's skills they've hired. Does this solution look good and will accomplish the task that was prioritized? Excellent! Pass it to IT to evaluate. If the task has specification - excellent, have somebody in IT look for a product that ticks all the boxes and let's choose it together.

fphhotchips 7y ago

How many years since The Phoenix Project and this conversation has barely moved an inch? CIO probably wins this battle and gets the VP fired, but will be mystified when they're reporting to the CFO or a Chief Digital Officer when it happens 3 more times by the end of the year.

scarejunba 7y ago
It is wonderful. Each crack in a behemoth’s process is a point of leverage for a startup. We will outcompete by being better.
finnthehuman 7y ago
>How many years since The Phoenix Project and this conversation has barely moved an inch? People are still fighting the lessons in The Mythical Man Month; it's gonna be a while.

hartator 7y ago

Isn't IT something from the past? I would expect people knowing how to use a computer and what they need to do their job.

kleborp 7y ago
As a lowly help desk technician perusing this thread, you couldn't be any further from the truth. The software devs never open tickets, everyone else does and its for the most banal problems
- rhinoceraptor 7y ago
  I've opened plenty of tickets as a developer, mostly the tickets are IT's garbage software or horrible network setups preventing me from doing my work.
- silverfox17 7y ago
  A lot of people who post here don't seem to realize that a lot of companies still have IT departments.. a lot of people here are also developers who don't realize they can easily do things to compromise data, even if they think they know better.
Macha 7y ago
When MacOS borks itself in an update because the endpoint management software does something funky with partitions, I'd rather delegate that to a team that's handled three cases of that this week and get on with my job description than diagnose the exact steps to resolve a problem that'll often require access I don't have anyway.
jillesvangurp 7y ago
My thoughts exactly. It's been a while since I last saw a company with an actual IT department and even longer where they had an actual clue. The reality is that IT in most SMEs simply sucks and no longer requires a college education. Working in IT for an SME is not a career plan. You're at constant risk of being outsourced and essentially all you do is done better by a gazillion companies as a service that probably cost less than a few months of your salary. Frankly, most SMEs would be better off doing exactly that. Most startups I work with do this from day one for obvious reasons. Once you hit a certain scale different dynamics may kick in but even then, outsourcing is an option. As a freelancer these days, I bring my own laptop and am granted access to stuff for the duration of the project. It's understood and expected of me that I do such things as encrypt disks, use 2FA, and don't use "secret" as the password. Most stuff I access for these projects is SAAS based. I'd probably walk away from projects where that wasn't the case.

protomyth 7y ago

Its the user who downloaded a program they "needed" which had malware which sent out a lot of spam email because this was a user that did announcements which basically got an e-mail server listed on blacklists that creates these IT policies. You want to treat people like responsible adults, but they aren't the ones who have to deal with the fallout. Developers know the score for the most part, so full privileges are expected with the caveat, if it all goes bad, we are wiping the machine[1], not doing a recovery. IT dreads the moment we are called to account for something some user decided they needed to do. 1) most developers understand backup tools and code control - those that don't, well...... with great power comes great responsibility

dx87 7y ago
Yep, a company I worked at hired a tech writer that downloaded some cracked version of software that included ransomware on their first day of work because they said they didn't want to wait for the company to get them a legitimate copy.
- toyg 7y ago
  Well, that used to be common practice... in the '90s. Thank $deity for the rise of opensource.
  - protomyth 7y ago
    OpenSource still isn't a force in that particular area. Its Microsoft or Adobe there.
    - toyg 7y ago
      Yeah, what I meant is that, these days, the culture is such that one assumes there will be an OSS tool somewhere, before one even considers a sketchy binary. Maybe the OSS option will be inferior, but it's almost guaranteed that it will get some stuff done and not nuke your machine. That's a significant improvement (of course we know that having a github repo is no guarantee and blablabla, but it correlates well enough for most purposes).

thomasjudge 7y ago

I work in an IT organization & I see (in the sense of witness) both sides of this. We are over-tasked and under-resourced and new projects/ideas/initiatives that come in the door go into a backlog of requests. So I see business/end users signing up on their own for SAAS solutions to solve their problems.

argd678 7y ago
Right, the CIO is also being held to a lower standard than a P&L. The CIO could have planed out an interim solution to meet the business needs quicker. If the tables were turned, say the CIO needed to deliver a service and didn’t have a big enough budget, then what?

sgt101 7y ago

Well, that cleared that up then! Gosh I had no idea that the solution would be so simple. It does shock me that the people who've had their whole infrastructure compromised and held to ransom by viruses and the people who've been held over a barrel by suppliers or had vast amounts of money burned by being locked into a dozen vendor contracts for the same service are so silly and hysterical about it when the solution is as simple as "identify when you need to be best in class and stay small everywhere else".

thisisit 7y ago

As a born and bread corporate (mostly banks) corporate IT guy, I used to frown upon this behavior. Then I got one of my bigger career breaks because the finance team went behind IT, bought a software and installed it in a machine which kept under their desk. They further hired people from an IT service company to configure the machine. The configuration was so bad that it exposed the company's network to whole wide world. Google contacted the company and after searching high and low IT security managed to track the pc down and take it away. Finance team promised to hire someone with skillset required to run the software in a closed environment. And that's how I ended up getting my job.

lapnitnelav 7y ago
Really interesting story, especially if you put it in the context of being corporate world and a finance team, both of which aren't the type to rock the boat. It says a lot about the struggles of enabling change in structures with strong silos that (what are probably among the most risk averse type of) people would go to such lengths.

vinay_ys 7y ago

Does your IT team use key loggers or other employee monitoring software at your company? I hear some big trillion dollar companies do this. Is that true?

jrjarrett 7y ago

This thread hits home. I switched jobs a few years ago because the IT policies on workstations were being ratcheted down to make my job as a developer difficult to impossible. Now, the company I work for, ostensibly a _software_ company, got its ISO certifications, which meant policies and procedures that make developing hard or impossible again. How does a software business _successfully_ implement stringent access controls while still allowing for efficient software development? I'd like to see/hear what works.

CrossWired 7y ago
I'm heading down this path right now. How do I obtain my certs while also allowing enough freedom for the dev teams to operate. We have to deal with the fallout when they screw something up, there has to be a happy medium somewhere.
- user5994461 7y ago
  What's the issue specifically? Developers don't need admin rights for much of anything in this decade. No need to bother with that. Common software has to be made available in self-service, so developers can install development tools like notepad++ or visual studio. Deployment is usually the challenge because you have to store binaries somewhere, copy it to some random servers and finally execute it, each step causing numerous security headaches, so there has to be some approved tooling to handle that.
  - stefan_ 7y ago
    You can't get past "docker run" without admin rights. No, the challenge is access to production data without auditing.
  - jrjarrett 7y ago
    Developers don't need admin rights for much of anything in this decade. No need to bother with that. Please defend this position. My experience, mostly with Linux-like tools, is that those tools are built with the assumption they are being used by someone who knows what they are doing, and that they have the appropriate level of control of the machine -- they are tools for professionals to build tools. If you don't have rights to install or execute them, you're done. You can't make any forward progress.

davvolun 7y ago

yebyen 7y ago
When a person says "we need this infrastructure project" and a project is commissioned, acknowledging the need, it is in my experience that unfortunately that person's job function is rarely placed on hold until the appropriate infrastructure has been made available. "Who cares about your pie-in-the-sky infrastructure project, my boss continues to measure our real performance with basic accounting, and is expecting to be able to report on growth each quarter, which I can't help without tools" seems to be a bit closer to the argument posed here, IMHO.
- davvolun 7y ago
  Of course that's the real question with the actual story in the actual article -- could they have implemented their security on the temporary infrastructure, or "good-enough" security, if the CIO knew about it? In that story, is the blame on the VP for going ahead instead of getting dialogue started between CEO, VP and CIO? Is it on the CIO for just saying "no" instead of recognizing the need and the value? Is it on the CEO for failing to empower the VP and CIO to get that conversation started themselves? And then, it's all well and good to worry about the bottom line first, until you're sitting in Equifax's shoes right.
  - yebyen 7y ago

rsuelzer 7y ago

Our IT security department was incentivized to deny everything from new tools to new internal applications. We had an outside firm making security decisions and if there were any security issues it would end up being on them. So as long as they did not allow us to release any products and or install any software they could not be held responsible. I made friends with a lower level contractor who told me off the record to use my judgement on what to install to get the job done, because the security department would never approve anything new unless directly instructed to by the CEO. Fast forward three months, there was a major security flaw on our website (also built with outsourced labor) which allowed anyone to access private data without a login. A few of us had reported to the security department that the code running the website was so poorly written that the odds of being insecure were close to 100 percent. We suggested upgrading the website and rewriting the code, and management was on board with this but security department refused to allow us to use any new frameworks since they were not approved. Of course in a matter of a few months the site was hacked and millions were spent as a result. I quit this job after we were unable to release several products after a year even though we jumped through every hoop we needed to. That department killed all innovation.

closetohome 7y ago
I think it really depends on the company. If you're something like a nontechnical non-profit, sure, turn that decision making over to IT. In that case IT is performing a vital, skilled function. But in most software shops, the workers are probably more qualified than the IT department to be making decisions about what applications to use, and what kind of security they need. IT is just there to make things run and fix them when they break. They don't really need to offer guidance.

la_barba 7y ago

Hehe, if you think this is nuts, come to pharma. We can't do jack shit with our machines. If you so much as change the time on your machine, that is a 'data integrity breach', and if your actions are determined to be malicious it can result in a firing.

GuB-42 7y ago
To be honest, changing the time on a machine is a very serious concern. Accurate timekeeping is crucial in security, that's how you connect events together.
- la_barba 7y ago
  Well, all the rigid policies like no dropbox or no FTP or no whatever, also arise from serious concerns. I just wanted to point out another seemingly innocuous one. Most of our equipment is not internet connected, and we need to manually change the time for daylight savings or other corrections. We have a company policy and procedure to do that periodically so that our audit trails are accurate. Sometimes folks get busy and the shop floor guys take matters into their own hands.

alkonaut 7y ago

Security training focuses way to much on email phishing and not enough on this kind of stuff. Actually getting your work done, managing your own computer. Of course people can't be trusted if they havem't been trained. How to handle USB drives. What and from where you can download and run programs. What actually IS a program and what isn't. Many of us learned this the hard way by playing lots of cracked games in the 90s. But not everyone did that. Try explaining to a non-technical person how how a desktop background image isn't a program so it's basically safe to grab from anywhere, while a screen saver is definitely a program and usually unsafe to get from most places, and a word document is some times a program that might eat your computer. Training could involve things like "which of these 5 webbpages would you consider it safe to download and run executable from"? Having too cumbersome rules around security just means it's ignored or circumvented, increasing risks.

JakeTheAndroid 7y ago
It's tough, I give security awareness trainings myself and I completely agree with what you're saying. However, that's a lot of information to give to a group of new employees that can span any department and technical understanding. I actually was talking today with a customer during a logical assessment about if I talked about downloading malware in the training. I dedicate an entire section to downloading documents, but I don't really give people the information you're talking about. I tell them how to avoid ever having to download anything, and if they must do it, how to try and do it properly. All of this is ended with the process on how to report incidents because eventually something bad will happen. As a company you kind of expect this to be solved at a number of layers. Endpoint management should hopefully help resolve this issue. Restricting web access where it makes sense can help. Sec Awareness Training helps keep people aware. Etc, etc, etc. You hope your controls are what save you from incidents, because there is no way you can effectively train your entire company on security topics to a degree that they can make good, security conscious decisions. That said, many of these SAT's are really just checking compliance requirements, because thats the real need. I put my own training together starting with what I know needs to be covered for compliance (pii handling, passwords, acceptable use policy, common threats, security incident response reporting, etc). Anything else that makes it in is purely because I have extra time and I know it to be important.

noja 7y ago

There's a difference between using software that IT has approved and shipping customer data outside of the company.

TheRealDunkirk 7y ago

I just witnessed a very similar situation, on a smaller scale, but there are many of these in my company, and they add up. Boss: "We need access to the database of our primary application that you wrote for us so that we can pull the data into this new tool to track progress." IT: "No. Not only can we not give you access to YOUR data in YOUR application that we wrote on YOUR dime, we will not allow you to have this new application written by someone who isn't in our group. If you wanted something like this, why didn't you just ask us? We would have written this for you." Boss: "We had a meeting about this over a year and a half ago, and you told me that you didn't even have the time to discuss it further." IT: "... Well, we're still not going to let you do this." IT is effectively holding the rest of the company hostage, and the corporate technical debt is becoming epic. So skunkworks solutions will continue to be developed.

dingo_bat 7y ago

In our company people just started using free slack en mass, boycotting the horrible IT approved Skype for business. When it was discovered that thousands of employees were using slack, the CTO had to step in and tell IT to fuck off, and started paying for the full version.

jmspring 7y ago

This brings back a memory. The only time I was fired "for cause". Summer after my freshman year at college, I was temping and got an assignment doing real estate purchase comps with a company in the East Bay. At the time, there were laser printers, but often printing sucked up CPU time and let's just say multitasking was still not a widespread thing. I found myself tired of sitting around. I found a TSR / print spooler that would use RAM and offload the process of printing. This allowed me to keep working. My productivity (as a temp) was higher than many others including the person I was "reporting to" at the company. They found the print spooler, labeled it "unapproved software", and I was walked out the door. The funny thing is, a friend at the time (and I didn't realize it) was higher up in the management. He reached out to me on a multi-line BBS that was popular in the area and offered me a full time job a few days later. I was in school and obviously declined. Working the rest of the summer for a Chemical Engineer in Martinez/Benecia ended up being incredibly more interesting. So it was a net win.

jccalhoun 7y ago

I just wish I had a decent computer at work that didn't have 3rd party antivirus that would just slow the software to a crawl.

analog31 7y ago

Something that's crossed my mind is John Gall's observation that complex systems operate in failure mode 100% of the time. I understand "failure mode" to mean that built-in guards have been bypassed in order to enable the system to do anything at all. Germane to this thread, the "guards" are IT approvals. I suspect that if a business is complex enough to have IT policy, that policy is always being bypassed in some way, at any given time. Somebody is using unofficial software, or using official software in an unofficial way.