14 ms·

One thing I have learned as a sysadmin who has had the privledge to see the inside of hundreds of companies from medium sized law firms to F500 oil companies:

One thing I have learned as a sysadmin who has had the privledge to see the inside of hundreds of companies from medium sized law firms to F500 oil companies: There is a lot more incompetence than you would ever want to believe, and it's not always where you think. I've traced most of it to a failure of connection/communication between IT departments and C-levels/boards. The CTO/CIO and the person immediately below them (and the person immediately below them) are the "buck stops here" people for these kinds of issues, but often are either one of two types. 1) Too much MBA, not enough tech. 2) Too much tech, not enough MBA. Both tend to have pretty similar results.

stcredzero 7y ago
The CTO/CIO and the person immediately below them (and the person immediately below them) are the "buck stops here" people for these kinds of issues, but often are either one of two types. 1) Too much MBA, not enough tech. 2) Too much tech, not enough MBA. 1) Knows how to get people to do things, but doesn't know what needs to be done. 2) Knows exactly what would fix security, but doesn't know how to get people to do those things.
- 1000units 7y ago
  MBA types are great at conning others into self-serving false insights like (2) isn't perfectly capable of enforcing sound security policies given the appropriate authority. Their jobs are directly about justifying their own existence, usually by song and dance.
- kevin_thibedeau 7y ago
  1) Has been told what needs to be done and decided to nix the idea because security is a cost center that cuts into their bonus potential.
  - mox1 7y ago
    Eh, while this happens its getting rarer and rarer. Execs know that if a cybersecurity issue is brought to them they kinda have to take it seriously (in 2020 anyways) or be the next Experian. Edit: whoops Equifax, not Experian
    - mandelbrotwurst 7y ago
      Well, risk being the next Equifax, not necessarily BE the next Equifax. If taking on that risk gives someone a 90% chance at a massive comp boost and a 10% chance at a more negative outcome where they're likely still well compensated, many execs would consider that.
    - twinkletwinkle_ 7y ago
      What exactly happened to Experian? Their stock is up almost 100% since the breach announcement, and no one went to jail. If I were an executive looking to Experian as an example I would conclude that the market as a whole does not care about security.
      - jaywalk 7y ago
        Experian is a pretty unique case, because the people whose data they lost (you and I) don't have a choice to not do business with them. And the people who actually do have a choice (businesses) realistically don't have a choice either.
      - donarb 7y ago
        It was Equifax that had the data breach, not Experian. Equifax was fined $700 million. Experian benefitted from the breach though since Equifax provided Experian credit monitoring services to those whose data was released. Equifax stock has gone up since the 2017 data breach, though because consumers are not customers of Equifax, it's banks and lenders who are not concerned with data they don't control. If hackers had broke into Equifax and changed people's credit scores, the banks might have been a bit more concerned.
        LiquidSky 7y ago
basch 7y ago
Part of this is just an org structure problem in general. Paywalled, but https://blogs.wsj.com/cio/2014/06/19/why-you-should-eventually-fire-your-cio/ https://blogs.wsj.com/cio/2014/06/19/why-you-should-eventual... They make the argument that every C suite employee should be tech literate, and if thats the case, then all the "tech decisions" dont fall to one bottleneck on the team, and it isnt that one persons job to persuade everyone else of something they may choose not to understand. The organizational structure itself, perpetuates silos, and holds companies back from evolving and growing. "One of the key reasons the C-suite has not yet developed tech maturity is the presence of the CIO. Executives feel that they can deflect technology responsibilities to the CIO in part because the very existence of a technology executive provides an excuse to do so (why else is he or she there?) In the same way that no self-respecting firm would hire a chief quality officer today (quality should be everyone’s job), firms risk perpetuating the technology equivalent of a chief quality officer when they hire a CIO." "Even if the CIO’s role endures, expect it to involve more facilitation, coordination and strategic planning rather than implementation and operation, while the rest of the executive team handles day-to-day IT decision-making." CIO should own the process of governance building, and collaboration enablement. Figuring out how to facilitate communities within the company coming together, and supplying them with the resources they need to succeed. Just like any other sort of resource planning.
jordanbeiber 7y ago
Something that I’ve seen as a root to a lot of problems and inertia at big co IT is that theres been a couple of cycles where people where internally recruited from business to IT - the mainframe era and more recently the client/server era. Nothing wrong with recruiting internally per-se, it’s more about the fact that many lacked basic knowledge. I’m painting with a big brush here, but I’ve seen effects of this throughout my so called career. - 40 years ago career-paths was opened as cobol programmers. Knowledge was scarce and security was in many ways non-existing. - 20 years ago guys got shifted to IT because they knew how to fix the printer. Client/server knowledge was scarce and security non-existing. You breed a segment of IT workers, many which are in ways clueless, but it’s big business. It’s enterprise IT. Massive breaches, slow to change, and downtime by the buckets. Tickets churning along in “automated” workflows. Queue the outsourcing strategies! Might seem a sane choice considering the above - only... it’s difficulty to outsource something you’re not in control of. This obvious lack of control have led to the ITSM field with it’s CABs, ITILs and other acronyms, all designed to lull everyone into a false sense of security. I could go on, perhaps write a book. Anyway, I totally agree with what you’re saying, just wanted to share my perspective.
- HenryBemis 7y ago
  And this is why IT Audit and IT Security came forward, because IT staff (1st line) will do IT stuff and won't take into account design effectiveness, operating effectiveness for controls/processes. IT Risk (2nd line) is mostly non-IT people that try to convince IT people "what is right". That never works. One of the advantages I have when I work for the 2nd line, is that I've been the 1st line and I know the shit the IT staff have to deal with, and I don't pretent I know better, I do know (((EDIT: I do know exactly what they go through because I've gone through that myself))). And my "risk mitigation advice" is actual, tangible, and with real examples. There is also the 3rd line (internal audit) that in most orgs they completely miss the mark. ZERO IT knowledge, probably former Big4 that have never seen a console in their lives, wearing a police hat barking orders. No wonder that the IT landscape is suffering. I also understand that COO/CEOs don't allow CIO-CTOs the time to have a freeze period, STOP running forward and give them the time to pause, breathe, think, repair. You can't be making and perfecting at the same time. Nobody has the resources for that, so it's all accidents waiting to happen. I'm not fearmongering, I've just seen enough crap on IT systems (mainly on LARGE corporations) that have made my skin crawl..
falans 7y ago
> There is a lot more incompetence than you would ever want to believe I work for a company that you've definitely heard of, and a lot of people have actively used our product. We're a very big company. We have domain admins browsing the public web via interactive RDP sessions on the domain controllers themselves. That is one of many horror stories of the security at this company.
- AnimalMuppet 7y ago
  Heh. 15+ years ago, I was working for an internet security company, and we got hit with a virus, because our CEO opened the wrong email.
  - MaximumYComb 7y ago
    Look at the statistics on phishing, it's remarkably effective.
BiteCode_dev 7y ago
I work with a F500 oil company from time to time. Half of the devs from there that I was in contact with were not capable of: - googling a solution to a problem efficiently. When they hit a wall, they turned to me with an empty look like they were lost. - read an error message to troubleshoot. A stack trace is utter mystery. - use effectively the UI of their laptop. Some can't even Ctrl + S to save, they look up the "save" entry in the menu. We are talking about people writing code every day, in several programming languages: fortran, c, c++, java, Python... Because I'm a freelancer, I don't care. I'm paid extremely well to be very nice to them and solve all their problems. But I'm very glad I don't have to be held responsible for anything those people end up putting in production. And I have no reason to believe it's different in their security department. However, and this is a good lesson to all of the geeks like me that think work is about doing the right thing: the output they produce is good enough in our society. Its cost/value hits the sweat spot. Business is not about doing things right, it's about being profitable. If you have one scandal a year, but it costs you less than making sure you have a secure system, and you are not legally challenged, then you are golden. In fact, the chances to have even one scandal are very low. Actual risks of failure or attack are low. And consequences in case of crisis are low too. People don't care that much about privacy, cyber-security, etc. And policy makers won't enforce their laws anyway, at least not to any extent that will endanger the company. So if the software allows people to do their job IRL at a reasonable price, under an acceptable deadline, good enough. In fact, David Goodenough is a very funny French meme: https://www.youtube.com/watch?v=ho4W5LnFl6s https://www.youtube.com/watch?v=ho4W5LnFl6s
- cpach 7y ago
  “the work they produce is good enough in our society” Gah. I wish I could rebut this but I’m not sure I can. How does one find work for a tech/IT company that actually embrace quality?
  - namibj 7y ago
    I'm curious, too. It's sad to be the only one even using proper commit messages.
Veserv 7y ago
For any of these multi-billion dollar organizations you have worked at or have sufficient trustworthy information about, do you think any of them would be able to withstand a competent pentesting organization with a $1 Million contract to breach their systems? As a corollary, if they implemented all of the suggestions of a type 2, technical CTO/CIO, do you believe they would be able to withstand such a pentest, and if so what would you estimate the cost to totally compromise their systems be? All of my outsider knowledge indicates that none of the multi-billion dollar organizations can protect themselves from adversaries 1/1000th their size, and I would like to know if this is consistent with the knowledge of people with more direct firsthand experience. Thanks.
- Enginerrrd 7y ago
  I'd venture to guess this is almost a natural law. Look at the pathogen/organism relationship in life. A small attacker with low overhead can cripple a large complex organism. That's been evolving for billions of years without resolution. In essence, I imagine that security vs cost is an exponential curve with security being the independent variable, and cost the dependent variable. At the lowest levels, this is great because there's some very easy and cheap fixes to get large gains in security, but at the high end, each incremental gain increases in cost until it reaches beyond the realm of feasibility. Air gap all your computers? Cool, you just killed productivity which massively increased costs. Actively supervise every user? There's some niche systems that probably require this level of security, but it's rare. Even then, I don't think such a thing as perfect security exists.
- arminiusreturns 7y ago
  I don't think any of them could withstand a 50k contract to breach TBH. The only ones that really could generally tend to be DoD/E related. That's not necessarily the point though, in an era of APT. I like to say breaches are inevitable, the real question is how fast do you know about it so you can mitigate/respond? HIDS/NIDS, auditing, monitoring and logging are the core of this imho.
chaps 7y ago
Oh my, yes. I've had very similar privileges, and it's almost universally a complete shitshow. It doesn't really ever matter what the age, size, skillsets, etc, of the company are, either. It's just.... all bad. Example: one of the first questions I try to ask when starting a new job is, "Can I have an accurate list of the physical hosts and the IP of the management NIC?", and the answer is usually some form of, "We don't maintain a list like that, you'll have to grab everything from [pick two]: (DNS|Spreadsheets|Monitoring System|Configuration Management|nmap|ARP tables).". Then, because of course, lists don't match with each other, so literally no source of truth for what hosts exist in a network exists. Even for production! It's truly maddening.
- jiggawatts 7y ago
  My follow-up question is: "If you don't have a list, how do you apply critical security updates to the lights-out management firmware?" Answer: "The light-ou... what?"
  - chaps 7y ago
    I'm stealing that. :)
  - anchpop 7y ago
    I think I'm missing the pun here, can someone explain?
    - chaps 7y ago
      Not a pun - OP's talking about these[1], which essentially gives an IP address to a serial port, often internet-facing. The joke is about how clueless management is about their infrastructure. https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out
Stranger43 7y ago
It's not just incompetence, what we are seeing is far to systemic to be the fault of mere individuals it have to be rooted in the way modern business culture deals with IT. Almost every real world security post mortem i have ever seen ends up as untimely application of patches due to the business either rejecting downtime/changes or simply not having the tools and staffing in place to keep up with changing vendor recommendation and security updates. And despite every evidence based report placing the blame for our current mess on "systematic maintenance failures" the only thing we heer hear from the MBA mills and business leaders is a call for more snakeoil products that can, do what nobody have yet to demonstrate under real world conditions, and add security to an insecure environment. This is not incompetence/ignorance it's denial likely caused by an "prisoners dilemma" explained elegantly in H.C. Andersen's "The Emperor's New Clothes" where nobody wants to be the first to admit that they aren't keeping up with best practices for fear of ridicule or worse.
smacktoward 7y ago
Perhaps this is less a knock on those people than it is with the position of CTO/CIO itself. A role that only works effectively when it's filled by a special unicorn candidate is a broken role. I can think of some reasons why this might be: - It takes a particular kind of personality to ascend to a C-suite role, and that personality isn't a good fit for solving this particular problem - Concentrating responsibility for this problem in a single C-suite role lets the rest of the C-suite ignore it as "somebody else's problem" - Concentrating responsibility for this problem in a single C-suite role serves mostly to provide a handy scapegoat when things inevitably go wrong - This is a cross-cutting problem that can't effectively be siloed into a single role, even a C-suite one; unless the CEO/board also buy in, the CTO/CIO will lack the authority required to override objections or inaction on the part of other officers
dgzl 7y ago
I used to contract with state governments and can say the same thing about them...
anonsivalley652 7y ago
I was a sysadmin since 17 and a client-facing, on-prem AWS consultant for F200 and startups in the early days. A lot of big companies outsource their IT to the cheapest vendor, who, in turn, outsources roles to onshore/offshore. There's not necessarily incompetence by virtue of people from elsewhere in the world, but there is sufficient incompetency, second-stringers and lack of accountability because of all the bureaucratic layers and indirection in the responsibilities. As such, technical debt, confusion and substandard work thrives because of all the vampiric technology "ticks" consulting companies who maximize profit before customer value. The way-out is for companies to directly hire fewer but better technical employees, employee-ownership and not jumping on the IPO bandwagon because of the perverse motivations of publicly-traded companies.
peterwwillis 7y ago
I don't think it's a "Tech vs Business" problem. Tech is just another part of the business. Every part of the business may have bottlenecks; sales, marketing, support, finance, etc. C-level has to learn to identify the bottleneck in the value chain and improve it until it goes away. You don't have to become an expert in every field to do that, but you do have to ask a lot of questions and use an improvement loop. It's the same whether you're in manufacturing, distribution, health care, etc. Yes, you have to be able to learn a bit about the field. But just being a domain expert ("knowing about tech") often does not substantially improve the overall output of the value chain, as experienced engineers learn eventually.
- cat199 7y ago
  > Tech is just another part of the business. while this is completely true, many people on the 'business side' don't see it this way at all. thankfully this is changing in the culture, but at a glacial pace.
eunos 7y ago
I can understand the peril of "Too much MBA, not enough tech" but what are the perils of "Too much tech, not enough MBA" ?
- cat199 7y ago
  "lets postpone delivery of the system to rewrite everything again from scratch this month because now the coolest tech is X and I really don't want to be stuck using last months technology even though it has 0 impact to users and actually negatively impacts the business case in terms of skillset staffing and operational overhead" for starters..
- spookthesunset 7y ago
  > Too much tech, not enough MBA Yup. Left to their own devices, engineers will chose to do what is “coolest” using the hippest technology stack and the most pure architecture imaginable. And they’ll build something and it will be cool.. to them.... but it won’t solve any business needs or will solve the wrong ones and not deliver as much value as it could have if it had somebody with a good business head on. ...or they’ll go reinventing the wheel and build massive costly systems that should have been purchased off the shelf from a vendor. Nope. You need a balance. It isn’t enough to just have tech smarts. You need somebody with business smarts to keep things grounded and solving real business problems that deliver value to the customer.
erikbye 7y ago
> There is a lot more incompetence than you would ever want to believe I'm currently in ops/sysadmin. Large org. Yes, the incompetency you see is sometimes incredulous: I have helped developers install Visual Studio; how to use RDP; had to explain how saving to local disk is different from a networked folder ... I can't think of better examples at the moment, though they exist and are plenteous. Thing is, I am a hundred percent sure these same people could have turned the tables on me and be aghast by my lack of knowledge in many disciplines. There are more things to know in this world than any one person can and it is thinking you are beyond this that is truly delusional. Even with a "verified expert" in a subject, whatever that means, we can find ample holes in their knowledge.

stcredzero 7y ago

The CTO/CIO and the person immediately below them (and the person immediately below them) are the "buck stops here" people for these kinds of issues, but often are either one of two types. 1) Too much MBA, not enough tech. 2) Too much tech, not enough MBA. 1) Knows how to get people to do things, but doesn't know what needs to be done. 2) Knows exactly what would fix security, but doesn't know how to get people to do those things.

1000units 7y ago
MBA types are great at conning others into self-serving false insights like (2) isn't perfectly capable of enforcing sound security policies given the appropriate authority. Their jobs are directly about justifying their own existence, usually by song and dance.
kevin_thibedeau 7y ago
1) Has been told what needs to be done and decided to nix the idea because security is a cost center that cuts into their bonus potential.
- mox1 7y ago
  Eh, while this happens its getting rarer and rarer. Execs know that if a cybersecurity issue is brought to them they kinda have to take it seriously (in 2020 anyways) or be the next Experian. Edit: whoops Equifax, not Experian
  - mandelbrotwurst 7y ago
    Well, risk being the next Equifax, not necessarily BE the next Equifax. If taking on that risk gives someone a 90% chance at a massive comp boost and a 10% chance at a more negative outcome where they're likely still well compensated, many execs would consider that.
  - twinkletwinkle_ 7y ago
    What exactly happened to Experian? Their stock is up almost 100% since the breach announcement, and no one went to jail. If I were an executive looking to Experian as an example I would conclude that the market as a whole does not care about security.
    - jaywalk 7y ago
      Experian is a pretty unique case, because the people whose data they lost (you and I) don't have a choice to not do business with them. And the people who actually do have a choice (businesses) realistically don't have a choice either.
    - donarb 7y ago
      It was Equifax that had the data breach, not Experian. Equifax was fined $700 million. Experian benefitted from the breach though since Equifax provided Experian credit monitoring services to those whose data was released. Equifax stock has gone up since the 2017 data breach, though because consumers are not customers of Equifax, it's banks and lenders who are not concerned with data they don't control. If hackers had broke into Equifax and changed people's credit scores, the banks might have been a bit more concerned.
      - LiquidSky 7y ago

basch 7y ago

Part of this is just an org structure problem in general. Paywalled, but https://blogs.wsj.com/cio/2014/06/19/why-you-should-eventually-fire-your-cio/ https://blogs.wsj.com/cio/2014/06/19/why-you-should-eventual... They make the argument that every C suite employee should be tech literate, and if thats the case, then all the "tech decisions" dont fall to one bottleneck on the team, and it isnt that one persons job to persuade everyone else of something they may choose not to understand. The organizational structure itself, perpetuates silos, and holds companies back from evolving and growing. "One of the key reasons the C-suite has not yet developed tech maturity is the presence of the CIO. Executives feel that they can deflect technology responsibilities to the CIO in part because the very existence of a technology executive provides an excuse to do so (why else is he or she there?) In the same way that no self-respecting firm would hire a chief quality officer today (quality should be everyone’s job), firms risk perpetuating the technology equivalent of a chief quality officer when they hire a CIO." "Even if the CIO’s role endures, expect it to involve more facilitation, coordination and strategic planning rather than implementation and operation, while the rest of the executive team handles day-to-day IT decision-making." CIO should own the process of governance building, and collaboration enablement. Figuring out how to facilitate communities within the company coming together, and supplying them with the resources they need to succeed. Just like any other sort of resource planning.

jordanbeiber 7y ago

Something that I’ve seen as a root to a lot of problems and inertia at big co IT is that theres been a couple of cycles where people where internally recruited from business to IT - the mainframe era and more recently the client/server era. Nothing wrong with recruiting internally per-se, it’s more about the fact that many lacked basic knowledge. I’m painting with a big brush here, but I’ve seen effects of this throughout my so called career. - 40 years ago career-paths was opened as cobol programmers. Knowledge was scarce and security was in many ways non-existing. - 20 years ago guys got shifted to IT because they knew how to fix the printer. Client/server knowledge was scarce and security non-existing. You breed a segment of IT workers, many which are in ways clueless, but it’s big business. It’s enterprise IT. Massive breaches, slow to change, and downtime by the buckets. Tickets churning along in “automated” workflows. Queue the outsourcing strategies! Might seem a sane choice considering the above - only... it’s difficulty to outsource something you’re not in control of. This obvious lack of control have led to the ITSM field with it’s CABs, ITILs and other acronyms, all designed to lull everyone into a false sense of security. I could go on, perhaps write a book. Anyway, I totally agree with what you’re saying, just wanted to share my perspective.

HenryBemis 7y ago
And this is why IT Audit and IT Security came forward, because IT staff (1st line) will do IT stuff and won't take into account design effectiveness, operating effectiveness for controls/processes. IT Risk (2nd line) is mostly non-IT people that try to convince IT people "what is right". That never works. One of the advantages I have when I work for the 2nd line, is that I've been the 1st line and I know the shit the IT staff have to deal with, and I don't pretent I know better, I do know (((EDIT: I do know exactly what they go through because I've gone through that myself))). And my "risk mitigation advice" is actual, tangible, and with real examples. There is also the 3rd line (internal audit) that in most orgs they completely miss the mark. ZERO IT knowledge, probably former Big4 that have never seen a console in their lives, wearing a police hat barking orders. No wonder that the IT landscape is suffering. I also understand that COO/CEOs don't allow CIO-CTOs the time to have a freeze period, STOP running forward and give them the time to pause, breathe, think, repair. You can't be making and perfecting at the same time. Nobody has the resources for that, so it's all accidents waiting to happen. I'm not fearmongering, I've just seen enough crap on IT systems (mainly on LARGE corporations) that have made my skin crawl..

falans 7y ago

> There is a lot more incompetence than you would ever want to believe I work for a company that you've definitely heard of, and a lot of people have actively used our product. We're a very big company. We have domain admins browsing the public web via interactive RDP sessions on the domain controllers themselves. That is one of many horror stories of the security at this company.

AnimalMuppet 7y ago
Heh. 15+ years ago, I was working for an internet security company, and we got hit with a virus, because our CEO opened the wrong email.
- MaximumYComb 7y ago
  Look at the statistics on phishing, it's remarkably effective.

BiteCode_dev 7y ago

I work with a F500 oil company from time to time. Half of the devs from there that I was in contact with were not capable of: - googling a solution to a problem efficiently. When they hit a wall, they turned to me with an empty look like they were lost. - read an error message to troubleshoot. A stack trace is utter mystery. - use effectively the UI of their laptop. Some can't even Ctrl + S to save, they look up the "save" entry in the menu. We are talking about people writing code every day, in several programming languages: fortran, c, c++, java, Python... Because I'm a freelancer, I don't care. I'm paid extremely well to be very nice to them and solve all their problems. But I'm very glad I don't have to be held responsible for anything those people end up putting in production. And I have no reason to believe it's different in their security department. However, and this is a good lesson to all of the geeks like me that think work is about doing the right thing: the output they produce is good enough in our society. Its cost/value hits the sweat spot. Business is not about doing things right, it's about being profitable. If you have one scandal a year, but it costs you less than making sure you have a secure system, and you are not legally challenged, then you are golden. In fact, the chances to have even one scandal are very low. Actual risks of failure or attack are low. And consequences in case of crisis are low too. People don't care that much about privacy, cyber-security, etc. And policy makers won't enforce their laws anyway, at least not to any extent that will endanger the company. So if the software allows people to do their job IRL at a reasonable price, under an acceptable deadline, good enough. In fact, David Goodenough is a very funny French meme: https://www.youtube.com/watch?v=ho4W5LnFl6s https://www.youtube.com/watch?v=ho4W5LnFl6s

cpach 7y ago
“the work they produce is good enough in our society” Gah. I wish I could rebut this but I’m not sure I can. How does one find work for a tech/IT company that actually embrace quality?
- namibj 7y ago
  I'm curious, too. It's sad to be the only one even using proper commit messages.

Veserv 7y ago

For any of these multi-billion dollar organizations you have worked at or have sufficient trustworthy information about, do you think any of them would be able to withstand a competent pentesting organization with a $1 Million contract to breach their systems? As a corollary, if they implemented all of the suggestions of a type 2, technical CTO/CIO, do you believe they would be able to withstand such a pentest, and if so what would you estimate the cost to totally compromise their systems be? All of my outsider knowledge indicates that none of the multi-billion dollar organizations can protect themselves from adversaries 1/1000th their size, and I would like to know if this is consistent with the knowledge of people with more direct firsthand experience. Thanks.

Enginerrrd 7y ago
I'd venture to guess this is almost a natural law. Look at the pathogen/organism relationship in life. A small attacker with low overhead can cripple a large complex organism. That's been evolving for billions of years without resolution. In essence, I imagine that security vs cost is an exponential curve with security being the independent variable, and cost the dependent variable. At the lowest levels, this is great because there's some very easy and cheap fixes to get large gains in security, but at the high end, each incremental gain increases in cost until it reaches beyond the realm of feasibility. Air gap all your computers? Cool, you just killed productivity which massively increased costs. Actively supervise every user? There's some niche systems that probably require this level of security, but it's rare. Even then, I don't think such a thing as perfect security exists.
arminiusreturns 7y ago
I don't think any of them could withstand a 50k contract to breach TBH. The only ones that really could generally tend to be DoD/E related. That's not necessarily the point though, in an era of APT. I like to say breaches are inevitable, the real question is how fast do you know about it so you can mitigate/respond? HIDS/NIDS, auditing, monitoring and logging are the core of this imho.

chaps 7y ago

Oh my, yes. I've had very similar privileges, and it's almost universally a complete shitshow. It doesn't really ever matter what the age, size, skillsets, etc, of the company are, either. It's just.... all bad. Example: one of the first questions I try to ask when starting a new job is, "Can I have an accurate list of the physical hosts and the IP of the management NIC?", and the answer is usually some form of, "We don't maintain a list like that, you'll have to grab everything from [pick two]: (DNS|Spreadsheets|Monitoring System|Configuration Management|nmap|ARP tables).". Then, because of course, lists don't match with each other, so literally no source of truth for what hosts exist in a network exists. Even for production! It's truly maddening.

jiggawatts 7y ago
My follow-up question is: "If you don't have a list, how do you apply critical security updates to the lights-out management firmware?" Answer: "The light-ou... what?"
- chaps 7y ago
  I'm stealing that. :)
- anchpop 7y ago
  I think I'm missing the pun here, can someone explain?
  - chaps 7y ago
    Not a pun - OP's talking about these[1], which essentially gives an IP address to a serial port, often internet-facing. The joke is about how clueless management is about their infrastructure. https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out

Stranger43 7y ago

It's not just incompetence, what we are seeing is far to systemic to be the fault of mere individuals it have to be rooted in the way modern business culture deals with IT. Almost every real world security post mortem i have ever seen ends up as untimely application of patches due to the business either rejecting downtime/changes or simply not having the tools and staffing in place to keep up with changing vendor recommendation and security updates. And despite every evidence based report placing the blame for our current mess on "systematic maintenance failures" the only thing we heer hear from the MBA mills and business leaders is a call for more snakeoil products that can, do what nobody have yet to demonstrate under real world conditions, and add security to an insecure environment. This is not incompetence/ignorance it's denial likely caused by an "prisoners dilemma" explained elegantly in H.C. Andersen's "The Emperor's New Clothes" where nobody wants to be the first to admit that they aren't keeping up with best practices for fear of ridicule or worse.

smacktoward 7y ago

Perhaps this is less a knock on those people than it is with the position of CTO/CIO itself. A role that only works effectively when it's filled by a special unicorn candidate is a broken role. I can think of some reasons why this might be: - It takes a particular kind of personality to ascend to a C-suite role, and that personality isn't a good fit for solving this particular problem - Concentrating responsibility for this problem in a single C-suite role lets the rest of the C-suite ignore it as "somebody else's problem" - Concentrating responsibility for this problem in a single C-suite role serves mostly to provide a handy scapegoat when things inevitably go wrong - This is a cross-cutting problem that can't effectively be siloed into a single role, even a C-suite one; unless the CEO/board also buy in, the CTO/CIO will lack the authority required to override objections or inaction on the part of other officers

dgzl 7y ago

I used to contract with state governments and can say the same thing about them...

anonsivalley652 7y ago

I was a sysadmin since 17 and a client-facing, on-prem AWS consultant for F200 and startups in the early days. A lot of big companies outsource their IT to the cheapest vendor, who, in turn, outsources roles to onshore/offshore. There's not necessarily incompetence by virtue of people from elsewhere in the world, but there is sufficient incompetency, second-stringers and lack of accountability because of all the bureaucratic layers and indirection in the responsibilities. As such, technical debt, confusion and substandard work thrives because of all the vampiric technology "ticks" consulting companies who maximize profit before customer value. The way-out is for companies to directly hire fewer but better technical employees, employee-ownership and not jumping on the IPO bandwagon because of the perverse motivations of publicly-traded companies.

peterwwillis 7y ago

I don't think it's a "Tech vs Business" problem. Tech is just another part of the business. Every part of the business may have bottlenecks; sales, marketing, support, finance, etc. C-level has to learn to identify the bottleneck in the value chain and improve it until it goes away. You don't have to become an expert in every field to do that, but you do have to ask a lot of questions and use an improvement loop. It's the same whether you're in manufacturing, distribution, health care, etc. Yes, you have to be able to learn a bit about the field. But just being a domain expert ("knowing about tech") often does not substantially improve the overall output of the value chain, as experienced engineers learn eventually.

cat199 7y ago
> Tech is just another part of the business. while this is completely true, many people on the 'business side' don't see it this way at all. thankfully this is changing in the culture, but at a glacial pace.

eunos 7y ago

I can understand the peril of "Too much MBA, not enough tech" but what are the perils of "Too much tech, not enough MBA" ?

cat199 7y ago
"lets postpone delivery of the system to rewrite everything again from scratch this month because now the coolest tech is X and I really don't want to be stuck using last months technology even though it has 0 impact to users and actually negatively impacts the business case in terms of skillset staffing and operational overhead" for starters..
spookthesunset 7y ago
> Too much tech, not enough MBA Yup. Left to their own devices, engineers will chose to do what is “coolest” using the hippest technology stack and the most pure architecture imaginable. And they’ll build something and it will be cool.. to them.... but it won’t solve any business needs or will solve the wrong ones and not deliver as much value as it could have if it had somebody with a good business head on. ...or they’ll go reinventing the wheel and build massive costly systems that should have been purchased off the shelf from a vendor. Nope. You need a balance. It isn’t enough to just have tech smarts. You need somebody with business smarts to keep things grounded and solving real business problems that deliver value to the customer.

erikbye 7y ago

> There is a lot more incompetence than you would ever want to believe I'm currently in ops/sysadmin. Large org. Yes, the incompetency you see is sometimes incredulous: I have helped developers install Visual Studio; how to use RDP; had to explain how saving to local disk is different from a networked folder ... I can't think of better examples at the moment, though they exist and are plenteous. Thing is, I am a hundred percent sure these same people could have turned the tables on me and be aghast by my lack of knowledge in many disciplines. There are more things to know in this world than any one person can and it is thinking you are beyond this that is truly delusional. Even with a "verified expert" in a subject, whatever that means, we can find ample holes in their knowledge.