10 ms·

In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes. 1. Massive security breaches af

In the tech-related (but now fully mainstream, unlike a few years ago) news these days, there are a number of recurring themes. 1. Massive security breaches affecting major corporations, governments, and so on. 2. Massive IT outages, affecting corporations, governments and so forth. It doesn't take a lot of incidents to mark them as 'massive' since things are strongly centralized in this world. Meanwhile, in the last 15 years of my IT career, never have I seen such a strong push towards offshoring. One would think that with such a vulnerable IT landscape mixed with an unprecedented dependence on IT infrastructure, that CTO's would want to spend more and not LESS on operational costs. I'm slightly baffled.

codeulike 9y ago
The reason is simple: short term thinking driven by short term investors
- ams6110 9y ago
  Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Perhaps it really is cheaper to have a few outages than it is to built the sort of systems that never go down. Every "nine" you add gets more and more expensive. Or perhaps the systems are so complex that it's just impossible for them to be fully reliable, and this is just one of the costs of having the ability to fly anywhere in the world, cheaply and at the drop of a hat, and to have it work 99% of the time.
  - mancerayder 9y ago
    Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Is that what you tell hundreds of thousands of passengers sitting in a terminal with canceled flights and confused airline employees during one of the biggest holiday weekends? If your credit card number gets stolen or worse, your identity, and it's up to you to get that straightened out, are you just going to shrug your shoulders and write that off as the inevitable impossibility of perfection?
    - ams6110 9y ago
      I'm saying yes, maybe that is the case. Maybe those scenarios are either a) cheaper to handle on occasion than building systems which will never fail, or b) unavoidable (with some low probability). Air travel itself is not 100% safe. Planes crash. People die. Yet we feel that the probabilities are enough in our favor that we accept it.
      - toyg 9y ago
        What? "We" don't accept anything: strict regulations and processes were put in place to reduce the probability of accidents to the absolute minimum. Did this stop manufacturers and airlines to spearhead this or that new plane? Maybe - very few companies can carry that sort of compliance effort; but it's why the public trust air travel as fundamentally safe (statistically speaking, it's probably safer than cars).
rashkov 9y ago
It seems that companies would rather buy "insurance" rather than spend money on actual security. They have accepted that they will eventually be hacked or have a really expensive outage, so now they want a cushion of money to pay for any penalties and lawsuits. I get the impression that for the most part, no one really believes in security, and doesn't understand that it's a process and not a product. They already tried throwing money at the problem by purchasing security products, which ultimately failed to deliver security. Now they're trying the opposite by cutting costs. Meanwhile there are very few consequences to losing control of people's data.
- jdavis703 9y ago
  Why aren't there Security Process Engineers (SPE)? I was recently waiting on a massively delayed flight and observed three instances where employees left the computer behind the check in desks unlocked. I'm sure these computers checked off all the requirements for having a firewall turned on, anti-malware software, auto-updates turned on, etc. But the front door was left wide open because the employees were forced to move between their counters and the actual jet bridge. Since they get penalized for late flights, but not for security breaches it seemed as if they didn't care. This would be where a SPE would come in and propose something to remediate the process failure.
  - konceptz 9y ago
    There are certainly this type of thing out there. SSI or software security initiatives, when given greenlights based on buy in from governance can introduce secure processes. But before this happens to any meaningful level, risk based anaysis should prove it is cost effective. Guess what those analysis say?
- yeukhon 9y ago
  "They already tried throwing money at the problem by purchasing security products, which ultimately failed to deliver security." So true. I used to wonder why corporates can't build a real security team. One would spend $1M on an annual contract but perhaps you will do okay for $700,000 for a few security engineers. Then I realized that most corps are indeed looking for insurances. You see, managed security vendors have a team. You don't manage the hiring, the tooling building etc. Obviously you can't just sit back and watch them put out the fire. You train them to understand a bit about your environments, you work with them on triages and resolutions, and you work with them to integrate your systems with them (e.g install an agent etc) But it isn't most enterprises' interest to build a strong security team. Many of the in-house security team's role is to manage incident response. Many of them don't really have a say, they are often consulted and that's it. Many of the product demos sound exicting and if you are not careful during your POC you will end up with a medicore product. Even if you did your best during POC yo evaluate the product, you will realized the product is really medicore. It caught the low-hanging fruits and are full of false positive. You ask for better analysis but because they come in as black-box, there is a lot of back and forth before you can act on the issue. So at work I would get a ticket from the vendor and I would end up doing a lot of the analysis. That sucks because while I enjoy doing security triage, that's not my role, but the conflicting side is at least they caught something. I would have to engineer or deployment some solutions and monitor the whole thing, since no one man or team can do everyone's job. Since no one wants to be responsible for other's job, that's the whole point of having a secuirty team. It just happens that team, the people who are building tools and monitoring incidents are outsourced. Most importantly, there is very little control of what your vendor can do. Want a new feature? Want better reporting? Want to change some configuration? A lot of time you are out of luck, or you need to be patience. But, seriously, security practice is not magical. There are best practices such as SSH to server using key or cert authentication, not password. Least privilege to run processes, so system admin/devops should create a checklist on what's in place what isn't. It does take a serious committment from management to move forward though. And security is an iterative process. Evaluate the best most secure option possible for the moment being, and put a realistic plan together for the remaining unresolved concerns.
fauigerzigerk 9y ago
Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore? Arguably, delegating IT operations to a proper IT company could improve security and reliability. We use a low cost offshore IT provider and we haven't had any security or reliability issues so far (https://gsuite.google.com/ https://gsuite.google.com/)
- raesene9 9y ago
  The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible. So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded. An outsourcing company is not going to spend money it doesn't have to, simple as that. SAAS like gsuite is an entirely different prospect. you're buying a service from Google not outsourcing your IT to them. If google's service was insecure or unreliable no one would buy it.
  - fauigerzigerk 9y ago
    I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously. Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news. And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation is not being met.
    - sgt101 9y ago
      Here is the list of things that are worse for an IT outsourcer. 1. Being more expensive than the other guy.
      - fauigerzigerk 9y ago
        I doubt that anyone wants to compete solely on price. It's a margins game.
egwor 9y ago
In one way the CTO gets the 'purchase' of insurance here. Not against the company having issues but for their job because they can say, "Well I hired the best company to do this. If they couldn't do this, then we definitely couldn't!"
ajeet_dhaliwal 9y ago
Ignoring the offshoring part of this for a moment I think you're on point about the CTOs, it baffled me too at the beginning. When I started doing sales for a startup service and was targeting CTOs I discovered based on my interactions that outside of Silicon Valley companies or obviously tech centered companies, CTOs in other countries and industries are in many cases just the usual political shark type person interested in showing how they were able to cut costs to help improve profit and often there isn't the real interest in tech to understand why doing things better is so important. As an example a person in a tech leader role at a major company in the UK who I thought may be interested in hearing about a product was quite rude about me never contacting them again and refusing to take any calls or emails before even hearing about something that other CTOs has at least looked at and some had even become customers. She happened to have a personal blog that I checked out and all it mentioned was her passion for swimming, nothing at all, not a single thing about tech. Why is this person in this role then? Beats me but it is sadly commonplace, until you get people at the top that truly understands tech and was actually a software engineer themselves at the start of their career and are things won't change.
- justinclift 9y ago
  > As an example a person in a tech leader role at a major company in the UK who I thought may be interested in hearing about a product was quite rude about me never contacting them again ... You didn't spam them did you? Your description of their behaviour sounds a lot like the standard response to spammers. eg FOAD
  - ajeet_dhaliwal 9y ago
    No I did not, and that's the issue, a lot of these people are not actually interested in making things any better, they'd rather not be disturbed instead. And then something like what happened today happens. And then people are misdirecting blame to others. The leadership is in most cases to blame.
    - justinclift 9y ago
      Interesting. What approach did you use to tell them about the product they hadn't heard of before?
gaius 9y ago
that CTO's would want to spend more and not LESS on operational costs The incentives are the same as they always were. A well run IT organization can coast on momentum for a long time - everything is neatly automated. So they sack their good engineers and hire an outsourcing company and save an amazing amount of money just long enough to get that next bonus or promotion, and they're long gone when the momentum runs out and the wheels come off. Repeating the same scam at another company.
lordnacho 9y ago
I liken it to shorting options. Let me explain. In the derivatives world (that I started my career in) there's a common theme: you can sell options, and generally you will make money. The only issue is, now and again you lose a lot of money. Often enough to wipe out everything you made. But incentives matter. If you come in as a new guy, you want to impress by making money. And come up with some story about how you'll know when not to sell options. You also don't want to be the guy losing money when the other desks are making it. If there's a crisis and everyone loses, that's fine. Unforeseen, right? It happened to everyone. The CTO is facing a very similar choice. Save money by not upgrading, work the security guys a bit harder and let them update the OS a bit slower. Don't think too hard about redundancy. Or code review, that takes away dev time. You're saving money. But now and again, something happens. Maybe you have to leave, but you'll still have "saved money" for several years before. You'll land somewhere.
madeofpalk 9y ago
> never have I seen such a strong push towards offshoring As I've been working in the airline IT industry, I've observed that the reality is more nuanced. Core booking and ticketing systems are 'outsourced' to solutions from Amadeus and Sabre. This makes sense not only for many strategic reasons (easier to have airline alliances when the airlines are all on the one system), but also because airlines are in the business of flying planes, not building ticketing systems. Airlines are often full of loads of legacy and old systems which you just can't hire developers for (in the numbers, and costs, desired) - this is where Indian outsourcing firms come into the play and resolve the shortcomings. They're taking all the jobs and work that no one really wants to do anyway. However airlines, and many other industries (banks come to mind), are realising that their digital offering is just as much, if not more, a core part of their offering and those are skills they need to bring in house and own if they want to be competent. I'm seeing more and more web and app developers be brought in house so airlines can establish these competencies. Digital/Product agencies are 'stepping in' in the meantime to help out (like Virgin America and Work & Co).
- hinkley 9y ago
  Airlines are logistics companies that fly planes. They have to have a functioning plane at the right location - and only the right location - at the right time, or they bleed money. It baffles me that they ever thought outsourcing part of that logistics was a good idea. But then I've worked at tech companies that tried to outsource their IT departments too.
  - madeofpalk 9y ago
    Retailers are in the business of selling things, yet 'outsource' their Point of Sales systems. This is just the same thing.
    - hinkley 9y ago
      No, it isn't. The point of sale system is about taking payments. If your core business isn't 'the processing of financial transactions', then buying that is the sane thing to do. IBM being able to field a POS system doesn't make them a competent retailer. But they handle regulatory rules for you and let you be a competent retailer. Airlines aren't just responsible for flying planes. They are responsible for the planes. It's not enough for the plane to show up in Chicago by 5pm, it has to be there and all of the necessary maintenance and safety work has to be done by 5pm. Anyone who could figure that out for them would simply go into business for themselves and compete.
hinkley 9y ago
Don't manufacturers face the same sorts of issues with supply chains? I mean, if there's a train wreck or an ice storm, the people who needed those deliveries are pretty much screwed. I think we like to pretend that this is a problem unique to informatics.

codeulike 9y ago

The reason is simple: short term thinking driven by short term investors

ams6110 9y ago
Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Perhaps it really is cheaper to have a few outages than it is to built the sort of systems that never go down. Every "nine" you add gets more and more expensive. Or perhaps the systems are so complex that it's just impossible for them to be fully reliable, and this is just one of the costs of having the ability to fly anywhere in the world, cheaply and at the drop of a hat, and to have it work 99% of the time.
- mancerayder 9y ago
  Or maybe it's what is repeated here so often: don't let the perfect be the enemy of the good. Is that what you tell hundreds of thousands of passengers sitting in a terminal with canceled flights and confused airline employees during one of the biggest holiday weekends? If your credit card number gets stolen or worse, your identity, and it's up to you to get that straightened out, are you just going to shrug your shoulders and write that off as the inevitable impossibility of perfection?
  - ams6110 9y ago
    I'm saying yes, maybe that is the case. Maybe those scenarios are either a) cheaper to handle on occasion than building systems which will never fail, or b) unavoidable (with some low probability). Air travel itself is not 100% safe. Planes crash. People die. Yet we feel that the probabilities are enough in our favor that we accept it.
    - toyg 9y ago
      What? "We" don't accept anything: strict regulations and processes were put in place to reduce the probability of accidents to the absolute minimum. Did this stop manufacturers and airlines to spearhead this or that new plane? Maybe - very few companies can carry that sort of compliance effort; but it's why the public trust air travel as fundamentally safe (statistically speaking, it's probably safer than cars).

rashkov 9y ago

It seems that companies would rather buy "insurance" rather than spend money on actual security. They have accepted that they will eventually be hacked or have a really expensive outage, so now they want a cushion of money to pay for any penalties and lawsuits. I get the impression that for the most part, no one really believes in security, and doesn't understand that it's a process and not a product. They already tried throwing money at the problem by purchasing security products, which ultimately failed to deliver security. Now they're trying the opposite by cutting costs. Meanwhile there are very few consequences to losing control of people's data.

jdavis703 9y ago
Why aren't there Security Process Engineers (SPE)? I was recently waiting on a massively delayed flight and observed three instances where employees left the computer behind the check in desks unlocked. I'm sure these computers checked off all the requirements for having a firewall turned on, anti-malware software, auto-updates turned on, etc. But the front door was left wide open because the employees were forced to move between their counters and the actual jet bridge. Since they get penalized for late flights, but not for security breaches it seemed as if they didn't care. This would be where a SPE would come in and propose something to remediate the process failure.
- konceptz 9y ago
  There are certainly this type of thing out there. SSI or software security initiatives, when given greenlights based on buy in from governance can introduce secure processes. But before this happens to any meaningful level, risk based anaysis should prove it is cost effective. Guess what those analysis say?
yeukhon 9y ago
"They already tried throwing money at the problem by purchasing security products, which ultimately failed to deliver security." So true. I used to wonder why corporates can't build a real security team. One would spend $1M on an annual contract but perhaps you will do okay for $700,000 for a few security engineers. Then I realized that most corps are indeed looking for insurances. You see, managed security vendors have a team. You don't manage the hiring, the tooling building etc. Obviously you can't just sit back and watch them put out the fire. You train them to understand a bit about your environments, you work with them on triages and resolutions, and you work with them to integrate your systems with them (e.g install an agent etc) But it isn't most enterprises' interest to build a strong security team. Many of the in-house security team's role is to manage incident response. Many of them don't really have a say, they are often consulted and that's it. Many of the product demos sound exicting and if you are not careful during your POC you will end up with a medicore product. Even if you did your best during POC yo evaluate the product, you will realized the product is really medicore. It caught the low-hanging fruits and are full of false positive. You ask for better analysis but because they come in as black-box, there is a lot of back and forth before you can act on the issue. So at work I would get a ticket from the vendor and I would end up doing a lot of the analysis. That sucks because while I enjoy doing security triage, that's not my role, but the conflicting side is at least they caught something. I would have to engineer or deployment some solutions and monitor the whole thing, since no one man or team can do everyone's job. Since no one wants to be responsible for other's job, that's the whole point of having a secuirty team. It just happens that team, the people who are building tools and monitoring incidents are outsourced. Most importantly, there is very little control of what your vendor can do. Want a new feature? Want better reporting? Want to change some configuration? A lot of time you are out of luck, or you need to be patience. But, seriously, security practice is not magical. There are best practices such as SSH to server using key or cert authentication, not password. Least privilege to run processes, so system admin/devops should create a checklist on what's in place what isn't. It does take a serious committment from management to move forward though. And security is an iterative process. Evaluate the best most secure option possible for the moment being, and put a realistic plan together for the remaining unresolved concerns.

fauigerzigerk 9y ago

Is there any evidence that IT outsourcing has made matters worse? Or is it only worse if the provider is located offshore? Arguably, delegating IT operations to a proper IT company could improve security and reliability. We use a low cost offshore IT provider and we haven't had any security or reliability issues so far (https://gsuite.google.com/ https://gsuite.google.com/)

raesene9 9y ago
The key is incentives. An IT outsourcer's goal is generally to fulfill their contractual obligations (to the letter not the spirit) for as little money as possible. So if something like security or reliability isn't clearly stated in contracts (and it's extremely difficult to do that well) it is disregarded. An outsourcing company is not going to spend money it doesn't have to, simple as that. SAAS like gsuite is an entirely different prospect. you're buying a service from Google not outsourcing your IT to them. If google's service was insecure or unreliable no one would buy it.
- fauigerzigerk 9y ago
  I don't see why an IT outsourcing company would not have the greatest incentive to take security and reliability extremely seriously. Is there anything worse for an IT service provider than being blamed for a massive IT outage at a global corporation? This is headline news. And I don't see any difficult contractual issues at all. On the contrary. A massive outage, by definition, means that the contractual obligation is not being met.
  - sgt101 9y ago
    Here is the list of things that are worse for an IT outsourcer. 1. Being more expensive than the other guy.
    - fauigerzigerk 9y ago
      I doubt that anyone wants to compete solely on price. It's a margins game.

egwor 9y ago

In one way the CTO gets the 'purchase' of insurance here. Not against the company having issues but for their job because they can say, "Well I hired the best company to do this. If they couldn't do this, then we definitely couldn't!"

ajeet_dhaliwal 9y ago

Ignoring the offshoring part of this for a moment I think you're on point about the CTOs, it baffled me too at the beginning. When I started doing sales for a startup service and was targeting CTOs I discovered based on my interactions that outside of Silicon Valley companies or obviously tech centered companies, CTOs in other countries and industries are in many cases just the usual political shark type person interested in showing how they were able to cut costs to help improve profit and often there isn't the real interest in tech to understand why doing things better is so important. As an example a person in a tech leader role at a major company in the UK who I thought may be interested in hearing about a product was quite rude about me never contacting them again and refusing to take any calls or emails before even hearing about something that other CTOs has at least looked at and some had even become customers. She happened to have a personal blog that I checked out and all it mentioned was her passion for swimming, nothing at all, not a single thing about tech. Why is this person in this role then? Beats me but it is sadly commonplace, until you get people at the top that truly understands tech and was actually a software engineer themselves at the start of their career and are things won't change.

justinclift 9y ago
> As an example a person in a tech leader role at a major company in the UK who I thought may be interested in hearing about a product was quite rude about me never contacting them again ... You didn't spam them did you? Your description of their behaviour sounds a lot like the standard response to spammers. eg FOAD
- ajeet_dhaliwal 9y ago
  No I did not, and that's the issue, a lot of these people are not actually interested in making things any better, they'd rather not be disturbed instead. And then something like what happened today happens. And then people are misdirecting blame to others. The leadership is in most cases to blame.
  - justinclift 9y ago
    Interesting. What approach did you use to tell them about the product they hadn't heard of before?

gaius 9y ago

that CTO's would want to spend more and not LESS on operational costs The incentives are the same as they always were. A well run IT organization can coast on momentum for a long time - everything is neatly automated. So they sack their good engineers and hire an outsourcing company and save an amazing amount of money just long enough to get that next bonus or promotion, and they're long gone when the momentum runs out and the wheels come off. Repeating the same scam at another company.

lordnacho 9y ago

I liken it to shorting options. Let me explain. In the derivatives world (that I started my career in) there's a common theme: you can sell options, and generally you will make money. The only issue is, now and again you lose a lot of money. Often enough to wipe out everything you made. But incentives matter. If you come in as a new guy, you want to impress by making money. And come up with some story about how you'll know when not to sell options. You also don't want to be the guy losing money when the other desks are making it. If there's a crisis and everyone loses, that's fine. Unforeseen, right? It happened to everyone. The CTO is facing a very similar choice. Save money by not upgrading, work the security guys a bit harder and let them update the OS a bit slower. Don't think too hard about redundancy. Or code review, that takes away dev time. You're saving money. But now and again, something happens. Maybe you have to leave, but you'll still have "saved money" for several years before. You'll land somewhere.

madeofpalk 9y ago

> never have I seen such a strong push towards offshoring As I've been working in the airline IT industry, I've observed that the reality is more nuanced. Core booking and ticketing systems are 'outsourced' to solutions from Amadeus and Sabre. This makes sense not only for many strategic reasons (easier to have airline alliances when the airlines are all on the one system), but also because airlines are in the business of flying planes, not building ticketing systems. Airlines are often full of loads of legacy and old systems which you just can't hire developers for (in the numbers, and costs, desired) - this is where Indian outsourcing firms come into the play and resolve the shortcomings. They're taking all the jobs and work that no one really wants to do anyway. However airlines, and many other industries (banks come to mind), are realising that their digital offering is just as much, if not more, a core part of their offering and those are skills they need to bring in house and own if they want to be competent. I'm seeing more and more web and app developers be brought in house so airlines can establish these competencies. Digital/Product agencies are 'stepping in' in the meantime to help out (like Virgin America and Work & Co).

hinkley 9y ago
Airlines are logistics companies that fly planes. They have to have a functioning plane at the right location - and only the right location - at the right time, or they bleed money. It baffles me that they ever thought outsourcing part of that logistics was a good idea. But then I've worked at tech companies that tried to outsource their IT departments too.
- madeofpalk 9y ago
  Retailers are in the business of selling things, yet 'outsource' their Point of Sales systems. This is just the same thing.
  - hinkley 9y ago
    No, it isn't. The point of sale system is about taking payments. If your core business isn't 'the processing of financial transactions', then buying that is the sane thing to do. IBM being able to field a POS system doesn't make them a competent retailer. But they handle regulatory rules for you and let you be a competent retailer. Airlines aren't just responsible for flying planes. They are responsible for the planes. It's not enough for the plane to show up in Chicago by 5pm, it has to be there and all of the necessary maintenance and safety work has to be done by 5pm. Anyone who could figure that out for them would simply go into business for themselves and compete.

hinkley 9y ago

Don't manufacturers face the same sorts of issues with supply chains? I mean, if there's a train wreck or an ice storm, the people who needed those deliveries are pretty much screwed. I think we like to pretend that this is a problem unique to informatics.