17 ms·

God these "we're so ignorant!" responses are sooooo tiring. If people don't want to hire the hardcore nerds that know this shit, if they don't want to pay for t

by tomc1985 5y ago

God these "we're so ignorant!" responses are sooooo tiring. If people don't want to hire the hardcore nerds that know this shit, if they don't want to pay for the expertise to work the technology they build their business on, then maybe they deserve it. This modern attitude of compartmentalization, extremely localized expertise, and outsourcing everything to the cloud is going to be our downfall. Stop hiring muggles!

jiggawatts 5y ago
"No." Says a million finance people that prefer armies of compliant muggles over a few self-important wizards.
flyinglizard 5y ago
You can’t just man every brick and mortar out there with “hardcore nerds”. There simply isn’t enough of them.
g_p 5y ago
It strikes me that companies need to look back at covid and consider the value IT provided to their business during that period, and the extent to which it has grown their business since the 90s. Then take that and allocate a reasonable chunk of that growth (say 15%) to ensuring that can continue, through ongoing investment in IT. Their alternative is to abandon the internet, go back to paper, and dump their IT costs (and boost to business). Unfortunately though, as long as there are bean counters trying to cut every outgoing and outsource every cost, and play accounting bingo to turn everything into monthly opex, it's unlikely to change. Most companies need skilled technical people because the sheer aggregation of risk in a few outsourced providers (see Kaseya recently) shows that they won't be top priority when something hits the fan. If they want to be top priority, they need the people on payroll and on site. Not everyone needs a group of rockstar 10x-types, but we do definitely need better fundamental IT knowledge and ability to solve problems. And business needs to make clear this is what's needed in terms of job adverts and compensation - supply tends to rapidly learn to deliver what's valued... If you can convince bean counters to pay anything for it, that is...
- joe_the_user 5y ago
  Companies for whom a data-loss will cause significant distress/damage to the public should be penalized for that loss, up and including jail time for their officers. Companies for whom data loss doesn't impact the public should be able to screw around however they want.
  - lmm 5y ago
    The very nature of a limited-liability corporation encourages picking up nickels in front of a steamroller. Even holding the companies liable for the damages they do isn't enough, because they'll take risks that they can't ever repay (and jailing their officers might make us feel better but it can't un-steal your identity).
    - squiggleblaz 5y ago
      > they'll take risks that they can't ever repay (and jailing their officers might make us feel better but it can't un-steal your identity). I'm not sure exactly who the officers of a company are (genuinely) but if we're talking about the decision makers - the board and CEOs and whoever - not the employees, then most of them aren't going to take risks that make them genuinely likely to go to jail. Creating that genuine risk is probably the only way of manipulating their behavior, especially when they haven't got any real skin in the game (e.g. a CEO paid a few million and some irrelevant stock holdings in the company that make their net worth high, but which are safe to lose). The main issue is that such people are often well enough connected that they can spin a story that it's completely unreasonable to hold them responsible for their decisions. Personal responsibility is something for poor people. Someone will find a way to make the law that says legalese for "If personal data from a company is stolen because they chose to interpret the risks in way that unreasonably exaggerated the migitation costs and downplayed the restoration costs, or because they failed to consider the risks to the personal data they ought to have known they were collecting, then the CEO should go to prison for eight to eighteen months" mean "If the CEO personally steals personal data from a company and sells it to foreign agents, then they get twenty days in a luxury resort - but if they just use it to increase the cost of customers' insurance, then they get saddled with new high paying job, poor fellow".
void_mint 5y ago
Or...you know....as with all technology, just make doing the _right thing_ easier and cheaper. I don't want to hire the "hardcore nerds" that gatekeep expertise behind ridiculous rates and tribal knowledge, fwiw. I'd much rather pay for services, incrementally adoptable technology and clear roadmaps.
- csomar 5y ago
  > I don't want to hire the "hardcore nerds" that gatekeep expertise behind ridiculous rates and tribal knowledge, fwiw. Seriously? Most of the Archlinux/Linux (where nerds congregate) automation and customization stuff is open source. Lots of people sharing their configs, and you can copy from them. Also, why is it not okay for nerds to command high rates? Doctors do it. Lawyers do it. Politicians do it. And most of them suck at their job, by the way. So if a "hacker" could deliver on his promise, I think paying him a high white-collar rate is quite fair.
  - void_mint 5y ago
    > Seriously? Most of the Archlinux/Linux (where nerds congregate) automation and customization stuff is open source. Lots of people sharing their configs, and you can copy from them. Yep! So tell me again why I should employ what the previous poster referred to as "hardcore nerds"? > Also, why is it not okay for nerds to command high rates? Doctors do it. Lawyers do it. Politicians do it. And most of them suck at their job, by the way. So if a "hacker" could deliver on his promise, I think paying him a high white-collar rate is quite fair. I'm responding almost entirely to the idea that "hardcore nerds" are the people to employ, and "muggles" are not. Paying for expertise is great! But really, as with all technology, I want to pay for people to make things easier for those around them. It strikes me as painfully obvious that, if you're crying out to employ "hardcore nerds" because "muggles" can't handle the work, you're also probably not the type to employ at all, either way. This is just my take, however.
    - Retric 5y ago
      The point is you want hardcore nerds relative to your organization. At the coffee shop level that might just be paying someone 15$ an hour rather than winging it. It’s little different than calling an electrician rather than stringing extension cords everywhere. At the other end, Fortune 500 companies have unique needs which requirer significant expertise. At that level trying to outsource most stuff is perfectly reasonable, but they still need internal talent to avoid being ripped off or suffering major disasters.
rhizome 5y ago
>If people don't want to hire the hardcore nerds that know this shit Sorry, all those people had to get jobs at AWS, Digital Ocean, Heroku, etc. years ago if they didn't want to be Puppet jockeys for the next mumble years. Frankly I wouldn't be surprised if the shared-hosting "cloud" companies didn't actively push DevOps as a way of reconfiguring their hiring landscape.
ourmandave 5y ago
Butch Cassidy: If he'd just pay me what he's paying them to stop me robbing him, I'd stop robbing him.
emodendroket 5y ago
I think cloud-based businesses are way more likely than homespun solutions to actually be able to stand up a system from backups quickly, so I'm not sure what you mean about the cloud being "our downfall." Specialization is a sign of a maturity, isn't it? A few hundred years ago you could more or less know what there was to know in multiple fields of science, but this is no longer possible.
- simonh 5y ago
  Quite, when you build a system in AWS you are hiring a bunch of hard core nerds to do all this shit for you. That's the deal. You don't need to set up your own restore server to verify your tape backups every night, you just archive your snapshots to Glacier. You can even set up a restore environment in a new VPC for zero capital costs and just spin it up once a quarter to verify the process. It makes all of this stuff an order of magnitude easier and cheaper.
dkokelley 5y ago
1. Couldn't you argue the same thing about brick and mortar stores that get robbed? "If they don't want to hire hardcore commandos to protect their property, maybe they deserve to get robbed/looted?" (My point is that incompetence doesn't make it morally ok that a criminal thing happens to someone) 2. There's a risk/reward component. Nobody likes to buy insurance. Resource constrained organizations will almost always choose to invest their resources to get MORE resources, not protect against the chance that something bad will happen. A rational organization should only invest in protection when the risk is so great that it's likely to interfere with its primary business (beyond their legal/moral obligation to protect information they're trusted with). 2a. If a $2m ransomware attack hits your organization every 5 years, and it would cost you $1m/year in talent & resources to harden against this, you SHOULD let it happen because it's cheaper. Just patch the vulnerability each time it happens and try to stretch the next ransomeware attack to more than 5 years away. 3. Of course there are many irrational organizations that don't protect against ransomware for irrational reasons (e.g. due to internal politics). There's not much to say here except that at some level of management (including the CEO & board) where people are not paying attention to what's happening, and they should go hire those hardcore nerds and pay them what they need to.
- deleted 5y ago
  [deleted]
- DiffEq 5y ago
  On Point 1....Yes; and besides that most Brick and mortar stores have insurance for catastrophic loss...so essentially they have a working backup.
  - bash-j 5y ago
    And most physical items in a store can easily be replaced. If a criminal is holding your shop ransom, it's not for intellectual property.
  - deleted 5y ago
    [deleted]
- 5y ago
aaron695 5y ago
> If people don't want to hire the hardcore nerds that know this shit Put a cost to this then perhaps your argument will be believable, or you will see it's not a viable option. This is what companies like Accenture try to do. They are not cheap and I'm not sure if they have withstood a ransomeware attack.
jrockway 5y ago
I think people are afraid to take the first step. For example, you really want to continuously test your backups, and that's not a straightforward engineering problem. But, before you invest time in that, you could just test them manually every month or so. A lot of people let great become the enemy of good -- if you restored one backup successfully, you're way ahead of most of the industry. Sure stuff can break in between the manual runs, and disaster can easily strike while they're in the broken state. But, that's less likely than "oh, our backups missed a critical table" or something. I also think doing a disaster recovery exercise every few months is also highly valuable. You might think you know how everything works, and that you've covered everything, but remove permission from staging for everyone on your team and have them build it from scratch, and you'll figure out all the things that you forgot about that silently work in the background. (Last time we did this, we realized we didn't back up our sealed secret master keys -- they get auto-rotated out from under us. So we had to provision new passwords and recreate the sealed secrets to recover from a disaster. Now we back those up ;) (A corollary: if you've had really good uptime in the last year, your customers probably think that you offer 100% uptime. But you don't, and they're going to be mad when you have an actual outage that the SLO covers. So it might be worth breaking them for 5 minutes a quarter or something so that they know that downtime is a possibility.) One more point that I want to make... sometimes the cost isn't worth it. If you're a startup, you live or die by making the right product at the right time. If your attention is focused on continuously testing your backups, your time is taken away from that core challenge. While a hack where you don't have a backup is likely to kill your company, so is having a bad product. So, like everything, it's a tradeoff.

jiggawatts 5y ago

"No." Says a million finance people that prefer armies of compliant muggles over a few self-important wizards.

flyinglizard 5y ago

You can’t just man every brick and mortar out there with “hardcore nerds”. There simply isn’t enough of them.

g_p 5y ago

It strikes me that companies need to look back at covid and consider the value IT provided to their business during that period, and the extent to which it has grown their business since the 90s. Then take that and allocate a reasonable chunk of that growth (say 15%) to ensuring that can continue, through ongoing investment in IT. Their alternative is to abandon the internet, go back to paper, and dump their IT costs (and boost to business). Unfortunately though, as long as there are bean counters trying to cut every outgoing and outsource every cost, and play accounting bingo to turn everything into monthly opex, it's unlikely to change. Most companies need skilled technical people because the sheer aggregation of risk in a few outsourced providers (see Kaseya recently) shows that they won't be top priority when something hits the fan. If they want to be top priority, they need the people on payroll and on site. Not everyone needs a group of rockstar 10x-types, but we do definitely need better fundamental IT knowledge and ability to solve problems. And business needs to make clear this is what's needed in terms of job adverts and compensation - supply tends to rapidly learn to deliver what's valued... If you can convince bean counters to pay anything for it, that is...

joe_the_user 5y ago
Companies for whom a data-loss will cause significant distress/damage to the public should be penalized for that loss, up and including jail time for their officers. Companies for whom data loss doesn't impact the public should be able to screw around however they want.
- lmm 5y ago
  The very nature of a limited-liability corporation encourages picking up nickels in front of a steamroller. Even holding the companies liable for the damages they do isn't enough, because they'll take risks that they can't ever repay (and jailing their officers might make us feel better but it can't un-steal your identity).
  - squiggleblaz 5y ago
    > they'll take risks that they can't ever repay (and jailing their officers might make us feel better but it can't un-steal your identity). I'm not sure exactly who the officers of a company are (genuinely) but if we're talking about the decision makers - the board and CEOs and whoever - not the employees, then most of them aren't going to take risks that make them genuinely likely to go to jail. Creating that genuine risk is probably the only way of manipulating their behavior, especially when they haven't got any real skin in the game (e.g. a CEO paid a few million and some irrelevant stock holdings in the company that make their net worth high, but which are safe to lose). The main issue is that such people are often well enough connected that they can spin a story that it's completely unreasonable to hold them responsible for their decisions. Personal responsibility is something for poor people. Someone will find a way to make the law that says legalese for "If personal data from a company is stolen because they chose to interpret the risks in way that unreasonably exaggerated the migitation costs and downplayed the restoration costs, or because they failed to consider the risks to the personal data they ought to have known they were collecting, then the CEO should go to prison for eight to eighteen months" mean "If the CEO personally steals personal data from a company and sells it to foreign agents, then they get twenty days in a luxury resort - but if they just use it to increase the cost of customers' insurance, then they get saddled with new high paying job, poor fellow".

void_mint 5y ago

Or...you know....as with all technology, just make doing the _right thing_ easier and cheaper. I don't want to hire the "hardcore nerds" that gatekeep expertise behind ridiculous rates and tribal knowledge, fwiw. I'd much rather pay for services, incrementally adoptable technology and clear roadmaps.

csomar 5y ago
> I don't want to hire the "hardcore nerds" that gatekeep expertise behind ridiculous rates and tribal knowledge, fwiw. Seriously? Most of the Archlinux/Linux (where nerds congregate) automation and customization stuff is open source. Lots of people sharing their configs, and you can copy from them. Also, why is it not okay for nerds to command high rates? Doctors do it. Lawyers do it. Politicians do it. And most of them suck at their job, by the way. So if a "hacker" could deliver on his promise, I think paying him a high white-collar rate is quite fair.
- void_mint 5y ago
  > Seriously? Most of the Archlinux/Linux (where nerds congregate) automation and customization stuff is open source. Lots of people sharing their configs, and you can copy from them. Yep! So tell me again why I should employ what the previous poster referred to as "hardcore nerds"? > Also, why is it not okay for nerds to command high rates? Doctors do it. Lawyers do it. Politicians do it. And most of them suck at their job, by the way. So if a "hacker" could deliver on his promise, I think paying him a high white-collar rate is quite fair. I'm responding almost entirely to the idea that "hardcore nerds" are the people to employ, and "muggles" are not. Paying for expertise is great! But really, as with all technology, I want to pay for people to make things easier for those around them. It strikes me as painfully obvious that, if you're crying out to employ "hardcore nerds" because "muggles" can't handle the work, you're also probably not the type to employ at all, either way. This is just my take, however.
  - Retric 5y ago
    The point is you want hardcore nerds relative to your organization. At the coffee shop level that might just be paying someone 15$ an hour rather than winging it. It’s little different than calling an electrician rather than stringing extension cords everywhere. At the other end, Fortune 500 companies have unique needs which requirer significant expertise. At that level trying to outsource most stuff is perfectly reasonable, but they still need internal talent to avoid being ripped off or suffering major disasters.

rhizome 5y ago

>If people don't want to hire the hardcore nerds that know this shit Sorry, all those people had to get jobs at AWS, Digital Ocean, Heroku, etc. years ago if they didn't want to be Puppet jockeys for the next mumble years. Frankly I wouldn't be surprised if the shared-hosting "cloud" companies didn't actively push DevOps as a way of reconfiguring their hiring landscape.

ourmandave 5y ago

Butch Cassidy: If he'd just pay me what he's paying them to stop me robbing him, I'd stop robbing him.

emodendroket 5y ago

I think cloud-based businesses are way more likely than homespun solutions to actually be able to stand up a system from backups quickly, so I'm not sure what you mean about the cloud being "our downfall." Specialization is a sign of a maturity, isn't it? A few hundred years ago you could more or less know what there was to know in multiple fields of science, but this is no longer possible.

simonh 5y ago
Quite, when you build a system in AWS you are hiring a bunch of hard core nerds to do all this shit for you. That's the deal. You don't need to set up your own restore server to verify your tape backups every night, you just archive your snapshots to Glacier. You can even set up a restore environment in a new VPC for zero capital costs and just spin it up once a quarter to verify the process. It makes all of this stuff an order of magnitude easier and cheaper.

dkokelley 5y ago

1. Couldn't you argue the same thing about brick and mortar stores that get robbed? "If they don't want to hire hardcore commandos to protect their property, maybe they deserve to get robbed/looted?" (My point is that incompetence doesn't make it morally ok that a criminal thing happens to someone) 2. There's a risk/reward component. Nobody likes to buy insurance. Resource constrained organizations will almost always choose to invest their resources to get MORE resources, not protect against the chance that something bad will happen. A rational organization should only invest in protection when the risk is so great that it's likely to interfere with its primary business (beyond their legal/moral obligation to protect information they're trusted with). 2a. If a $2m ransomware attack hits your organization every 5 years, and it would cost you $1m/year in talent & resources to harden against this, you SHOULD let it happen because it's cheaper. Just patch the vulnerability each time it happens and try to stretch the next ransomeware attack to more than 5 years away. 3. Of course there are many irrational organizations that don't protect against ransomware for irrational reasons (e.g. due to internal politics). There's not much to say here except that at some level of management (including the CEO & board) where people are not paying attention to what's happening, and they should go hire those hardcore nerds and pay them what they need to.

deleted 5y ago
[deleted]
DiffEq 5y ago
On Point 1....Yes; and besides that most Brick and mortar stores have insurance for catastrophic loss...so essentially they have a working backup.
- bash-j 5y ago
  And most physical items in a store can easily be replaced. If a criminal is holding your shop ransom, it's not for intellectual property.
- deleted 5y ago
  [deleted]
5y ago

aaron695 5y ago

> If people don't want to hire the hardcore nerds that know this shit Put a cost to this then perhaps your argument will be believable, or you will see it's not a viable option. This is what companies like Accenture try to do. They are not cheap and I'm not sure if they have withstood a ransomeware attack.

jrockway 5y ago

I think people are afraid to take the first step. For example, you really want to continuously test your backups, and that's not a straightforward engineering problem. But, before you invest time in that, you could just test them manually every month or so. A lot of people let great become the enemy of good -- if you restored one backup successfully, you're way ahead of most of the industry. Sure stuff can break in between the manual runs, and disaster can easily strike while they're in the broken state. But, that's less likely than "oh, our backups missed a critical table" or something. I also think doing a disaster recovery exercise every few months is also highly valuable. You might think you know how everything works, and that you've covered everything, but remove permission from staging for everyone on your team and have them build it from scratch, and you'll figure out all the things that you forgot about that silently work in the background. (Last time we did this, we realized we didn't back up our sealed secret master keys -- they get auto-rotated out from under us. So we had to provision new passwords and recreate the sealed secrets to recover from a disaster. Now we back those up ;) (A corollary: if you've had really good uptime in the last year, your customers probably think that you offer 100% uptime. But you don't, and they're going to be mad when you have an actual outage that the SLO covers. So it might be worth breaking them for 5 minutes a quarter or something so that they know that downtime is a possibility.) One more point that I want to make... sometimes the cost isn't worth it. If you're a startup, you live or die by making the right product at the right time. If your attention is focused on continuously testing your backups, your time is taken away from that core challenge. While a hack where you don't have a backup is likely to kill your company, so is having a bad product. So, like everything, it's a tradeoff.