7 ms·

In my experience, certificate issues is a huge tell into organization and treatment of IT folks. Every place I've worked which has had issues with last minute c

In my experience, certificate issues is a huge tell into organization and treatment of IT folks. Every place I've worked which has had issues with last minute certificate changes or expiring certificates without renewal has had a systemic problem with underpaid and understaffed IT department. This is not a new problem, organizations will always choose guaranteed profits over possible loss of business unless the loss of business is catastrophic, I just wish that in this case instead of trying to make it seem like a big deal by writing an entire multipage excuse, a company for once would be honest and say 'The risk percentage did not fall in our favor this time, but we're not going to do anything about it because it didn't really impact our profits.'

PragmaticPulp 5y ago
> Every place I've worked which has had issues with last minute certificate changes or expiring certificates without renewal has had a systemic problem with underpaid and understaffed IT department. I've seen the opposite: Organizations who spent so much on the department that everyone was getting promoted to manager and hiring someone underneath themselves to manager things. Responsibilities being shuffled around as the department is constantly reorganized, until no one really understands who's responsible for what any more, but there are enough low-level employees to blame when things go wrong. I've seen enough variations of organizational dysfunction that I no longer pretend to be able to guess what's going on behind the scenes.
whalesalad 5y ago
In my experience there are a lot of IT departments full of people who know how to click around and hack shit together but aren’t what you’d call classically trained experts. Kinda like “I’ll get my nephew to make my website”
mdoms 5y ago
This seems a bit presumptuous. Epic's Glassdoor reviews[0] don't seem to list pay or staffing as systemic issues. [0] https://www.glassdoor.co.nz/Reviews/Epic-Games-Reviews-E266904.htm https://www.glassdoor.co.nz/Reviews/Epic-Games-Reviews-E2669...
- dijit 5y ago
  I really don’t know one way or the other, though as mentioned in another thread: I’m a devops in games and it pays less but not as poorly as it does for programmers. That said; Glassdoor is a terrible metric and has been widely criticised as a source of information due to the fact that bad reviews can be removed for payment; though “officially” they don’t accept payment to delete reviews; it’s part of one of their packages to clean up a companies image. It has also been gamed by employers- but that is obviously a problem for all review sites of this kind. https://www.reddit.com/r/sysadmin/comments/8tfhxv/glassdoor_removes_bad_reviews/ https://www.reddit.com/r/sysadmin/comments/8tfhxv/glassdoor_...
- Twirrim 5y ago
  About 5 years back they were going through a spell of getting lots of engineers from AWS to join them, offering way more than Amazon was. Some of the smartest and most capable systems engineers I know headed in that direction.
bartread 5y ago
Yeah, that's why Epic Games have been transparent enough to post this incident report: not to provide some explanation to their customers, or some information that the rest of us might be able to learn something from, but so that people on HN can make entirely unfounded accusations about the state of their organisation based on (at best) weakly correlated behaviours and symptoms. Be reasonable: you know nothing about how Epic Games treats their IT staff or whether or not the team is adequately resourced. I wouldn't say certificate expiry is something that happens particularly often, but I have seen it happen, and it's been simply an oversight rather than an indication of some serious systemic issue.
- Groxx 5y ago
  The fact that a company can't deal with a scheduled-far-in-advance, highly-public-if-failed event does tell you some things about their priorities / how well they do things they need to do.
  - bartread 5y ago
    OK, fine, I'll bite: what specifically are those things it tells you that you can verifiably claim are true about Epic Games, again, specifically?
    - Groxx 5y ago
      That they apparently sometimes fail to do these things. You can't verify anything internal unless you're internal or it has already failed publicly, so you of course have to draw on patterns seen elsewhere. Critical-process failures in one area correlate heavily with failures in others. Plus, Epic has not exactly shown themselves to be producing consistent quality in anything related to their store, or many internet-connected properties. If they were, this might be more attributable to "accidents happen, it's impossible to prevent them all". It could still be an abnormality, but they're edging further towards "... maybe not though" territory. --- Edit: lets add a concrete "kinda example, kinda counter-example". Google is a tech company that is pretty good at consistently renewing its many certificates. They recently failed to do so for Google Voice: https://www.bleepingcomputer.com/news/google/recent-google-voice-outage-caused-by-expired-certificates/ https://www.bleepingcomputer.com/news/google/recent-google-v... I think there's a reasonable argument to be made that this reinforces claims that Google Voice is low priority / at higher risk of future issues due to lack of care, i.e. systemic issues, compared to other Google properties. I have no proof, but that doesn't mean it's automatically unreasonable.
nickysielicki 5y ago
Video game developers are underpaid because they have an undying love for video games and are willing to work for less than they could make elsewhere. I suspect this becomes a problem in the context hiring devops people, because whereas you can make the argument that writing game engines and working on game logic is more fun and justifies working for less, it's hard to make the argument that a devops job at Epic running game servers and websites is any more exciting than running servers and websites anywhere else. This puts epic in the situation of having to pay market rate to attract devops people, but below market rate for attracting developers, which fucks up their pay scaling completely. What ends up happening is they just don't adjust their pay scale at all, which means they're hiring cheap devops people.
- rootsudo 5y ago
  This. I interviewed for a role with a game studio and the pay was 40% lower than what I'm making. I was just curious since they approached me and I had fun with the experience and saying I didn't play their games/had no idea. The recruiter had no idea of local wages.
- dijit 5y ago
  I work as a devops in the games industry. It’s true that it’s underpaid, and by quite a bit. But it’s not as bad as the programming teams, IME devops pays more.
- joana035 5y ago
  I interviewed with epic games and got all the questions answered, though I used generic term to describe each aws product and drilled down into specifics/fundamentals of the questions, protocols, configuration gotchas, etc. Got rejected with "no experience with aws". Now seeing this I'm sure I dodged a bullet.
- Impossible 5y ago
  Epic doesn't pay below market rate. They can't offer stock because they aren't public but offer cash bonuses 2x-4x salary. I do agree with OP that (some) game developers undervalue IT. Oculus had a similar and pay rate was equal to FAANG (because it is FAANG!), so it came from culture, not pay.
psanford 5y ago
> Every place I've worked which has had issues with last minute certificate changes or expiring certificates without renewal has had a systemic problem with underpaid and understaffed IT department. That's an interesting anecdote, but its quite easy to find examples of companies with well respected, well paid engineering teams that still have an occasional certificate expire. Microsoft[0], Spotify[1], Facebook[2], Apple[3] have all had embarrassing outages due to certificates expiring. [0]: https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status https://www.theverge.com/2020/2/3/21120248/microsoft-teams-d... [1]: https://www.theverge.com/2020/8/19/21375032/spotify-down-songs-loading-issues-outage https://www.theverge.com/2020/8/19/21375032/spotify-down-son... [2]: https://www.theverge.com/2018/3/7/17092084/oculus-rift-headsets-stopped-working-expired-certificate https://www.theverge.com/2018/3/7/17092084/oculus-rift-heads... [3]: https://www.theverge.com/2015/11/12/9721108/apple-mac-app-store-bug-security-certificate https://www.theverge.com/2015/11/12/9721108/apple-mac-app-st...
- machello13 5y ago
  The Apple issue was not a case of forgetting to renew a certificate, certain 3rd party apps just weren't handling the upgrade correctly. So maybe not quite as easy to find examples after all.
- xtracto 5y ago
  Right handling certificates is one of those chores that, particularly in a startup is easy to oversee. If the average turnaround for employees is 2 years, and a certificate can be bought for a bit more than 2 years. Normally someone will put it in their calendar and leave the company before it expired, so the new employee will be welcomed by an expired certificate and a not-so-clear list of places where to place it. That´s why things like AWS certificate manager + ELB kind of things are useful, so that they are mostly auto-renewed. It is a chore that had bit most of the places where I have worked.
renewiltord 5y ago
This is so overfit. Unbelievable anyone goes along with it. I was paid north of $400k total comp when I made this error last. Easy mistake to make.
spondyl 5y ago
We had this issue a few times at the place I previously worked. At first, it wasn't clear whose responsibility it was since back in the operations day, emails would go to someone's specific address or even a mailing group, where most of the employees who were on it had left while new employees weren't added to the list since they didn't know about it. After it happened once or twice, metrics were set up to track expiring certificates (they were mostly all migrated to AWS Cert Manager I believe) while a few key ones couldn't be. As a bit of background, we also follow the Google-esque model of not having a phone number for customer support and requiring customers to submit a ticket. We do have outgoing calls but no incoming phone number. I say that because those key certificates would generate an email that said something like "Press this button and we'll call you to confirm you want to renew" so as you can imagine, my first thought was "Well, how the fuck is shit gonna work?" I think in the end we just ended up calling the certificate provider to say we don't have a phone number and then we managed to get them migrated to DNS-based validation after some time. This too wasn't a case of being underpaid but rather having a lack of knowledge. It's the sort of task that some particular person did for a long time but then left so none of us newer folks even knew where these things were provisioned from. Additionally, you don't feel like you have the authority to ie; call up some multi-national provider and be like "Hi, we own this thing but umm, I have no idea how to go about renewing it". It feels like being a teenager calling up about a first job haha. It's just one of the casualities of "high growth" businesses mixed with humans being bad at seeing cause and effect when the gap between the two is super wide. Cause being people leaving and effect being "I forgot to ask how to do X or Y" I guess I would clarify that we were following a devops model but had transitioned from a classic dev/ops split so it's quite literally a generational thing where you conceptually don't know how to go about eg; renewing a certificate on the phone because you've entered the industry in the time of dns validation via lets encrypt (and because there literally are no phones anymore in the businessa)
pan69 5y ago
I don't think this has anything to do with the treatment of IT folks. It has more to do with the time frame validity of certificates. If your certificate expires every month you will have a system or process in place to deal with that (preferably an automated one). However, if your certificate expires every two years or so, someone will set a calendar reminder, leaves the company at some point, and there is your problem.
deleted 5y ago
[deleted]

PragmaticPulp 5y ago

> Every place I've worked which has had issues with last minute certificate changes or expiring certificates without renewal has had a systemic problem with underpaid and understaffed IT department. I've seen the opposite: Organizations who spent so much on the department that everyone was getting promoted to manager and hiring someone underneath themselves to manager things. Responsibilities being shuffled around as the department is constantly reorganized, until no one really understands who's responsible for what any more, but there are enough low-level employees to blame when things go wrong. I've seen enough variations of organizational dysfunction that I no longer pretend to be able to guess what's going on behind the scenes.

whalesalad 5y ago

In my experience there are a lot of IT departments full of people who know how to click around and hack shit together but aren’t what you’d call classically trained experts. Kinda like “I’ll get my nephew to make my website”

mdoms 5y ago

This seems a bit presumptuous. Epic's Glassdoor reviews[0] don't seem to list pay or staffing as systemic issues. [0] https://www.glassdoor.co.nz/Reviews/Epic-Games-Reviews-E266904.htm https://www.glassdoor.co.nz/Reviews/Epic-Games-Reviews-E2669...

dijit 5y ago
I really don’t know one way or the other, though as mentioned in another thread: I’m a devops in games and it pays less but not as poorly as it does for programmers. That said; Glassdoor is a terrible metric and has been widely criticised as a source of information due to the fact that bad reviews can be removed for payment; though “officially” they don’t accept payment to delete reviews; it’s part of one of their packages to clean up a companies image. It has also been gamed by employers- but that is obviously a problem for all review sites of this kind. https://www.reddit.com/r/sysadmin/comments/8tfhxv/glassdoor_removes_bad_reviews/ https://www.reddit.com/r/sysadmin/comments/8tfhxv/glassdoor_...
Twirrim 5y ago
About 5 years back they were going through a spell of getting lots of engineers from AWS to join them, offering way more than Amazon was. Some of the smartest and most capable systems engineers I know headed in that direction.

bartread 5y ago

Yeah, that's why Epic Games have been transparent enough to post this incident report: not to provide some explanation to their customers, or some information that the rest of us might be able to learn something from, but so that people on HN can make entirely unfounded accusations about the state of their organisation based on (at best) weakly correlated behaviours and symptoms. Be reasonable: you know nothing about how Epic Games treats their IT staff or whether or not the team is adequately resourced. I wouldn't say certificate expiry is something that happens particularly often, but I have seen it happen, and it's been simply an oversight rather than an indication of some serious systemic issue.

Groxx 5y ago
The fact that a company can't deal with a scheduled-far-in-advance, highly-public-if-failed event does tell you some things about their priorities / how well they do things they need to do.
- bartread 5y ago
  OK, fine, I'll bite: what specifically are those things it tells you that you can verifiably claim are true about Epic Games, again, specifically?
  - Groxx 5y ago
    That they apparently sometimes fail to do these things. You can't verify anything internal unless you're internal or it has already failed publicly, so you of course have to draw on patterns seen elsewhere. Critical-process failures in one area correlate heavily with failures in others. Plus, Epic has not exactly shown themselves to be producing consistent quality in anything related to their store, or many internet-connected properties. If they were, this might be more attributable to "accidents happen, it's impossible to prevent them all". It could still be an abnormality, but they're edging further towards "... maybe not though" territory. --- Edit: lets add a concrete "kinda example, kinda counter-example". Google is a tech company that is pretty good at consistently renewing its many certificates. They recently failed to do so for Google Voice: https://www.bleepingcomputer.com/news/google/recent-google-voice-outage-caused-by-expired-certificates/ https://www.bleepingcomputer.com/news/google/recent-google-v... I think there's a reasonable argument to be made that this reinforces claims that Google Voice is low priority / at higher risk of future issues due to lack of care, i.e. systemic issues, compared to other Google properties. I have no proof, but that doesn't mean it's automatically unreasonable.

nickysielicki 5y ago

Video game developers are underpaid because they have an undying love for video games and are willing to work for less than they could make elsewhere. I suspect this becomes a problem in the context hiring devops people, because whereas you can make the argument that writing game engines and working on game logic is more fun and justifies working for less, it's hard to make the argument that a devops job at Epic running game servers and websites is any more exciting than running servers and websites anywhere else. This puts epic in the situation of having to pay market rate to attract devops people, but below market rate for attracting developers, which fucks up their pay scaling completely. What ends up happening is they just don't adjust their pay scale at all, which means they're hiring cheap devops people.

rootsudo 5y ago
This. I interviewed for a role with a game studio and the pay was 40% lower than what I'm making. I was just curious since they approached me and I had fun with the experience and saying I didn't play their games/had no idea. The recruiter had no idea of local wages.
dijit 5y ago
I work as a devops in the games industry. It’s true that it’s underpaid, and by quite a bit. But it’s not as bad as the programming teams, IME devops pays more.
joana035 5y ago
I interviewed with epic games and got all the questions answered, though I used generic term to describe each aws product and drilled down into specifics/fundamentals of the questions, protocols, configuration gotchas, etc. Got rejected with "no experience with aws". Now seeing this I'm sure I dodged a bullet.
Impossible 5y ago
Epic doesn't pay below market rate. They can't offer stock because they aren't public but offer cash bonuses 2x-4x salary. I do agree with OP that (some) game developers undervalue IT. Oculus had a similar and pay rate was equal to FAANG (because it is FAANG!), so it came from culture, not pay.

psanford 5y ago

> Every place I've worked which has had issues with last minute certificate changes or expiring certificates without renewal has had a systemic problem with underpaid and understaffed IT department. That's an interesting anecdote, but its quite easy to find examples of companies with well respected, well paid engineering teams that still have an occasional certificate expire. Microsoft[0], Spotify[1], Facebook[2], Apple[3] have all had embarrassing outages due to certificates expiring. [0]: https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status https://www.theverge.com/2020/2/3/21120248/microsoft-teams-d... [1]: https://www.theverge.com/2020/8/19/21375032/spotify-down-songs-loading-issues-outage https://www.theverge.com/2020/8/19/21375032/spotify-down-son... [2]: https://www.theverge.com/2018/3/7/17092084/oculus-rift-headsets-stopped-working-expired-certificate https://www.theverge.com/2018/3/7/17092084/oculus-rift-heads... [3]: https://www.theverge.com/2015/11/12/9721108/apple-mac-app-store-bug-security-certificate https://www.theverge.com/2015/11/12/9721108/apple-mac-app-st...

machello13 5y ago
The Apple issue was not a case of forgetting to renew a certificate, certain 3rd party apps just weren't handling the upgrade correctly. So maybe not quite as easy to find examples after all.
xtracto 5y ago
Right handling certificates is one of those chores that, particularly in a startup is easy to oversee. If the average turnaround for employees is 2 years, and a certificate can be bought for a bit more than 2 years. Normally someone will put it in their calendar and leave the company before it expired, so the new employee will be welcomed by an expired certificate and a not-so-clear list of places where to place it. That´s why things like AWS certificate manager + ELB kind of things are useful, so that they are mostly auto-renewed. It is a chore that had bit most of the places where I have worked.

renewiltord 5y ago

This is so overfit. Unbelievable anyone goes along with it. I was paid north of $400k total comp when I made this error last. Easy mistake to make.

spondyl 5y ago

We had this issue a few times at the place I previously worked. At first, it wasn't clear whose responsibility it was since back in the operations day, emails would go to someone's specific address or even a mailing group, where most of the employees who were on it had left while new employees weren't added to the list since they didn't know about it. After it happened once or twice, metrics were set up to track expiring certificates (they were mostly all migrated to AWS Cert Manager I believe) while a few key ones couldn't be. As a bit of background, we also follow the Google-esque model of not having a phone number for customer support and requiring customers to submit a ticket. We do have outgoing calls but no incoming phone number. I say that because those key certificates would generate an email that said something like "Press this button and we'll call you to confirm you want to renew" so as you can imagine, my first thought was "Well, how the fuck is shit gonna work?" I think in the end we just ended up calling the certificate provider to say we don't have a phone number and then we managed to get them migrated to DNS-based validation after some time. This too wasn't a case of being underpaid but rather having a lack of knowledge. It's the sort of task that some particular person did for a long time but then left so none of us newer folks even knew where these things were provisioned from. Additionally, you don't feel like you have the authority to ie; call up some multi-national provider and be like "Hi, we own this thing but umm, I have no idea how to go about renewing it". It feels like being a teenager calling up about a first job haha. It's just one of the casualities of "high growth" businesses mixed with humans being bad at seeing cause and effect when the gap between the two is super wide. Cause being people leaving and effect being "I forgot to ask how to do X or Y" I guess I would clarify that we were following a devops model but had transitioned from a classic dev/ops split so it's quite literally a generational thing where you conceptually don't know how to go about eg; renewing a certificate on the phone because you've entered the industry in the time of dns validation via lets encrypt (and because there literally are no phones anymore in the businessa)

pan69 5y ago

I don't think this has anything to do with the treatment of IT folks. It has more to do with the time frame validity of certificates. If your certificate expires every month you will have a system or process in place to deal with that (preferably an automated one). However, if your certificate expires every two years or so, someone will set a calendar reminder, leaves the company at some point, and there is your problem.

deleted 5y ago

[deleted]