7 ms·

> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none ar

> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?

ChuckMcM 10y ago
> Did the person who set up that S3 job simply write a > script or something and just go "yep, it's done" and > walk away? I don't know of course, but one failure mode that has to be explicitly tested for is continual monitoring that the existing backup process is still working. We had a backup process at Blekko which stopped working once when an S3 credential that appeared unrelated was removed, as I recall it was a Nagios test that detected that the next set of backups were too small and got that fixed.
- problems 10y ago
  Yeah, the only solution is to test your backups - either automatically or manually through failure. My small business uses a global network of cheap servers to provide low latency to customers, so we have encountered significant lengthy failures of disk or network once every few months, but hey, we can be damn sure our backups and deploy scripts work because we're forced to restore them all too often.
Elizzy 10y ago
To be fair the script is probably the same as what is in their repos, it probably wasn't set to report to a logging service that'd raise red flags.
overcast 10y ago
Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be conducting the interview, find her/his job title on our team page. Qualified applicants receive a short questionnaire and coding exercise from our Global Recruiters The review process for this role can take a little longer than usual but if in doubt, check in with the Global recruiter at any point. Selected candidates will be invited to schedule a 45min screening call with our Global Recruiters Next, candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Candidates will then be invited to schedule a 45 minute technical interview with a Production Engineer Candidates will be invited to schedule a third interview with our VP of Engineering Finally, candidates will have a 50 minute interview with our CEO Successful candidates will subsequently be made an offer via email
- jpgvm 10y ago
  5 interviews isn't -that- insane, it's on the high end though. But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen. So maybe the process really isn't that great.
  - omgdresser 10y ago
    I'm biased because I went through the process and was rejected, but I agree. There was more focus on asinine 'behaviorial' questions than the real reason you're talking to someone: are you (relatively) sane, and more importantly are you capable, can you do the job? Everything else is secondary, in my opinion.
  - ibejoeb 10y ago
    I wondered why they always have so many job reqs out, and why they're out for so long. It always seemed like an interesting organization to me, and I like the product, but everything about their hiring process seems to be shoeing people off. Perhaps this will be incentive to try onboarding people or at least to bring in some consultants to shore them up.
    - jpgvm 10y ago
      I think part of the problem, as it has already been touched on here is true Ops people are of a different breed to standard developers. They usually have similar coding ability (but different areas of expertise, automation and systems coding over business logic) but drastically different personalities. This is an issue during hiring when companies have minimal Ops background. They (usually) attempt to interview the candidate as they would a normal dev position. This has 2 serious problems. 1) Real Ops candidates are going to seem lacklustre compared to web devs when giving web dev coding tasks and inteview questions. 2) You probably aren't going to ask them anything actually relevant to their ability to do Ops well because you just don't know. This can usually be solved by hiring an experienced SRE manager. However those people are expensive and hard to find. The next best thing is to have a gun SRE in your personal network that you can convince to join your team and help you build it out. If you can't achieve that for whatever reason then you really just have to hire based on experience and suck up the cost of hiring an ex-Google/Facebook/Netflix/Twitter SRE that you can be reasonably confident will be good. That is still hard though because those sorts of people know there is a ton of shit shoveling to be done if you are the first real SRE on board.. you have to clean up the mess left in the wake of "devs doing ops" which is not Devops btw. Devops is what is done by SREs. "devs doing ops" is how this sort of thing happens. /rant
TimWolla 10y ago
> is no way to be running a public service with paying customers Is it even possible to pay for the hosted GitLab.com instance?
- Shelnutt2 10y ago
  Gitlab.com is free, not payed. https://about.gitlab.com/products/ https://about.gitlab.com/products/ The only for pay versions are self hosted EE and independent cloud hosting.
  - btgeekboy 10y ago
    Very well; my apologies. I've updated my comment.

ChuckMcM 10y ago

> Did the person who set up that S3 job simply write a > script or something and just go "yep, it's done" and > walk away? I don't know of course, but one failure mode that has to be explicitly tested for is continual monitoring that the existing backup process is still working. We had a backup process at Blekko which stopped working once when an S3 credential that appeared unrelated was removed, as I recall it was a Nagios test that detected that the next set of backups were too small and got that fixed.

problems 10y ago
Yeah, the only solution is to test your backups - either automatically or manually through failure. My small business uses a global network of cheap servers to provide low latency to customers, so we have encountered significant lengthy failures of disk or network once every few months, but hey, we can be damn sure our backups and deploy scripts work because we're forced to restore them all too often.

Elizzy 10y ago

To be fair the script is probably the same as what is in their repos, it probably wasn't set to report to a logging service that'd raise red flags.

overcast 10y ago

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be conducting the interview, find her/his job title on our team page. Qualified applicants receive a short questionnaire and coding exercise from our Global Recruiters The review process for this role can take a little longer than usual but if in doubt, check in with the Global recruiter at any point. Selected candidates will be invited to schedule a 45min screening call with our Global Recruiters Next, candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Candidates will then be invited to schedule a 45 minute technical interview with a Production Engineer Candidates will be invited to schedule a third interview with our VP of Engineering Finally, candidates will have a 50 minute interview with our CEO Successful candidates will subsequently be made an offer via email

jpgvm 10y ago
5 interviews isn't -that- insane, it's on the high end though. But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen. So maybe the process really isn't that great.
- omgdresser 10y ago
  I'm biased because I went through the process and was rejected, but I agree. There was more focus on asinine 'behaviorial' questions than the real reason you're talking to someone: are you (relatively) sane, and more importantly are you capable, can you do the job? Everything else is secondary, in my opinion.
- ibejoeb 10y ago
  I wondered why they always have so many job reqs out, and why they're out for so long. It always seemed like an interesting organization to me, and I like the product, but everything about their hiring process seems to be shoeing people off. Perhaps this will be incentive to try onboarding people or at least to bring in some consultants to shore them up.
  - jpgvm 10y ago
    I think part of the problem, as it has already been touched on here is true Ops people are of a different breed to standard developers. They usually have similar coding ability (but different areas of expertise, automation and systems coding over business logic) but drastically different personalities. This is an issue during hiring when companies have minimal Ops background. They (usually) attempt to interview the candidate as they would a normal dev position. This has 2 serious problems. 1) Real Ops candidates are going to seem lacklustre compared to web devs when giving web dev coding tasks and inteview questions. 2) You probably aren't going to ask them anything actually relevant to their ability to do Ops well because you just don't know. This can usually be solved by hiring an experienced SRE manager. However those people are expensive and hard to find. The next best thing is to have a gun SRE in your personal network that you can convince to join your team and help you build it out. If you can't achieve that for whatever reason then you really just have to hire based on experience and suck up the cost of hiring an ex-Google/Facebook/Netflix/Twitter SRE that you can be reasonably confident will be good. That is still hard though because those sorts of people know there is a ton of shit shoveling to be done if you are the first real SRE on board.. you have to clean up the mess left in the wake of "devs doing ops" which is not Devops btw. Devops is what is done by SREs. "devs doing ops" is how this sort of thing happens. /rant

TimWolla 10y ago

> is no way to be running a public service with paying customers Is it even possible to pay for the hosted GitLab.com instance?

Shelnutt2 10y ago
Gitlab.com is free, not payed. https://about.gitlab.com/products/ https://about.gitlab.com/products/ The only for pay versions are self hosted EE and independent cloud hosting.
- btgeekboy 10y ago
  Very well; my apologies. I've updated my comment.