4 ms·

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed t

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quality be damned. Or maybe they're missing talent in areas that matter - like experienced postgres DBAs managing the database rather than having the developers and/or someone who has only ever used MySQL trying to wrangle it. What they went through is what you'd expect from a first-time pet project, not a professional business out to make money. I suspect they prioritized trying to scale with enough features to catch up with the competition above all else, to the point where sustainability is an afterthought. When desperately trying to gain market share is more important than the quality of the product, this is what you can expect.

sytse 10y ago
I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/ https://about.gitlab.com/jobs/specialist/database/
- BrentOzar 10y ago
  The word "backup" doesn't appear anywhere in the job description. Might wanna revise that today.
  - justinclift 10y ago
    They'd probably be better off hiring a "backup engineer" to go through all of their systems (end to end) ensuring full backups exist and work for everything. Instead of backing up every system in isolation, have well though out backup/restore processes for all parts of their operation. Saying that as someone who's done exactly this before (professionally, for mission critical places). ;) eg: · Inventory the systems (boxes, services, etc) · Determine what each needs (package dependencies, etc) · Create scripting (etc) for consistent backups · Work out the restore processes · Make it work (can take several test/dev iterations) · Document it And also (importantly): · Have the ops staff perform the documented processes, to reveal holes in the docs, and show up parts which need simplifying
- dijit 10y ago
  They want a developer who can do database stuff, not a DBA (based on that job posting.)
  - corobo 10y ago
    They are who you're replying to > I'm the CEO of GitLab https://about.gitlab.com/ https://about.gitlab.com/ More information about me is on http://sytse.com http://sytse.com
    - dijit 10y ago
      Fair enough, but I don't see him denying it.
- briffle 10y ago
  Just as an aside, you can hire a remote DBA to help out, until you get a full time person. We have a contracted 3rd party that does secondary monitoring around the clock, and a very, very good pro that we work with on projects. In our case, we have a 'retainer' for 15 hours a month. (and a fixed cost per hour after)
  - haylem 10y ago
    The entire GitLab team is remote, as far as I understand their company model. https://about.gitlab.com/2015/04/08/the-remote-manifesto/ https://about.gitlab.com/2015/04/08/the-remote-manifesto/ But maybe you meant part-time?
    - developer2 10y ago
      Wow, that explains quite literally everything. Never, ever, trust a company whose cost-cutting measures involves opting to not pay for office space to run a legitimate business. The attempt to tout the lack of a full office as being "modern, hip, and cool" is a blatant lie from management - it's the same concept as open-office floor plans, reimagined and magnified by a thousand. They get to save hundreds of thousands (possibly even a couple of million) dollars a year; meanwhile, you get the exactly the kind of product you'd expect from a bunch of people pretending to be productive from home and working in such a disjointed fashion. It is always a HUGE red flag when a company opts to have all or a majority of their workforce working remotely. It's a cost-cutting measure, nothing more. Cutting costs equates to cutting corners, and the business - and its customers - suffer the deserved consequences. This really explains the flippant "it's 11pm and I want to go to bed" reaction in their report. The guy doesn't have an office to go to when shit hits the fan. He's sitting at home, with a bloody ssh terminal open, trying to remotely debug critical engineering problems over a slow vpn connection. Alarmingly huge red warning flags.
      - anonymous_green 10y ago
        Cost-cutting measure don't just end at the office space level, they hire cheap developers too. There's quite a big divide between salaries they pay to their team in US and remote team and they get to do it just because they are "remote".
- developer2 10y ago
  >> As a database specialist not only will you work on the database, but also the application's usage of the database. As a result a significant amount of experience with both Ruby and PostgreSQL is a strict requirement. This is a mistake. The number of DBAs who are good with postgres is very small compared to something like mysql. The truly talented pool for such a position is too small to expect them to also be a developer. The very mention of terms like "ruby" and "programming" should be removed from that job post. It's not a realistic expectation.
- developer2 10y ago
  To be a bit more specific, it is extremely unlikely (nearly impossible) that you could ever find a single individual who is both a "postgres database specialist" and a "ruby developer". Do you even know what the word "specialist" means? It means specializing in one area - like postgres - without trying to be a Jack of all trades, ie: with ruby. "DBA" is a full-time position, not an addon to a developer's duties. They are separate skillsets; you don't get a "2-for-1" special by hiring an expert developer + DBA in one person for one lowly salary. Do you realize this event would have never happened if you had hired a pure DBA? Or do you really believe you can pin the blame on your ruby developers for not being able to wrangle a production postgres database? The oblivious or intentionally cheap "the DBA must be an amazing ruby developer" expectation shows your hiring staff - or the management guiding them - has absolutely no clue what they are doing. I can just imagine the internal discussion right now; pointing the finger at the developers with no postgres experience, or downplaying the significance of this event and pretending like it was simply bad luck, and lying to yourselves about how "it will never happen again". This job posting is completely outside the realm of reason. If that job posting has been up for months or years, I can see its description being exactly why you didn't have the right talent on board to avoid this incident.

sytse 10y ago

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/ https://about.gitlab.com/jobs/specialist/database/

BrentOzar 10y ago
The word "backup" doesn't appear anywhere in the job description. Might wanna revise that today.
- justinclift 10y ago
  They'd probably be better off hiring a "backup engineer" to go through all of their systems (end to end) ensuring full backups exist and work for everything. Instead of backing up every system in isolation, have well though out backup/restore processes for all parts of their operation. Saying that as someone who's done exactly this before (professionally, for mission critical places). ;) eg: · Inventory the systems (boxes, services, etc) · Determine what each needs (package dependencies, etc) · Create scripting (etc) for consistent backups · Work out the restore processes · Make it work (can take several test/dev iterations) · Document it And also (importantly): · Have the ops staff perform the documented processes, to reveal holes in the docs, and show up parts which need simplifying
dijit 10y ago
They want a developer who can do database stuff, not a DBA (based on that job posting.)
- corobo 10y ago
  They are who you're replying to > I'm the CEO of GitLab https://about.gitlab.com/ https://about.gitlab.com/ More information about me is on http://sytse.com http://sytse.com
  - dijit 10y ago
    Fair enough, but I don't see him denying it.
briffle 10y ago
Just as an aside, you can hire a remote DBA to help out, until you get a full time person. We have a contracted 3rd party that does secondary monitoring around the clock, and a very, very good pro that we work with on projects. In our case, we have a 'retainer' for 15 hours a month. (and a fixed cost per hour after)
- haylem 10y ago
  The entire GitLab team is remote, as far as I understand their company model. https://about.gitlab.com/2015/04/08/the-remote-manifesto/ https://about.gitlab.com/2015/04/08/the-remote-manifesto/ But maybe you meant part-time?
  - developer2 10y ago
    Wow, that explains quite literally everything. Never, ever, trust a company whose cost-cutting measures involves opting to not pay for office space to run a legitimate business. The attempt to tout the lack of a full office as being "modern, hip, and cool" is a blatant lie from management - it's the same concept as open-office floor plans, reimagined and magnified by a thousand. They get to save hundreds of thousands (possibly even a couple of million) dollars a year; meanwhile, you get the exactly the kind of product you'd expect from a bunch of people pretending to be productive from home and working in such a disjointed fashion. It is always a HUGE red flag when a company opts to have all or a majority of their workforce working remotely. It's a cost-cutting measure, nothing more. Cutting costs equates to cutting corners, and the business - and its customers - suffer the deserved consequences. This really explains the flippant "it's 11pm and I want to go to bed" reaction in their report. The guy doesn't have an office to go to when shit hits the fan. He's sitting at home, with a bloody ssh terminal open, trying to remotely debug critical engineering problems over a slow vpn connection. Alarmingly huge red warning flags.
    - anonymous_green 10y ago
      Cost-cutting measure don't just end at the office space level, they hire cheap developers too. There's quite a big divide between salaries they pay to their team in US and remote team and they get to do it just because they are "remote".
developer2 10y ago
>> As a database specialist not only will you work on the database, but also the application's usage of the database. As a result a significant amount of experience with both Ruby and PostgreSQL is a strict requirement. This is a mistake. The number of DBAs who are good with postgres is very small compared to something like mysql. The truly talented pool for such a position is too small to expect them to also be a developer. The very mention of terms like "ruby" and "programming" should be removed from that job post. It's not a realistic expectation.
developer2 10y ago
To be a bit more specific, it is extremely unlikely (nearly impossible) that you could ever find a single individual who is both a "postgres database specialist" and a "ruby developer". Do you even know what the word "specialist" means? It means specializing in one area - like postgres - without trying to be a Jack of all trades, ie: with ruby. "DBA" is a full-time position, not an addon to a developer's duties. They are separate skillsets; you don't get a "2-for-1" special by hiring an expert developer + DBA in one person for one lowly salary. Do you realize this event would have never happened if you had hired a pure DBA? Or do you really believe you can pin the blame on your ruby developers for not being able to wrangle a production postgres database? The oblivious or intentionally cheap "the DBA must be an amazing ruby developer" expectation shows your hiring staff - or the management guiding them - has absolutely no clue what they are doing. I can just imagine the internal discussion right now; pointing the finger at the developers with no postgres experience, or downplaying the significance of this event and pretending like it was simply bad luck, and lying to yourselves about how "it will never happen again". This job posting is completely outside the realm of reason. If that job posting has been up for months or years, I can see its description being exactly why you didn't have the right talent on board to avoid this incident.