5 ms·

Gitlab's 2021 Survey uncovers a new DevOps maturity model

supernovae 5y ago
Still looking for a home for AI/ML and they're saying its part of Testing. Not sure I buy it...
john_cogs 5y ago
You can download the full report here: https://about.gitlab.com/developer-survey/ https://about.gitlab.com/developer-survey/
ciguy 5y ago
Interesting timing given that their CI/CD pipelines are currently down and have been for hours.
- dnsmichi 5y ago
  Good timing, the pipelines seem to recover in this moment: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4473#note_567178755 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4...
- deleted 5y ago
  [deleted]
- mdaniel 5y ago
  > Code change related to creating partial indexes targeted to improve this query: gitlab-org/gitlab!60942 (merged) This is a band-aid solution until we break apart pending/running job information into a table separate from ci_builds. Damn, it seems a few of their (outages|degradation) centers around poor index management. I would think PG has better tooling to support surfacing that silliness, but I freely admit I'm not a DBA
candiddevmike 5y ago
The number one problem reported (testing) is something without an easy fix IMO. Testing is almost as custom as the code you write. There is no app you can buy or service you can use that will supplement writing tests or manual testing. Maybe firing the QA team "because DevOps" wasn't the right solution?
- inopinatus 5y ago
  Aye, anyone who did that missed the point, I think. There’s no more pivotal role in devops than QA, I’ve seen many takes on devops and the best outcome (so far) was when QA own the CI/CD pipeline and their job isn’t performing tests, but writing automated ones, and directly pairing with developers, BAs, and SREs. This tends to make QA a gateway job to dev, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and the reframing helps flatten the pecking order. In the long run there’s a virtuous cycle to accessing the talent pool this way.
  - boleary-gl 5y ago
    I completely agree - it's why I love the title "Software Engineer in Test". QA should be a critical engineering role that supplements the team - not a place that people see as a stepping stone for later roles. There should be the same career progression possible in your QA department as any other engineering department.
- perlgeek 5y ago
  Testing requires a mindset. It always takes longer to write tests than not -- at least in the short run. Some developers, at least, seem to get it now. Even if they don't write enough tests, they have some awareness that they should be writing tests. Ops? There might be exceptions, but most ops folks don't even think they should be writing tests. And yet, many ops folks do more automation than before, write more code or complex config. I agree there is no easy fix. Changing minds is always harder than finding a shiny tool.
throwaway823882 5y ago
Maturity models are just theoretical constructs designed to tell you what you aren't doing right. They aren't a roadmap, so their utility is about as useful as an academic research paper. Don't target what your organization should do around a maturity model.
- blacktriangle 5y ago
  And at worst trying to adhere to some maturity model ends up causing disasters. The Richardson Maturity Model for REST is just straight-up wrong. REST == HATEOS, it's not fuzzy logic, its boolean. The RMM is how a bunch of companies got away with half-assed glorified RPC APIs that they marketed as REST.
- numbsafari 5y ago
  I was really into Maturity Models when I was younger. They're easy to understand/communicate. But, as you say, they suck when it comes to actually achieving anything. The priorities and context are completely absent. It's funny to see anyone pitch maturity models for DevOps after the folks from DORA crapped all over them in their various works. Their idea of "capability models" is a lot more helpful and maps much more appropriately to outcomes based roadmaps.
carlsborg 5y ago
> 75% of teams are either using AI/ML or bots for test/code review, or they’re planning to – up 41% from 2020. What are some examples where you are using AI/ML for test/CR?
- monkpit 5y ago
  I was hoping for that information as well... would love examples of the tools referenced in the article, and whether they seem to work or not!
  - _joel 5y ago
    https://www.perforce.com/blog/vcs/ai-code-review-evolution https://www.perforce.com/blog/vcs/ai-code-review-evolution
  - john_cogs 5y ago
    We didn't capture that information but I've asked the person who runs the survey to consider including a question on this in next year's survey.
- NikolaeVarius 5y ago
  Combining AI/ML and boys into a single number seems... dishonest.
  - john_cogs 5y ago
    In the report, you'll find detail on the breakdown of the 75% number that is mentioned in the blog post: - 25% of respondents use bots to test their code - 16% use AI/ML to review code before a human sees it - 34% are exploring the idea of AI/ML
    - NikolaeVarius 5y ago
      I know, I read it. The fact of the matter is that the 75% number is the number at the top of the article, which is all most people will read, and tries to present a certain narrative about AI/ML for code.
numbsafari 5y ago
Where's the "maturity model"? I'm not seeing them propose any kind of "maturity model", just survey results. It's also very annoying that as I scroll through the survey report, I get a modal popup that cannot be cleared that tells me to fill out a non-existent form. This doesn't look like it was ready for release.
- sslalready 5y ago
  Use the download button in the top right of the pdf viewer thingy. That’ll render the actual pdf without html and annoying modals.
- boleary-gl 5y ago
  GitLab employee here...that's an issue with our learn.gitlab.com platform not the survey itself. I've filled an issue with our vendor to try and figure out what's going on with the modal.
  - boleary-gl 5y ago
    FWIW I don't get the popup in Chrome, only in Firefox. Confused face...
- whalesalad 5y ago
  Don’t ya love inbound marketing?

Still looking for a home for AI/ML and they're saying its part of Testing. Not sure I buy it...

You can download the full report here: https://about.gitlab.com/developer-survey/ https://about.gitlab.com/developer-survey/

ciguy 5y ago

Interesting timing given that their CI/CD pipelines are currently down and have been for hours.

dnsmichi 5y ago
Good timing, the pipelines seem to recover in this moment: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4473#note_567178755 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4...
deleted 5y ago
[deleted]
mdaniel 5y ago
> Code change related to creating partial indexes targeted to improve this query: gitlab-org/gitlab!60942 (merged) This is a band-aid solution until we break apart pending/running job information into a table separate from ci_builds. Damn, it seems a few of their (outages|degradation) centers around poor index management. I would think PG has better tooling to support surfacing that silliness, but I freely admit I'm not a DBA

candiddevmike 5y ago

The number one problem reported (testing) is something without an easy fix IMO. Testing is almost as custom as the code you write. There is no app you can buy or service you can use that will supplement writing tests or manual testing. Maybe firing the QA team "because DevOps" wasn't the right solution?

inopinatus 5y ago
Aye, anyone who did that missed the point, I think. There’s no more pivotal role in devops than QA, I’ve seen many takes on devops and the best outcome (so far) was when QA own the CI/CD pipeline and their job isn’t performing tests, but writing automated ones, and directly pairing with developers, BAs, and SREs. This tends to make QA a gateway job to dev, SRE, or product management. In some organisations the QA function is perceived as a dogsbody role, and the reframing helps flatten the pecking order. In the long run there’s a virtuous cycle to accessing the talent pool this way.
- boleary-gl 5y ago
  I completely agree - it's why I love the title "Software Engineer in Test". QA should be a critical engineering role that supplements the team - not a place that people see as a stepping stone for later roles. There should be the same career progression possible in your QA department as any other engineering department.
perlgeek 5y ago
Testing requires a mindset. It always takes longer to write tests than not -- at least in the short run. Some developers, at least, seem to get it now. Even if they don't write enough tests, they have some awareness that they should be writing tests. Ops? There might be exceptions, but most ops folks don't even think they should be writing tests. And yet, many ops folks do more automation than before, write more code or complex config. I agree there is no easy fix. Changing minds is always harder than finding a shiny tool.

throwaway823882 5y ago

Maturity models are just theoretical constructs designed to tell you what you aren't doing right. They aren't a roadmap, so their utility is about as useful as an academic research paper. Don't target what your organization should do around a maturity model.

blacktriangle 5y ago
And at worst trying to adhere to some maturity model ends up causing disasters. The Richardson Maturity Model for REST is just straight-up wrong. REST == HATEOS, it's not fuzzy logic, its boolean. The RMM is how a bunch of companies got away with half-assed glorified RPC APIs that they marketed as REST.
numbsafari 5y ago
I was really into Maturity Models when I was younger. They're easy to understand/communicate. But, as you say, they suck when it comes to actually achieving anything. The priorities and context are completely absent. It's funny to see anyone pitch maturity models for DevOps after the folks from DORA crapped all over them in their various works. Their idea of "capability models" is a lot more helpful and maps much more appropriately to outcomes based roadmaps.

carlsborg 5y ago

> 75% of teams are either using AI/ML or bots for test/code review, or they’re planning to – up 41% from 2020. What are some examples where you are using AI/ML for test/CR?

monkpit 5y ago
I was hoping for that information as well... would love examples of the tools referenced in the article, and whether they seem to work or not!
- _joel 5y ago
  https://www.perforce.com/blog/vcs/ai-code-review-evolution https://www.perforce.com/blog/vcs/ai-code-review-evolution
- john_cogs 5y ago
  We didn't capture that information but I've asked the person who runs the survey to consider including a question on this in next year's survey.
NikolaeVarius 5y ago
Combining AI/ML and boys into a single number seems... dishonest.
- john_cogs 5y ago
  In the report, you'll find detail on the breakdown of the 75% number that is mentioned in the blog post: - 25% of respondents use bots to test their code - 16% use AI/ML to review code before a human sees it - 34% are exploring the idea of AI/ML
  - NikolaeVarius 5y ago
    I know, I read it. The fact of the matter is that the 75% number is the number at the top of the article, which is all most people will read, and tries to present a certain narrative about AI/ML for code.

numbsafari 5y ago

Where's the "maturity model"? I'm not seeing them propose any kind of "maturity model", just survey results. It's also very annoying that as I scroll through the survey report, I get a modal popup that cannot be cleared that tells me to fill out a non-existent form. This doesn't look like it was ready for release.

sslalready 5y ago
Use the download button in the top right of the pdf viewer thingy. That’ll render the actual pdf without html and annoying modals.
boleary-gl 5y ago
GitLab employee here...that's an issue with our learn.gitlab.com platform not the survey itself. I've filled an issue with our vendor to try and figure out what's going on with the modal.
- boleary-gl 5y ago
  FWIW I don't get the popup in Chrome, only in Firefox. Confused face...
whalesalad 5y ago
Don’t ya love inbound marketing?