What I learned from Mabl’s State of Testing in DevOps report 2021

Here we go again.

A few months back, I blogged about a testing report which aims to cast light on the state of testing. That blog was weirdly popular. So much so that when I saw Mabl had released a similar report, this time focussing on DevOps in relation to testing, I thought it would be remiss of me not to talk about it (love that phrase!).

So, what is the Mabl report all about? This is the third year running that the popular automation tool has surveyed those in and around the QA profession about how, or indeed if, they work with DevOps. 600 people were surveyed, and there are some great insights. You can find the detail here, curtesy of Software Testing Weekly.

Now, if I’m letting my imposter syndrome hat drive my pre-report thinking, I’d have been expecting it to paint a picture of pretty much everybody being fully DevOps. Think multiple deployments a day, 100% automation at all levels of the stack, perfect pipelines and testers (if they even exist anymore right?) scratching their heads thinking “what can we do now?”.

However, as with so many things, the reality lives down to the hype.

Some of the facts that struck me were:-

• only 11% of those surveyed classed themselves as “fully DevOps & Automated”. The majority were Aspiring, or Striding (in order words, not there yet).
• there was a 14% decline year-on-year with those who said they had adopted “CI” (continuous integration) principles. 33% said they were in the process of “Transitioning” to utilising CI.
• Just 27% of respondents said they had adopted Continuous Deployment.
• How about regular deployments? Almost half of teams still deploy new code less than once a month – so if you’re in a team where that is happening trust me you are not alone. Just 15% deploy daily or more, and this %age has actually decreased year-on-year.
• When it comes to what layer of the stack to automate, unit, UI and regression are the firm favourites – but all of those have only around 40% of teams saying they automate them.

Perhaps the key one for me was this.

One very interesting revelation from this year’s data is that most teams – regardless of where they are in the DevOps maturity – rely on a combined automated and manual testing approach.

Mabl, State of Testing in DevOps Report 2021, page 17.

We often here “manual testing is dead” type themes from those who want to emphasize the silver bullet type benefits that automation can supposedly bring. I’m with Melissa Fisher here, click this to listen to her insightful Racket on this topic. Automation is a useful tool that can help testers, but it can’t replace them, and nor should it try to. Speed should not come at the expense of quality.

Whilst the companies who are nailing DevOps are more likely to have a culture of quality, and even though they are smaller in number than we might have realized, they will undoubtedly be destination companies for testers keen to embrace future patterns of working. Customers of these companies seem to be happier too, which is a great measure of success in my book.

So DevOps does appear to be on the horizon for a lot of testers. We’re just not all there yet.

It won’t be an overnight move, and if you happen to work somewhere which hasn’t fully embraced DevOps ways of working, whether that’s due to cost, politics or a plethora of other reasons, you don’t need to consider yourself over the hill and unemployable. If you’re somewhere that does bits of DevOps, but is still trying to slowly do more, you’re not alone.

And if you think you have to only do 100% automation to survive in the world of testing, you might want to think again.

Cover image: Online illustrations by Storyset