Over the past several years, we’ve heard a lot about “shift-left” and “shift-right” testing. We’ve seen the benefits of having testing specialists involved in testing-related activities on both sides of the “DevOps loop.” Inspired by Dan Ashby’s Continuous Testing graphic, Janet Gregory and I came up with our own visualization of a continuous testing loop (figure 1).
Testing activities happen throughout the infinite cycle of software development. Many testers now find it natural to test feature ideas during feature planning discussions. More and more teams guide development with business- and technology-facing tests, using practices like test-driven development and behavior-driven development. And, more teams embrace the idea of working together – including testers – on testing activities that happen on the right side of the loop.
The whole team approach to quality is at the heart of DevOps. We don’t throw release candidate artifacts over the wall to an operations team. We work together with SREs (site reliability engineers) and platform engineers to take care of our production environment and help prevent customer pain. Many testing specialists hesitate to get involved on the Ops side of DevOps. They aren’t familiar with the tools used for monitoring, alerting and observability. They may never have worked with operations specialists. And who wants to get paged in the middle of the night?!
I was fortunate to work with system administrators doing operations activities for much of my career. I enjoyed configuring continuous integration, and learning how to automate deployments to test environments. I discovered the value of learning how our product works by examining log data. As platforms progressed to virtual machines and then to the cloud, infrastructure as code became a thing, and new technology enabled gathering and analyzing huge amounts of data about our systems, I got both more interested and intimidated! What really caught my attention was the advent of observability. Capturing enough events so that we can ask questions about our production system that we didn’t anticipate having to ask makes so much sense to me. We testers are great at asking questions!
As a testing specialist, how can you get involved with DevOps? How can you learn the skills, how can you add value with the skills you already have? Here are a few tips that have helped me.
Build Relationships
When I first read Katrina Clokie’s excellent book, A Practical Guide to Testing in DevOps, I was struck by how much of the book she devotes to the importance of building bridges not only within your team but to other teams in the company. Collaboration is the secret sauce. I’ve worked to get over my shyness to reach out to people who can help me as well as my team.
Here’s a recent example. I started a new job this year and found that people on other teams were quite willing to schedule regular 1:1 meetings with me. I took advantage! This paid off right away. We needed to migrate the deployment pipeline for one of our products to a new infrastructure. I had just learned about this new infrastructure from one of the platform team managers in our 1:1. I scheduled another meeting with him to ask about the potential risks and learn where we should focus our testing. This helped me put together an effective test strategy. I worked with a developer teammate and an SRE (aka platform engineer) embedded in our team to do the testing and we confidently completed the migration.
As part of this first project, I joined a weekly “DevOps refinement” meeting with developers from my team, our embedded SRE and others involved with platform infrastructure. The monitoring and alerting tools used by our organization were all new to me. Getting to know the people who could help me learn about them was a huge help since we are starting to implement some observability in our product. This is a huge area of interest for me, and building these relationships has given me a step up in learning about it. Which leads me to…
Offer to Help
If you hear of a DevOps-related initiative such as a migration to a new infrastructure, or creating dashboards to monitor a new feature, put your hand up to help! On my current team, we now write user stories to create monitoring and observability dashboards as well as alerts so that we can watch new features in production. If the data looks bad, we can turn the feature flag off while we investigate. I make sure to get involved with those stories, so I can learn the tools we use for that work.
Accepting help seems obvious, but it could escape your radar, so be on the alert for it. A platform engineer specializing in one of our tools used to have a weekly “office hour” to help people learn it but I never seemed to make time for that. Everyone else did the same, so he quit offering it. Recently, he mentioned that he’d be happy to have a weekly session with a couple people on my team to build dashboards that would give us observability into some crucial parts of our product. Well, heck, yes! I scheduled 30 minutes with him, our embedded SRE, and one of my developer teammates, to meet weekly. When we need more time, we schedule it.
Help Establish a Practice
DevOps and observability are still new to most software organizations. If you are a testing practitioner, you may think, “I don’t know much about that even though I am interested, and I cannot help build those practices across that organization.” If you think that, you may be wrong. Small nudges can have big impacts.
At an earlier job, our quality organization was under the same leadership as monitoring and observability, and I had an opportunity to help build an observability practice. It was a much larger engineering organization than I was used to, with about 40 teams. This was at the start of the pandemic, so it was all remote. I started researching what was happening on teams across the organization that related to monitoring and observability. I put my findings on a big mind map, and then I met with individuals on different teams to go over the mind map. Two good things happened – they expressed surprise at learning what other teams were doing, and, and I learned what their team was doing as they explained it to me.
I wondered what could I do with that information? I started a page on our engineering organization wiki with a table of what all the teams were doing related to observability. I captured who was leading the effort, what tools were they using, and what were their results. Some teams did proof of concepts with OpenTelemetry and Honeycomb. Other teams were working in ElasticSearch and Kibana. Still others were using Snowflake.
I also started a Slack channel dedicated to observability practices, and socialized the wiki page. Individuals on different teams took the initiative to update the wiki page with their observability-related activities. The teams started talking to each other. A teammate and I started holding observability practice sessions where people could share their experiences. When other teams wanted to try to build in observability, they had access to people with experience who could help them get started.
Is it a tester’s job to start a community of practice around DevOps activities? Well, why not? Ask questions, build relationships, bring the right people together – these are our tester super-powers. We need those relationships because we know we cannot test everything prior to release. We know that in these days of complex systems in the cloud, our test environments will not look like production. We know that we cannot know what our customers will do – unless we start looking at the production use data.
We testers have work to do on both sides of that DevOps loop. By engaging in the entire development cycle, we can help to delight our customers.
