How Do I Build A Culture of Quality Excellence?

How Do You Build A Culture of Quality Excellence?

Greg Sypolt, VP of Quality Assurance at EverFi, led an hour-long webinar for Applitools titled “Building A Culture of Quality Excellence – Understanding the Quality Maturity Rubric.” Let’s review his key points and his core slides.

About Greg Sypolt

Greg has been writing blogs about his experiences as a quality advocate inside his company. He wants to learn how to make testing more efficient and share what he learns with others. He loves to geek out about testing and technology, but he’s also a sports fan, he loves doing DIY projects, and he’s a husband and father of three.

Overview

Greg’s webinar focuses on you – someone who is looking to improve the quality culture at your organization. No matter where you are on the journey, from just starting out to making significant progress, Greg has useful thoughts and ideas for you.  His webinar breaks into several key pieces:

• Vision
• Journey
• Three Pillars of App Quality Excellence
• How To Measure The Quality Maturity Rubric
• Building A Quality Platform
• Reaping The Savings

Vision

Greg’s key message starts with a key idea: Quality is a journey, not a destination. Your tools and software constantly evolve, and your processes and procedures evolve as well.

How do you build your culture of quality excellence? To start, Greg says, envision an organization where each member has the mindset that he, she, or they own quality. This vision differs drastically from the legacy idea of a “QA team”. By getting on that path towards broad quality ownership, Greg says, each team member can focus on becoming the voice of quality for the organization.

At the same time, Greg says, understand your long-term goals. You want to build an organization where quality aligns with your organization’s other goals. You want to provide both speed and efficiency, in addition to reliability. Your organization likely needs a higher level of collaboration. And, by speed, he means focusing on the speed of testing – making sure that your actual tests run quickly. Quality should not be your bottleneck.

Quality Journey Objectives

Next, Greg shows the following slide:

Greg describes each of these steps in turn.

• Everyone understands how we make and test things. In this step, you have a good understanding that design requirements involve validation requirements. You know that you want unit tests, API tests, integration tests, and UI tests. You want to handle functional cases and failure cases. These cases need to be part of the design.
• Building a quality mindset. Here, you recognize the importance of quality in every aspect of your process to produce a quality product.
• Understanding our current testing coverage. Provide transparency to your team about what you cover and what you do not. Prioritize areas for test development.
• Reduction of manual testing, advocating for automating the right things. Slow, inconsistent and inaccurate manual tests impede your testing speed. But prioritize. Know which tests benefit from automation.
• Build a balanced testing portfolio. Understand how well you cover behavior across unit, API, integration, UI, performance, and security tests. Know your deficiencies and address them.
• Shift-left mindset by embracing quality in every stage of the CICD pipeline. As you build, test. Ensure that your pipeline gates work well to check each code build against expected behavior.
• Transitioning to exploratory charters and fully automated testing. This step gets you to automating the right things and beginning to expand your thinking to uncovered cases.
• Building in quality velocity for developers. Now you deliver ways to help your development team create with a quality mindset by designing with test, testability, and validation results in mind.
• Quality visibility and accountability. Create the tools and the feedback loop to show discovered defects and their origin – not as a path to punishment but to aid improvement.
• Remove the traditional QA silo mentality. As part of all the other steps, quality becomes a team metric, rather than the responsibility of a subgroup tasked with assurance.
• Not a destination. The journey continues – improvements continue along your path.

Greg’s Journey As An Example

Greg shows some data from his experience at his previous company. Greg had worked there for five years.  When he started, the company had no test automation. Each release took weeks to accomplish.

At the end of his journey, the company:

• Ran 4000 or more builds per week
• Regularly generated 600 Selenium, 55 Visual, and 75 Integration automated tests
• Ran an average of 5,000,000 or more tests per month with an average 99.2% pass rate
• Authored tests in less than 15 minutes and ran all their tests in less than 5 minutes.

By creating all this test infrastructure to go along with the software development builds, Greg’s team created the infrastructure to validate application behavior through test automation. They also built the tools to automate test generation. Finally, they created a discipline to build tests that would run quickly with comprehensive coverage, so testing would allow those 4000+ weekly builds.

Clearly, the process before and after required an evolution at the company. He is working with his team at Everfi with the goal of achieving a comparable outcome.

The Three Pillars of Application Quality Excellence

Next, Greg speaks about the core pillars of building a culture of quality.

First, Greg describes “Quality over quantity.” The first step here is to test the right things, but not everything. Every line of test code you write becomes test code you need to maintain. New features get added, and your test code will likely have to change. You need your developers to learn to think from an end-user’s perspective – how a user will use the application? Analytic tools can help your team understand the paths that users take through  your application to know what’s important. That data can help you get a robust test infrastructure that targets how your users use your products.

Quality over quantity also focuses your team on transparency for quality. If a developer makes a change and kicks off a build, the build result provides immediate feedback to the developer and the team if the added feature causes test failures. Hold everyone accountable for the quality of their code.

Second, Greg describes “Work smarter not harder.” At Greg’s previous company, they had a dedicated developer solutions team providing self-service solutions for testing. That team made it easy for developers to build the app and tests at the same time. They also built a central repository for metrics, so testing results across behavior, visual, and integration tests could be viewed and analyzed in one place. They worked on tools to author tests more quickly and more effectively. The result made it easier to create the right tests.

They also looked at ways to include more than just developers in helping to both understand the tests and to help create tests.

Third, Greg explains what he means by “Set the right expectations.” He makes it clear that you need to set the right expectations with the development team. When developers begin to develop tests, they need to understand what you expect from them – both from a coverage as well as velocity perspective. You need to make sure that everyone understands who can ensure quality and performance of the product – and it’s the people who write the code who have the greatest influence on the quality of that code.

Greg also makes it clear that the you need a clear plan to move to a more mature quality structure. Everyone needs to know what is expected, what will be measured, and what outcomes the team wants to achieve.

Measuring Quality Maturity Rubric

At this point, Greg shares his rubric for analyzing quality maturity.

He mentions that the metrics involve the people on your teams, the technology you use, and the processes you put in place. Your people need to be ready and organized appropriately. You require the right technology to increase your automation. And, you need to put processes in place that allow you to succeed at improving your quality maturity level.

As a preface, Greg describes these metrics and this rubric as based on his experience. He expresses an openness to update these based on the experience of others.

The Quality Maturity Rubric

His full rubric involves 23 different maturity attributes that each have four range values:

• Beginner – getting started on maturity
• Intermediate – taking steps to organize people, use technology, or apply processes
• Advanced – organizing effectively to achieve the maturity measure outcome
• Expert – having automated the process to be achieved with people used at their most efficient.

Greg goes through several attributes in detail, giving examples of beginner through expert for each.  For example:

This table shows the “Culture” attribute.

• Beginner – No shared quality ownership, siloed development and QA, losing sight of the larger quality picture.
• Intermediate – Identify and define specific levels of quality involvement for individuals on teams, enable shared responsibilities
• Advanced – Teams are finding problems together
• Expert – Machines are identifying problems

To use this table, compare your organization to the description of each value that most closely matches yours. Give yourself a “1” if you’re a beginner, “2” if you’re Intermediate, “3” if you’re advanced, and “4” if you achieved expert.

Greg has created tables for each of the attributes. For example:

For the Environment attribute, the Advanced level involves automated builds and staging, with push-button deployment.

Using the rubric, you go through each attribute and assign a numeric value based on how close you are to one of the levels.

As you go through the analysis, you can also use the table to set your goals for improvement over time, as you look to increase the quality maturity of your organization.

Greg posted his grading rubric online – you can find it at:

http://bit.ly/qmc-grading-rubric

Once you use the grading rubric, you can start to figure out the next two steps:

• Create an improvement map. You can’t move everything all at once. Focus on the key attributes that matter. Greg points out that culture allows you to move everything else. Figure out where you can go with culture maturity.
• Move to implementation. Once you know what you want to do, move forward with appropriate steps. Do you have the right people? Do you have process changes you need to deploy? New technology? Move forward in steps appropriate to your organization.

Quality Roles

Now that you have people, processes and technology thought through, along with your approach to maturity, it’s time to think about your people and their skills.

Greg presents a map of QA role clarity that helps you think about your existing technical skills among your team. Greg created this table based on his experiences in quality engineering.

1. Level 1 is a QA specialist, who focuses on manual and exploratory test
2. The second level is an automation engineer, who has the Level 1 skills plus the ability to develop automation test scripts.
3. At Level 3 you find an Industry SDET (software development engineer in test) who does the work of the first two levels, plus can write code and develop test algorithms.
4. Level 4 defines the need of EverFi – an EverFi SDET. In addition to the first three levels, an EverFi SDET must be proficient with DevOps. Greg hopes to get his team to this level in the next 12 to 18 months. A DevOps SDET can help make the development team proficient integrating test with development.
5. The top level, Level 5 – the Next Gen SDET, incorporates the prior four levels. On top of that, a Next Gen SDET brings proficiency in security, data science, and machine learning. This level is more aspirational. Greg expects that, over time, more quality engineers will obtain these skills.

Greg sees this table as another rubric you can use to evaluate your people. You can evaluate where you are today. And, you can start to think about skills you want to add to your team. The people on your team will help you execute your vision, so you need to know where you are and where you want to go.

You can look to hire people with these skills. You likely can find engineers with Level 3 skills of SDET. More than likely, though, you, like Greg, will be building these skills among your team over time.

Building A Quality Platform

Once you have thought through your quality maturity, reviewed your existing processes and technologies and begun to evaluate your team, Greg wants you to think about your current and future states as a “quality platform”.

He reminds you that your quality platform serves key outcomes:

• Building a culture that embraces quality at every stage from intake, discovery, execution and release
• Enabling and driving continuous improvement and adoption of quality practices
• Giving teams the ability to lead with a sense of purpose, openness and trust.

The combination of people, processes, and technologies that make up your quality platform can help you deliver major quality improvements.

At the core of your quality platform is the Developer Solutions team – working to create quality solutions among the software development team.

Next, you need data insights that help create visibility across the organization.

Third you develop turnkey solutions that simplify the deployment of key functions or processes. For example you can easily deploy Selenium or Cypress test automation through a set of well-defined code structures, and. You can easily build new tests and structures in your code. For example, at EverFi, Greg has deployed Applitools to easily add visual validation to tests – simplifying overall test development.

Fourth – your platform team serve as quality ambassadors. They represent quality practices and endorse effective change within the organization.

Fifth – you focus on functional values of the platform, making testing better and easier for everyone.

Lastly, you focus on non-functional behaviors, like monitoring the critical paths inside your application. You make sure to understand the critical paths and ensure they work correctly.

Reaping The Savings

Finally, Greg gets to the bottom line.

How does the quality platform deliver savings to your organization? Greg shows this handy table.

Each element of the quality platform contributes savings.

Greg gives the example of Data Insights providing cost savings because they help you know where you can add tests as well as providing data that can be consumed by the team. Developer Solutions helps you reduce the time to deploy pre-existing or new solutions. Functional tests can help improve your quality by ensuring you run tests on every pull request and get instant feedback, instead of waiting for savings. Or, a turnkey session with an external vendor can help you get up to speed quickly with the vendor’s technology.

Building Your Own Quality Platform

As Greg points out, moving to your own quality platform results in a journey of constant improvement. Your begin with your current state, envision a future state, and move to deliver that future state through people, technology, and process improvements. Each step along the way you end up with measurable savings.  

Here’s to your journey!

For More Information

• Measure Your Test Automation Maturity
• How To Ace High-Performance Test for CI/CD
• The Impact of Visual AI on Test Automation Report
• How Do You Catch More Bugs In Your End-To-End Tests?
• Sign up for a free Applitools Account
• Request an Applitools demo