Continuous testing can significantly improve time to market, but what effects does doing so have on the rest of your organization?
When adopting a new architecture or process for software development, the organizational requirements are just as important as the technical ones. So if you want to succeed in implementing continuous testing in your organization, take into account your timing and your team, and create a solid plan.
Here's what you need to know before taking the leap into continuous testing, and best practices for when you do.
Timing considerations
To determine the optimum timing for implementing continuous testing (CT), you must consider a range of factors, but these three are the most important:
- Where you're at in your delivery pipeline process
- The infrastructure you have in place
- Your current and future business needs
The delivery pipeline process
Two key components of delivering value from your CT pipeline are to have a way to drive the execution of those tests and a way to tie the outcome of a build to the test results. While you can start creating CT-based tests in the early development stages, if you don't have a continuous integration (CI) pipeline within which to execute them, you won't get the full value.
Given that implementing a CI pipeline for your delivery process is a large effort in and of itself, make sure it is either completed first or is in the final stages of development. Jumping the gun in this area can result in a test infrastructure that isn't compatible with your resulting pipeline process.
Infrastructure: What you need vs. what you have
An often disregarded aspect of implementing CT (and most automation as well) is the infrastructure you need to support test execution. Frequently, the environments used for any automation testing is either hand-me-down hardware after an upgrade to production, scaled-back virtual hardware, or old desktop machines from the IT storage room. (Nearly every company I've been at had this scenario when I got there.)
This is usually a result of a lack of perceived value for automation.
To get the most value from CT, you must have the right infrastructure. You don't need a full clone of your production environment, but you will need to stand up all of the components that you have in production. Some of the biggest CT implementation failures I've seen resulted from a lack of infrastructure for testing, all because the organization didn't understand the need.
Your business needs
While CT is always valuable, you need to weigh that value against the cost of implementation. And to make that comparison, you must understand both the business need and the cost of failure.
In a small organization or one that's still building its user base, the business risk associated with production failures or bugs may be minimal. If you drive the majority of your revenue through your website, however, a failure could be a major problem.
The point is that you need to understand the actual business risk. That risk could be legal, financial, reputational, or something else entirely. Evaluate each risk in the grand scheme of things to determine the fallout should an issue occur.
Team structure
Once you confirm that the timing and environment are right to start implementing CT, you need to figure out who is going to be building it out.
To succeed, you need to fill a few key roles, including:
- Automation architect: Defines the framework or test harness that you use for the actual test execution. This process is typically done with the DevOps/release engineer and should be designed to fit into existing systems where possible.
- DevOps/release engineer: Works with the automation architect to wire up the test framework or test harness to the delivery pipeline.
- Test developer: Creates the actual tests that will run at execution time.
- Business/product owner: Responsible for defining the areas of risk and what the verification points should be. These artifacts can be in the form of tests cases, additional information about the user stories, or documentation on a team wiki page.
While there are multiple job functions, these roles often overlap. The point in specifying each of the roles is to help you understand what deliverables you need and how to get them created. While having a single person take on all four roles isn't advisable, having the same person as automation architect and test developer is a reasonable, and standard, approach.
Existing versus new resources
Based on the positions outlined above, you might determine that you have all of the workers (resources) needed to get the ball rolling. Before you press the start button, however, make sure that using those resources for the project will not adversely affect your existing deliverables, specifically, your current automation.
If redirecting your automation team to work on CT means automation suffers, factor that in. It may be that your existing automation isn't adding value and that pointing them in a new direction would be the best use of their time. Just make sure you do that analysis before making a decision.
Hiring a new team or other resources for anything automation-related isn't easy, and when you start looking for people with experience building out CT, it gets even harder. Sometimes, though, it is the right way to go, and it pays to find the right person for the job. The last thing you want is to try to shoehorn people into areas where they don't fit, as this rarely turns out well for anyone.
Craft an implementation plan
If you've determined that the timing is right and you've identified the right team, it's time to create your implementation plan. While every organization is different and will have unique needs and requirements, there are some general lessons and best practices to keep in mind.
Document early and often
This one sounds pretty straightforward, but you would be amazed at the number of times a team fails to document the plans it worked so hard to build.
Best practice: Document as you develop your plan, and create a road map to keep you on track.
Don't ignore the business
Because CT is all about identifying business risks, it makes sense for the business stakeholders to be heavily involved. Unfortunately, this rarely occurs. Either the business does a bunch of work up front, or the stakeholders come in at the last minute. Neither approach leads to success.
Best practice: Make sure the team building the tests works closely with the business throughout the tests' lifecycle.
Don't ignore your team
When a team fails, the reasons are usually easy to identify. Don't ignore the warning signs. If a team is struggling, talk to the team and work together to determine the cause.
Best practice: Set your team up for success by providing the resources and support it needs to do what you ask.
Don't boil the ocean
Trying to do it all at once is deadly to any effort, and implementing CT is no different. Lack of direction is what leads to a bunch of half-working tests or a framework that works only half of the time.
Best practice: Prioritize the low-hanging fruit, then move on to the more complex needs. Finalize one thing before moving to the next.
Don't forget the cost of maintenance
Perhaps the single biggest reason for failure in any automation effort is the failure to account for test maintenance. As features or business requirements change, update your tests or framework to account for them.
Best practice: Create a plan for managing changes early and understand how to resolve them. Define this process early to avoid trouble down the road.
How will continuous testing affect your organization?
Understanding the organizational impact of taking on CT is a key factor in determining whether you are ready to move forward. This step is overlooked far too often, since most of the planning tends to happen at the operational level, without the full involvement of leadership.
By following the basic guidelines above, you will greatly increase your chances of success.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
