How to pick the best tests for automation
POSTED : October 24, 2017
BY : Jason Jobe

I was invited recently to speak about Concentrix Catalyst’s hands-free test pipeline for T-Mobile at the STARWEST software testing conference, one of the longest-running and most respected software testing and quality assurance conferences. We’ve found this conference to be the birthplace of some of the best ideas in the industry. Taking part helps us advance the industry as a whole and also bring innovative ideas back into our client work.

After showcasing the solution, I was lucky enough to connect with Paul Merrill, Principal Software Engineer in Test and Founder of Beaufort Fairmont Automated Testing Services. My discussion with Paul challenged a few remarks I made about automation and the usefulness of the testing pyramid as a standard model.

We came to the conclusion that there is no one-size-fits-all model for software development and testing. Paul shared several examples where it would make more sense to have an ice cream cone (more UI, less unit test) or diamond model (less UI, more integration, less unit). This reaffirmed something most in the industry already know – that software development is a case-by-case practice.

That’s why it’s important to keep an open mind when deciding what to automate in your development process. Get all of the facts, then use your own experience to help guide the approach to take. It’s normal to have a go-to approach, but don’t let bias cloud your judgment.

There are, however, several groups of guidelines based on testing throughout a tech stack that can help you determine the best practices for a particular project. When applied to your product, development model, organization maturity, tools and technology, they can help you determine the “right” set for you.

Fundamentals

Automation Selection: FundamentalsLevel of Testing
Item: DefinitionUnitIntegrationUI
Deterministic:
Tests that clearly pass or fail; subjective tests aren’t ideal automation candidates.
xxx
Repeated:
Tests that occur often.
xxx
Time:
Tests that take a long time to perform manually.
 xx
Precision:

 

Tests that are subject to human error and/or require high precision.

 xx
Complexity:
Tests that have many steps or complex steps to properly complete.
 xx

Priority

Automation Selection: PriorityLevel of Testing
Item: DefinitionUnitIntegrationUI
Impact:
Failure severity, data loss, etc.
xxx
Likelihood:
Higher possibility of failure, more exposure to risk, more automation importance.
xxx
Data:
Require high data combinations, high quantity.
xxx
Importance:
Tests that are critical to the business; “Key Business Scenarios.”
xxx
Fixes:
Difficult or impossible to fix vs. easily and quickly fixed.
xxx
Conditional:
Tests that simulate practices that are costly or impossible to recreate manually e.g. Perf/Load.
 xx

Return on investment (ROI)

Automation Selection: ROILevel of Testing
Item: DefinitionUnitIntegrationUI
Configurable:
Automation can be reused by changing data, no code change.
xxx
Stability:
Code, environment stability.
xxx
Test time:
Automation development vs. manual execution time, project time remaining.
xxx
Maintenance:
Cost of automation maintenance.
xxx
Dependencies:
Amount of components that need to work together in a specific way.
 xx

These aren’t checklists that can be applied to every situation, and there is no single right answer, but these lists can serve as a starting point for your automation process. Realizing the full potential of automation can have substantial benefits for your organization, and having the right partner to lead the way can help you achieve integration, efficiency and effectiveness faster.

Learn more about Concentrix Catalyst’s Automation and Operations service line.

Tags: , , , , ,