Abstract
Large IT transformation programs typically have multiple levels of testing as part of their best practice quality control gating processes performed for every release. In a build to deploy delivery phases, there are four functionality centric testing such as – Unit testing, E2E Integration testing (aka SIT or System Integration Testing), User Acceptance testing, Business Verification Testing.
While this process ensures the quality of the IT systems being deployed, there are times when the organization must look at alternative testing approaches based on the circumstances that arise around time to market, testing efforts/budget, and many such challenges such as – High priority Requirements that come in as late entrants into a release cycle, technical enhancements, to name a few.
This blog talks about one such method called “Combined Acceptance Testing” employed by certain clients in their IT delivery programs, to enable quick deployment of business use-cases/epics/features without affecting the quality.
Testing in Large Transformation projects
The development team performs Unit test for all enhancements in a release and hands over to SIT. Entry and Exit gating criteria are defined, agreed, and enforced for every phase of testing. The criteria include items like % Functionality coverage, % Defects fixed, Number of open High/Critical defects, to name a few.
Software Development Lifecyle in IT projects:
As seen in the flow, the testing phases are executed in a sequential manner. This is necessary for efficient testing, avoid rework and ensure the quality of the software. A certain degree of overlap is acceptable in some organizations to achieve delivery deadlines; however, it is kept to the minimal.
End to end Integration test is performed by the transformation team (typically within IT department). When the defined criteria are met, it is then handed over to UAT, which is performed by Business. In general, there is considerable overlap in test cases performed between SIT and UAT phase.
Above depiction is solely focused on business functionality delivered from Build to Production, and does not cover other DEVOPS centric testing such as Env setup smoke test, sanity test after deployments, etc. these are technical testing to ensure the hygiene of release code baselines. This blog does not touch upon technically intended testing phases.
Combined Acceptance Testing
Combined Acceptance Testing (CAT) is an alternate to performing SIT and UAT sequentially. It is not a replacement to these test phases, instead CAT co-exists with SIT & UAT. Organization cherry-picks the EPICs/features that can go through CAT, thus bypassing the SIT & UAT only for those features.
Combined Acceptance Testing addresses 2 types of business needs.
- Larger Releases to deploy multiple sprints.
- High priority, late entry requirements
At times, business needs to push through enhancements in a fast-track testing route. For e.g., an urgent regulatory requirement to Mobile service providers that a customer can have only up to 5 prepaid cards. Such requirements must be accommodated in the middle of a release when the testing has already started.
Typically, every release has a few technical features included in it that doesn’t really require both SIT and UAT testing to be done. E.g., Personally Identifiable Information of customers (PII) must be encrypted at rest.
Combined acceptance testing addresses such needs. It skips SIT and UAT phase for such candidates, while putting such requirements through a CAT phase, thus reducing the time & effort needed to test this.
Low risk enhancements (less functional impacts) can be candidates for CAT. During test planning for a release, candidates for CAT are identified. In addition, late entrant enhancements are also discussed and agreed for CAT, if they fit the criteria. Enhancements with considerable functional impacts are not recommended for CAT, as they pose a higher risk of escaped defects.
While CAT is performed as a collaborative effort between IT and Business, it is typically owned by business team and is run in the UAT environment. Organizations may also choose to have the IT team own the CAT phase and use the SIT environment for running CAT test cases; If done so, a gating process must be defined & agreed with Business to accept the risk of bypassing UAT for these requirements.
Application and Benefits of CAT
Below enhancements/changes are ideal candidates for CAT.
- Technical enhancements that require very less functional testing in bigger releases
- Late entrant low risk requirements with a small number of functional test cases. The size depends on the type of business and transformation project.
It is important to control the number of requirements identified for CAT. Allowing more enhancements to CAT increases the risk of escaped defects, leading to customer dissatisfaction.
Having a Combined Acceptance Testing phase provides below benefits to the project:
- Gives leeway to push through urgent but low risk requirements into a release, late in the test cycle.
- Avoids spikes in testing team size (and consequently the development team’s effort in supporting the testing), by optimizing the testing effort spent across Integration Test & UAT, especially for big releases.
- For organizations that run multiple releases in parallel, CAT helps in optimizing the testing effort (and the development team’s effort in supporting the testing) across the releases.
Important Points to consider when using CAT
Depending on the type of projects, it is recommended to agree on a ceiling of test cases identified for CAT phase (E.g., 10-20% of UAT test cases). This avoids business to get carried away and push large number of requirements to CAT, which will lead to bad quality software.
Establish an entry criterion for allowing only low risk requirements to CAT. Any requirement that has bigger number of functional test cases are recommended to go through the usual testing cycle (not CAT). Similarly, define a deadline for new requirements to be accepted into CAT phase, and publish it in advance to the stakeholders for each release.
Setup CAT phase in systems (like JIRA) that is used to create & track defects, test cases. This will help measuring the effectiveness of testing.
Summary
Combined Acceptance Testing is not a replacement to SIT & UAT phase, however it is an alternative to the usual testing approach that suits some circumstances. Implementing the CAT testing approach in large transformation projects helps handle late entrant high priority requirements. It also gives a cushion to the testing teams to accept sudden spike of testing requirements in a large release, without having to ramp-up the team size for that release.
In order to realize the benefits of CAT, it is important that organizations define clear criteria for what requirements are suitable for CAT, how much of testing can be covered in CAT instead of the usual SIT+UAT and enforce these regulations religiously.
