Resourcing Your Testing Function
By Mark Schnauffer, 18th December 2020
When it comes to resourcing your project team it is essential you have the right roles identified as well as the right number of people for those roles. Not having the right roles will ultimately cause delays due to not having the right skills in place, too few or too many of any one role will likely result in bottlenecks resulting in delays and ultimately unnecessary additional cost.
In this article we will be examining the Testing or Quality Assurance function as part of a digital transformation project and which steps you can take to ensure you make the right decision for your project and budget. Essentially, you want to be certain you have the right number of resources, with the right skills, sufficient test coverage and a workable testing strategy. There are many considerations to take into account and each will be discussed to understand what impacts these may have when considering your resourcing requirements for testing.
It should be noted that project resourcing is a large topic to cover and won’t be fully covered by this single article, however, additional supporting articles will follow, if you would like to be notified of when these become available then follow us Twitter, LinkedIn or subscribe to our newsletter below. We value your feedback or share with us your experiences at firstname.lastname@example.org.
This section covers the project specifics and how these might impact testing. The project specifics are usually defined as part of a Business Case, Project Brief or Project Initiation Document (PID) depending on the method you are following and should include the desired project approach, budgets, teams, etc. all of which will have an impact on the project’s testing function.
Project approach relates to how the project will be managed and rolled out, generally this will be either Waterfall or Agile depending on what best fits the project or in some circumstances mandated by the organisation. Quite often larger digital transformation projects operate a hybrid approach taking the best ‘bits’ from both approaches, Agile for iterative development cycles and quick and often releases, whilst using Waterfall for overall governance structure and future planning for example. Choosing the right project approach or method is extremely important not just for resourcing but the overall success of your project. Depending on your chosen approach, test resources will either be aligned to a scrum team (Agile) or part of a resource pool that will be assigned to testing tasks throughout the project (Waterfall).
If your budget has already been agreed then obviously this will determine how many project resources (and for how long) your project can afford. There are several options available to reduce costs and these should be investigated further, for instance off-shore, near-shore, different resourcing options (internal, contractors), project scheduling, fixed cost or time and materials. Some organisations may also have a policy regarding offshoring which may limit the options.
Projects with a date dependent go-live (slippage is not an option e.g. legal obligation) or high business criticality (revenue impacting or service impacting) then building additional contingency and checks into your project is advisable. It is not advisable to rely on development and testing progress alone, but allow for additional testing in key areas such as testing of migrated data, integration testing, release dress rehearsals, data dry run(s) and extending your UAT duration.
Consideration should be taken when assessing the level of testing required for your digital transformation project, some projects quite rightly require more testing than others. Take projects that are revenue generating or will have a direct impact on your clients, or have legal implications, or integrate to many up/downstream systems, combine that with a high-level of coding and configurations, new hosting infrastructure and with complex business change. Any of these are likely to be warning signs that the amount of testing that you should consider for your project should increase.
Other areas to consider when it comes to solution complexity:
- Data - How can the new solution be tested with data? What type of data will be required for each test and can this be automatically created? Can automation or scripting be used? Is there special skills or knowledge required to successfully test the new solution?
- Business change - Projects that make changes to fundamental business processes should also be dealt with carefully, these changes can mean adaptations to integrations, automated and manual processes, changes to product rate cards, approval routing, etc.
The working model describes the teams and interaction between them, how requirements are created and priorities agreed as well as the daily ‘ways of working’ between teams to complete project work or Sprint. This also takes into consideration the team structure and capability as well as the number of project team members to fulfill the project tasks. Whilst the focus is on development and testing teams, other project functions need to be assessed included PMO (Project Management Office), business analysts, product owners and scrum masters (for Agile projects), and finally and most importantly internal business teams and stakeholders.
Your project team, who they are and how they are organised is probably one of the most important factors in successfully delivering your project. Whether you decide or the organisation mandates teams to be onsite, remotely working, nearshore, offshore or mixture will have a significant impact on the outcome of your project, especially if you do not have the right governance and management in place. Language barriers and communication limitations can also play a significant role in the success of a project. Another consideration is that it’s likely there will be multiple teams that perform different functions, for example analysis, development and configuration and finally testing. Whichever the working model and team structure you decide will impact the number and skill level of the testing resources required for your project.
Your digital transformation project may include compliance or legal challenges, for instance in 2018 the introduction of the General Data Protection Regulation (GDPR) for European individuals means more attention had to be given when controlling and processing personally identifiable information and includes the transfer of data outside the European Union. Obviously, other regions and countries outside of the European Union may have their compliance measures that need to be considered, failing to be compliant can result in substantial fines being imposed. Where projects involve data sensitive information more thorough testing should be considered.
In this section we will discuss the many different types and phases of testing to consider when planning your digital transformation project. Some types of testing e.g. functional and data will be essential to the success of the project whilst others may only be advisable or simply a waste of time and money. Take fully hosted platforms for example, the need to carry out scalability, stress or load testing generally isn’t something I would always advise (for smaller clients especially) when there’s been no code changes or minimal API calls and when dealing with non-critical business solutions. Whereas, data migration testing is essential for projects that require any level of data migration. Understanding the test scope (reach) and test coverage (depth) for your project will directly impact the roles and the number of resources you will need.
In-Sprint or Functional Testing
This is the on-going activity performed by testing resources, for Agile projects this type of testing is usually documented as part of each user story, for Waterfall projects this can be a specific task that is detailed on the Work Breakdown Structure (WBS). Either way, as mentioned in the introduction it’s essential to have the right number of testing resources. When considering the number for functional testing resources, one approach is to determine the ratio of testers to developers and functional consultants whilst also taking into consideration the solution complexity.
System Integration Testing (SIT)
The majority of digital transformation projects will have some level of integration with other systems, this could be calling APIs, data exports/inputs, notifications using email or SMS services, document generation, etc. the list is almost endless. Experience has told me that system integration testing should never be left to the end of the project, but ongoing as part of In-Sprint or Functional Testing, whether tests can be fully or only partially tested it’s still a worthwhile exercise. In addition, for complex or business critical projects carrying out a thorough SIT phase is advisable.
User Acceptance Testing (UAT)
UAT is quite often the last major milestone and stakeholder approval before you prepare for your project release. The solution has to deliver to the requirements both functionally and non-functionally. Data needs to be accurate and should have been migrated as part of a data migration dry run and business users need to be fully prepared to carry out user acceptance testing and have the confidence to sign-off this important stage gate.
Before your business users have access to the new solution and start performing their UAT tasks it is best practice to carry out the tests beforehand. Testing resources as well as business analysts are usually best placed for this activity since they know and understand the system and end-to-end processes in most detail, however, this can sometimes be performed in part by other project team members.
This phase of the project shouldn’t increase the number of testing resources required for your project as it will use the same testing resources that performed functional testing and they will continue to support this phase. However, for larger projects you may want to engage a dedicated UAT manager whose primary responsibility is to make sure UAT is executed successfully. This also can include the following:
- Presenting the UAT kick-off to business users as many people will not be accustomed to what UAT is.
- Vetting of business users and making sure they are suitable and available.
- Managing the creation of test scripts to make sure they are accurate, easy to understand and have the correct coverage (reach and depth).
- Daily management of UAT as well as sharing regular progress of scripts completed and defects raised.
Where projects require full or partial data migration it is essential the testing and ultimately the sign-off of migrated data is thoroughly covered. A project that is expected to deliver exceptional benefits will quickly fail if the data cannot be trusted. On few occasions the question of who should carry out ‘data migration testing’ has been raised; the person writing the data migration scripts, data testing resource with the relevant migration skills or the business. The only answer to this is simple...ALL should carry out a level of testing to fully satisfy and provide the confidence needed by Senior Executives that the data is fit for purpose.
Here are some of the activities that a data testing resource can assist with:
- Review mappings and rules required for transformations
- Review dataset to be migrated
- Review the data migration code
- Create ‘test’ data to support ongoing development
- Write data migration test scripts
- Contribute towards data migration sign-off process
Testing automation for many projects is a luxury, for others it’s a necessity, when to use testing automation for your digital transformation project is out of scope of this article, however it will be covered in a follow-up article in the near future. For the purpose of this exercise, you simply need to consider whether testing automation (and to what extent) it will be used in your project.
NFR (Non-Functional Requirements)
What is a NFR? Non-Functional Requirements relates to the working operation of a system rather than the operational behaviour, for example, performance, disaster recovery, load, scalability, etc. Over the last 20 years testing NFRs has significantly decreased with Cloud technology and SAAS solutions, but NFR should not be disregarded or their importance as part of the testing scope.
The amount of regression testing (making sure existing functionality still works following latest changes) depends on a number of project factors. Your release approach for an Agile project that has frequent deployments will need more regression cycles than a project that has a single release. That being said if your single release project is being developed on a shared code base with the potential of impacting other parts of the business, regression could require more effort. Here are some areas to consider when deciding the level of regression required:
- Level of data & integration testing
- Level of regression testing, number of times
- Automation (the more automation the simpler regression should become)
- Shared code base
Nothing can prepare you better than experience when it comes to resourcing for large digital transformation projects, whether that is for the testing function or other project functions or roles. There are many areas to consider when planning the testing function for any project. This article has tried to highlight some of the main areas to be considered, however, every project and organisation is different and unfortunately there is no one size that fits all.
Planning (of course) plays a fundamental role in all projects, understanding the work, sequencing, prioritisation and fundamentally the resources that are required to deliver on time and to budget only comes with detailed planning. The increase in business confidence comes with detailed planning and making sure that all aspects of the project have been covered. This article has only covered resourcing your testing function, but there are many other areas to consider,
if you would like to be notified of when these become available then follow us Twitter, LinkedIn or subscribe to our newsletter below. We value your feedback or share with us your experiences at email@example.com.