Software Test Management (ISTQB) Cheat sheet

Software Test Management

Software test management in general is not a precise science. Most organisations have developed their own methods for tackling this problem.

However, there are institutions in the software testing industry that are trying to standardize some of the software testing practices. International Software Testing Qualification Board ISTQB is one of such organizations.

I am summarizing some topics related to test management from ISTQB’s foundation guide. This summary should provide a nice checklist for you and bring some method to the madness.

We test software to mitigate its risks. Risks can have impact right from the initial stages of a project. So early testing saves time and resources and the higher the risk, the higher the coverage required.

There are generally two types of risks that we focus on, project related and product related. Managing the project risk helps preserve your capability to deliver. Project related risks include: project issues, delays, inaccurate estimates, changes, staff issues, suppliers issues, and technical issues.

Product risks on the other hand, are risks to the quality of the product. It includes: failure-prone software, inadequate system architecture, happy-path solutions, defects causing individual or property harms.

Test Organization

Check 1 — Is your testing process independent enough?

Authors of code get to read their own code over and over multiple times and as time passes they are less able to see their own errors. This is where independent testing is important. It is important to involve a pair of fresh eyes and a head whose sole purpose in a job is to find defects in an application. Independent testers are likely to recognize different kinds of failures and challenge the author’s assumptions.

Of course independent testing is not the silver bullet. It comes with a higher collaboration and communication cost. It could delay feedback, developers may lose sense of responsibility because they assume someone else is taking care of that. And independent testers may lack some important contextual information. But, a balanced independence could solve a lot of issues. In a paradigm, a company with no software tests, has the lowest test independence score and a company with outsourced testing has the highest independence score.

Check 2 — What test strategy and approach do you need to employ?

There are multiple approaches to how an organization sees its applications to be tested. Following are some of the approaches:

1. Preventative: Writing the tests first and then making the solutions(applications) pass them.
2. Reactive: Develop the solution first and then test it for defects.
3. Analytical Tests: Test each feature based on the analysis of some factor. I.e. risk based.
4. Model based: test based on models. Such as based on statistical information about the failure rates of a feature, or usage model (what feature is used the most).
5. Process-compliance: Testing for compliance to international, national, or industry standards IEEE, MISRA, ISO standards.
6. Heuristic: Exploratory testing, execute a test and evaluate it.

Test Planning and Estimation

Test planning ensures that there are a list of measured tasks and milestones to track progress against. The main document out of this stage is a project test plan. The project has a high level plan for test-related activities. The details of test-level activities are documented in the test-level plans. It is developed in the beginning of the project and updated via change control throughout the life of the project.

Factors to consider in the test planning process are: risks, constraints, resources, criticality, test policy, scope of testing, test objectives, and testability etc.

The ISO 20119–3 Software Testing Standard outlines that there should be a minimum of 15 sections present in a test plan.

Check 1: What information goes into the test plan?

1. Overview: Introduction and a birds eye view of the document.
2. Unique Identification Number: Test plan is a formal document, it needs a unique identification number and proper versioning.
3. Issuing Organization: Who is the owner of this document.
4. Approval Authority: Whose approval and final review makes this an official document.
5. Change History: The whens, whys, and whats of the changes brought to this document.
6. Introduction: Introduction to the content of the document.
7. Scope: Defines the areas of coverage.
8. Reference: Lists referenced documents and software & test information etc.
9. Glossary: Defining terms, abbreviations, jargon etc.
10. Context of testing: includes: Details of the sub-processes covered by the plan, test items, test scope, assumptions and constraints, stakeholders, communication lines inside and outside the test team
11. Risk register: List of the known risks.
12. Test Strategy: includes: what test process and sub-process will be conducted, test deliverables, test design techniques, test completion criteria, metrics to be collected, test data requirements, test environment requirements, retesting and regression testing approach, suspension criteria, deviation from the organisational test strategy
13. Test Activities and Estimates: Documenting the activities to be completed and and time required to complete those activities.
14. Staffing: Details of the staffing requirement, roles and responsibilities, and training requirements.
15. Schedule: Milestones on a timeline.

Check 2: How do you develop the plan?

1. Sit with the project manager and subject matter expert to determine the scope and the risks that need to be tested.
2. Understand the delivery model (waterfall, Agile, etc.)
3. Make sure your testing activities adhere to the delivery model.
4. Design: the development of testing software (framework).
5. Development: coding.
6. Implementation: bringing the test framework into live environment.
7. Identify responsibilities and distribute them to resources.
8. Deciding what needs to be documented, which plans, how the test cases will be documented and so on. Ensuring the test documentation generates repeatable test assets.
9. Defining the management information, metrics, processes for monitoring and controlling test processes.

Check 3: What is your entry criteria and exit criteria (Define “ready” and “done” — Agile terminology).

Determine when a test activity can start. This could include test activity planning and when test design and when test execution for each level of testing is ready to start.

Ready: A test could be ready if:

1. Testable requirements, user stories, or models are available.
2. Test tools, environment, code, and data are ready for use.
3. Previous tests have completed and met their exit criteria.

Done: A test is done if:

1. All planned tests have been executed.
2. A certain level of coverage has been achieved.
3. The number of unresolved defects are within threshold.
4. All high-risk areas have been fully tested.

Check 4: What does your schedule look like?

All test cases, both manual and automated could be organized into test suites and their execution scheduled. The schedule could take into consideration prioritizations and dependencies.

Factors that could influence the test effort include: product characteristics, development process, personnel and staff, results of the tests

Check 5: How do you estimate your tests (time and effort)?

Two of the most common estimation techniques are metrics-based and expert based.

Metrics-based looks at data, i.e the number of: test conditions, test cases written, test cases executed, defects found and environment outages.

Expert-based approach is the subjective decision of an expert. They could be business experts, test process consultants, developers, etc.

Test Monitoring and Control

Check 1: Monitoring

After the test plan is complete, all activities are plotted over a timeline, it is time to closely monitor your progress against your goals. You monitor your progress based on the metrics and estimations you have already developed.

Metrics could include: progress against the plan, current quality, effectiveness of the test activities, percentage of planned work done, number of tests executed, defects found, subjective confidence of testers, milestones met, costs.

Check 2: Control

For the control part, you can tweak priorities, minor schedule re-adjustments, re-evaluating “ready” and “done” criteria, and altering the scope of test activities. Even delaying release into the production environment.

Check 3: Reporting

At this stage you can prepare a concise report showing your test metrics, tests undertaken, a test progress if the tests spread over multiple reporting periods, a test summary if tests meet their “done” criteria.

Test Progress Report:

1. Overview: Introduction and a birds eye view of the document.
2. Unique Identification Number: a unique identification number and versioning system.
3. Issuing Organization: Who is the owner of this document.
4. Approval Authority: Whose approval and final review makes this a complete document.
5. Change History: When, why, and what of the changes brought to this document.
6. Introduction: Introduction to the content of the document.
7. Scope: Defines the areas of coverage.
8. Reference: Lists referenced documents and software and test information etc.
9. Glossary: Defining terms, abbreviations, jargons etc.
10. Test Status: includes: reporting period, progress against the plan, factors blocking the progress, test measures, new and changed test risks, testing planned in the next period

Test Summary Report:

1. Overview: Introduction and a birds eye view of the document.
2. Unique Identification Number: a unique identification number and versioning system.
3. Issuing Organization: Who is the owner of this document.
4. Approval Authority: Whose approval and final review makes this a complete document.
5. Change History: When, why, and what of the changes brought to this document.
6. Introduction: Introduction to the content of the document.
7. Scope: Defines the areas of coverage.
8. Reference: Lists referenced documents and software and test information etc.
9. Glossary: Defining terms, abbreviations, jargons etc.
10. Testing performed: includes: summary of performed tests, deviations from plan — if any, completion evaluation — were all tests performed, if not why not?, factors that blocked progress, residual risks, deliverables, reusable test assets, lessons learnt

Check 3: Defect Management

If your actual results don’t match the expected results — it is a defect. There needs to be a process that tracks all the defects through closure. The purpose of tracking defects is to provide clear feedback and reproduction steps for a defect and ideas for its improvement.

ISO 29119–3 defines a test defect report. A defect report could include:

1. A unique ID
2. Title of the defect
3. Date of its report, issuing author
4. Environment specification
5. The development phase it was identified in
6. Reproduction steps
7. Expected and actual results
8. Urgency
9. State of the defect report: is it open, closed, deferred, etc.

Configuration Management

Configuration management involves managing products, facilities, and processes. If a change is needed in any of these, it has to have a proper change management pipeline. It ensures that a stable environment is maintained throughout the lifecycle of a project. If to smoothen the side-effects of any change that has to happen.