NOTE： The English version of Agile Testing Manifesto can be found after this Chinese version.
Agile Testing Manifesto
Full Lifecycle Testing OVER Isolated Testing Phase
Team Shared Responsibility OVER Testers Ensure Quality
Continuous Targeted Automation OVER Widespread Regression Testing
Quality Built-in OVER Defect Detection
The Agile Testing Manifesto expresses our beliefs and values about agile testing, including four dimensions: process, teamwork, automation, and core values.
As the software release cycle was long, the frequency of requirement changes was low, and the barriers to cooperation between different roles were high, the traditional development model adopted the isolated testing phase, the testers independently checked the quality, full regression automation and quality detection, which are beneficial. However, with agile development, agile testing puts more emphasis on the values on the left.
Full Lifecycle Testing
Agile testing advocates shifting testing to the left and right. From the early stage of the software life cycle (left side) to the production environment after the product is released, testers’ involvement and testing activities are essential.
Shifting left, the goal is to better understand and clarify requirements in order to reduce waste caused by inconsistent requirement understanding, while shifting right is to make full use of the data of the production environment in order to optimize the development and testing process so that the software can handle a variety of unexpected behavioral patterns.
Shifting left and right is not only about moving the testing activities to the endpoints on both sides, but also emphasizing the effect of each stage, that is, full life cycle testing, which is the key to ensuring high-quality software delivery from the process.
Team Shared Responsibility
With agile, the roles of the team are no longer clearly defined, the collaboration between different roles is closer, and the team is responsible for the quality together, which is the agile testing's guiding principle.
The team needs to share a unified understanding of quality objectives, and various roles participate in different stages of the agile software life cycle. It is each role's responsibility to achieve quality objectives.
Continuous Targeted Automation
Automated testing is the foundation of agile testing and a critical means of rapid feedback. Automated testing should not pursue coverage blindly, but should pursue purposeful and accurate coverage. That is to say, Automated testing must first be effective and based on business risks in order to truly achieve rapid feedback.
The automated tests must run continuously in the CI pipeline, providing feedback for each code submission, to ensure system functions will not be broken because of new code submissions. As new functions continue to iterate, it is necessary to update and add more automated tests so that new features are covered by automated tests.
Quality built-in is the core of agile testing. It is necessary to combine the entire process of testing, the team's responsibility for quality, and continuous and accurate automated testing altogether, to prevent defects in every phase of the agile software life cycle, and to integrate quality into the product being developed and built.
Ten Principles of Agile Testing
- Our goal is to deliver high-quality software as quickly as possible with the team.
- Testers engage as early as possible in the software development process, collaborating with all roles in the team to ensure a consistent understanding of the business value.
- Testers monitor quality in the production environment, collect data, then analyze the results to optimize the process of requirement analysis, development, and testing.
- Rather than having separate testing teams or departments, testers and developers are part of the same product project team.
- Testers are responsible for exploratory testing and work with developers to design, implement and maintain automated tests.
- Automated tests are executed continuously and accurately in the CI pipeline, and the impact of each new code submission on existing functions can be easily identified.
- Test data is sufficient for automated testing and is available on demand.
- Tests are documented as living documents and versioned as knowledge assets along with the product code.
- Automation tests require effective layering.
- Rather than focusing on the number of defects found, prevent them.
1. Our goal is to deliver high-quality software as quickly as possible with the team.
It is the goal of agile testing, around which all testing activities are conducted. Agile testers should change their mindset, no longer focus on finding more bugs, but think and act accordingly to help teams deliver software faster and with higher quality.
2. Testers engage as early as possible in the software development process, collaborating with all roles in the team to ensure a consistent understanding of the business value.
Shift testing to the left and articulate requirements to ensure the team is doing the right thing and avoiding detours while embracing change.
3. Testers monitor quality in the production environment, collect data, then analyze the results to optimize the process of requirement analysis, development, and testing.
Due to the particularity of the production environment, shifting testing to the right cannot simply move the testing activities to the production environment, but only collect and analyze the data of production environment to optimize development, testing and business value, so that the production environment and the SDLC forms a virtuous circle.
4. Rather than having separate testing teams or departments, testers and developers are part of the same product project team.
Testers are part of the development team, tearing down communication barriers between roles, and testers work closely with developers.
5. Testers are responsible for exploratory testing and work with developers to design, implement and maintain automated tests.
Automated test is not just the responsibility of testers. Unit tests and API tests are mainly the responsibility of developers, and UI automated tests also requires developers to participate in design, implementation and maintenance, which is a more efficient way to free testers to do something more valuable, i.e. exploratory testing.
6. Automated tests are executed continuously and accurately in the CI pipeline, and the impact of each new code submission on existing functions can be easily identified.
In addition to pursuing complete coverage, automated tests should also run efficiently and provide feedback quickly. They need to be run in the CI pipeline and can be executed precisely on demand.
7. Test data is sufficient for automated testing and is available on demand.
Automated tests should contain a complete data set, including covering both happy paths and unhappy paths, meeting the execution requirements of different environments or platforms, etc.
Similarly, it should include data creation and management functions so as to obtain relevant test data.
8. Tests are documented as living documents and versioned as knowledge assets along with the product code.
Requirements documentation should be tied to automated tests, become executable living documents, and be versioned with product code as a knowledge asset.
9. Automation tests require effective layering.
Automated tests should be effectively layered according to different test objects.
Low-level tests have lower cost, higher execution efficiency, and more accurate problem location, but it cannot be well related to the business.
High-level tests are close to the business and can reflect the business value, but the execution speed and stability are poor, and it is difficult to locate the problem if a test fails.
It is necessary to plan a reasonable proportion of each layer of tests according to the specific conditions of the project such as system requirements and technical architecture, and cannot blindly pursue high coverage.
10. Rather than focusing on the number of defects found, prevent them.
The core value of agile testing is quality built-in, which is essentially defect prevention.Team should focus on preventing defects and improving the built-in quality of software.
Focusing only on the number of defects, or even making it a KPI, goes against this core value.
Tracking defects trends and analyzing the root causes are effective methods that can help prevent defects and should be recommended.