Note: You can find the English version after the Chinese one.
【中文版 Chinese version】
1.1 价值驱动 vs. 指标驱动
1.2 鼓励创新 vs. 追责文化
"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."
-- Norm Kerth, Project Retrospectives: A Handbook for Team Review
1.3 全员负责 vs. 独立割裂
—— M. Conway（马尔文·康威）
—— Naomi Stanford，摘自《组织设计指南》（Guide to Organization Design)
【英文版 English Version】
0.1 Three levels of Systematic Thinking for Testing
Do testing personnel lack systematic thinking?
How to systematically test a newly established product?
How to build a unified testing system at the organizational level?
Many of us have come across countless articles or training courses on testing and quality, which are not lacking in testing practices or technical content, but it is difficult to build our own testing system. Based on similar doubts from many friends and my own team practice and consulting experience over the years, I will discuss the construction of systematic thinking for testing at three different levels: basic, intermediate, and advanced.
- "Building Systematic Thinking for Testing (Basics)"
- "Building Systematic Thinking for Testing (Intermediate)"
- "Building Systematic Thinking for Testing (Advanced)"
0.2 Review of the Basic Level Content
In January 2022, "Building Systematic Thinking for Testing - Intermediate" was published, which introduced the corresponding practical activities and methods for the five basic responsibilities of testing.
- Understand and clarify business requirements
- Develop strategies and design tests
- Implement and execute tests
- Defect management and analysis
- Quality feedback and risk identification
In March, "Building Systematic Thinking for Testing (Intermediate)" was published, which was about building systematic thinking for testing from a quality perspective.
- From testing to quality: Move beyond testing and consider the construction of testing systems from the perspective of quality.
- Quality assurance and quality built-in: Quality is inherent in products, and quality built-in is the fundamental guarantee for quality assurance.
- In-depth practice of quality built-in: How to implement quality built-in is the most concerned issue, and this article will introduce in-depth practices for reference.
0.3 Overview of Advanced Level Content
This article is the Advanced and will discuss the systematic thinking from building a testing system at the organizational level.
- From team to organization: Move beyond product/project teams and consider the systematic construction of testing at the organizational level.
- Organizational testing system map: Based on the "one center, four directions" map, explore the construction of organizational testing systems comprehensively.
- Organizational quality empowerment: Based on the testing system map, empower the entire organization from an organizational level perspective.
01 From Team to Organization
1.1 Why Move from Team to Organization?
In the "Basic" article, we discussed the basic responsibilities of testing, and in the "Intermediate", we discussed quality built-in, which were mainly within the scope of the team/project. However, to truly build a testing system or systematically consider and carry out quality assurance work, the team's capabilities are still limited. For example, have you encountered familiar scenarios such as:
- My organization has many KPIs, and work is driven by the KIPs, but I feel that some work does not bring value.
- I am in an independent testing department, and the performance is inconsistent/contradictory with the development team's.
- The business department is far away from us, and all development and testing work can only be carried out based on the requirement specification document.
- The organization's various review processes are very strict, with high requirements for documents, and a lot of time is spent on writing documents and going through review processes.
- The software products developed have dependencies on both upstream and downstream, and a lot of waiting is needed for test data preparation and test environment readiness.
- I am not clear about the specific details of the tools and testing techniques used by other teams.
The problems in the scenarios above are difficult to solve within a single team. Therefore, the ultimate goal of building a systematic testing system is to look at the construction of testing systems from an organizational perspective.
1.2 What Should Be Considered in An Organizational-level Testing System?
At the organizational level, the focus of the testing system should be on the people, processes, and technologies related to quality assurance. Typically, there are two aspects to consider:
- Those closely related to quality and testing, including quality objectives, process specifications, testing strategies, and practice standards;
- Those seemingly unrelated to quality and testing, but have a critical impact on the implementation of quality assurance work, such as organizational structure, corporate culture, and the way departments collaborate.
02 Organizational-level Testing System Map
The organizational-level testing system is inevitably much more complex than the team level. To better understand and discuss it, the aspects that need to be considered are organized into the following map:
The map consists of "one center and four directions." To build a complete organizational-level testing system, it is recommended to focus on "one center" and make efforts in the four directions:
- One center: core values
- Four directions: efficient collaboration, standardized guidance, normative implementation, and automated support.
2.1 Core Values
Core values are the cultural beliefs of an organization that deeply influence the behavior of every member within it. The construction of a testing system should be guided by the core values.
2.1.1 Value-Driven vs. Metric-Driven
The vast majority of organizations use metrics to assess the work of each individual because it is a simple way to measure the quality of the work done. However, this approach is also quite crude because "not everything that can be measured is important, and not everything that is important can be measured."
If we focus too much on metrics, we can become metric-driven, where work is done to meet a certain metric. Whatever is measured, something will surely be obtained. Once a certain metric is established, employees will try their best to meet it in their work, but it becomes difficult to determine whether the truly important work has been completed. This metric-driven approach to work will inevitably ignore the true value behind the work. This is extremely dangerous! In the opening of the book "" two classic examples are given: the crime clearance rate of police officers and the success rate of surgeries performed by obstetricians and gynecologists.
Let's take an example of metric-driven software development. The fallacy of "velocity drives everything" mentioned in the article "Velocity Is Not Responsible for It" is an instance of incorrectly using a metric (development velocity) to drive development. It is shown in the following:
I believe that everyone can easily come up with examples of metric-driven software development, so I will not go into detail here.
So, is it not necessary to measure at all? No, appropriate measurement is still necessary. Measurement can serve as a traction to help the team to improve.
However, work cannot be driven by any metric, but should be value-driven. Software development requires delivering software that can bring business value and is of high quality, and work done around this value goal is meaningful. Therefore, software quality assurance and software testing should also be driven by business value. For more on this topic, please refer to the article "Business Value-Driven Testing".
At the organizational level, it is necessary to build a culture that is value-driven, guiding employees to focus more on value and weakening the negative effects of metric-driven work, to prevent falling into "".
2.1.2 Encouraging Innovation vs. Blame Culture
The core consequence of performance appraisal is blame culture, which is closely related to KPIs. Whether employees are blamed or improvement measures are taken when performance indicators are not met or accidents occur will directly affect their behavior, attitudes, and problem-solving method.
In the event of a serious production issue, should the responsible person be blamed and punished, or should the root cause of the issue be analyzed and improvement measures be taken to avoid it in the future?
If the former approach is taken, all relevant personnel will be anxious and it will be difficult to objectively analyze and solve the problem. Instead, certain factors may be concealed, leading to the possibility of similar problems occurring again. Regarding the latter approach, there was a detailed practice of defect analysis in the article "How Defect Analysis Helps Build Quality In" . Here, it is important to emphasize that defect analysis only needs to analyze the objective reasons for the problem, without the need to blame individuals.
A very typical example of non-blame culture is the prime directive followed in agile retrospective meetings:
"Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."
-- Norm Kerth, Project Retrospectives: A Handbook for Team Review
The other side of blaming is encouraging innovation. Success comes from constant trial and error, and to do things well, one needs to constantly try, experience multiple failures, and summarize experience and lessons learned from each failure to make progress. For example, high-tech companies like Google and Amazon are very encouraging of innovation.
Organizations need to promote this culture of innovation, give employees appropriate space for innovation, encourage everyone to actively try new ideas, and continuously improve the software development process to improve software quality.
2.1.3 Collective Responsibility vs. Independent Separation
For software development teams, "team's responsibility for quality" is almost a household name. However, from an organizational perspective, how should a team be defined? While the principles are understood, how are they implemented at the operational level?
Based on my knowledge of the current situation, there is still a significant gap between departments and different roles within a team. Many people believe that quality is primarily the responsibility of testers (departments/teams/individuals), while business people are responsible for business requirements, developers are responsible for coding, and operations is responsible for the online environment.
I have written several articles on team responsibility for quality before. You are welcome to read them:
- "Is Team Responsibility for Quality Enough, or Do I Need to Be Responsible Too?"
- "Guiding Principles for Agile Testing: Team Responsibility for Quality"
- "Is Everyone Responsible for Quality?"
From an organizational perspective, it is necessary to clarify quality goals and make all members aware of them to drive collaboration between functional departments based on quality objectives and enhance collective responsibility for quality among all team members.
2.2 Efficient Collaboration
The emphasis of efficient collaboration lies in the organizational structure and the collaborative methods among different parts within the organization. This is a crucial part of building a testing system at the organizational level.
2.2.1 Organizational Structure
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." - Melvin E. Conway (Conway's Law)
Software development is a complex social activity that requires sufficient communication and collaboration among business, developers, testers, operations, and other teams. According to Conway's Law, it is necessary to adjust the organizational structure to meet the needs of software development.
However, traditional enterprises typically have thick departmental walls, with business and technology, developers and testers, and operations belonging to different departments, and these walls are difficult to break. This is undoubtedly a major contradiction.
For example, traditional enterprises usually have independent and large testing departments/testing teams, which undertake testing work in independent testing phases. In the new situation that requires continuous testing, how to adjust the testing department to better cooperate efficiently with the development department is a matter of concern and also a problem that needs to be solved in establishing a comprehensive testing system in the organization.
How to solve this problem? There are usually two common ways:
Keep the original independent testing department, with testing personnel managed by the testing department but assigned to the development team to carry out testing work within the development team;
No longer have a testing department, disperse all testing personnel and integrate them into the development center, deeply integrating with the development team.
Which way is better? It's hard to say.
Formally, the second way of completing integration seems to be better, which can achieve deeper integration of developers and testers. However, this also requires high demands for the team, such as doing well in testing shift left, team responsibility for quality, and other aspects, to provide reassuring quality assurance. Therefore, for industries that have very high quality requirements, such as the financial, the first way may be more acceptable. One reason is that breaking through traditional departmental walls is too difficult, and another important aspect is that there is still an independent testing department to "check" the quality, which seems to make everyone more at ease.
As for which form is suitable for your organization, it still needs to be explored by your organization itself.
2.2.2 Team Topologies
"Organizations should be seen as complex, adaptive organisms, rather than as linear, mechanistic systems."
Naomi Stanford, excerpt from "Guide to Organization Design"
The book "" points out the trap of organizational structures: traditional organizational charts are relatively static and do not help us understand the real communication patterns within an organization. In software development, there may be many communication patterns that are not reflected in the organizational chart. These adaptive communication structures that are critical and valuable to software development are emphasized. Therefore, the concept of team topologies is proposed.
The team topologies approach revolves around the effective team structure for enterprise software delivery and brings a new way of thinking. It offers four fundamental team types: Stream-aligned teams, Platform teams, Enabling teams, and Complex subsystem teams, and three core team interaction modes: Collaboration, X-as-a-Service, and Facilitating. These concepts aim to help organizations create efficient and effective teams that are optimized for delivering software products. It emphasizes focusing more on actual communication paths outside of the static organizational structure (team structure) and supplementing the dynamic and perceptual capabilities that are lacking in traditional organizational design for technical organizations.
Looking back at the issue of how to organize the testing department mentioned earlier, based on the team topologies concept, we may think of different solutions. Strengthening the interaction between testing and various departments/roles is the key. We need to build a flexible and adaptable team topology structure based on product development needs. The specific organizational form may not be the most important.
2.2.3 Communication and collaboration
Team Topologies emphasizes the importance of communication structures within an organization, but just building teams with the right communication patterns doesn't guarantee the desired communication needs will be met.
For example, you may have experienced or heard about situations such as:
- Only "upper-level" leaders know some business goals and quality objectives and are not transparent to the entire organization or team.
- Some companies have business and development teams sitting together, but communication still happens through requirement documents, just as before.
- In some teams, testers and developers belong to the same team, but testers are not involved in technical discussions, developers don't inform testers of technical updates, and developers don't participate in the formulation of testing strategies and plans.
- Different small teams of the same product or project operate independently, without knowledge or information sharing, leading to repeated wheel reinvention.
There may be many similar situations. These are seemingly fine on the surface but lack effective communication and collaboration.
Regarding communication, it's important to value the content of communication, making information transparent throughout the organization or team, especially regarding vision, goals, and risk information, which can enhance members' sense of mission and subjective initiative. In my article "Guiding Principles of Agile Testing", I introduced the characteristics of efficient and collaborative teams, and in "Beware of 'Mushroom Farming' in Teams", I shared the importance of information sharing by examples.
On the other hand, it's also important to pay attention to the forms of communication, as different forms of communication have vastly different effects. Typically, face-to-face communication is the most effective way, as shown in the figure below:
Collaboration is built on the premise of adequate communication. Once communication is in place, collaboration will be smoother.
2.3 Standardized Guidance
I have previously written an article titled "Quality Assurance Empowerment for Agile Teams", in which I investigated the software development and delivery processes of large companies such as Google, Amazon, and Facebook, and summarized that team quality assurance empowerment needs to achieve standardized processes throughout the entire process.
By guiding the standardization of work methods, teams can better ensure quality at every stage and take responsibility for quality as a team. It is important to note that standardization should not be limited to written forms only, as described in "Factual Standards and Written Standards" in the article "Quality Assurance Empowerment for Agile Teams)".
2.3.1 Standardization of Development Processes
Testing and development are inseparable, and testing strategies are influenced by different development processes.
Whether it is a traditional waterfall development process, an iterative agile development process, or a combination of the two, the corresponding process strategies need to be standardized, with clear definitions of the activities in each stage and the responsibilities of different team members at the organizational level.
With standardized guidance for the development process, teams should conduct detailed analyses of bottlenecks in any stage and make adjustments and improvements if the process no longer fits the current team's situation.
2.3.2 Standardization of Testing Strategy
For different development processes, there needs to be a standardized testing strategy at the organizational level. The organizational-level testing strategy serves as a unified guiding direction, and different product lines and project teams also need space for autonomous adjustment to better customize and adapt their strategies.
In addition to providing space for teams to adjust their strategies, it should also be emphasized that strategies are not set in stone and need to be adapted and adjusted based on quality risks and team status during different periods. It should be evolutionary.
Regarding testing strategy, it is strongly recommended to refer to "One-Page Test Strategy" and customize it based on the characteristics of the organization.
2.3.3 Standardization of Quality Objectives and Measurement Strategies
Clear quality objectives can better drive the construction of the organizational-level testing system. Organizational-level quality objectives should not only be quality requirements for pure system use but also closely related to business objectives and driven by business value. Regarding the relationship between business value and testing, you can refer to the following articles:
When it comes to quality objectives, corresponding measurement indicators and strategies are necessary. Organizational-level measurement indicators should be defined for quality objectives and measured through a unified measurement strategy. In terms of measurement, it is especially important to not focus on a single indicator but rather to use a combination of indicators for measurement. Indicators should not be used as a standard for horizontal comparison between product lines/teams but rather as a benchmark to guide product lines/teams' continuous improvement. I strongly agree with my Thoughtworks colleague Ben WU's statement in his article "":
"In agile IT development and delivery, measurement plays a role in identifying the biggest bottleneck in the value stream, in order to identify the biggest bottleneck (and direction error) in end-to-end value flow among the dimensions of 'accurate value, fast flow, and good quality,' and make it the next improvement point for improvement, in order to maximize the effectiveness of improvement."
Regarding quality measurement, my Thoughtworks colleague YU Xiaonan has shared very detailed insights from both qualitative and quantitative perspectives and has written a series of related articles. You can refer to her "" for more information.
2.4 Normative Implementation
Under the guidance of standardized strategy, it is a challenging task to ensure that each practical activity is well-implemented. Some practices may be implemented incorrectly from the beginning, or they may start off correctly but eventually become outdated.
The implementation plan for standardization is a guarantee for the landing and implementation of standardized strategies. Organizational levels need to consider the effective implementation of each practical activity from the following aspects: defining practice standards, precipitating new excellent practices, customizing maturity models, and conducting regular governance and continuous improvement.
2.4.1 Define Practice Standards
Clear definitions should be provided for each practical activity, including the practice's objectives, applicable conditions, participating roles, inputs and outputs, and it is best to introduce the entire practice activity in detail using examples. Canvas can be used to introduce the above items more clearly and intuitively.
The defined practice standards are the reference for the team's implementation and landing. During the team's practice process, the team is encouraged to improve each practical activity based on their experience, achieving continuous optimization.
2.4.2 Precipitate New Excellent Practices
During the practice process, the team may also discover new practical activities. It is recommended to share these new practical activities with more teams, recommend them for trial, and once they are recognized by multiple teams, standardize them and store them in the organizational practice library.
At the same time, practices that are no longer applicable should also be eliminated and removed.
2.4.3 Customized maturity model
Based on the practice standards, a maturity model can be developed to evaluate the implementation status of each stage of the team's practices and formulate corresponding improvement measures to help the team grow continuously. The organization needs to develop a guiding maturity model for each team to customize and guide their improvement. For more information, please refer to my article on "Testing Assessment".
It is important to emphasize that the purpose of the maturity model is not for evaluation but for guiding improvement. If maturity is to be associated with team performance, it must be approached with caution, or else it may fall into .
2.4.4 Regular governance and continuous improvement
Just having a maturity model is not enough. Regular governance is important to guide improvement. At the organizational level, a quality governance strategy should be defined and applied to each team. Based on the maturity model, teams should periodically review the implementation status of their practices, assess the current situation, and develop new improvement measures based on the review and evaluation results, combined with current quality goals.
For example, teams can be encouraged to conduct regular reviews and governance based on iteration and release cycles. Also, different improvement measures may require different cycles for governance and improvement.
In summary, the organization should have a sense of regular governance and continuous improvement, combined with the maturity model, to encourage teams to achieve continuous improvement.
2.5 Automated Support
A strong infrastructure can provide automated support for the implementation of quality practices, simultaneously improving quality and efficiency.
From the perspective of building a testing system, organizations need to consider automation support in several areas:
- Process management: The software delivery process throughout the entire lifecycle usually requires pipeline support.
- Testing framework: Various automation testing frameworks and their association with continuous integration pipelines.
- Data analysis: Visualization of various data and analysis results, including log information and metric data.
- Monitoring and warning: Real-time monitoring of team and product status, intelligent prediction, and warning of quality risks and severe defects.
Whenever efficiency is mentioned, various efficiency support platforms and tools come to mind. However, given the enthusiasm for tool platforms, it is essential to remind of a few points:
Emphasize the use of tool platforms, but personnel lack the corresponding empowerment, are not clear about the principles behind the tool platforms, and cannot use them efficiently. Both platform vendors and purchasing enterprises have talked to me about this phenomenon, indicating that it is not an isolated case. Even if the tool platform is excellent, it can only achieve twice the result with half the effort if used correctly; otherwise, it will become a burden.
Standards and specifications remain on paper and have not been implemented. This is what was mentioned earlier in "Quality Empowerment of Agile Teams", the section of "'real standards' and 'written standards'", and tool platforms can help turn written standards into real standards.
Different teams use different tool frameworks, and the organization does not have a unified tool platform. This will not be conducive to collaboration and integration between different teams, will not be conducive to collecting organization-level data, and will increase unnecessary costs. Do not reinvent the wheel. Tool platforms that can be shared within the organization should be shared, and a consistent tool platform will bring benefits to data collection and visualization. At the same time, the issue of team priority adaptation must also be considered. For example, it is not appropriate to force the use of tool B if team A's best solution for automated testing is tool A, even if there is an emphasis on organizational-level unity. All things require balance.
03 Quality Enablement at the Organizational Level
Based on the test system map introduced earlier, overall, quality enablement at the organizational level should focus on the following aspects:
3.1 Establishing a value system that adapts to the new era's software quality requirements and fostering an organizational culture where everyone values quality and value delivery
With the continuous development of technology, the software ecosystem is becoming increasingly complex, and new era software quality has higher requirements. Traditional quality assurance and software testing approaches may not fully adapt to the changing landscape. Therefore, it is necessary to change traditional perceptions, achieve deep integration of multiple roles, collaborate, and take responsibility for quality and value delivery.
3.2 Moving towards standardized processes throughout the entire process and reducing quality costs
Standardizing processes throughout the entire process is the trend, but it is also challenging to accomplish in one step. A step-by-step approach is more feasible:
- Firstly, define organization-wide process strategies and form written standards;
- At the same time, develop practice activity specifications to guide the implementation of written standards;
- Gradually promote and standardize by verifying through practical tests of some teams to form organizational standards;
- Utilize tool support to set up quality gates and other methods to ensure that written standards can truly be implemented, and continuously improve and optimize strategic standards and practice specifications.
3.3 Strengthen quality infrastructure construction, expand automation scale, and improve R&D efficiency
Large-scale automation is a helpful way for achieving standardized processes throughout the organization. The organizational-level testing system needs to strengthen infrastructure construction in both testing automation and process automation, as both are essential.
3.4 The capability of quality personnel is crucial, and enhancing capability building cannot be ignored.
On the one hand, it is essential to establish organizational-level talent development mechanisms, to establish corresponding capability models and development paths for different roles, to form a talent gradient, and to implement personnel capability development in a planned manner. At the same time, community building should be encouraged, and a learning culture should be created to promote personnel's autonomous learning and development.
Regarding the improvement of quality personnel capabilities, you can refer to the following articles:
Following the Basic level that focused on the basic responsibilities of testers and the Intermediate level that focused on quality built-in throughout the entire lifecycle, this Advanced level introduces the "one center, four directions" organizational test system map to help enable organization-level quality.
With this, the series of systematic thinking for testing at the basic, intermediate, and advanced levels are all complete, hoping to provide a testing system framework that can help everyone think about testing.