Saturday, August 11, 2018

What shouldn't we automate in an agile environment? | Supreme Agile

If you're familiar with my blog you already know that I'm a great believer in automation frameworks that allow the team to create a more productive, efficient and reliable process of coding and testing. However, some aspects of the testing process still need human eyes, common sense, and intelligence.

Usability Testing

Usability testing is very different from other test types that determine the quality of the software. As such, it cannot be automated because it requires someone actually work with the software to determine how it was experienced and where the gaps are in the user experience.

GUI Testing (Is it worth the ROI?)

GUI testing is one of the most difficult areas to automate. I’ve seen just too many organizations that invested thousands of human hours to automate the GUI of their products but at the end found it was a waste of time that didn't really provide the expected ROI.

Some GUI tests can be used just to make sure there are no unexpected changes in the GUI, but you should ask yourself whether it’s worth the costs and investment that can go instead to improving other quality issues that will provide a better ROI and reduce the risks in that area.

Tests that are not worth the investment

I used to be involved in an automation project where the test team automated a thousand tests that provided (at least on paper) great test coverage. So what is wrong here? They automated almost all the available test scenarios without really thinking about the ROI.

The team invested weeks on automating tests that were marked as low risk, tests that would never fail, and tests whose failure had a very low chance of impacting the software. The entire automation process was based on the spirit of "let's automate everything" instead of asking the simple question of what is the ROI that this automation project provides.

In some cases, there are tests that are written without real thought as to whether they are important or not, and once the automation project starts, the team will automate these tests just because they never want to run them again (because they know that there is 0% chance that it will actually make a difference). 

Tests that need to be executed only once

The main goal of automated testing is to allow the team to focus on the things that are really important in the software development lifecycle. Automating test scenarios that will run only once are not worth the time that the team needs to invest in the design, creation, and execution of these tests. 

Exploratory Testing

In my opinion, exploratory testing is the best and most efficient method that agile teams use in any testing process. Exploratory testing can be used for learning purposes (you learn more about the software when actually testing it) or to provide a fast way to evaluate the overall quality of the software. However, when it comes to real testing effort, a skilled tester is required to design and execute tests.

Exploratory testing should be done by humans and not by automated scripts because automated scripts will not let the tester take in new information he generated from the exploratory session and use it to improve future testing and development processes.

In addition, although exploratory testing is a great testing approach, there is a real need for automated tests that will allow the team to focus on their exploratory sessions without worrying about any regressions that automated tests should cover.   

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile

Thursday, August 9, 2018

Cloud testing: the right way to do it | Supreme Agile

In this article, I will review some important insights on cloud computing. In order to achieve the most from this article, we need first to understand the cloud-computing concept, and how the cloud is different from any other infrastructure.

The basics of cloud computing

Cloud computing is the result of one of the biggest and most important revolutions we’ve witnessed in the software industry, the technology known as virtualization that has changed how organizations around the world manage their computing resources.

This advanced technology creates a completely new methodology of how organizations can share computer resources across multiple systems in order to reduce costs and deployment time, increase scalability, and facilitate the IT department in managing their infrastructure.  

The virtualization technology becomes even more important once evolved in the new form of cloud computing. Cloud computing is an internet-based platform that uses the virtualization technology and its various computing services like hardware, software and any other computer-related services that provide a total solution of resources based on demand across the internet. 

To summarize it, let us review the main points that you should keep in mind what is cloud computing:
  • Cloud computing is a general term for the delivery of software/services over the internet
  • Enables companies/people to consume on-demand resources, such as a virtual machine, storage, and applications.
  • Cloud allows access to services without the user need to have technical knowledge or control of supporting infrastructure
The cloud structure is based on three types of delivery models (aka components) that provide the “as a service” solution: 

Infrastructure as a service (IaaS)

This is the fundamental layer of the cloud solution. It focuses on physical resources such as computing services, networking, and data storage space. IaaS resources are usually billed on-demand based on customer usage.

  • Microsoft Azure
  • Google Engine (GCE)
  • Amazon Web Services (AWS)

Platform as a service (PaaS)

This is the second layer of the cloud solution, which provides organizations with a platform with the main advantage of removing their need to manage the underlying infrastructure. An organization that uses this layer will not need to worry about resource procurement, capacity planning (you can simply set it to grow dynamically as long as you have the budget) and the maintenance of both hardware and software.

  • Google App Engine (GAE)
  • Apprenda
  • Amazon E2C

Software as a service (SaaS)

The top and the most cost-effective layer of the cloud platform, which provides a complete product, is often referred to as an end-user application, run and managed by the service provider. In this layer, applications are available to the end users on demand via the internet. Using SaaS, a customer can access their applications without installing the software on a personal device (workstation/server). The entire processing effort is conducted on the vendor’s datacenter.

  • Salesforce
  • Google Apps
  • Gmail

Types of cloud

There are three types of available cloud formations: public, private and hybrid.

Services are available to everyone, resources are allocated and consumed dynamically as per the tenant request
Managed under the security restriction of a particular organization and available to this customer only
Mixture of both public and private clouds. The mixture depends on the organization's decision on what services to expose to all and what services they want to expose to specific users.
Owned and operated by a service provider
Owned and operated by the organization's IT
Allows IT organizations to become brokers of services

Very high
Very high

Depends on the security measures of the service provider
Most Secure, all storage is on-premises
Very secure, integration option add an additional layer of security

Connectivity over the internet
Connectivity over the internet, fiber and private network

Combination of both
Technical Difficulties

Technical knowledge required

You get the Basic setup but still, the knowledge of the subject is required.

Customers do Not need to worry about technicalities; The SaaS provider company handles everything.

Cloud concerns

Cloud is a great technology that has already started to change the industry and the way companies manage their data. However, the world is not perfect and there are still some concerns that we should take into consideration:

Interoperability – A universal set of standards and/or interfaces have not yet been defined, resulting in a significant risk of vendor lock-in.

Latency – All access to the cloud is done via the internet, introducing latency into every communication between the user and the provider.

Regulations – There are concerns in the cloud computing community over jurisdiction, data protection, fair information practices, and international data transfer—mainly for organizations that manage sensitive data.    
Reliability – Many existing cloud infrastructures leverage commodity hardware that is known to fail unexpectedly.

Resource control – The amount of control that the user has over the cloud provider and its resources varies greatly between providers.

Security – The main concern is data privacy: users do not have control or knowledge of where their data is being stored.

Cloud testing 

Cloud testing refers to testing of cloud resources (both hardware and software) that are available on demand. Cloud testing must be conducted to ensure that the product under test meets both its functional and non-functional requirements.

SaaS Software Development Lifecycle (SSDLC)

      Requirements - Gathering and prioritizing business needs/stories by the customer/PO for the product as well as capturing them in a central location.

Design - Building a technical blueprint of how the proposed system/feature/model will work. It includes elements such as system features, models, technical architecture, integration points, interfaces, UX, etc.

Development - This is the physical building and coding of the product’s features/model including database based on the design and requirements.

Testing - Verifying the feature/component of a product works as expected and meets all of the business requirements. It also includes writing test conditions and executing test scenarios.

Go-live & maintenance - Implementing the feature/component of the product in the production environment as well as the day to day maintenance of the application (including updates).

Types of cloud testing 

There are four different types of cloud-based testing. Each type has its own objectives.

Testing SaaS in a cloud (testing an application) - This type of testing is used to validate the quality of the application in the cloud. Functional and non-functional requirements of the particular application are verified.

Testing of a cloud - The cloud is tested as a whole entity and based on its functionality. This type of testing is used to validate the quality of the cloud from an external (end users) point of view (its capabilities and service features).

Testing inside a cloud (infrastructure testing) - This type of testing is carried out by the cloud vendor and checks the quality of a cloud from an internal view or feature, based on the internal infrastructure and capabilities of the cloud (e.g. automatic capabilities, security, management, monitoring).

Testing across clouds (services testing) - Testing an application is done over various clouds (private, public, and hybrid). It is based on application service requirements.

Cloud testing Enlivenments:

There are two types of cloud testing environments that can be used by development teams  for testing activities:
  1. A test lab that simulates a cloud-based environment, where the application is deployed and tested.
  2. A hybrid, public or private environment, where the application is deployed and tested as it will be available for the customers.  

Challenges of cloud testing

Quality control - How do we maintain quality products in an area that demands fast, high turnover of deliverables with no bugs? This is the world of cloud, which can be very complex for those who do not invest the time to learn it.

Data security and privacy - One of the biggest advantages of cloud infrastructure is multi-tenancy support. Although multi-tenancy support is great, there is still a major challenge to ensure that the customer’s data is not compromised, security standards are applied and the privacy-related regulations are enforced. 

Upgrades with a short notice period - Cloud providers give existing customers a very short notice prior to upgrades. This is a big problem when manually validating changes to your SaaS application and is another major consideration when thinking about conducting manual testing in cloud projects.

Data migration - Data migration, the process of moving customer data from one cloud provider to another. During this process, the risk increases dramatically because both providers involved must ensure that the data is migrated without losing any critical data.

Upgrade testing - Cloud testing’s biggest challenge is to ensure live upgrades do not influence the existing connected cloud users. Think about a multi-tenant environment that uses the same cloud environment, when the application is upgraded for a specific customer. Sound simple? Unfortunately, in some cases, the upgrade process may influence the user experience of the other tenants due to latency, networking issues, and their shared resources.

Bugs - Bugs are no longer isolated; once seen they can be seen by all and exploited.

Frequent releases - Frequent releases provide less time to run tests, less time for regressions and as a result more unexpected defects and higher risks.

Cloud testing vs. conventional testing

Test parameters
 Conventional testing
Cloud testing
High costs due to major
investment in hardware 
and software
Lower costs, payment per use of the cloud services.
Test environments
Test labs (pre-fixed and
configured test environment)
An open public test environment
with adjustable resources
Impact of bugs
Bug isolation and low visibility
 (per customer)
Each bug is a bug for everyone. Fixing a problem for one customer fixes it for everyone
Security tests
Tests are done based on server type and policy of the organization
Testing is done in the vendor’s cloud-based configuration.
Performance, load,
and scalability
Performed on a fixed, isolated
test environment
Performed on both real-time and virtual online test data.
Time to delivery
Internal software releases
once every 1-12 months
Internal software releases 
multiple times a week
(sometimes even more).
Monitoring and 
Reactive software monitoring
(downtime reported to
customers in hours, days )
Proactive software health-monitoring (downtime reported to customers in
seconds, preventive actions
taken at defined procedures)

The main types of testing performed in a cloud environment

During cloud testing, teams must validate that their tests cover both aspects of functional and non-functional testing. Let us review some of the more common test types that are part of a cloud testing project:

Disaster recovery (DR) testing – The cloud as a service must be available to customers at all time, therefore, it’s important that a replicated site will be available in case of critical failure. While executing DR tests, the team must ensure that the app can recover in case of a massive failure(restore to the last available point, no loss of data, minimum downtime, etc.).

Availability testing – This type of test is usually owned by the cloud vendor that ensures that the cloud is available to customers at all time without any downtime.

Capacity testing – This verifies that current and future hardware supports expected usage as determined by the specification of the product (such as adding or removing resources to or from a customer).

Multi-tenancy testing – This type of testing is very important in any cloud-testing strategy. During these tests, the cloud services are tested by multiple users from different tenants (each service can serve multiple customers). Testing must be performed to guarantee there are no security incidents such as access (control or data leaks) and that there is no degradation in performance once multiple customers access the same service.

Functional testing - This tests the app delivers the required functionality.

Reliability testing – To ensure that the app is capable of performing failure-free for a specific period of time in a specific environment.  

Security testing – As discussed earlier, the cloud environment provides access to multiple customers who can use the same services. As a result, we must ensure that there is no unauthorized access to the data within the SaaP application, no privacy leaks, and that customer data integrity is kept under strict security gates.

Common test Guidelines:
  • Validate that data integrity is not compromised by unauthorized access
  • Validate that only the authorized customer can access the data
  • Validate that data migration is made through secured (encrypted) channels
  • Validate that all user data is removed in case of dropping the service
  • Validate that only the relevant ports are opened
  • Validate that there is a clear separation between tenants

Scalability testing – Cloud services are relevant to both small and large organizations; as a result, there must be tests to ensure that the business can scale up or down its resources based on the customer’s need. 

Load and stress testing – To identify the stability of the system beyond its operational capacity to see how it reacts to different loads. 

Live upgrade testing – To ensure that we can deploy new versions on the cloud without affecting customers’ user experience.

Performance testing – To ensure that the SaaS application can manage different traffic loads that depend on the number of customer requests. The main factors that we want to validate in this type of tests are network latency, the response time of the application and the workload balancing (NLB) in case of massive use. 

Common test guidelines:
  • Response times should not be affected due to the actions of other tenants
  • Failures in one tenant should not affect other tenant performance  
  • Scaling process should not cause any degradation in performance factors

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile

Wednesday, August 1, 2018

Barriers to incorporating automation in agile teams | Supreme Agile

As I’ve written numerous times, agile teams must automate many aspects of their work, such as testing and build creation, as well as any other process whose automation would save time for the team.

However, despite it being clear to me that it’s very logical for agile teams to use automation for any product backlog item (PBI), I’ve come to the conclusion that it's simply not as obvious as I thought. As a result, I decided to write this article about the main reasons agile teams fail when trying to implement an automated solution. 

Developers’ attitudes towards automation

To understand developers’ attitudes toward automation solutions, we need to look at how they see automation in traditional environments, where separate QA teams do all the testing work without involving developers in the testing process. The direct result of this environment leads developers to become less involved in the testing process (why bother testing if there are dedicated QA teams to do the work for us).

In addition, the waterfall development process is built with different phases for development and testing, which make testing even more remote to developers who do not need to do much after their phase is done and they have already moved on to the next project.

In an agile environment, there is no separate QA department that provides a safety net for developers. The agile development team must ensure the quality of the product, and developers must become involved in all aspects of the testing process, from the unit test level till system testing.

Developers must change their attitude, mindset, and culture to allow the team to succeed in an agile working environment. Developers who fail to do this will impede the team in being able to deliver on their commitments.

Unrealistic maintenance costs

The main goal of automated frameworks is to boost the team’s ROI while increasing the efficiency of the test execution, build creation and overall process. So if we think about it, one of the main goals of using automated frameworks is to free engineers from performing manual work so they can focus on other aspects of the project.

But what happens when the team selects the wrong automated framework and uses poor test design that consumes most of their time in maintenance and stabilization of the written tests? This leads to an unrealistic situation where the team spends hours and days of manual work on frameworks that were supposed to free up their time.

A classic example of this scenario that I often see is agile teams that do not want to spend time on user-interface (UI) testing. The first thing they do is to develop or purchase a third-party vendor capture tool to record their tests, expecting it to solve all their automation problems. Well, this will not work. The creation of thousands of lines of UI test scripts that (usually) don’t follow code practices will create a situation where, over time, no one knows what they are supposed to do or why they were written at all. This leads to unrealistic maintenance time on tests that are maybe no longer relevant.

To overcome this barrier, the team must choose the right framework for the automation they want to achieve. Someone who is capable of seeing the big picture must invest time in great test design and the future roadmap, and the team must use the relevant code practices that will make the code readable and easy to maintain over time.

Previous bad experiences

One barrier that is very common among engineers is actually very basic and that is a previous bad experience in automation projects that didn't pay off.

There are many challenges in automation projects and therefore more opportunities that can cause teams to fail, such as poor design, unstable automated frameworks, and many others. In this case, the organization must analyze the reasons for the failures and ensure they will not recur in future projects. 

Legacy code

This is a simple fact: most engineers prefer to write their own code, rather than getting legacy code that was written by other programmers. When writing automated tests, we have another layer of testing, which makes it even harder for a developer to succeed with the project.  

The first barrier is the code itself. If a developer needs to work with code that he didn’t write or participate in the design of, it may be hard for him to understand the code itself and what tests should be created to provide good test coverage.

The second barrier is legacy code that isn't designed for testability, which makes it almost impossible for an engineer to create automated test scripts without needing to refactor the legacy code.  

Letting testers do the work

If I need to select one reason that will lead to failure again and again, I would probably say that it is letting testers write the automated tests when they do not have the necessary knowledge and experience in the coding field.

This is just one classic example of how all team members of an agile team should be working as a single unit without separating between programmers and testers. Otherwise, the entire project can fail by letting the testers know that if they cannot handle basic test scripts they have nothing to offer in the agile world.

In the opposite direction, a strong agile team will not let testers do this job without a supporting framework from the rest of the team. The team must understand that any automation project is the team’s problem, not the responsibility of the testers just because they were responsible for quality in the old traditional environment. 

Job security

Agile teams contain testers that in many cases were added to the team as a legacy from the previous environment. These testers often do not have coding experience and therefore focus on manual testing, which is less suitable in an agile environment. These testers may reject the idea of using automation processes for testing for fear of it making them less relevant in the future.

Fear of failure

Due to challenges inherent in automation projects, from determining the goals to the implementation itself, automation projects can be scary to engineers. As I learned over the years, programmers may have the knowledge to write great production code, but once they focus their efforts on writing automated tests, they will face many logical and technical issues that they do not have in their day-to-day work.

Lack of support and knowledge 

I think that every engineer who has common sense about automation’s benefits will want to use it to simplify work. But what happens when this engineer has neither the knowledge nor the time to invest in the creation of this framework? How can he free time, in an already stressed environment, to learn new tools such as automated frameworks and design practices such as TDD and refactoring?

To allow the team to gain the knowledge they need to master this area, the organization must step in to provide the necessary coaching. An external expert is a great option to help the team get set up and save a great amount of time. Coaching is needed in both the theoretical and technical aspects of automation practices. More important is to free the team so they can really learn and adopt this new approach in their day-to-day activities.

An investment that will not pay off right away

The team will need to invest time to create, plan, and design the automated solutions that will reduce the manual work in the long term. But although the benefits are clear, we need to remember that even with the entire agile team working on the automated solution, it still requires a big up-front investment that will reduce their ability to deliver functional PBIs in the first few sprints. This can be a big problem in an agile environment.

There is a huge psychological barrier that agile teams and the organization run into when they understand the investment that they need to make at the beginning of the automated project, an investment that will not pay off right away. Both the team and the organization must know that it takes time to decide on which processes and tests should be (and can be) automated as well which frameworks to use.

As I’ve seen in almost any automated project, the team must show senior management how automated solutions will help the organization increase the ROI. Although they will not see an increase in ROI in the first few iterations, without knowing the benefits, there is no chance that the organization will allow the team to invest the time they need to succeed with their automation challenges.

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile

Thursday, July 26, 2018

The agile testing mind-set | SupremeAgile

To understand the agile testing mindset, we first need to determine what makes a team "agile." To me, an agile team is one that continually focuses on becoming self-organized and cross-functional to be able to complete any challenge they may face during a project. In addition (and based on the agile spirit), the team must put the customer first and ensure that at the end of each iteration the customer receives the best possible product.

In my experience, there are some aspects that differentiate "agile" teams from other teams, such as:

Continuous learning. As in any other field in life, agile team members must grow as individuals to be able to contribute to the team effort to succeed. They must learn agile practices, new technologies, tools, and technical excellence as well as master all aspects of the product.

Discipline ensures that each team member will follow the rules and guidelines of the agile manifesto and the organization (as each organization has its own rules, boundaries, and goals).

Great HR skills that will allow team members to work with one another, help in case of trouble, and most importantly, allow for team dynamics that focus on both the team and on the individual.

Embracing continual improvement that will allow the team to improve the process and product each iteration.

Sharing knowledge to promote team members who do not have the relevant skills, knowledge or experience that the team needs to complete their goals.

Trust and respect, which are fundamental aspects we want to see in any team and in agile teams in particular. Team members who do not trust and respect their co-workers will become the biggest impediment of the team. 

Now that we understand the characteristics of a great agile team, we can see that successful agile projects are the result of teams built from great people who have the freedom to utilize their skills to generate great work.  

The new tester in agile teams

Agile teams are built from both developers and testers that work together towards a common goal. (The best teams I created were built without testers at all, but this is for a more advanced article.) Now that we’ve discussed the characteristics of a great agile team, it’s time to define the same thing for testers and ask what makes someone a success as a tester on an agile team? The basic answer is that what makes a tester succeed on an agile team is usually the same characteristics that make a highly respectful tester on any other team.

An agile tester doesn't see himself as the team’s quality authority (for more information: What does it mean to be an agile tester?), protecting the customers from the programmers as he did back in the old days when he worked in a dedicated test department, usually at a separate physical location from the development teams (for more information: Testers are not the "Quality" officers in an agile team). 

For me, there is no difference between tester and developer; they are all equal and should follow the same team culture, mindset, and principles. Based on that, I can say that great testers that really succeed in agile teams are those who are willing to share their knowledge and test experience among other team members. They are willing to work hand in hand with developers and the product owner to promote the team and the overall quality of the product, using a collaborative approach that states that the entire team is responsible for quality.

Agile testers (and probably any other testers) that have the right HR skills, technical skills and mindset promote the team and themselves by providing continuous feedback about the overall quality of the product. This allows the team to see the big picture and what should be done to increase quality.

In addition, agile testers are continually searching for new ways to help the team increase their efficiency and produce high-quality products. What does it mean to search for new ways? Here are three examples:
  1. Testers can search for new technical tools that will improve the efficiency of the testing process and reduce manual efforts (such as automated frameworks that will shorten regression cycles).
  2. Testers can create local group meetings or roundtables with other teams to learn and adapt success stories that worked in their team.
  3. Testers can share their knowledge with the rest of the team to increase the quality ownership. During this process, the tester can act as a quality consultant that will help other members remove quality barriers they are unfamiliar with. 
The bottom line is that testers in agile teams should have a different mindset than the one they had in the environment of traditional software development. They must understand that their agile teammates (which include more programmers than testers) share the same goal as them. Testers should like their teammates and enjoy learning from them (especially coding skills that will help them add more value). Testers should never limit themselves to solving only quality issues.

But wait, there’s more. Agile testers should also help the team address issues that arise during the iteration and afterward, especially regarding technical aspects of the product. In addition, testers should know that the team trusts them to understand the big picture of the whole product, rather than concentrating on a specific component.

Great testers are the ones that the team rely on because they know that they have an instinct and knowledge for where and how software might fail and what solutions the team should implement to reduce the risk of failure.

But wait, I think that testers can do even more! Testers should use their expertise, knowledge, and experience in testing and help the team in other layers of the development process. A great tester isn't afraid of being involved in design discussions, giving suggestions that will improve testability and reduce future risk.

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile

Sunday, July 22, 2018

The changing role of QA managers in the Agile environment | SupremeAgile

Many questions arise when an organization starts the transformation to an Agile environment, some related to the adoption of the new Agile culture and some related to the dramatically changing organizational structure.

One area that creates confusion among managers is, of course, what is going to be their new role once the organization becomes Agile. In this post, I will focus on QA or test managers and describe the transformation from traditional QA manager. I’ll discuss new opportunities in the Agile environment while reviewing some of the most commonly asked questions, such as:
  • What is the role of the QA manager in an Agile environment?
  • Do we really need a QA manager in an Agile project? 
  • Who is responsible for ownership on quality?

A few facts on the current software industry 

Before we start, I want to mention a few major facts that will help you understand the context for this post.

Fact 1: The current challenges in the software industry are different from a few years ago. Just think about the new platforms that did not exist 10-15 years ago such as web, cloud, and mobile.

Fact 2: Agile frameworks are here to stay; we are seeing more and more companies worldwide making the transition to Agile development processes (such as Scrum and Kanban).

Fact 3: The current software industry pushes organizations to their limits, and as a result, organizations must deliver quality products fast and early to stay relevant.

Fact 4: Employees’ welfare is critical to the organization’s success. Organizations that treat their employees as resources will not be able to keep them.

Fact 5: Agile transformation affects all layers of the organization, but the role of managers (and QA managers in particular) is the one affected most.

QA managers pre-Agile

To best provide the answers to the questions above, I’ll start with a quick review on what it meant to be a QA manager in the pre-Agile world. I will review the QA manager’s role in traditional software development while focusing on the Waterfall methodology created in the early 70s. 

The management style was different…

The traditional manager’s management style is mostly based on the “Command & Control” method, defined, according to the Cambridge Dictionary, as “a situation in which managers tell employees everything that they should do, rather than allowing them to decide some things for themselves.

Here are five primary characteristics of this kind of manager:

  • Controls all major aspects of the project (timelines, pace, task assignments)
  • A strict hierarchy of authority (the manager at the top of the pyramid)
  • Micromanagement led by a lack of trust between the manager and his employees
  • Zero tolerance for mistakes made by his employees
  • Commands must be followed, or discipline is applied

The interaction between the manager and his employees is based on five phases:

Phase 1: Identification of the organizational “needs” (budget, resources, and requirements)
Phase 2: Providing employee demands, expectations, and goals
Phase 3: Employees follow the manager’s instructions without question
Phase 4: Monitoring employee progress of task completion
Phase 5: Comparing employees’ deliverables to the preliminary goals

The outcomes of “Command & Control”:

  • Employees are treated as resources and not as human beings who have their own dreams and personal goals
  • There is less room for an employee’s creativity and innovation (how can there be, as the manager provides all the instructions)
  • The manager is at the center of the team (team empowerment is less important)
  • The manager’s success is more important than the success of the team
  • Employees are unmotivated
  • Employees are less involved in high-level decision processes

More opportunities as managers 

QA managers had their own departments, groups, and teams that they could manage, depending on the hierarchy of the organization. Roles like a quality director, test group leader, and the test team leader were very common in traditional software development processes.

In addition, in the Waterfall model, testers worked together on the same team (usually at a separate physical location from the development team) which meant that QA managers managed only testers (there were no programmers on their teams).

Testing projects were challenging but had some major advantages  

Most of the QA managers who have ever worked in the Waterfall SDLC are familiar with the different challenges, disadvantages, and deficiencies of this model. However, compared to an Agile SDLC, the Waterfall model has some major advantages that helped QA managers do their job:
  • There was a dedicated testing phase that started only when the development phase was completed, which reduced the risk of multiple code modifications as occurs in an Agile environment.
  • QA managers and their teams received comprehensive requirements documentation (specs, SRS, etc.) that allowed them to see the big picture of the whole project. In Agile development, one does not use large comprehensive docs.
  • QA managers and their teams had more time to prepare comprehensive test documentation (STP, STD, and STR), which are less relevant to Agile software development.
  • QA managers had the chance to determine both the entry and exit criteria for the testing process. 

Quality ownership

In the pre-Agile era, the ownership on quality was the responsibility of QA managers. As a result, they had more power to affect the different quality aspects of the project. Let’s see some classic examples:
  • QA managers had the power to determine the defect life cycle (DLC)
  • QA managers established the quality metrics used to track different quality aspects and progress
  • QA managers chose how to manage the testing risks based on their own knowledge and experience.
  • QA managers controlled how to split and prioritize the work among their testers.
  • QA managers decided on the test coverage for the different areas of the product.

QA managers post-Agile

After we looked at the world of QA managers pre-Agile, it is time to see how Agile development methodologies changed the roles and core responsibilities of QA managers. In this part, we will review the main reasons QA managers often feel confused about their future once the organization starts an Agile transformation.

There is no such thing as test departments

As we saw earlier, back in the old days of the Waterfall development process, QA managers had their own departments, groups, and test teams. When the organization transitions to an Agile environment, there is no such thing as a pure testing department where testers, managed by a QA manager, sit together, usually at a different physical location from the development team.

Let us examine the most common Agile framework: Scrum. The Scrum framework describes three main roles: Scrum master (SM), product owner (PO) and Scrum development team. The Scrum team is self-organized (does not need managers to tell it what to do) and is composed of different engineers such as programmers, DBA, designers, and testers.
In summary:
  • In Agile, testers and programmers work together on the same team.
  • The Scrum framework does not define the role of a QA manager.
  • QA managers do not manage teams in an Agile environment.
  • There is no day-to-day management of testers (testers are integrated into the new scrum teams).

No day-to-day direct management of testers

In an Agile environment, testers are integrated into the Scrum teams, and Scrum teams are self-organized; as such, there is no need for a QA manager to manage the testers as in the traditional working environment.

Another important aspect in this regard is that the Scrum teams should be able to determine what and how to test each PBI, and therefore there is no need for a QA manager to tell them what to do during the testing effort.

There is less room for comprehensive documentation 

In an Agile environment, there is less room for comprehensive documentation, as described in the following table:
Traditional SDLC
How and when are requirements delivered?
Heavy SRS/specs that specify all the project requirements at the beginning of the project
Small user stories that describe a small requirement. Not all stories are written at the beginning
Test documentation
QA managers and their teams create two main documents: Software Test Plan (STP) and Software Test Design (STD), which specify all aspects of the testing project (test environment, test strategy, risks, etc.)
All tests are written at the PBI level, and there is less room (if any) for traditional test documentation.

Quality is everyone’s responsibility

As we saw earlier, in the pre-Agile world, QA managers mainly held the accountability on quality. As the leaders of the test departments, QA managers were known as the main focal point for all quality issues that were found during the SDLC and for defects found in customer environments. In Agile, QA managers are not the quality officers anymore. Scrum teams are now accountable for the quality of their deliverables as a group and not as individuals.

Let’s take one classic example of a defect found in a production environment. In a traditional SDLC,` the QA manager was the one responsible to provide answers on why his team failed to find this issue and what mitigation plan he has to ensure that similar defects will not occur. Now, in an Agile environment, a similar issue is handled differently. Once the defect is discovered, the entire Scrum team (or specific people in the team) will try to understand what was missed during the iteration and how similar cases can be avoided.

This is why QA managers do not belong to Scrum teams and cannot be team members with this title. I saw many teams use this role in their team to try to keep some old habits. The direct result was that the QA managers—as team members—held the responsibility for failures of quality instead of letting the team take responsibility and grow from it. 

Progress and monitoring

In Agile teams, there is no need for a QA manager to monitor the testing progress. The Agile framework allows the organization and any other stakeholder to see the team progress using visual boards, backlogs, and burn-down charts.

QA managers lost their ability to see the big picture of the testing project as they did in a traditional testing project. Scrum teams are now accountable for communicating and reporting their progress, providing their estimations as a mandatory part of the Agile methodology, and there is no need for a QA manager to act as an observer. 

New roles for QA managers in the Scrum framework

In Scrum, the leading Agile framework, there are three main roles, with no room for managers: development team, Scrum master, and product owner. The Scrum team is self-managed and cross-functional, so they can take responsibility for all aspects of the project, including that of delivering high-quality products. This eliminates the need for a traditional QA/Test manager. However, all three of the roles specified by the Scrum framework are available to QA managers.

QA manager -> product owner

The first option, which is less common, is QA managers who become the team’s product owner. This new role is completely different from what they did as QA managers. Although it is less common, I’ve met QA managers who become very efficient POs, and some who failed miserably.

For more information about the role of the product owner:

QA manager -> Scrum development team member

Yes, you will be amazed to know that in many cases QA managers (especially ones who managed small teams in the traditional environment) decide to become one of the members of the development team. This means that they act as equal tester/developer (depending on their technical skills) without any authority on other team members.

For more information about the role of the development team:

QA manager -> Scrum master

This is probably the most common option for QA managers. During an Agile transformation, organizations do not know how to handle this role called SM, but because this role is a valid option for a QA manager, why not keep things calm and let the QA managers do it. This allows QA managers to take a real and active part in the transformation with minimal impact to their ego.

For more information about the role of the scrum master:

What other options do QA managers have in an Agile environment?

While there are three main roles that QA managers can take in the Scrum framework, that may not be an option for some who want to keep their ability to influence more, as they did in the traditional world.

Based on my experience, I will share some other options that I see in organizations without determining if they’re right or wrong. Remember that every organization has its own needs and what fits one organization may not be suitable for another.  

Integration team manager

Organizations have multiple Agile teams, each responsible for specific features. The problem here is that there is no team that can see the big picture of the whole product.

The solution is to create an integration team that will get the deliverables of all the Scrum teams and test them as a whole using end-to-end testing. This type of team is managed by a QA manager as in the traditional working environment.

Automation team manager

Depending on the technical skills of the QA manager, in some cases, they become the manager of the team that is responsible for building the automated frameworks that Agile teams use to automate their work and processes.

Quality Consultants

This role is may be titled differently from one organization to another. The common names for this role are a technical leader and quality architect. In this role, QA managers use their vast experience in the quality field to help the new Agile teams handle all quality issues as external consultants.

QA managers in an Agile team

This is the worst thing that can happen to the organization, the team and the QA manager, but it still occurs when the organization does not know how to handle the transformation process. The organization will want to ensure that the product quality is not affected once the test department is broken down and integrated into small Agile teams.

As a result, they will want to keep their ability to affect the team by keeping the QA managers in the new Agile teams. This is a major point, so I want to add another factor to the equation and that is the organization and team maturity in Agile, AKA “Scrum Level,” as described in this simple flow chart:

Defining the role of the NEW QA manager 

As we‘ve seen, the shift from a traditional working environment to Agile will have a major impact on QA managers. QA managers can add real value to the business if they understand that they must change to remain relevant in a world that is completely different from what they used to know.

Therefore, the big question here remains what can QA managers do in an Agile environment. I will review some of the ways QA managers can add value to the business if they have the personality that allows them to make the personal transformation.  

Help build new cross-functional teams

Self-organized Agile teams are the center of the Agile concept, but this does not guarantee that they have the necessary technical skills to handle the challenges that may arise during a development project.

To ensure that Agile teams perform at the highest level, the organization must ensure that they build their Agile teams with the necessary expertise that will allow them to handle all the different challenges of a development project.

However, this is not an easy task at all. We need to remember that each Agile team has its own area of expertise. One team may focus on the user experience of the product (UX), while another team focuses on the performance and functional aspects of the software. Therefore, each team should be built with people who have the relevant knowledge and skills that will allow the team to meet the project goals.

QA managers can have a major contribution in building these teams due to their vast knowledge in testing and in the personalities of their testers. QA managers will also help testers integrate into their new teams, as well as help them understand their new responsibilities, roles, and boundaries.

Process improvement and optimization

Great QA managers should be able to allow the organization to maximize the benefits of Agile. Based on their experience and knowledge, QA managers should monitor the Agile processes and strive towards improving them.

For example, QA managers could:
  • Participate in technical meetings to see how to reduce risks
  • Help the team create functional user stories
  • Work with the PO to define DOD criteria
  • Create and implement quality standards
  • Offer suggestions that will lead to a robust development process
  • Implement new test/development practices to improve testing/coding standards

QA managers as transformation leaders

QA managers can become the leaders of or the biggest impediments to an Agile transition. An Agile transition may take several months or even years to complete. In fact, I have never seen an organization that stopped continually improving the process. During this time, there are many issues and impediments that will affect the organization’s ability to maximize the advantages of Agile.

QA managers can be the change agents of the organization by removing impediments that prevent the teams from reaching their full potential. In addition to facilitating a Scrum implementation within the organization and training their team on Scrum practices, they can:
  • Help teams acquire resources (hardware/software)
  • Remove impediments that are beyond the ability of the team
  • Create a safe environment that will increase confidence among the team
  • Motivate employees by identifying their unique motivation triggers
  • Make room for failure and use it as a growth engine for the team
  • Promote continuous improvement

Promote testers’ professional development

QA manager should ensure that all Agile teams and especially their testers continue to grow as quality owners. By being the quality expert, the classic work of a QA manager is to ensure that testers continue to learn and develop even when divided among the different teams.

QA managers should provide technical lessons about quality practices, support testers in times of crises, mentor testers on how to perform their jobs in a new environment and ensure that they do not lose their identity. 

Lay the ground rules for testing cross-organization

QA managers do not have any authority over the Agile teams and especially not on the testers that may be under their direct management prior to shifting to Agile. QA managers will have to learn to let Agile teams grow in their path to becoming self-organized, independent and strong teams.

However, this is not anarchy; if in the past there were two main departments (QA and development) now we have tens and even hundreds of new Agile teams that are self-managed. To allow the business to ensure that all the teams are keeping to the same quality standards as they do in the traditional environment, there must be clear guidelines and ground rules that all teams must follow.

This is where the QA manager can help and use his experience, knowledge and technical excellence to set specific quality guidelines. The guidelines may include what testing methods the teams should use, testing tools, automation strategies, testing standards and the overall test methodologies to apply.

Cross-functional testing

In an organization of multiple Agile teams, each team is usually focused on their own feature without knowing what other teams are doing. This means that each Agile team will develop, test and deliver an incremental working functionality at the end of each iteration.

The approach that each Agile team is responsible for a specific part of the project has many advantages but still one major problem. The problem arises when there is no one who sees the big picture and how all teams’ deliverables are integrated into one complete product.

QA managers (or any other new title) have the capability to perform cross-functional testing (AKA end-to-end testing) to ensure that the integration between teams works well as a whole product.

In addition, QA managers can provide insights into the overall product quality based on their test results and this is one major responsibility.

The authority for all quality issues

It is hugely important that the business has an authority who can quickly resolve all quality issues that may arise during the Agile development process. QA managers are the best candidates to take this role and act as the quality focal point of the organization. As part of this role, QA managers will hold the following responsibilities:
  • Act as the point of contact for all quality issues
  • Mitigate quality issues that arise during retrospectives
  • Mitigate all quality issues that arise from lack of communication
  • Provide quick feedback on the quality of the team deliverables
  • Resolve conflicts that related to the defect life cycle (DLC)
  • Create learning sessions on technical gaps

Set the quality metrics

One thing that is very common among organizations that started an Agile transformation is the fear of downgrading the quality of their deliverables. This is a very logical fear because the quality ownership that was under the direct responsibility of the QA manager is now spread among the multiple Agile teams.

To overcome this fear, the organization should ensure that it has the right quality KPIs, which will provide the crucial ability to measure the major quality aspects of an Agile working environment.

This is where QA managers help the business by defining the quality KPIs and trends among the different teams. Let us review some classic examples:
  • The number of bugs and their severity per iteration
  • Team velocity compared to team capacity
  • How many stories meet the DOD
  • Defects found in production/customer environment
  • Time to market
  • Number of effective meetings
  • Defects found due to miscommunication between teams
Quality metrics have another great advantage: they allow the organization to identify the weakest and most problematic areas in the process and make the necessary changes to ensure that they meet the quality standards.

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile

My Presentations