Sunday, January 15, 2017

Negative testing: Complete guide | David Tzemach


תוצאת תמונה עבור ‪negative software testing ‬‏A quality assurance process contains many activities such as learning, designing, and testing that are performed to validate that the customer will receive the application with the highest quality as possible.

During these activities, the testing teams will test the application with different test techniques, methodologies, and tools that will allow them to increase the chances to find the maximum defects as possible to ensure that the client can use the software as expected and based on the preliminary requirements and specifications.

Now, finding defects in any application is probably the main task for any tester, due to the importance of this task, the tester must cover two main aspects:

The Positive side
Test the application against the preliminary requirements and specifications, this tests will determine if the application performs as expected.

The Negative side
While the positive side of testing is crucial and important, the negative side is sometimes even more important, during this tests, we will validate that the application can handle failures or any abnormal situations that can cause unnecessary risks or affect the requirements and specifications.

Covering these two aspects is mandatory for the success of any testing process, a good tester will always aim to combine both positive and Negative test scenarios to make sure that the user can use the software without any risk.

So what is “Negative” testing anyway?

Negative testing is a testing process where the tester will use an invalid data as an input to the application functionalities. During this tests, the tester will validate whether the application can handle this type of inputs and if the application behaves as expected and defined in the preliminary requirements and specifications. 

Goals of Negative testing

It is very important to understand the real purpose of negative testing, and why you must combine them in any testing process, this list will allow you to understand what negative testing is all about:
  1. Validate that the application will not do anything that it is not supposed to do.
  2. Validate how the software will recover from failures that caused by invalid data.
  3. To determine under which scenarios the application can crash.
  4. Identification of the weaknesses points of the application.
  5. Validate that the application can abnormal situations where the user uses an invalid input (Notifications, Response time, Errors, Logs….).
  6. Validate that the different aspects that are not documented in the positive requirements and specifications. 

Negative testing process

Knowledge Gathering
The tester will gather all information that is needed to design the negative test cases, during this activity, the tester should read the application documentation, review the application design and gather all information that he can use prior to design the test cases.

Test Design
Based on the information gathered in the first activity, the tester has the relevant information that he can now use to design the negative test scenarios including the testing data that he will use throughout the test process.

Test Execution
The tester will run the tests and report any incident that was found during this process.

Examine the Test Results
Once all tests are done, the test results are reviewed and analyzed to determine if further tests are needed. 

Advantages of Negative testing

  1. Negative testing will allow you to increase the confidence in the product, the quality of the application is always determined by both positive and negatives scenarios.
  2. Using negative testing, we can show that the application does not work.
  3. Negative testing will demand more creativity and thinking from the testers to think about test cases that are not specified in the basic requirements and specification.
  4. Once a negative scenario that caused the application to fail is fixed, we will remove one more risk that will increase the stabilization of the software.
  5. Negative testing will allow us to understand the limitations of the application.
  6. We can use Negative testing in all layers of testing (Unit, component, integration, system and End-to-End).
  7. Negative tests are used to cover many aspects that are not covered in the process of positive testing. 
  8. Negative testing will allow us to gracefully handle errors and notifications that may pop-up to the user. 

When to stop Negative Testing?

There is no “End” date to stop negative testing, similar to any other testing process, you will stop testing only when the risks are removed and there are no remaining issues that can affect the user experience and the stability of the application.

Another testing stopper that is well known is the project deadlines, although you may have thousands of test cases you will never succeed to run them all due to the time limitations and the release commitments, therefore you will need to prioritize your tests in order to find the most important issues in the specifically available time frame. 

Why you cannot ignore Negative testing?

If you read all paragraphs above, you probably understand that you cannot perform a test process that does not include both “Positive” and “Negative” testing, to enhance the importance of the negative side lets review few more examples:
  1. Negative testing will allow you to find problems that are unique for specific integration.
  2. Positive testing will not allow you to understand how external artifacts can affect the application functionality (Low Disk Space, Low Memory, Low Bandwidth, Service crushes etc.)
  3. As many exceptions that are found and remove, the better quality you achieve.
  4. Negative testing will allow us to improve the notifications in case of failures.
  5. If you fail to find a software exception, the system will be released as if no problem has been found, this can lead to a major issue on all system levels.
  6. Using negative testing, we will enforce the application guidelines and demands, these enforcements will prevent unnecessary and unexpected scenarios that may raise will the customer interacts with the application.
  7. Negative testing will allow you to stop the application execution in case of a failure until the problem is fixed.
  8. If you run only “Positive” scenarios, you will fail to understand what are the application Upper and lower limits.
  9. A small problem that we fail to find during the testing cycles can (and probably will) be accelerate and become a major issue. 

My Presentations