Tuesday, July 25, 2017

Top 10 Benefits of automated testing | David Tzemach

תוצאת תמונה עבור ‪best quotes about software testing‬‏

 

Execution Time

The creation of the test scripts may take some time, but once done, we can execute them faster than any human being that will try to execute them manually.

In addition, the tests can run at night and 24/7 via schedule, this is a Hugh advantage because it does not consume the daily work hours of the resources. 

Quick and detailed feedback

One biggest advantage of carrying out automated testing is the ability to get a quick and detailed feedback on the status of the application at any given time.

This capability is crucial to any development methodology but especially on Agile teams where the code is modified numerous times a day and can be validated with a simple test run.

No more Regression tests

let’s think about a testing process that does not include any automated tests, where testers perform the same regression tests on any testing cycle and version. What is the benefit? None, Testers become frustrated, the testing time is consumed and there is no thinking about advanced and complex test scenarios.

Now, let’s think about a testing process that includes automated tests, What the benefit? Testers will no longer execute regression tests, they will gain more time to improve the test design and focus on exploratory testing.

 

Automation on all levels of the application development

There are few mandatory phases involved in any process of application development, using automation frameworks, we can automate each phase, which will help us to find defects sooner (Both Functional and Logical).

To demonstrate this point, I created this table that demonstrates how automated tests can affect the quality per development level.

Development & Testing levels
Description
How? 
Unit
We will test each function with unit tests
Automated tests
Component
Tests are done when the component is tested as “Black-Box” with preliminary inputs and expected results.
Automated tests
Integration and Interface
We will test the integration and interfaces among two or more components, this is can be done using Stubs and Drivers.
Automated tests
System
The entire system is tested on isolated environment when all components are integrated together
Automated tests
End To End
The application is tested with external artifacts (File Servers, Networks, Etc.)
Automated tests
Manual
The human tester will run any tests that are not automated on the previous levels
Manual execution by testers

Continuous integration(CI)

Using automated tools we can implement a process of “Continuous integration” where each code modification is tested (Automatically J) before any check-in to the main branch. Implementing such process will increase the effectiveness of engineering in the following ways:
  • CI detects problems early, therefore any defects will have a smaller impact.
  • Developers receive a fast feedback about the status of the application.
  • Developers can submit their code multiple times a day.
  • The code is tested before conducting a new build.

Reduce the testing Costs

Yes, carrying out an automation process is expensive, but if you think about the full picture, the overall advantages will reduce the overall costs compared to the same process that conducts manually.

Examples:
  • The testing Resources can be reduced after the initial implementation.
  • The entire development process becomes more efficient.
  • The manual testing effort is reduced
  • Regression effort is narrowed. 

 

Shared Responsibility

Automated tests are can be written by both developers and QA engineers, therefore, we can split the responsibility between the two sides that will increase the responsibility and contribution of the development team.

 

Test Coverage

Once built, the testing coverage is increased dramatically, Just think about each test case that you automated, you can run the same test(Simultaneity) on many supported environments, Browsers, and hardware that you just cannot run manually due to narrow timelines.

 

Reduce the Human resources

After the first implementation, you will no longer need the large number of manual testers that will run the same tests over and over again.

 

Reliability

Automated tests are created from a test scripts that will follow the same steps over and over again, this is far more reliable than a human tester that can skip or ignore one step which will lead to a different result. 

Thursday, July 20, 2017

How to write an effective Test summary report? | David Tzemach

תוצאת תמונה עבור ‪test results‬‏

Any testing effort contains several documents prepared as part of the testing process, each document is prepared to answer a specific request/demand during the software development life cycle such as:
  • Software Requirements specification (SRS)
  • Software High Level Design (HLD).
  • Risk Management Plan (SRM).
  • Software Test Design (STD)
  • Software Test Plan (STP)

For more examples, please use this link containing a great PPT that summarizes the entire list of test documentation:


What is a test summary report?

As you probably know, the SDLC contains many phases including testing which is used to determine the quality of the application before releasing it to the company clients.

The testing phase contains many activities such as test planning and test execution which have their own specific documentation. 

Once those phases are completed and the testing team approves the quality of the application, they will need to summarize the details, activities and any problems they encountered during the testing process and present it to the senior management, project owners and sometimes even to the company clients.

What sections should be included in a Test Summary Report?

Before examining the content of this report, know that each company may have a different test report format and content, but in general, each report must include a list of mandatory sections and data that I will review in the following paragraphs (Please note that you can change the order of artifacts as you wish). 


#1 –Title

The title should include an informative description about the project name.

#2 – Unique Identifier

Each test report should include an unique identifier that associated to each particular testing cycle, using this identifier we will increase the tractability once needed.

#3 - Objective

This section is used to describe and explain the main activities used during the testing process, besides a short description of what type of testing was conduct (Unit, System, Etc.).

#4 – Description of the Application

Under this section, we should add a very short description of the application that was tested, the main goal here is to let the readers a short review about the application complexity, goals and the business case behind it. 

#5 – References

Add any important links to the documentation used in the testing project and can be used to support this report, the mandatory references are:
  • Test plan used during the testing process (STD / STP).
  • Software requirements specifications (SRS).
  • High-level Design (HLD) of the application.

#6 – Resource allocation  

Add a short description of the resources what were involved in the testing process, the description should include the following details:
  • Name and Title
  • Area of responsibility.
  • Time spent during the project (house per Day/week).
  • Planned vs unplanned resource modification during the project.

#7 – The application Testing Scope

This section summarizes the Modules/features that were tested/untested by the testing teams, the general idea is to let the readers understand which areas are covered in the testing process and which are not.

When you create this section, think about the next bullets:
  • The integration among the application components.
  • The application individual components.
  • If a module is not tested, add the cause and reasons. 
  • Main flows that were tested.

#8 –  Testing Activities

Define the testing activities and methodologies that were used during the testing process (It will be more efficient if you include a basic information about each activity). 

#Testing Methodologies

  • Scrum.
  • Waterfall.
  • Extreme Programming (EP)

#Testing Activities (In-Scope)

  • Functional testing.
  • Exploratory testing.
  • Usability testing.
  • Component and unit testing
  • Load and Scale testing.

#Testing Activities (Out of Scope)

  • Negative testing.
  • Security testing.
  • Integration testing.
  • Endurance testing.
  • Stress Testing.

#Test Coverage

  • How many test cases are successfully executed?
  • How many test cases are were failed?
  • Describe the total number of tests that where planned at the beginning of the project against the actual tests that executed.

#Risks

  • Determine the modules that were excluded from testing.
  • Determine the modules with higher risk.
  • Determine the modules with low risk.


#9 – Defects Statistics and related information

This is the place to provide some statistics about the number of defects found during the testing effort, the information should help the readers to understand the test execution results.

Make sure that your report covers the following statistics:
  • The Number of defects divided based on their severity (Blocker, Critical, Major etc.).
  • A Total number of defects found during the version and per each testing cycle.
  • Defects that were not fixed in the current version (Deferred to future version).
  • How many bugs are not resolved and deferred to future versions.
  • The Total number of bugs found by automation/manual testing.
  • List of the most important bugs found during the version.
  • Defects per application module (Defect patterns).
  • The Number of defects divided based on their statuses (Invalid, Duplicate, Cannot Reproduce, Etc..).


#10 – Automation 

Add a short description about the automation framework used during the tests, the information should include:
  • The number of defects found using the automation framework.
  • A general Description of the automation framework.
  • The effort invested in creating the automation tests.
  • The number execution hours per automation.
  • The test coverage of the automation.

#11 – Testing Tools

Add a short description about the 3rd party / In-house tools used during the tests.


#12 – Test Environments

This area should include the test environments used by the testing team during the testing process, the information should include the following:
  • Details about the environment configurations.
  • The environment availability during the project.
  • Details about any special configuration.
  • Details about the operating systems.

#13 – Definition and Approval of the Exit criteria

Description of the testing exit criteria that were defined at the beginning at the testing project, the main goal is to be able to determine if all conditions are fulfilled. Example:
  • All bugs with Major and above severity are fixed and verified – Approved/Declined.
  • 90% from all tests are moved to automation framework – Approved/Declined.
  • All planned test cases are executed – Approved/Declined.
  • All important modules should be tested – Approved/Declined.
  • All major Risks are removed -– Approved/Declined.
  • There is a 90% code coverage– Approved/Declined.
  • All development activities are done and no more testing is required – Approved/Declined.

Monday, July 17, 2017

Security and Penetration test checklist for Web-Applications | David Tzemach

During the testing process of any application (including Web-Based application), we must run tests that will help us to understand how well the application is secured before release it to the market, this phase of testing, will allow us to determine the vulnerabilities, Security Gaps and to determine how well do we keep the confidential data of the tested application.

In this post, we will review the main points and consideration that you should follow when executing this kind of tests, please make sure that you adjust this checklist for your own needs and based on the testing effort that is needed. 


This testing levels will include different testing methods, such as:
  • Authentication tests.
  • Penetration testing.
  • Brute Force testing.
  • SQL injection tests.

Checklist

General

  • Validate that there is no sensitive data that is stored as a record in the registry.
  • Validate that any security issue is documented in a corresponding log file.
  • Validate that the site will run on a specific list of browsers (specified by the user), this list should not support old and insecure browser versions.
  • Validate that the security logs are written and maintained with the relevant access permission.

A Browser related tests

  • Validate that the user will not see any sensitive/encrypted data in the web page URL.
  • Validate that the user cannot manipulate the URL of the site with invalid Attributes.
  • Validate that the “View source code” function does not reveal sensitive data.
  • Validate that all data that stored in the browser cookie is encrypted.
  • Validate that all secured pages are configured to use HTTPS protocol (instead of HTTP).

Malicious and 3rd part attacks

  • Validate the resilience of the user passwords in case of “Guessing” attacks.
  • Validate that your site can handle Simple Object Access Protocol (SOAP).
  • Validate that the network traffic between the client/Server is secured.
  • Validate the system can handle Denial of Service attacks (DOS).
  • Validate the system can handle Document Object Model (DMM).
  • Validate the system can handle brute Force attacks.
  • Validate the system can handle XPath injection.
  • Validate the site against HTTP header injections.
  • Validate that your site can handle script attacks.
  • Run SQL injection attacks.


Security of the site host (Back End server)

  • Validate that there is no authentication information that is hard coded in the site.
  • Validate that the server is configured to run with the latest security updates.
  • Validate that the communication between the client/Server is secured with the relevant certificate (in any case that the site is using this authentication method).
  • In any case of a failure (Client/server Side), you must validate that the information that displayed to the customer will not reveal the back end server information or any other sensitive data (404 error page will be just fine).
  • Validate that the Back End server will decline files with a potential to cause damage (exe/bat).


Authentication and authorization test scenarios

  • Validate that the authentication fields do not allow the auto complete mechanism.
  • Validate that the user answers the security questions before restarting his password.
  • Validate that the authentication process is performed with an encrypted channel.
  • Validate the authentication process that uses “impersonation” method.
  • Validate that every refresh of the site will trigger a new Captcha code.
  • Validate that the security answers are not saved in DB as plain text.
  • Validate that the data that entered in the password field is masked.
  • Validate that the user authentication data is not stored in cookies.
  • Validate that the session tokens are transmitted in secure channels.
  • Validate that the user authentication password is created based on predefined quality rules (Complexity, length etc.).
  • Login access should be prohibiting when the user exceeded the number (usually 2-3) of unsuccessful attempts.
  • Validate that when the user lost/change a current password, he cannot access the site with the old pass.
  • Validate that the user can perform operations on the site based on the Role and permission rights.
  • Validate that the “Reset” password function is working when the user as lost is credentials.
  • Validate that your site contains Captcha validation so the spam bots do not spam your site.


Site Runtime session

  • Validate that there is no trace of the user credential when he logout from the system.
  • Validate that the cookie session is terminated in a defined time frame/log out.
  • Validate that the security policies are enforced.
  • Validate that in any case that the user logged out of the site, he cannot navigate the site without re-authentication.
  • Validate that security of the site when the user is moving from Secure to insecure pages.

Input Fields

  • Validate that the user cannot overwrite the application files.
  • Validate that the user cannot upload Folders (Files only).
  • Validate that the site can handle empty inputs.
  • Validate that the site can handle partial inputs.
  • Validate that there is a filter between the Client/Back-end servers that filter any malicious files that are uploaded by the user.
  • Validate that the site can handle a malicious attack attempts on input fields (Usually via Scripts/HTML Tags).

My Presentations