Friday, July 17, 2015

Installation Testing - The ultimate checklist


Overview

This is the second article regarding “installation” testing, in the first article (Add Link) I explained the importance of this testing method and the different testing aspects that involve during the test execution.

In this article, I will review some basic and more advanced test cases/testing ideas that should help you to achieve more from this testing method, please take a long breath because this list is a massive one.


Group 1 – The Environmental Parameters

Disk Space


  • Validate that the required “Minimum” Disk Space is the accurate one.
  • Validate that the application can calculate and report the needed Disk Space at the prerequisite step (Before we will start the actual implementation).
  • Install the application when the Disk space is ‘0’.
  • Validate that the installer can handle situations that require more Disk Space during the installation process.
  • Install the application when the Disk Space is reduced to ‘0’ in the middle of the installation process.
  • Etc.

Registry

  • Validate that the installation/Uninstallation process does not cause a registry corruption.
  • Validate that the registry contains the relevant keys with the suitable configuration.
  • Validate that all Keys are removed from the registry when the software is removed from the system.
  • Etc.

File System


  • Validate that the File system structure contains the relevant application files (After installation).
  • Validate that the application is installed in the default location (Based on Requirements).
  • Validate that each file on the File System contain the relevant version number.
  • Validate that all the TMP files that we use during the implementation process are located in a dedicated folder (Usually the operating system ‘temp’ folder).
  • Validate that the TMP files that are used in the implementation process are removed (Cab Extraction, TMP Files Etc.)
  • Validate that the application can be installed on another path that the default location (Based on user configuration).
  • Validate that the File system structure doesn’t contain the application files (After Uninstallation).
  • Etc.


CPU/RAM


  • Validate that the installation/Uninstallation process doesn’t cause any memory leaks.
  • Validate that the installation/Uninstallation process is performed without heavily consuming the server CPU/RAM.
  • Validate that the installation/Uninstallation process release the consumed resources after the process ends.
  • Etc.

Services


  • Validate that the application service is up and running after implementation.
  • Validate that the service “Startup Type” is set with the predefined requirements.
  • Validate that the service is configured with all the associated dependencies.
  • Validate that the service is displayed with a valid Name and Description.
  • Validate that the service can communicate with the application console.
  • Validate that the service installed with the predefined credentials.
 

Group 2 – Implementation & Removal flows

Implementation Flow:


  • Validate that you can implement the product/Components with automated tool (If supported).
  • Validate that you can implement a specific components, while using the API interface.
  • Validate that you can implement the product when older version is already installed.
  • Validate that the installer is capable to repair an existing corrupted installation.
  • Validate that the user cannot open two instances of the installation window.
  • Validate that you can implement the product while using the API interface.
  • Validate that you can implement the product/Components manually.
  • Validate that you can implement the product on a clean environment.
  • Validate that the installer can identify an existing installation.
  • Delete some mandatory installation files and run a repair.
  • Validate that you can install a specific Update.
  • Validate that you can install a specific patch.
  • Validate that you can implement the product with distributed configuration (All components are located on different servers).
  • Validate that you can implement the product with consolidated configuration (All components are located on a single server).
  • Etc.


Removal Flow


  • Validate that you can uninstall the product when it was installed on distributed architectures.
  • Validate that you can remove the software while using the system Add/Remove feature.
  • Validate that you can uninstall the product when it was installed on consolidate server.
  • Validate that you can uninstall a specific component.
  • Validate that you can uninstall a specific Update.
  • Validate that you can uninstall a specific patch.
  • Validate that you can remove the software while using the “Uninstall” file that’s located under the path of the installed software.
  • Etc.

Agent (If Any):


  • Validate that the agent doesn’t affect other 3rd party software that configured on the server.
  • Validate that the communication between the agent and the corresponding HUB.
  • Validate that the agent reports to the corresponding Event Viewer container.
  • Validate that you can implement the agent on different operating Systems.
  • Validate that you can implement the agent on different platforms.
  • Validate that the agent can start is job after implementation.
  • Validate that the agent doesn’t cause any server errors.
  • Etc.


Group 3 – General Guidelines for effective test design   


  • Make sure that you design your tests to support both the negative and positive test scenarios.
  • Make sure that you design your tests based on the predefined requirements.
  • Make sure that you design your tests to support as many End to End scenarios as possible, the End to End testing, should be able to validate that all work flows are covered.
  • Make sure that you design your tests to validate all notification/Error popups that may raise during the installation/Uninstallation process.
  • Make sure that you design your tests to support any case of force stop during the installation/Uninstallation process.
  • Make sure that you design your tests with the idea that different localizations can affect the implementation process.
  • Make sure that you have a valid trace regarding the installation process (Dedicated Logs, EventVwr Etc.)
  • Etc.

Group 4 – User Experience


  • Validate that the user receives a summary of the installation/Uninstallation process.
  • Validate that the user can install the application without breaking is teeth.
  • Validate that the user can follow the installation instructions.
  • Validate that the user receives a confirmation notification on critical/complex operations that may affect the software/User environment.
  • Validate that the user has the ability to track the installation/Uninstallation process (Progress Bar, Counter, graphical interface Etc.).
  • Validate that the user receives and informative popups when he try to perform invalid operations.
  • Etc.

Group 5 – Networking and Privileges 


  • Try to install the software on distributed servers and block the communication port.
  • Try to use different user on different components (Impersonation Validation).
  • Try to install the software on a protected/No permission folder.
  • Install the product with the Minimum privileges as possible.
  • Try to install the product with invalid user credentials.
  • Try to install the software on distributed servers when there is a low bandwidth that cause communication delays during the installation process.
  • Try to install the product when the user is disabled.
  • Try to install the software, when the Firewall is on.
  • Etc.
 

Group 6 – API (If any)


  • Verify that each method that should return values reproduced the expected results.
  • Verify that all common operations that exposed to the client are functional.
  • Verify that all objects, functions and properties are tested.
  • Verify that all Internal/External APIs are
  • Verify that all objects, functions and properties are reviewed and approved by the project owner.
  • Etc.

Group 7 – Documentation


  • What are the minimum requirements that we need to apply before implementation can start.
  • What are the supported devices can we use to run the application (PC, Phones Etc.)?
  • What are the supported operating Systems that we need to use in our tests?
  • Is there any 3rd party software that should reside with the installed software?
  • Is there any known issues that may reduce the productivity of the software?
  • How can we installed the application (Consolidate, Distributed Etc.)?
  • Do we support both X86 and X64 architectures (Hardware)?
  • Do we support both X86 and X64 architectures (Software)?
  • What are the supported Web Browsers that we can use?
  • Etc.

Group 8 – Licensing


  • Validate that the user cannot install the software without a valid license.
  • Validate that the user can only install components that are licensed.
  • Validate that the user can install a new license for expired product.
  • Try to install components that are no licensed.
  • Validate that the license is visible/Encrypted after implementation (Depends on the company policy).