Wednesday, January 15, 2014

STP (Software Test Planning) – Part 3

Section 4: Testing methodologies
Every testing process should base on testing methodology, this section should explain the one we are going to use on current testing process.

Main Bullets:
  • Selected Methodology – describe the testing methodology (Waterfall, Agile-Model…).
  • Benefits – Describe the benefits we are going to achieve when using it.
  • Flow – Display a short flow for the methodology implementation. 


Low detailed Examples:
V model:
Requirements - > STP - > STD->Unit testing - > Integration testing….

Waterfall:
Requirements -> Design-> Implementation->Verification…..

Note!
My example contains the headers, you should add additional information per section. 

Section 5: Detailed information regarding your test strategy

These sections should provide detailed information regarding the test strategy we are going to use in the testing process, this strategy should be based on the selected methodology, Available resources (Human, Hardware and more), limitations and requirements.

In a very simple words, this section should describe how the testing process will confirm the actual results against the software requirements (SRS).

For my opinion and based on my actual experience, this section should contain the followings bullets: 

Part 1: Documentations
Every Quality Assurance process contains few mandatory docs (Requirements, STP, STD and STR), this section describe the relevant details for each doc type.

Example: 

Document
Each doc must Contain :
Owner
Reviewer
SRS
Detailed information based on customer or business requirements
Rob
David
STP
I hope you understand it after reading current doc…
Dan
David
STD
Highly detailed testing scenarios based on STP, this doc includes the test case and the expected results based on Requirements.
Vera
Dana
STR
Written after the STD compilation (In some cases you can start written it while executing the STD), this doc must include the testing results, opened bugs, Risks, suggestions and more
Guy
May

Part 2: Separate the software components and determine the integrations between them
The first thing you should do is to ‘Separate’ the software components, 
it’s easier to control your testing process when you design it to multiple
components and not to one big elephant called ‘Software’,
using this strategy you can see the list of integration tests 
you need to test for different components.

Example:
Let’s give example for a bank account system:
  • Components: User interface, Login procedure, login engine, Database
  • Integrations: DB and Engine, UI and Engine, DB and UI…

Part 3: Determine the testing cycles
This part should provide the number of testing cycles involved on the testing process, each cycle should be described with the testing types and levels you are going to use, Example:

Cycle 1: Exploratory testing + validating the fulfil of the Entry criteria’s

Cycle 2: Functionality testing + Bug Verifications

Cycle 3: Regressions (Tester 1) + Non functionality tests (Tester 2)

Cycle 4: Sanity

Cycle 5: You got the point….


Part 4: Testing types involved in the testing procedure
This section will be used for describing the testing types we are going to use on the current testing procedure.

Example:
Unit Testing – performed by the development team before QA can start Integration tests.

Computability testing – software will be tested on Environment A and B, Operating Systems B and C…

System Tests – the tests will be made after all components and integration tests completed.

International Testing – the product will be tested on different OS language packs because it’s going to be installed worldwide.

Stress Tests – tests will be made on servers that possess massive resources, in addition, the tester will use tools A and B.

In Addition, This section is the place to describe also the testing methods that we cannot accomplished due to lack of resources.

Example:
Non functionality testing like Load, Performance and stress that you need specific Tools and Hardware to executing them.

Section 6: Completion criteria’s (Exit Criteria)
This section should specify the exact criteria for success,
if one of the criteria’s not accomplished 
the Risk become larger, Example:
  • All bugs with severity greater than ‘Normal’ verified and closed.
  • All test cases on STD with priority greater than one must be executed.
  • Client requirements covered.

Section 7: High level testing details 
You all know that STP is the Key Stone for a great STD.Therefore this section will hold the main areas you need to test and will be expend on STD documentation.
Main Purpose: 

  • All bullets will be used as guidelines when writing the STD
  • Owner can review the bullets and determine if the coverage is good enough.

Note!
In some companies this section called TRD (Test Requirement Document), and will be written as completely different doc.

Example:
We have new car for testing, I’ll provide few bullets for testing:


·       Functional Testing :
1.      Test the car switch (Electricity on half switch).
2.      Test the car switch (Engine ignition on full switch).
3.      Test the car switch when Power off.

·       Usability Testing:
1.      Test that the drive shift is smooth and easy to manipulate.
2.      Test the dashboard buttons location and size.
3.      Check if the rear camera is easy to use based on the wheel buttons.



STP (Software Test Planning) – Part 2

Your STP structure may be different per company, in some companies the STP may include different sections that not relevant to other companies,but must in the local QA process.

This section will provide my suggestion for the “Key Stones” that every STP must have, feel free to comment my suggestions I’m always eager to learn new ways to improve myself.




Section 1: Project Review
This section will cover the main bullets relevant for the current project, someone who read this section should be able to understand the main concept of this project and the way it’s going to be tested.

Example:

Introduction (Project) - Describe the project in few words (You can also describe the main features and benefits we gain when using STP docs).

Software Review - you should add detailed information regarding the tested software, I suggest few fields that cover this task:

  •  Description – Describe the software and the main functionalities.
  • Benefits – what’s the benefits for user when using this software.
  • Requirements – Describe the requirements that used as development guidelines.
  • Targets – What are the main targets for this software?


Document Structure – Here you should add a list of all sections (Timelines, owners, Testing Types, External tools, environment and more...).


Data Resources – add table of resources this STP based on, this table is very important when other owner/tester wants to check your data (SRS, DDD and more…).

Terminology – explain the term’ used on this document, not all readers can understand the meaning of SRS, DDD and TBD….

Example:

TBD
To be defined
STR
Software Test Result
STD
Software Test Description

Owners – add list of owners, other users who read this doc should know all employees involved in the testing procedure (Product manager, Developers, testers...).

Example:

Owner
Title
Omer .S.
DB Manager
Dan .e.
QA Manager

Timelines - you should add the timelines for this project, the following table demonstrate it perfectly:

Task
Review
Estimated Times
Entry Criteria
QA team should prepare list of tests executed by development team, in addition QA team will define set of criteria that must be accomplished before the software can be delivered from Dev. to QA.
Start Date :1/1/2013
End Date : 1/2/2013
STD
Detailed design for How we are going to test the software
Start Date :1/2/2013
End Date : 1/10/2013
Exploratory Testing
After Software delivered the QA team will execute Exploratory testing
Start Date :1/10/2013
End Date : 1/12/2013
Functional Testing
Execution for Functionality testing
****
Load Testing
Load Testing using External Testing Tools
****
Bug Verification
Closing all opened bugs
****
Sanity
Short sanity before Alpha
****
STR
Testing results
End Date : 2/1/2013

Section 2: Testing Environments 

This section describes the testing environments used to test the software, this section should be very detailed so everyone who wants to read this section can understand the testing architecture and the limitations we have on tests due to missing Hardware/Software for tests.

Example:

Testing Type
Environment
Detailed Information
Unit Testing
Developer Testing environment
All unit tests will be performed to check units integration before validating components integration (QA Side)
Integration Testing
Testing execution on QA environment
All tests performed on QA testing environment before starting System and Computability testing
Servers : Win 2012 ,Celera,  NTAP
compatibility Testing
QA testing environment
Different DB : Oracle, SQL ...
Different Web Browsers: Firefox, Explorer…
Alpha Testing
Company production
Installation made after all Exit criteria accomplished on last build



Testing Limitations:
Describe the limitations based on available resources, for example:
  • Load Testing cannot be performed due to lake on Cup’s and Storage limitations.
  • Compatibility testing on Exchange 2013 cannot be executed due to missing license.



Section 3: Testing Tools
This section should cover two important issues:
  • Relevant tools that we are going to use in the testing process.
  • Tools we need for testing and not available on current timelines.

This section is very important because testing tools are decisive part in almost every testing process, in addition this is the place to describe the limitations we have if some testing tools will not be available.

In addition, the project owner should understand how the testing tools can help the QA teams to reduce the risks and expand the testing coverage.

STP (Software Test Planning) – Part 1

Overview
The Software Test Planning (STP) is one of the most important documents when designing a software testing process, usually each project contains a single STP that based on information allocated from different resources (Reviews, PM, SRS and more).

STP should (Probably ‘must’ is the correct word…) contain all the relevant information regarding Test procedures for specific software, every STP must provide detailed information regarding the test process (Owners, Timelines, Requirements, budgets, procedures and many more…).

STP can be different per company (you probably get 100 different templates from 100 different companies…). Although, each STP must be based on few mandatory criteria’s that would be examined on the next few paragraphs.




STP – When should we start it..?
STP documentation is a big part on every software testing process, but the question is when should we start to write it..? 

Well,
The answer is very simple,
STP become relevant only after few criteria’s accomplished, the following flow will demonstrate this point: 

1.    Client should send a request for a software development based on specific requirements.

2.    After the client requests approved by the company owners (every company should make money…).

3.    Product management team (PM) prepare the software design Documentation (SRS) based on user requirements.

4.    Technical teams (Development, DB...) prepare a detailed Design documentation (DDD) based on SRS requirements.

5.     QA teams have the relevant documentations to start the test plan (STP), this step is dedicate to determine the “What” to test in the Quality procedure.

6.    Based on STP, QA teams prepare the actual test plan (How we are going to test the software).

7.    STR should be available after the testing procedure accomplished.


Note!
If you’re not familiar with STD/STR docs, I already started new posts that going to be available in the next few weeks.


STP – Allocating the data
As explained the STP is crucial for your testing process and will be used as bridge for other documentations that based on it. Therefore, the research must be highly effective and relevant resources must be investigated and used. 

The following resources should be used: 
Software Requirements Specification (SRS) – we can declare that SRS is probably the most important resource you should use, in other words we can say that SRS is “The” foundation stone for the entire life cycle (From design and until release).

The SRS Written by the company employs (PM could be great example) and based on the software requirements (usually come directly from the client).

The SRS must be detailed as possible because both development and QA teams will use it (Developers for development and QA for testing documentation (STP/STD...)).

Development– every development process contain unique documentation that handles the develop process, we can use it to understand exactly how the developers develop the software and probably discover high risk areas that could affect the entire testing approach.

Sales/Operations Team – they know the clients and all the week points, they may supply great information regarding main points that may affect the client experience.

Company knowledge base – it’s not something new, but the company KB may supply different point of view or different approach for testing. You can search information on Bug tracking system and learn about issues that may affect the current process or ask experienced manager that already worked on similar project and may add useful information.

The following tools may be used to allocate the data from your resources:
Personal reviews – schedule meetings with every engineer/other employs that relevant, in the reviews you must come with predefined Questions and document all the answers. In my personal experience I cannot start my STP before reviewing all team owners that relevant for my project.


Examples:

Product manager – provide the “key stones” for the entire project, pay attention to his words…

Database/ Software Engineer – different departments that involved in the development process, here you should find useful info regarding developing issues and weak points that may occur.

Product Manager – understand the priority, timelines, Teams involved and more.

Sales/ Support Personal – learn about your clients and what are they expecting from the end product, investigate older release issues, software recalls and every other detail that may affect the current process.



Team Meetings – from previous example we can see that every project contain multiple teams involved, each team has different project and different agendas, your reasonability is to make sure that all teams synchronized with each other so the info you get is the most relevant and accurate. Furthermore, in team meeting you always find new ideas that may affect the product design and your STP afterwards. 


        Personal Investigation - make your own investigation, to accomplish this task you can use your previous experience and KB, just for example:

  • Investigate your bug tracking system, search for relevant bugs that may reproduce on current version.
  • Search on older documentation for relevant test cases that may be used on current tests.
  • Think about previous projects conclusions (STR’s) and try to understand the failures.

Wednesday, January 8, 2014

All you need to know about Regression testing

Overview


The Problem we have is that the life cycle of a software is a very dynamic. Therefore, every part in code may change or new code may be added. According to this, you should know that every change has risks behind him. The risk occurs when new code modifications may affect other code that already tested.


The Solution for this issue is using “Regression Testing” that provide great solution and coverage that reducing the risks if performed with great design and thinking behind it.



The internet contains many different definitions for this testing type, but the main idea is to make sure that Functionality that already worked and tested by the QA team still works after the new code added. 

Author View:

The main point for this testing type that I hope you all understand at the end of this article is that “Regression Testing is crucial to Re build the confidence that new changes doesn’t affect the great work you did in your original test plan execution.

You must understand that every change in code contain risks, regression testing is what you need to do to reduce the risks and dedicate your time for the new code testing. 

Time to use

Well, based on my overview section you can understand that regression tests executed only after the main functionality is tested and approved, furthermore, the Regression tests based on previous tests that executed while the software originally tested.

When to use:

Well it’s easy, you use regression tests whenever you have risk that the original code that already tested may be  affected, the risk could be arrive from different modifications, Example:
  •           New Feature that added to an application.
  •           Bug Fixes that may cause additional influence
              on nearby code.
  •          Changes in original requirements.
  •          New Requirements added.
  •           New patches or updated.
  •           Multiple builds that based on specific version source.

Regression tests Flow

Regression testing is part of predefined Flow, to make regression tests you should understand and master this flow, the following steps will demonstrate my point when new feature is added:

Flow Steps

Predefined Steps:
  • All tests for a specific code are done (Functionality, System, GUI and more).
  • New Feature added, this feature effect the already tested code.


Regression Flow Start her:

Step 1: Analyze and understand the impact
This step is critical for the entire flow, you cannot create a good Regression tests if you skip it, few question you should answer: 

  • How the feature combined with the current code?
  • Which areas are most affected by the change?
  • How the new feature affect the Non-Usability part for this feature (Performance, Load, stress and more...)?
  • How this change effect predefined tests for the original code?
  • How the change effect the testing data of the original code test?
  • How this change effect the software documentation?
  • How this change effect the current automation flow?
  • How this change effect other components that not directly relevant for the change?


Step 2: Select the relevant testing strategy
After understanding the impact and risks, it’s now the time to select the regression testing strategy and testing guidelines, as you probably know you cannot run tests forever. Therefore, your regression tests are limited in time, Strategy examples:


·      Tests the entire code– This strategy is the best when you have the Time, Money and highly trained automation team that allowing you to run the entire test plan including Old and new tests. Practical? From those criteria’s you can understand that using this strategy is not practical (Tell me if you have such resources and you’re not working in google…).

·       Testing made on Highly Risk areas– this strategy will be used when the testing team identifies the areas that most likely affected from the change and probably contain the bigger risks.

·        Bugs, Bugs and Bugs– this testing strategy will hold all test cases that involving bugs, every case that found bug will be part of the regression testing.

Step 3: Creating the actual test plan
Very easy, after you understand and analyze the change (Step 1), selecting the testing strategy (Step 2) it’s now the time to create the actual test plan.

Step 4: Executing regression this steps
Execution can be accomplished in 3 different ways:

Manual testing – executing regression just like every other STD, a tester or group of testers will executing the test plan based on priorities. This execution type is the less preferred for me, I’m not familiar with testers who like to repeat previous tests every testing cycle and after each modification, it’s also have massive costs for the company(Time,Money , Resources.…).

Automating Testing – My preferred way, all tests running automatically without any help from the tester, you need to remember that once regression tests are build ,this test plan will run multiple times with small modifications , automation is simply the best way to executing them.

Personal view:
Use automation and more automation, don’t run those tests manually.

Automation and Manual testing – Combination of the previous execution types, and may be a great solution because some cases cannot be executed with automation.

Step 4: Reporting
The testing results per regression cycle should be reported and analyzed, according to failures we can add/Remove tests and gain the best efficiency per testing cycle.

Guidelines and suggestions

·        Prioritize (Guidelines) – regression tests may include thousands of test cases, in some situations you simply don’t have the time to execute them all. Therefore, you must prioritize the highly important test cases that without them the Exit criteria will be failed.

To control your prioritization simply add an additional column on your regression document, Example:
Priority 1 – Must be tested
Priority 2 – Not include any Critical/Showstoppers, Risk reduced if executed.
Priority 3 – Only if time allows


Personal View:
For me this step is the one that makes the differences between testers, prioritization must be made only after a tester really understand the test case and how it’ effect the total risk for the tested software (We all know that some testers will mark many cases with priority 1 just to make sure that tests will be executes to cover their a!@#.

·        Documentation (Guidelines) – just like every other testing types,  you must create a dedicated Testing document for regression testing, as you probably understand this Doc will be built from other test cases that already tested in the software testing life cycle.
Please remember that creating such doc should be easy when you really understand the targets you want to achieve, The Hard part is the maintenance that for most testers isn’t relevant after the creation and will never be modified when modifications should be made.

·       Regression per modification (Suggestion) – you’re a QA, part of your job is not to trust the developer that says to you “The change will not affect the code, nothing should be tested”, if you can, run regression tests per change.

·        Functional and Fields tests (Suggestion) – regression tests are simply great test method to get the most efficient coverage for functionality and fields.


·        All Areas that affected from the modifications(Suggestion)  – make sure that all areas that relevant for the change are covered, in addition, make sure that lower risk areas are not the main cases you use (Short sanity tests will be enough).
  • Boundary testing – regression tests are perfect for such tests, it’s also easy to accomplished when using automation (Inputs Vs. expected results) make sure your regression contains such tests for every field that may be affected by the change.

·       Cases that found bugs (Suggestion)   - Select the cases that failed on the original run and verified after they fixed by the R&D team.

·        Duplications (Suggestion) – make sure that every test is unique, regression execution is by default the longest testing type, believe me that duplication is not something you need.

Considerations when building Regression tests

When building a regression test plan there is some main bullets you should remember and follow, please make sure that this once are part of your considerations:


·        Coverage  - Regression tests are used to cover code (New Change + Old code) , you must consider the coverage you want to achieve and the resources you will spend to accomplish it, in some cases the cost are too large.

·        Costs, Costs and Costs – the one question you must answer is this one “How regression tests save time and money…?”  


·        Maintenance – creating great STD for regression is probably the easy part, the hard part is to make great maintenance, and you must make sure that you understand the impact of the maintenance and how hard it will be to do it.

My Presentations