Tuesday, May 17, 2016

User Acceptance Testing Checklist | David Tzemach


After the application developed and approved by the engineering team (R&D/QA), the application is now available to be delivered to the customer for UAT that will help him establish confidence in the application prior to approve the purchase of the application.

The main goals of UAT

  • Allow the customer to validate that the system is compatible with his environment.
  • Allow the customer to gain confidence in the application.
  • Allow the customer to validate that the application is developed based on his preliminary requirements and specifications.

Who can run the UAT?

There are a few common scenarios that will work:

  • The client with a predefined list of tests that were provided by the company.
  • The company users that get the application and send the feedback for review.
  • The UAT team executes the tests, the client approves the results.
  • The client with the assistance of the UAT team/support.

The basic phases of UAT process

  1. Planning and strategy.
  2. Build the ATP team.
  3. Design the ATP.
  4. ATP Execution and Documentation.
  5. Comparing of the actual Vs client expectation
  6. Defects Fixing and re-Testing (If Needed).
  7. ATP Sign-off.

Acceptance Test Review Checklist

Testing that are made to validate that the developed system meets the client requirements.

Phase 1: Entry Criteria (Examples)

  • The application code was fully developed and tested by engineering teams (R&D/QA).
  • All bugs are fixed and verified (Blockers, Critical and major).
  • The client business requirements are available.
  • The client environment is ready for tests.
  • Exit criteria for the testing process met.
  • Cosmetic/minor bugs can be approved.
  • All risks are examined and removed.
  • The testing cycles provide a full Code coverage (Beside the sections that were not tested as part of the Risk Assessment process).

Phase 2: Preliminary Investigation

  • Prepare a timetable that will contain the ATP (ATP phases, ATP Review, ATP sign-off etc.).
  • What is the testing level that the user asks to run prior to approve the application
  • Prepare time estimations for releasing the ATP.
  • In any case that the user will test the system without the team, is there any Step-by-step test instruction that the client can use?
  • Who will validate that the client has the relevant systems to support the testing process (System requirements and specifications)?
  • Have you defined the process for reporting errors or any other issues that the client may have during the testing process?
  • Is there any other 3rd party applications that the client need to install prior to the implementation of the application?

Phase 3: UAT Team

  • Make sure that the UAT team understands the responsibilities during the UAT process.
  • Validate that the UAT will verify the performance on business main process.
  • Ensure that the UAT team will document every input/results of a test case.
  • Make sure that the UAT team understands the expectations of the client.
  • Determine the people that will be part of the UAT team.
  • Determine test team training (if required).
  • Make sure that the UAT team understands what are the tests to be executed and what are the expected outcomes that will satisfy the client.
  • Make sure that the UAT team has the knowledge to take any required action for each defect that was identified during the test execution.
  • Validate that the UAT team has a support team that can advise and resolve any issue during the process of test execution.

Phase 4: Test preparation

  • Validate that the test plan includes the main Use cases that were requested by the client.
  • Make sure that the test plan covers all possible interactions with the client environment.
  • Has acceptance criteria was defined regarding the complication of the test plan?
  • Validate that the test plan is compatible with the client time expectations.
  • Has the test plan for the UAT process was sent to review?
  • What tests will be executed manually/automatically?
  • Has the test plan for the UAT process was approved?
  • Validate that the team agreed to the test cases that are intended to run on client environment can cover the main aspects of the application.
  • Validate that the test plan documentation is good enough, so both the stockholders and the client can review and understand.

Phase 5: Test Execution & Test Analysis

  • Validate that the team will execute the tests without a deviation from the original test plan.
  • Validate that the application is not causing any downtime/Failures on client site.
  • Validate that all phases of the testing process documented.
  • Validate that the test results reviewed by the client.
  • Validate that all defects identified and resolved.
  • Validate that the test results are documented.
  • Validate that all defects are documented.
  • Validate that you analyze the testing process (What was wrong? How can you improve the future process?).

Friday, May 6, 2016

Guidelines for estimating project effort | David Tzemach


I really don't know what is your experience is, but when it comes to estimate a work effort, I can say that based on my experience I saw some really sketchy estimation processes that let me think that some manager that was involved in the decision making where to estimate the work based on a pure guess that ignores some major considerations like available resources, work effort, the skill of the team and few more considerations that we will review later.

When you have a manager that estimates a work effort without calculating all the factors, it will probably lead to a known outcome of a team that fails to accomplish the work that is expected from them during the time frame that was given by that manager.

In this article, I want to discuss a few considerations and guidelines that you can use to determine the work estimations that you can stand behind and be sure that you can really accomplish the work during this time.

Do Estimates are really necessary...?  

It’s a Rhetorical question, but Prior to reviewing my guidelines, you first need to understand why work estimations are a critical phase of any project.

Work estimations allow the team/managers and the Organization to understand when the work effort that is needed to be done, this information has a direct effect on the company releases and commitment against the customers.

Think about a company that has a great product, but they cannot commit to a specific time where they can deliver it, now as a customer, so you are really going to pay for it? I really don’t think so. 


Think about a scenario where a single company has 10 projects that they need to accomplish during a specific time frame. With a work estimation that is made per project, the decision makers can answer these critical questions: 
  • Which project are we going to start first?
  • How each project is affecting the other projects?
  • Can we start multiple projects at the same time (parallel execution)?
  • Do we have a project that we cannot accomplish throughout this time?
  • How can we create the most efficient plan to accomplish the most valuable projects first? 

Return of investment (ROI)
In the best scenario, the organization can estimate the project ROI even prior to the start of the project, this will allow the decision makers to understand if the ROI is worth the effort.

Resource allocation
The project resource allocation has a direct impact on the work estimations, it depends on the number of the available resources, their quality, and experience. This allows the decision makers to change the team to reduce the original estimations (Add more resources, replace people based on their knowledge…).

Commitment and Size
As a project manager, you need to ask yourself this simple question, "The work can be done or not?" Based on a project time estimation, we can determine the Size of a project and decide if we want/can commit to it. If the project is just too big for the company to handle they will probably don’t commit to it. 

Estimation red flags

Estimation is made without calculating all factors
When you are calculating an estimation of a work effort, you need to examine all factors that may affect the project timelines, these factors are related to different aspects of the project, just for example: 

  • Are there any dependencies on another teams/external support?
  • What are the resources that we have to accomplish the task?
  • What is the experience and knowledge of those resources?
  • What is the buffer that we have in case of delay?
  • What is the development and testing strategy?
  • What is the urgency of this project? 

Estimation is made with a single authority
In some cases, I saw a work estimation that was determined by a team manager, that's fine and probably the expected way in most cases, but in other scenarios, these managers determined some unrealistic timelines that jeopardize the entire project, so what can go wrong when a single manager determines the work effort?

  • A manager that does not consult with the team that is going to do the actual work.
  • A manager that knows the expertise of the team but does not have the experience in the new technology that is going to be in use during the implementation process.
  • A manager that thinks he knows everything and ignores other people experience.
  • A manager that does not understand all factors that involve in the process.
  • A manager that fails to understand/predict the project risks.
  • A manager that wants to impress other people without thinking about the team (remember the Titanic…).

The "Expert" estimation
In this scenario, the work effort is determined by an expert/System architect that has a flawless knowledge in the system, but not on the team and the people who will do the actual work.

During my career, I saw experts that estimate the work effort without even knowing who is the executing team will be, and the time estimation was based on the Expert view that has all the experience, but without knowing that the team is built from a group of junior engineers that don't have the same knowledge that he has. 

The Historical performance
A big no-no! You should always look at the past and learn from it, in our case this approach is critical to the accuracy of the estimations, the past can reveal a great information that you can use, examples:

  • Work estimation that was given on for previous projects.
  • The Performance and capabilities of the team members.
  • Projects that were failed due to miscalculated risks. 

My Guidelines for estimating work effort

In this section, I will review a list of guidelines that I used to estimate a work effort, these guidelines have been great for me for many major projects so I'm 100% positive that you will find them useful during your estimation process.

Determine how accurate your estimate needs to be
Prior to estimating a work effort of a project, you need to ask yourself how accurate the estimate need to be. The answer to this question will determine how long does it take to determine the work effort (Accurate -> More details -> More time to estimate).

Task and Work breakdown
It’s a basic thing, you must understand the tasks that are needed to be done to accomplish the targets of the project, during the process of the task breakdown you will gain these benefits: 

  1. You will have the ability to determine the work effort per small tasks which make your estimations more accurate.
  2. You will see the barriers and Risks of the project; this information will help you during the planning process.
  3. You will get the full information regarding the direct and indirect work that needs to be done to accomplish each task.
  4. The work effort is determined by Hours and days instead of weeks and months. 

Specialist working hours
In any case that you have a specialist (usually as an outsource) that involved in the project, you should calculate the work hours while calculating these factors:

  • How the specialist affects the overall hours of the project?
  • How many hours the specialist will take an active part?
  • Is there any training that needs to be done?

Estimate the activities that are not directly related to the implementation process
During my career, I saw project owners that estimate the work effort while ignoring a set of activities that have a major effect on the project timeline, among these activities I can think about:
  • Sleekness and vacation days of the team.
  • Preparation of the Project Environment.
  • Creation of the work Reports.
  • Training hours for the team.
  • Rework that has to be done.
  • Project management time.
  • Review with other teams.
  • Review meetings.
  • Project buffers.
  • Training hours. 

Trust the history
When estimating a project effort, you should always look at the past history, the past will help you understand what is possible and what is not, in addition, the past history can teach you how to overcome some barriers that already handled before.

When I need to determine a work effort, I will always use the experience that I gathered at previous projects to support the estimations that I need to provide now, my guidelines are:

  • What mistakes were done in the older projects that affected the timelines?
  • I always compare my team to a project that was made by a team that possesses the same knowledge, education, and experience.
  • What was the technology that is used in older projects, is there a new technology that we can use to reduce the current timelines?
  • What was the buffer that the team consumed?
  • The best comparing that you can do, will be against a project that was similar to yours (scale, time frame and available resources).

Estimate with the people that actually do the work
The work estimations should always involve the people that are going to do the actual work, you cannot estimate project timelines without consulting with the team.

When I need to determine a work effort, I always consult with my team, this brainstorming provides the following benefits:
  • The team can see technical barriers that are hidden.
  • The team understands the scale of the project.
  • The commitment of the team is increased.
  • Estimations are more accurate.
Document your estimates
I saw too many project owners that provide time estimations without the ability to answer the basic question of "How did you determine this estimation...?" well in my opinion, you control the estimations only when you have the ability to explain how you arrived at them.

Therefore, I always document my time estimation with the calculation that I used to determine them, this basic rule allows me to answer the question above at any time prior, during and when the project has finished.

Always include the "Out of scope" tasks
Sometimes people estimate the work effort of the project but fail to deliver the sections that are not covered in the project. Therefore, always add the "out of scope" tasks that are not included in the estimations that you provide.

Add a buffer factor
No matter how good your estimates are, there is always a chance that you will not foresee some problems that have a good chance to occur, to overcome this issue, you will need to add a "Buffer" factor that will allow your team to work without a major pressure that may arise and affect the original estimations.

Trust your experienced people
You should always use the experience of your people, their experience will help you in the estimation process, these people have known that sometimes can add another point of view that may change the time estimations that you thought until this moment.

Have different people estimations
Another way to estimate a work effort is to ask different people to estimate the same work and compare the time estimations.   

My Presentations