Beta testing is both essential and significant phase in any application life cycle, in this type of test, the company releases a “Beta” version of the app to a specific list of customers that will implement it on real production environments, once installed, the company users can work with the software and obtain a real feedback about the behavior, usability and quality of the application that will be used to improve the App.
Despite the fact that “Beta” is the official name for this testing method, there are a few more name alternatives that you can encounter:
- Customer acceptance testing (CAT).
- Customer Product Testing (CPT).
- User Acceptance Testing (UAT).
- Customer Validation(CV).
- Early release version.
- Pre-Release version.
- Beta Testing.
- Field Trials.
Types of Beta Tests
The application is distributed to a specific set of companies with a specific intention to allocate data
The application is distributed online; the company will use any feedback that returned from the field
The application is distributed in a specific market, with the intention to allocate data about a specific aspect of the application
The application is tested by the company employees (not Engineering) that will examine the application prior to release.
The application is distributed to a specific market, with the intention to allocate data about all aspects of the application
Beta testing flow
You may find different flows that companies use to run beta tests, most of them will include this simple steps:
- Select the customers that want to use the software as beta.
- Select the customers that can send the most effective feedbacks.
- Release the Beta version and validate that the customer gets the support he needed.
- Allocate the customer feedbacks.
- Transform the feedbacks into action items (FR, Defects and more).
- Prepare and distribute the Beta feedback results.
Why should you use beta test?
There are a few significant reasons that should motivate you to execute Beta test:
- The application is tested in real environments (Not a limited test environment...).
- The application is tested with multiple devices, architectures and configurations.
- Any returned feedback is important because it come from real user experience.
- The application is tested by the end users that does not have any limitations.
- Allowing customers to express what they really think about the app.
- The application is tested prior to the release to the general public.
- Cost effective (Testing is done without the in-house employees).
- Increase the customer’s satisfaction and confidence in the app.
- We can remove many risks that we had prior to this phase.
- Any defect/issue that found in beta will allow the company to make the fix without affecting the company reputation in the general public.
- The organization can ensure the release readiness.
When the tests should happen?
Beta testing is the final phase of testing and therefore it’s executed after the end of the thoroughly testing cycles (Alpha and QA test cycles), and prior to releasing the application to the general public as GA.
Simple: Alpha -> Beta -> GA
Who performs the beta test?
Beta tests are performed by external users on their own sites,these sites are not related to the in-house engineering team, therefore any feedback that you receive from them is priceless (Just remember that you cannot monitor the user “tests” at this phase, as a result the users can ignore defects or not report them as expected from in-house employees).
What happen next?
There are few obvious paths that the companies may use based on the collected feedbacks, in some cases the company will fill decide to fix issues (Only the critical once or deal breaker requests), but in other cases the feedbacks are translated into feature requests (FR) that will be implemented in future versions.
Entry Criteria for Beta Testing
- There is a mechanism that can allocate the user feedbacks and take decisions.
- You have customers that are willing to install the software as Beta.
- All tests are done in the regular testing cycles.
- There are no remaining critical/major bugs.
- There is a working build that used as Beta.
- Alpha tests are done and approved.
- Distribution team has the knowledge to release and mentor the customers about the product.
Exit Criteria for Beta Testing
- All customer feedbacks are reviewed and translated into action items.
- There are no remaining bugs (Major and above).
- Beta summary report is created and approved.
Common Beta Mistakes
- There is no real mechanism that can allocate the customer feedbacks.
- Ineffective communication between the company and the customers.
- The application is released without the approval of the Entry criteria.
- There is no support that the customer can use during the Beta test.
- Feedbacks are not translated into real action items.
- There are no specific timelines for the Beta period.
- The Version for release (Last Build) is not tested prior to release (Automation runs/Basic Sanity).