Thursday, August 9, 2018

Cloud testing: the right way to do it | Supreme Agile

In this article, I will review some important insights on cloud computing. In order to achieve the most from this article, we need first to understand the cloud-computing concept, and how the cloud is different from any other infrastructure.

The basics of cloud computing

Cloud computing is the result of one of the biggest and most important revolutions we’ve witnessed in the software industry, the technology known as virtualization that has changed how organizations around the world manage their computing resources.

This advanced technology creates a completely new methodology of how organizations can share computer resources across multiple systems in order to reduce costs and deployment time, increase scalability, and facilitate the IT department in managing their infrastructure.  

The virtualization technology becomes even more important once evolved in the new form of cloud computing. Cloud computing is an internet-based platform that uses the virtualization technology and its various computing services like hardware, software and any other computer-related services that provide a total solution of resources based on demand across the internet. 

To summarize it, let us review the main points that you should keep in mind what is cloud computing:
  • Cloud computing is a general term for the delivery of software/services over the internet
  • Enables companies/people to consume on-demand resources, such as a virtual machine, storage, and applications.
  • Cloud allows access to services without the user need to have technical knowledge or control of supporting infrastructure
The cloud structure is based on three types of delivery models (aka components) that provide the “as a service” solution: 

Infrastructure as a service (IaaS)

This is the fundamental layer of the cloud solution. It focuses on physical resources such as computing services, networking, and data storage space. IaaS resources are usually billed on-demand based on customer usage.

  • Microsoft Azure
  • Google Engine (GCE)
  • Amazon Web Services (AWS)

Platform as a service (PaaS)

This is the second layer of the cloud solution, which provides organizations with a platform with the main advantage of removing their need to manage the underlying infrastructure. An organization that uses this layer will not need to worry about resource procurement, capacity planning (you can simply set it to grow dynamically as long as you have the budget) and the maintenance of both hardware and software.

  • Google App Engine (GAE)
  • Apprenda
  • Amazon E2C

Software as a service (SaaS)

The top and the most cost-effective layer of the cloud platform, which provides a complete product, is often referred to as an end-user application, run and managed by the service provider. In this layer, applications are available to the end users on demand via the internet. Using SaaS, a customer can access their applications without installing the software on a personal device (workstation/server). The entire processing effort is conducted on the vendor’s datacenter.

  • Salesforce
  • Google Apps
  • Gmail

Types of cloud

There are three types of available cloud formations: public, private and hybrid.

Services are available to everyone, resources are allocated and consumed dynamically as per the tenant request
Managed under the security restriction of a particular organization and available to this customer only
Mixture of both public and private clouds. The mixture depends on the organization's decision on what services to expose to all and what services they want to expose to specific users.
Owned and operated by a service provider
Owned and operated by the organization's IT
Allows IT organizations to become brokers of services

Very high
Very high

Depends on the security measures of the service provider
Most Secure, all storage is on-premises
Very secure, integration option add an additional layer of security

Connectivity over the Internet
Connectivity over the Internet, fiber and private network

Combination of both
Technical Difficulties

Technical knowledge required

You get the Basic setup but still, the knowledge of the subject is required.

Customers do Not need to worry about technicalities; The SaaS provider company handles everything.

Cloud concerns

Cloud is a great technology that has already started to change the industry and the way companies manage their data. However, the world is not perfect and there are still some concerns that we should take into consideration:

Interoperability – A universal set of standards and/or interfaces have not yet been defined, resulting in a significant risk of vendor lock-in.

Latency – All access to the cloud is done via the internet, introducing latency into every communication between the user and the provider.

Regulations – There are concerns in the cloud computing community over jurisdiction, data protection, fair information practices, and international data transfer—mainly for organizations that manage sensitive data.    
Reliability – Many existing cloud infrastructures leverage commodity hardware that is known to fail unexpectedly.

Resource control – The amount of control that the user has over the cloud provider and its resources varies greatly between providers.

Security – The main concern is data privacy: users do not have control or knowledge of where their data is being stored.

Cloud testing 

Cloud testing refers to testing of cloud resources (both hardware and software) that are available on demand. Cloud testing must be conducted to ensure that the product under test meets both its functional and non-functional requirements.

SaaS Software Development Lifecycle (SSDLC)

      Requirements - Gathering and prioritizing business needs/stories by the customer/PO for the product as well as capturing them in a central location.

Design - Building a technical blueprint of how the proposed system/feature/model will work. It includes elements such as system features, models, technical architecture, integration points, interfaces, UX, etc.

Development - This is the physical building and coding of the product’s features/model including database based on the design and requirements.

Testing - Verifying the feature/component of a product works as expected and meets all of the business requirements. It also includes writing test conditions and executing test scenarios.

Go-live & maintenance - Implementing the feature/component of the product in the production environment as well as the day to day maintenance of the application (including updates).

Types of cloud testing 

There are four different types of cloud-based testing. Each type has its own objectives.

Testing SaaS in a cloud (testing an application) - This type of testing is used to validate the quality of the application in the cloud. Functional and non-functional requirements of the particular application are verified.

Testing of a cloud - The cloud is tested as a whole entity and based on its functionality. This type of testing is used to validate the quality of the cloud from an external (end users) point of view (its capabilities and service features).

Testing inside a cloud (infrastructure testing) - This type of testing is carried out by the cloud vendor and checks the quality of a cloud from an internal view or feature, based on the internal infrastructure and capabilities of the cloud (e.g. automatic capabilities, security, management, monitoring).

Testing across clouds (services testing) - Testing an application is done over various clouds (private, public, and hybrid). It is based on application service requirements.

Cloud testing Enlivenments:

There are two types of cloud testing environments that can be used by development teams  for testing activities:
  1. A test lab that simulates a cloud-based environment, where the application is deployed and tested.
  2. A hybrid, public or private environment, where the application is deployed and tested as it will be available for the customers.  

Challenges of cloud testing

Quality control - How do we maintain quality products in an area that demands fast, high turnover of deliverables with no bugs? This is the world of cloud, which can be very complex for those who do not invest the time to learn it.

Data security and privacy - One of the biggest advantages of cloud infrastructure is multi-tenancy support. Although multi-tenancy support is great, there is still a major challenge to ensure that the customer’s data is not compromised, security standards are applied and the privacy-related regulations are enforced. 

Upgrades with a short notice period - Cloud providers give existing customers a very short notice prior to upgrades. This is a big problem when manually validating changes to your SaaS application and is another major consideration when thinking about conducting manual testing in cloud projects.

Data migration - Data migration, the process of moving customer data from one cloud provider to another. During this process, the risk increases dramatically because both providers involved must ensure that the data is migrated without losing any critical data.

Upgrade testing - Cloud testing’s biggest challenge is to ensure live upgrades do not influence the existing connected cloud users. Think about a multi-tenant environment that uses the same cloud environment, when the application is upgraded for a specific customer. Sound simple? Unfortunately, in some cases, the upgrade process may influence the user experience of the other tenants due to latency, networking issues, and their shared resources.

Bugs - Bugs are no longer isolated; once seen they can be seen by all and exploited.

Frequent releases - Frequent releases provide less time to run tests, less time for regressions and as a result more unexpected defects and higher risks.

Cloud testing vs. conventional testing

Test parameters
 Conventional testing
Cloud testing
High costs due to major
investment in hardware 
and software
Lower costs, payment per use of the cloud services.
Test environments
Test labs (pre-fixed and
configured test environment)
An open public test environment
with adjustable resources
Impact of bugs
Bug isolation and low visibility
 (per customer)
Each bug is a bug for everyone. Fixing a problem for one customer fixes it for everyone
Security tests
Tests are done based on server type and policy of the organization
Testing is done in the vendor’s cloud-based configuration.
Performance, load,
and scalability
Performed on a fixed, isolated
test environment
Performed on both real-time and virtual online test data.
Time to delivery
Internal software releases
once every 1-12 months
Internal software releases 
multiple times a week
(sometimes even more).
Monitoring and 
Reactive software monitoring
(downtime reported to
customers in hours, days )
Proactive software health-monitoring (downtime reported to customers in
seconds, preventive actions
taken at defined procedures)

The main types of testing performed in a cloud environment

During cloud testing, teams must validate that their tests cover both aspects of functional and non-functional testing. Let us review some of the more common test types that are part of a cloud testing project:

Disaster recovery (DR) testing – The cloud as a service must be available to customers at all time, therefore, it’s important that a replicated site will be available in case of critical failure. While executing DR tests, the team must ensure that the app can recover in case of a massive failure(restore to the last available point, no loss of data, minimum downtime, etc.).

Availability testing – This type of test is usually owned by the cloud vendor that ensures that the cloud is available to customers at all time without any downtime.

Capacity testing – This verifies that current and future hardware supports expected usage as determined by the specification of the product (such as adding or removing resources to or from a customer).

Multi-tenancy testing – This type of testing is very important in any cloud-testing strategy. During these tests, the cloud services are tested by multiple users from different tenants (each service can serve multiple customers). Testing must be performed to guarantee there are no security incidents such as access (control or data leaks) and that there is no degradation in performance once multiple customers access the same service.

Functional testing - This tests the app delivers the required functionality.

Reliability testing – To ensure that the app is capable of performing failure-free for a specific period of time in a specific environment.  

Security testing – As discussed earlier, the cloud environment provides access to multiple customers who can use the same services. As a result, we must ensure that there is no unauthorized access to the data within the SaaP application, no privacy leaks, and that customer data integrity is kept under strict security gates.

Common test Guidelines:
  • Validate that data integrity is not compromised by unauthorized access
  • Validate that only the authorized customer can access the data
  • Validate that data migration is made through secured (encrypted) channels
  • Validate that all user data is removed in case of dropping the service
  • Validate that only the relevant ports are opened
  • Validate that there is a clear separation between tenants

Scalability testing – Cloud services are relevant to both small and large organizations; as a result, there must be tests to ensure that the business can scale up or down its resources based on the customer’s need. 

Load and stress testing – To identify the stability of the system beyond its operational capacity to see how it reacts to different loads. 

Live upgrade testing – To ensure that we can deploy new versions on the cloud without affecting customers’ user experience.

Performance testing – To ensure that the SaaS application can manage different traffic loads that depend on the number of customer requests. The main factors that we want to validate in this type of tests are network latency, the response time of the application and the workload balancing (NLB) in case of massive use. 

Common test guidelines:
  • Response times should not be affected due to the actions of other tenants
  • Failures in one tenant should not affect other tenant performance  
  • Scaling process should not cause any degradation in performance factors

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile


My Presentations