Saturday, May 17, 2014

Software Testing – guidelines and principles


A few years ago when I started my way in the testing field I thought that testing is all about coverage, I always want to test the full testing matrix no matter the time frame, “Risk Management” was one of those terms that I master now, but back then it looks like the worst enemy, how can I release a software that I wasn’t able to test the full available scenarios..? For a beginner, I was really afraid that after release this one case that I failed to test is going to be the one who will discover on the client environment.

This issue was my worst fear on every testing cycle no matter the quality of the job I supplied, therefore when I decided that QA is the profession for my life, I make a deep investigation on few main fields that for me every QA/Tester must know and keep in mind before he touch any application, the fields are relevant for “Risk Management” and the main testing principals.

The next few paragraphs will cover the second part “Principles”. In addition, the list below contains principles and guidelines that existing in the software world for more than 30 years, my job is to add my personal view and how this principle can help in your daily job when the time frame is narrow and the pressure is high.

Software Testing Guidelines and Principles

Principle 1: Developers should avoid testing their own software

Before I start the discussion about this principle, I must say that this principle is a great example of the changes we had in the software industry in the last few years, the knowing that QA teams are crucial for the organization success and the understanding that testing is a profession and not a sidekick of a development team.

But this part is about principles, and if I want to analyze this issue from this view I can say that developers cannot test their own code (I don’t talk about unit tests..), it’s a really bad idea for every organization that has respect for his clients because in the end developers are here to build the application and not here to destroys her (that’s our job…).

When the developer build the software he can see each function and know how it’s supposed to work, but for a developer to investigate the full system or different integrations it’s simply too complicated and not a good idea, in addition, think about a developer that you familiar with, now ask yourself how this developer response when you as a tester find a “Logical” bug in his code (you know one of those bugs that do the difference between the specialist and the novice testers), the response in 90% will be that he failed to think about it or probably failed to understand how this code affects the code it’s integrated with.

An additional aspect is related to the philological side and says that testers simply doesn’t want to find the errors in their code because they spent hundreds of hours in hard work to construct something, for a developer to search for his own errors it’s simply against his own nature. In addition, the programmer who builds the application will want to do the best coding as possible like the tester who wants to reduce the defects to the minimum, as a result the programmer may ignore issues in his code that may cause future errors in real-time environments because he simply doesn’t want to expose the basic fact that his code is variable in some scenarios.

Now, think about it… do you really think that all developers are smart enough to report such bugs to the manager, project owner… and demonstrate code parts that coded with errors in the first place..? I don’t think so, a developer will want to show a flow lace code execution and if an error will be found in real environments, it’s the QA fault that they simply failed to do their job, sound familiar anyone? 

Before I end this section I want to add additional thinking point, as you can see the principle is dedicated to the reasons that make developers not useful for testing, but if you think about this little further, that’s doesn’t mean that a developer (that probably learned one or two throughout in his life) cannot make tests on his own software, the major key for this guideline is related to the fact that if that’s the case you will probably should expect for multiple failures that would be discovered on any simple testing cycle performed by the QA teams.

Principle 2: Do not design your tests based on the assumption that the software free from bugs

This principle is simply classic for the testing world, my idea here is to give a real example from the real testing world and demonstrate how the timelines factor may lead to the wrong assumption that the software is free from bugs.

How many times you received a software or Feature for testing that in the optimum world you will need 20 days for testing, but in the narrow timelines for the current project you have only 2-3 days to finish your tests? 

Well, I’m sure that your answer will point that the timeline factor is relevant for almost any project, and why this point is so important? Because it has a direct impact on the tester judgment that based his tests on the wrong assumptions, the classic failure is the one that usually made by a project owner that will make the classic mistake and tell you that all you need to do is few “End to End” tests + some basic sanity tests because you already test this version on previous testing cycles and the risk now is minor and probably doesn’t exist. This assumption will lead to the second classic mistake that says that all we need to do in this case is to show that the software performed with the main functionality and nothing more, well this is wrong!

Our job as a professional is to test any given software from the perspective that any software’s will have its own bugs. Therefore, we must make our tests from the point of view that the bugs are out there and the main task we have is to find them.

You need to remember that when testing a software that has the new feature we have three main tasks: 

  • Approving that the software works as defined in the SRS doc.
  • Testing the old code and understand how the new feature effects it.
  • Reduce the bug counter to the minimum necessary. 

Most managers will most likely fail to understand that those three tasks are must be combined together on any testing effort (The timelines issues will cause the manager /tester to focus on the first task and most likely to ignore the other two tasks), any testing cycle that based only on one of them will not be effective and probably put the software on greater danger when deployment occurs in real-time environments.

Principle 3: Test results must be inspected thoroughly

Some of you will probably say that this principle is obvious, but although it is, many testers are failing to understand the importance of this issue and as a result, they failed to reduce the software failures. It’s basic, in 95% of the total testing cycle testers design their tests based on SRS and DDD documentation, as a result, most of the test scenarios will contain the expected results for a given operation, the tester who execute those tests must follow the expected outcomes and make sure it’s valid against the predefined expected results designed in the test case.

In addition, you always need to remember that validating the end results is only one half of this principle, sometimes the expected results from test cases that written based on pure documentation That in many cases has missing information regarding complex test scenarios due to the limit technical knowledge of the product manager, as a result you must make a deep investigation to understand if the outcome in the “Dynamic” testing is based on the PM expectations, you should never make the assumption that the result is valid if it’s not written in the original documentation. 

Principle 4: Software Test cases must be designed with unexpected and Negative inputs

That’s classic, when testing a software our STD must include test cases that are written with both invalid and valid inputs, most testers ignore the first part (Invalid Inputs) and goes directly to the second part (Valid inputs), in that way the testers will cover the demands from the SRS documentation and that’s fine, but the Invalid inputs (In many cases) are the once that going to make the difference regarding the software stability and resilience.

Based on my personal experience in testing enterprise software’s I can say for sure that most bugs and the one that I really like to find as a specialist are discovered when my inputs are not the one that expected based on spec. Therefore, a decisive part from my test design is dedicated to writing complex tests with Invalid inputs that really pushing the system to the limits and as a result my developer got additional working hours, you need to remember that when a developer wrote is code he simply do it (in most cases) like a blind horse, as a result, the requested functionality will probably work as expected, but when the same functionality is tested with inputs that our developer failed to understand and think  in the coding stage the errors will pop-up in a way that we couldn’t do with Valid inputs.

Principle 5: Test cases must be written with definitions for the inputs and the expected result

Although this principle may look obvious for most of you, it’s really important to understand the psychology behind it, when a tester that doesn’t familiar with is software or testing new feature based on predefined STD that failed to address the relevant expected results, you may fail to judge the actual results that you receive when executing the test case, some testers will take such output and say, Ok, that’s look like the correct outcome, we can approve this case and continue to the next one, well, such judgment is wrong, a tester will want to see the correct results may fail to understand that what’s look right is still far away from the PM expectations.

Because testers are humans that make mistakes based on this basic psychology, we must remove this factor from our tests, to make such a thing, we must reduce the “Human Eye” factor. Therefore, each test case must contain a detailed list of the expected results and inputs, in a way that a tester will execute the case without the option to think that “what he sees is probably the right thing”.

Now I want to clarify something, the human eye of a tester is crucial, otherwise testing could be made by robots, but when executing a test case that designed specifically based on SRS doc that contains the client design specifications, we cannot have deviated from the expected results, the human eye will help us in different ways such other impact we have while executing the case or any other issues that we see that simply doesn’t look right.

To summarize this principle I want you to think about the next point, if the test case designed with predefined inputs that led to a specific outcome, what the % for failure when a tester runs this case with detailed steps? Well, I think that when a tester can follow a detailed test case that contains the relevant outcome, the % for failures will be reduced to the minimum (based on the assumption that the tester who designed this case made is the investigation).

Principle 6: No matter the number of tests you made, you cannot prove that the software is free from bugs

I hate this principle, from the tester perspective it simply says that no matter the hard job you do, you still cannot guaranty that your software cleared from all failures, that’s really annoying because you always have the chance that failures will occur after the testing cycle is done and the software delivered to the company clients. The solution? Change the state of mind, although you cannot promise that the software cleared from bugs you can point your finger to the bugs you discover within the testing process and say that those failures are removed and will not affect the clients.

In addition, you already know that for me a great testing process cannot be accomplished without the knowledge about the risks (Risk Analysis), when a tester knows the risks he can simplify his test plan and cover the areas that most likely contains the critical/Blocker bugs (Risk Management), that leads to an efficient testing process that most likely reduced the risk for bugs after the end of the testing process.

Personal view: Mastering the art of “Risk Analysis” and “Risk Management” is the keyStones for a tester, without them you’re testing cannot be efficient and relevant in the software industry. 

Principle 7: Start your tests as early as possible

It's simple fact that the company costs of fixing bugs that found on later stages in the “Software Life Cycle”, will be much higher than the once that found on the earlier stages. To overcome this problem we need to start our tests as early as possible in the “Software life cycle”, why? Because early testing focuses on the “Static” side of testing that is crucial when we want to find the defects on earlier stages of the tested software.

After I made the connection between this principle and the “Static” testing side, we can understand that the benefits are Hugh, just a few examples of the benefits you gain when starting your test early:
  • Find design bugs on earlier stages.
  • Perform the first stage of the validation process.
  • Understand the big picture and the complexity of the project.
  • Determine the test objects and scope.
  • Save the company time and money.
  • Design our test cases per each level of the software life cycle.
  • Understand the user requests and the company business case.

Principle 8: Software testing has two different sides, First, Examine that the software does what it’s supposed to do, Second, examine that the software doesn’t do anything that it’s not supposed to do

This principle is highly important when testing a software, a tester that failed to accomplish one of the two sides of this principle will be probably failed to deliver a high-end product. Now let’s simplify things; the first half as I mention is dedicated to the verification that all the requirements and specifications design are functional and working without any deviation from the SRS documentation, the second part I mention is relevant when we want to see how the software performed when the functionality is implemented correctly, but in some situations the software performing in unexpected way that affect the basic requirements we tested on the first half of this principle (Think about an ATM that release money and on the same time update the account of another client, the ATM does what is supposed to do (Releasing money) but in the same time he does something that he is  not supposed to do).

Principle 9: testing is limited in time, you must design effective test cases

It’s obvious that any testing process is limited in time, as a result, the test plan should be written only after the tester understands the “big” picture. Let me explain, based on my experience on major testing projects I become familiar with the fact that testers create their test plans without the simple idea about the company business case (Why do we need this software?)  And in the worst case without the full info about the user expectation that leads to poor design of the test plan.

Based on this simple fact, we can say that an effective test plan will be determined by the information we gain in the SRS documentation, if the spec defines the precise and full requirements of the software we can design our test cases without the need to add the missing information that in many cases will lead to irrelevant test cases and waste of critical testing time.
To simplify this principle, I added a few examples that should lead to better efficiency when creating the test plan:

  • Test cases should be written only after the tester becomes familiar with the SRS documentation.
  • Missing SRS information should be completed by the PM and NOT by the tester assumptions.
  • It’s critical to understand user expectations and the company business case.
  • The majority of your Test plan must be written before the actual testing execution.
  • The tester must know the software architecture and supported platforms.

Principle 10: Defect clustering

Defect clustering…? Very simple, every software is built from a few integrated components, based on this principle we can say those specific modules will have most of the defects we will find on the pre-release testing. 

Based on my large experience I can say that this principle is really important because it’s can change the testing effort in a way that if you found such module you probably need to add additional testing effort to remove the risk.

Principle 11: software testing an endless process that should be stopped

We all know that testing procedure that executed on a complex software can take an unlimited amount of time (the testing matrix is endless), as a result, we need to decide when to stop the tests and what the consequence of our decision.
The available options for stopping testing are:
  • Timelines
  • Project budget
  • Acceptable level of risk
  • Testing coverage based on requirements

From the list above, you can see that every bullet affects the other once, and why it’s so important? Because when we need to stop our testing, we should understand the outcomes, the classic example is to stop the testing cycles due to pressure timelines and taking the chance that the software will have bugs that may influence the user experience (Lower quality against product release). 

1 comment:

My Presentations