Saturday, August 11, 2018

What shouldn't we automate in an agile environment? | Supreme Agile

If you're familiar with my blog you already know that I'm a great believer in automation frameworks that allow the team to create a more productive, efficient and reliable process of coding and testing. However, some aspects of the testing process still need human eyes, common sense, and intelligence.

Usability Testing

Usability testing is very different from other test types that determine the quality of the software. As such, it cannot be automated because it requires someone actually work with the software to determine how it was experienced and where the gaps are in the user experience.

GUI Testing (Is it worth the ROI?)

GUI testing is one of the most difficult areas to automate. I’ve seen just too many organizations that invested thousands of human hours to automate the GUI of their products but at the end found it was a waste of time that didn't really provide the expected ROI.

Some GUI tests can be used just to make sure there are no unexpected changes in the GUI, but you should ask yourself whether it’s worth the costs and investment that can go instead to improving other quality issues that will provide a better ROI and reduce the risks in that area.

Tests that are not worth the investment

I used to be involved in an automation project where the test team automated a thousand tests that provided (at least on paper) great test coverage. So what is wrong here? They automated almost all the available test scenarios without really thinking about the ROI.

The team invested weeks on automating tests that were marked as low risk, tests that would never fail, and tests whose failure had a very low chance of impacting the software. The entire automation process was based on the spirit of "let's automate everything" instead of asking the simple question of what is the ROI that this automation project provides.

In some cases, there are tests that are written without real thought as to whether they are important or not, and once the automation project starts, the team will automate these tests just because they never want to run them again (because they know that there is 0% chance that it will actually make a difference). 

Tests that need to be executed only once

The main goal of automated testing is to allow the team to focus on the things that are really important in the software development lifecycle. Automating test scenarios that will run only once are not worth the time that the team needs to invest in the design, creation, and execution of these tests. 

Exploratory Testing

In my opinion, exploratory testing is the best and most efficient method that agile teams use in any testing process. Exploratory testing can be used for learning purposes (you learn more about the software when actually testing it) or to provide a fast way to evaluate the overall quality of the software. However, when it comes to real testing effort, a skilled tester is required to design and execute tests.

Exploratory testing should be done by humans and not by automated scripts because automated scripts will not let the tester take in new information he generated from the exploratory session and use it to improve future testing and development processes.

In addition, although exploratory testing is a great testing approach, there is a real need for automated tests that will allow the team to focus on their exploratory sessions without worrying about any regressions that automated tests should cover.   

Thanks to Tally Helfgott for proofreading :) 

Linkedin Profile

1 comment:

  1. I'm unsure why the title is "What shouldn't we automate in an agile environment?"? It kinda implies that we _should_ automate these types of tests in other environments. :)


My Presentations