I attended a webinar last night on “Decreasing False Positives in Automated Testing”. This event was conducted by Sauce Labs and webinar presented by Anand Ramakrishnan, QA Director, QASource
There was very good learning in it and I would like to share in brief various things which were discussed.
All attendees of the meeting were initially asked to fill a small survey question…
Q. In your organization, what is the primary objective of using automation?
Options: A. Save money
B. Save time/ release faster –>This option got the maximum no. of votes
C.More QA coverage
My vote too was for the same option. Looks like the industry on the whole is facing a similar challenge of meeting faster release times.
Moving on to the topic…
Q.What are False Positives?
Ans. Tests that are marked as failure, when in reality they should have passed. In other words, they are false alarms.
We had another survey question just then :-
- What is the percentage of false positives within your respective automation tests?
Options: A. <5%
C.15-25% –> This option got the maximum no. of vote
My vote was for option B. Though we are on the better side of things, we still need to further reduce these false positives.
Reasons why tests encounter false positives : –
- Flawed automation approach
- Choosing the wrong framework
- Inadequate time to accommodate a test plan/design.
- Hard-coded waits/delays
- Less modularity of code
- Relying on co-ordinates & xpath of objects
- Change in UI element id, classname etc
- Shared environment for QA as well as automation.
- Slow performance of application, or particular environment
- Manual intervention prior to execution of automation scripts
- Browser incompatibilities.
Impact of False Positives :-
- Frustration within engineering team
- Sooner or later, test failures are ignored by stakeholders
- Risk of overlooking potential bug
- Babysitting of automation tests
- Maintenance cost of automation increases
Now the most important part of the discussion.
Ways to Reduce False Positives :-
- Deploy application on optimal configurations, for automation
- Keep tests short and simple. Avoid trying to do too many things in a single test case.
- Keep tests independent. No sequencing of tests
- Provide unique identifiers while developing application itself.
- Using right locators i.e in decreasing priority of id>classname>css locator
- Tear-down approach. Bringing test machine to base state before/after every testcase
- Dynamic object synchronisations. No hard-coded waits
- Re-execution capability of test framework. In case test fails, re-execute test and if it passes to ignore the previous failure.
Benefits of Eliminating False Positives:-
- Will not miss potential bugs
- Certainty of application health
- Increase in productivity
- Save time not babysitting
- Decrease cost of automation
Summary: It was an awesome webinar and also brings perspective on how critical automation is to meet current day SDLC challenges and how to make it more effective.
Your feedback both good & bad are always welcome.