Decreasing False Positives in Automated Testing

Hey all,

I attended a webinar last night on “Decreasing False Positives in Automated Testing”. This event was conducted by Sauce Labs and webinar presented by Anand Ramakrishnan, QA Director, QASource

There was very good learning in it and I would like to share in brief various things which were discussed.

All attendees of the meeting were initially asked to fill a small survey question…

Q. In your organization, what is the primary objective of using automation?

Options: A. Save money

                B. Save time/ release faster –>This option got the maximum no. of votes 

                C.More QA coverage

My vote too was for the same option. Looks like the industry on the whole is facing a similar challenge of meeting faster release times.

Moving on to the topic…

 Q.What are False Positives?

Ans. Tests that are marked as failure, when in reality they should have passed. In other words, they are false alarms.

 We had another survey question just then :-

  1. What is the percentage of false positives within your respective automation tests?

Options: A. <5%

                B. 5-15

                C.15-25%        –> This option got the maximum no. of vote

                D. >25%

My vote was for option B. Though we are on the better side of things, we still need to further reduce these false positives.

Reasons why tests encounter false positives : –

  1. Flawed automation approach
  2. Choosing the wrong framework
  3. Inadequate time to accommodate a test plan/design.
  4. Hard-coded waits/delays
  5. Less modularity of code
  6. Relying on co-ordinates & xpath of objects
  7. Change in UI element id, classname etc
  8. Shared environment for QA as well as automation.
  9. Slow performance of application, or particular environment
  10. Manual intervention prior to execution of automation scripts
  11. Browser incompatibilities.

Impact of False Positives :-

  • Frustration within engineering team
  • Sooner or later, test failures are ignored by stakeholders
  • Risk of overlooking potential bug
  • Babysitting of automation tests
  • Maintenance cost of automation increases

Now the most important part of the discussion.

Ways to Reduce False Positives :-

  • Deploy application on optimal configurations, for automation
  • Keep tests short and simple. Avoid trying to do too many things in a single test case.
  • Keep tests independent. No sequencing of tests
  • Provide unique identifiers while developing application itself.
  • Using right locators i.e in decreasing priority of id>classname>css locator
  • Tear-down approach. Bringing test machine to base state before/after every testcase
  • Dynamic object synchronisations. No hard-coded waits
  • Re-execution capability of test framework. In case test fails, re-execute test and if it passes to ignore the previous failure.

Finally,

Benefits of Eliminating False Positives:-

  • Will not miss potential bugs
  • Certainty of application health
  • Increase in productivity
  • Save time not babysitting
  • Decrease cost of automation

Summary: It was an awesome webinar and also brings perspective on how critical automation is to meet current day SDLC challenges and how to make it more effective.

Your feedback both good & bad are always welcome.

Regards,

VJ

Advertisements

Responsive Design: How device usage is changing the way we build websites

Nice article …Rebloggin this one..

Seven hours behind.

How many web enabled devices do you own? A very simple question. A phone? A tablet? A laptop? A desktop? These are just the basics. The most common. There are also web enabled TV’s, games consoles, and even fridges. Slowly more and more devices are connecting to the web.

As a developer or stakeholder, you will want your website to be visible and fully functional on as many devices as possible. The problem is that devices have different sized screens with different numbers of pixels.

If you write your website to work on desktop devices you don’t want to have to write another whole website to do exactly the same thing for mobile devices. Doing so would be a massive waste of time and resources that could be better used somewhere else.

That being said, you don’t want mobile device users to see a broken site that just doesn’t…

View original post 887 more words

JQuery After() And InsertAfter() Example

For example,

<div class="someRandomClass">Yo Yo!</div>
<div class="someRandomClass">Yo Sup!</div>

1. $(‘selector’).after(‘new content’);

$('.someRandomClass')
.after("<div class='someOtherRandomClass'>Fight Club!</div>");

2. $(‘new content’).insertAfter(‘selector’);

$("<div class='someOtherRandomClass'>Fight Club!</div>")
.insertAfter('.someRandomClass');

Result

Both methods above are doing the same task, but with different syntax, the new contents after after() or insertAfter() will become

<div class="someRandomClass">
   Yo Yo!
</div>
<div class='someOtherRandomClass'>Fight Club!</div>
 
<div class="someRandomClass">
   Yo Sup!
</div>
<div class='someOtherRandomClass'>Fight Club!</div>

Python: csv.DictReader

csv.DictReader(csvfile, fieldnames=None, restkey=None, restval=None, dialect=’excel’, *args,**kwds)

Create an object which operates like a regular reader but maps the information read into a dict whose keys are given by the optional fieldnames parameter. The fieldnames parameter is a sequence whose elements are associated with the fields of the input data in order. These elements become the keys of the resulting dictionary. If the fieldnames parameter is omitted, the values in the first row of the csvfile will be used as the fieldnames. If the row read has more fields than the fieldnames sequence, the remaining data is added as a sequence keyed by the value of restkey. If the row read has fewer fields than the fieldnames sequence, the remaining keys take the value of the optional restval parameter. Any other optional or keyword arguments are passed to the underlyingreader instance.

A short usage example:

>>> import csv
>>> with open('names.csv') as csvfile:
...     reader = csv.DictReader(csvfile)
...     for row in reader:
...         print(row['first_name'], row['last_name'])
...

So if you have a csv with First Name & Last name as the headers in the first row , the command row[‘first_name’] would give the field under that keyword(First Name) row by row. This was very very useful for my implementation.

Python: With command

Given below is an example of opening a file, manipulating the file, then closing it:

with open('output.txt', 'w') as f:
    f.write('Hi there!')

The above with statement will automatically close the file after the nested block of code. The advantage of using a with statement is that it is guaranteed to close the file no matter how the nested block exits. If an exception occurs before the end of the block, it will close the file before the exception is caught by an outer exception handler. If the nested block were to contain a return statement, or a continue or break statement, the with statement would automatically close the file in those cases, too.

Python Tutorial: Dictionaries (Key-value pair Maps) Basics

Nice one..Rebloggin this..

Shafiul Azam's Weblog

Dictionary in Python is a dat type also known as map. You may be already familiar with them, if using Associative Arrays in PHP or Hash-tables in C++ or Java. You may imagine a dictionary as an array, where instead of numericalindexes (the first element of array is indexed as 0, the second element indexed by 1 and so on) you can use stringindexes. Each element of a map is accessed (or, indexed) by an unique key, and so they are known as key-value pairs. Dictionaries are not sequences, hence maintains no left-to-right order.

Dictionaries are not sequences, hence maintains no left-to-right order.

Operations

You can create a map by the declaration: var_name = {key:value}

Code after “>>>” in following examples means: you need to type the code in Python command line. Any line not beginning with “>>>” indicates an…

View original post 37 more words