Parameterised Scheduler Plugin

“VJ, I want you to run the automation job on all our three environments on a nightly basis. So that we can compare the results & find out any discrepancies across environments.”

My team lead said this to me one day out of the blue.A small background on this – we have automation scripts used to test Rest APis on our cloud. This normally runs on a single environment every night, mainly pre-prod.Now the requirement was to run the same set of scripts on all environments i.e staging/dev, pre-prod & prod.

This was a new and exciting task for me and I got on with it right away!

Till now, I only knew how to schedule a job periodically as I had earlier written a blog post regarding it. But I did not know how to schedule the same job, multiple times with different parameters.

I found exactly what I needed in the Parameterised Scheduler Plugin!!

So once you install this plugin,restart the jenkins application for it to take effect. Very important. I tried scheduling it without restart and the plugin did not work.

Here’s how you can schedule the same job multiple times throughout the day with different parameters:-

Build_periodically_with_parameters

The % symbol separates the cron notation from the parameters(if any) that you want to give for your run. ‘env’ is my parameter which my scripts pick up.

Another example of scheduling the same job with multiple parameters :-

Scheduling_With_Multiple_Params

The parameters have to be separated by a ; for it to take effect. Here ‘env’ and ‘param’ are 2 params that my automation scripts accept.

So there’s the solution for it.Hope this post was helpful to some.

I love Jenkins for it’s plug-ability & wide range of solutions to commonly faced problems/requirements. Simply  superb!

Alright guys.. Have fun… Enjoy 🙂

This is VJ signing off!

 

 

 

Advertisements

Key Parameters while choosing an Automation Testing Tool

Hey all,

Hi Techies! How are you doing? Everything alright? How is the software industry treating you?

I hope it is keeping you all at the tip of your toes, as always. At least here in India it is! The industry is evolving at a blazing pace. There are start-ups propping up in every nook and corner. There are huge rounds of funding for those which show promise & all my buddies working in such start-ups are earning big bucks. Older more established companies are either withering away or are reinventing themselves to stay relevant & competitive.

Software Development  has moved on from the ‘Steam Engine phase’ to the ‘Bullet Train phase’. Cloud Technologies like AWS, Azure are enabling companies to set shop quicker,cheaper,lighter & to scale their systems more easily. Release times are getting shorter and shorter.To aid these faster release cycles there is a new set of Software Ninjas called Devops Engineers who are bringing a paradigm shift to the way the industry was working uptil now. There is a cultural and ideological change happening and conventional boundaries of Dev, QA and Support are slowing fading away for the better.

Coming straight to the subject of this post…

There is a huge necessity of software automation for various tasks & processes in today’s industry.Each company needs to have a robust automation framework in place. This is to match up to the speed of software development and test out the iterative builds. A quick reliable feedback loop on the code quality thus ensures moving to a ‘Continuous Delivery’ model where builds are deployment ready. It is close to impossible to test the quality of every single release with manual testing alone.I’m sure you’ll agree with me on this. Hence automation is the pressing need of the hour.

Automation requirements are massive in the industry – from UI automation for Desktop Applications,Browsers and Mobiles, to API level automation for backend architecture, to machine level automation for various tasks. There are a vast many automation tools out there in the market which can be used for different automation requirements. In my 5 years of experience in the Software Industry, I’ve come across many instances wherein I’ve had to select a tool for a type of automation & I’ve always been spoilt for choices. I had very little time to decide on the tool & design my automation framework. To add to this, an engineer is always expected to deliver fast tangible results. Simply put, you have the opportunity to make it or break it.

It becomes very important that you make a quick but well balanced decision while selecting your tool. Else sooner or later you will be in a situation that you have to scrap the tool you selected for more reasons than one. The tools all look similar in the beginning, but on careful reading and research, you will be able to identify the variations in each tool & the limitations of each one.

In this article I would like to share few key parameters to keep in mind while selecting an Automation Tool for your software quality assurance needs :-

1.Ease of setup- setup must be pretty straightforward and all dependencies must be handled internally during installation of the tool.

2.Ease of implementation-the APIs provided by the tool must be simple to understand. There must be good documentation for beginners.

3.Execution time of the scripts- each command in the script must be executed fast and overall time needed to execute a test-suite must be in acceptable limits.

4.Reliability- the tool must be reliable and must not give different results for the same test. Execution times & results must be consistent when run on a day to day basis. The tool must also work as per the documentation. Any deviation from that is a sign of unreliability.

5.Big developer/user community-Very important parameter. The bigger the community of users, the faster you will be able to resolve any roadblocks.

6.Support for issues/upgrades-There must be frequent updates to the tool with bug fixes. This shows that there is an active development team working to make the tool more and more robust. See the release notes too if possible and get an idea of the issues that have been fixed and how quickly they were fixed.

7.Integration with Continuous Integration- The tool must have hooks to be integrated with CI tools like Jenkins,Bamboo,Travis etc. Without that a user must then create his own hooks to plug the automation scripts with these CI tools which again requires time and effort.

8.Ease of maintenance-The tool must have a good overall framework which can handle maintenance related work. Eg: An update version of the tool must not break existing workflows. Google for such issues and if the occurrences of such issues are frequent, then that is a warning signal.

9.Open Source- I would prefer to select a tool that is open-source. Period.

10.Choice of Language- Lastly, select a tool which supports a programming language you are familiar with so that script development can be faster for you.

So there it is! These are the key parameters which came to my mind while selecting automation tools for my workflows. Hope it helps!

As you venture out to select your automation tool, it is best to create a table with all these parameters and evaluate all of your prospective tools against these parameters carefully. Believe me this definitely helps your decision making process and you will come out with a well balanced, level headed decision.

Until next time… From the Silicon Valley of India… Love to all 🙂

Regards,

VJ

 

NameError: undefined local variable or method `null’

I bumped into this error while trying to implement API validation tastcases using Airborne tool :-

NameError:
undefined local variable or method `null’ for #<RSpec::ExampleGroups::ApiProgramProgramid:0x000000030d2f20>
# ./spec/my_test_name.rb:9:in `block (2 levels) in <top (required)>’

Background of the issue:-

In the ‘expect_json’ part I had given the exact response of my API request as test data for future executions.

So for eg: if this was part of my API response:-

{“id”:17537987,”show_type”:”SM”,”series_id”:17537987,”season_program_id”:null}

I had given this as part of my Airborne validation script :-

expect_json({id:17537987,show_type:”SM”,series_id:17537987,season_program_id:null})

While trying to execute this script, I was continuously encountering this error saying “NameError: undefined local variable or method `null'”. It sure was getting on my nerves as I was’nt able to proceed with my implementation. And we all know how deadlines work! hehe… Anyways…

After a fair bit of trials & failures I found out the issue :-

Ruby as a language stores null values as nil. I was unaware of this as this is the first time I’m working with Ruby & had already started my implementation of Airborne Tool(ruby based), without getting sufficient time to ramp up on the language. We techies are required to deliver fast results you see.. hehe..

So finally, I gave this as my expected json value :-

expect_json({id:17537987,show_type:”SM”,series_id:17537987,season_program_id:nil})

And voila! The script worked!! 🙂

Hope this helps someone to resolve this specific issue.

Regards,

VJ

 

 

 

 

Jenkins ERROR: Failed to parse POMs : java.io.IOException: remote file operation failed: Caused by: java.lang.NoClassDefFoundError:

Hi all,

I have recently faced a peculiar error with my Jenkins+Selenium Integration.The scripts which were running fine for many weeks suddenly started failing.The scripts would not even get invoked.

Initially before invocation of the scripts I got a failed to parse POM error. How could that happen when I have’nt changed the setup at all myself??

This was the error I saw in the Jenkins console logs :-

parsing pom error-Jenkins

Anyhow after reading a few Stack overflow posts, I got a few clues that it must be somewhere related to the default Maven installations done by Jenkins in the respective slaves.

In the Jenkins slave default path I saw there are few jars/ packages that Jenkins configures/copied on to the slave :-

Temp Maven Files copied on to Jenkins Slave

The tools folder contains the following Maven Package :-

Temp Maven Files copied on to Jenkins Slave- inside Tools Folder

I deleted the 3 jars(maven3-interceptor-commons.jar, maven31-agent.jar, maven31-interceptor.jar) and the complete ‘tools’ folder and once again re-run my automation job.

And… Voila!! The scripts were invoked fine once again! 🙂 All those files were again copied fresh to the Jenkins Slave once again.

So it looks like maybe one of the Maven jars got corrupted for some reason and were breaking the execution. The reason this could have happened is still hazy to me. But atleast we have a solution for it :).

Let me know if you come across such an error and if you were able to resolve it with this post.

Regards,

VJ

Steps to get Safari Webdriver running on Mac OSX

Since this topic has not been well documented on the net and I struggled myself to get Selenium tests running on Safari browser, here are the complete set of steps to get Selenium tests up and running on Safari browsers. :-

Creating “Safari Extension” Developer Certificate

  1. Create an Apple developer account.
  • Go to https://developer.apple.com/
  • Click on “Member Center” on top panel.
  • Select “Create Apple ID”.
  • Go through the sign up forms and provide valid fields for a succesful sign up

2.  Login to created Apple account.

3.  Sign up for “Safari Extensions” developer

  • Click on “Certificates, Identifiers & Profiles”
  • In the “Safari Extensions’ section, click on “Join Now”
  • Go through the following steps and provide valid fields to successfully sign up for the “Safari Extensions” program

4.  Create “Developer Certificate”

  • Now inside “Certificates, Identifiers & Profiles” > “Click on Create/Add Safari Certificate”
  • You will see the following page – “About Creating a Certificate Signing Request (CSR)” Screen Shot 2015-04-15 at 1.40.24 pm
  • Click on “Continue”

You will see the following page :-Screen Shot 2015-04-15 at 1.40.42 pm

  • Upload the CSR file and Click on ‘Generate’ to generate the certificate
  • Now download the certificate once it is generated.

5. Download the certificate and install in the machine

  • Download the certificate in the download tab.
  • The certificate is downloaded as “safari_extension.cer”.
  • Double-click on the file to install the certificate in the Mac client/OSX.

6.  We have installed the safari extensions certificate for developers. Now we need to install the Safari Webdriver extension for the Safari Browser.

 Installing the Safari Webdriver extension in the Safari Browser

  1. Download latest Selenium Safari extension.

2.  Install the Safari Extension

  • Double-click on the “SafariDriver.safariextz” file.
  • You will get a prompt asking “Are you sure you want to install the extension “WebDriver”?“.
  • Click on “Install”

3.  Provide the default setting for the Selenium Webdriver Extension.

  • Click on “Safari” > “Preferences” > “Extensions” > You will find Selenium extension
  • Select “Enable Webdriver”

Now all the settings are done and now we should be able to launch our Selenium scripts using Safari Webdriver.

Launching Safari Webdriver

Webdriver driver = new SafariDriver();                

driver.get( http://www.google.co.in” );

builder = new Actions(driver);

This should launch your safari browser with the Safari Webdriver Extension 🙂

Hope this article comes of use to you all.

UPDATE (18th June,2015): With the latest update of Yosemite 10.10.3 & Safari 8.0.6, the execution on Safari Browsers has become unstable & unreliable.

Selenium Scripts work best on 10.10.2 , Safari 8.0.3 & Selenium 2.45.0 combination.

So if OSX prompts you for updates, please DO NOT install them if you want to run automation on Safari Browser

Regards,

VJ

Decreasing False Positives in Automated Testing

Hey all,

I attended a webinar last night on “Decreasing False Positives in Automated Testing”. This event was conducted by Sauce Labs and webinar presented by Anand Ramakrishnan, QA Director, QASource

There was very good learning in it and I would like to share in brief various things which were discussed.

All attendees of the meeting were initially asked to fill a small survey question…

Q. In your organization, what is the primary objective of using automation?

Options: A. Save money

                B. Save time/ release faster –>This option got the maximum no. of votes 

                C.More QA coverage

My vote too was for the same option. Looks like the industry on the whole is facing a similar challenge of meeting faster release times.

Moving on to the topic…

 Q.What are False Positives?

Ans. Tests that are marked as failure, when in reality they should have passed. In other words, they are false alarms.

 We had another survey question just then :-

  1. What is the percentage of false positives within your respective automation tests?

Options: A. <5%

                B. 5-15

                C.15-25%        –> This option got the maximum no. of vote

                D. >25%

My vote was for option B. Though we are on the better side of things, we still need to further reduce these false positives.

Reasons why tests encounter false positives : –

  1. Flawed automation approach
  2. Choosing the wrong framework
  3. Inadequate time to accommodate a test plan/design.
  4. Hard-coded waits/delays
  5. Less modularity of code
  6. Relying on co-ordinates & xpath of objects
  7. Change in UI element id, classname etc
  8. Shared environment for QA as well as automation.
  9. Slow performance of application, or particular environment
  10. Manual intervention prior to execution of automation scripts
  11. Browser incompatibilities.

Impact of False Positives :-

  • Frustration within engineering team
  • Sooner or later, test failures are ignored by stakeholders
  • Risk of overlooking potential bug
  • Babysitting of automation tests
  • Maintenance cost of automation increases

Now the most important part of the discussion.

Ways to Reduce False Positives :-

  • Deploy application on optimal configurations, for automation
  • Keep tests short and simple. Avoid trying to do too many things in a single test case.
  • Keep tests independent. No sequencing of tests
  • Provide unique identifiers while developing application itself.
  • Using right locators i.e in decreasing priority of id>classname>css locator
  • Tear-down approach. Bringing test machine to base state before/after every testcase
  • Dynamic object synchronisations. No hard-coded waits
  • Re-execution capability of test framework. In case test fails, re-execute test and if it passes to ignore the previous failure.

Finally,

Benefits of Eliminating False Positives:-

  • Will not miss potential bugs
  • Certainty of application health
  • Increase in productivity
  • Save time not babysitting
  • Decrease cost of automation

Summary: It was an awesome webinar and also brings perspective on how critical automation is to meet current day SDLC challenges and how to make it more effective.

Your feedback both good & bad are always welcome.

Regards,

VJ

7 Best Practices for Agile Test Driven Development

Test Driven Development (TDD) is a minimalistic software development process that is driven by an automated test case which defines the desired enhancement to the system. The test is first executed to fail. The developer then comes up with a minimal code that will pass the test case. Once the new code is tested, it is refactored to adapt to standards and retested. The cycle is then repeated to add further enhancements.

Designed as an offshoot of extreme programming, TDD follows the agile method of building software in iterations and involves clean, simple designs and code.

We present some of the best practices to be followed in TDD projects:

1.    Avoid functional complexity

Keep functionality to be achieved simple. Deliberate on it with the whole team to ensure the test case covers the desired functionality in every way possible. As the test case is the driver here, it should be reviewed for correctness and completeness.

2.    Focus on what you need to achieve

Be sure you understand where the code needs to be called and frame the test suite accordingly.  Ensure test cases follow standard naming conventions and clearly depict what needs to be achieved by the end of development process. This is crucial as functionality keeps getting added with iterations. Future developers should be able to look at the test and easily deduce the intended functionality.

3.    Maintain code austerity

Ensure your code has just enough meat to satisfy your test case. This is a basic tenet of TDD. This minimizes the chances of defects and also simplifies review and testing processes. However, do ensure the code is understandable and allows future enhancements.

4.    Test repeatedly

Test before coding and after coding. Then test once again after code refactoring. This is to reinforce that no code is broken in any of the steps. During refactoring, ensure the new code is maintainable and adheres to standards. The rule of thumb here is repeat testing whenever there is a code change or code move or code merger.

5.    Maintain code sanctity

Use version control tools to check out and check in code. This is important, specifically when more than one developer is working on the code. Using continuous integration tools like Jenkins can avoid code merger issues.

6.    Application knowledge

In TDD, coding needs to be limited but effective in that it achieves its purpose without breaking anything else. Also, the new code should ideally pass the test case in the very first run. Maintaining adequate system documentation, including a repository of test cases, and engaging team members with good application knowledge can ensure a smooth and successful project execution.

7.    Know when to use TDD

Last but not the least, TDD, like any other development concept, works best in certain scenarios. Use TDD for developments that can be quickly tested. Any testing that is prolonged or complex defeats the purpose of TDD.

With TDD, development is more controlled and, as a result, defects are considerably reduced. Repetitive testing ensures each component in the system is working correctly at every step.

Gallop is a front runner in providing agile testing services. Our expertise in various agile methodologies like Test Driven Development (TDD) and Behavior Driven Development (BDD) combined with the benefit of co-located test services can help you get the maximum benefit out of your agile projects. Contact us today to see how we can help you.
Courtesy: http://www.gallop.net/blog/7-best-practices-for-agile-test-driven-development/#pFa5KeizqggsiHKE.99