I did a Selenium meetup talk couple of days ago. I decided to talk about exploratory testing and automation.

For the meetup, we asked attendees to answer following questions when confirming their attendence via meetup.

1. Which programming language do you use for automation?
2. Why do you automate?

The answers to the second question was not surprising at all. 98% of the users said that they automate the tests for regression testing and based on their answers I can say that following are the main objectives for an automated testing effort.

• save time & money vs. manual testing
• improve software quality
• enhance testing efforts
• increase testing coverage
• eliminate error prone human behaviour
• faster feedback aka continuous integration

Test automation is viewed as developing automated tests for the system based on the acceptane criteria defined by the business. The teams developing these features use different tools and techniques to show the progress and meet the acceptance criteria. Once all the tests/checks are green and the application is deployed to an environment then it acts as a definition of done for the team and the team can then move on to a set of new features. This is what most of the agile projects aspire to acheive and some successfully do.

There are different types of automated test/checks developed by the team such as unit, integration, acceptance/functional, api contract tests and some times a form of performance tests if performance is something business cares about. These tests/checks are valuable to improve the testing cycle, reduce human effort and help the team while refactoring. In some cases these tests/checks act as the business specification and if you can read code then you can see what the code should do and how it should behave in certain scenarios.

This is all good stuff and it helps the team to deliver quality code much quicker. I feel that testing is more than just automating some checks using a tool. Automation should be viewed as a tool that can assist testing and I am glad to see growing list of people in the testing community who share the same opinion.

Exploratory testing is simultaneous learning, test design and test execution. As a tester you ask questions to the software, you learn from the answers and you ask more questions. There are different techniques to do exploratory testing and I am a big fan of session based testing. While keeping in mind this definition of exploratory testing, I have come up with the characterstics of an automated exploratory session and these characterstics are.

• context driven
• structured
• focussed
• adaptable
• short lived, low maintenance
• no assertion based failures
• can potentially run in Production (do not set up test data)

These tests or scripts or checks (it does not matter what you call them) are very specific and looking to explore the software to find out about an aspect of it. These are structured and you need to plan them accordingly. They are adaptable to change based on the answers you get from the software. Most of the times they are short lived and therefore very low maintenance, in some cases you can use them for specific purposes and throw them away. They do not produce standard passed or failed result. You can use assertions but these are meaningless because the purpose of these scripts are to find information from the software under test.

Following are some of the examples where I have used automation as a tool to explore software and found issues that have helped the team deliver quality software.

a newspaper article release

On a given date, we wanted to serve the article pages on the site from the new tech stack that the team had been developing for some months. The business decided to migrate 3 months of articles prior to the release date so that the site would have a consistent look and feel on the go live date. At the same time they wanted to migrate 7 years of articles.

I decided to develop a script that would run in the production environment, it will go to the different pages on the site and collect different article urls. Once we have the URLs then we can check based on the article published date that either this should be served from the new tech stack or from the legacy stack. I started running the test once the release was live and soon found that some of the articles published after we have went live are being served from the old stack. This was not suppose to happen. It tooks us a long time to find out why this was happening but we found the root cause of the problem and fixed it on the same day.

a newspaper content api

I participated in one of the hackathons for a media organisation and instead of developing something cool, I decided to test their api in the production environment because as a tester how often you get the chance to run tests in the production environment without justifying it. The expectation from the API was to mirror the browser based site because the source of the data was the same. I wrote an API client using Groovy, httpbuilder and jsonbuilder and got the data back from the API. I also used web driver to browse the site and scrap the data. Once I have the data from the pages and the articles then I could compare and see if there are any differences. There were 512 pages on the site and each page had 20-25 different articles so the script ran for some time but at the end I found 9 different problems where the content coming from the API was not the same as displayed on the site.

In both of the these examples, the development teams were using all differnet types of testing activities that an agile project uses these days. They had unit tests, integration tests, acceptance tests and all sort of other testing activities but you cannot possible develop regression tests for all different scenarios that might happen in the production environment. People will use your software in ways that product owners or the business analyst cannot think off.

Other than that we can use automation to help testing. Following are some scenarios where I have used automation to help me in testing effectively.

• explore different types of data
• set up different sets of data
• support for database operations
• testing queues and broker messages

I hope this helps fellow testers realising that automation is more than just automating bunch of checks that run may be every build and most of the time do not find the defects.

Property files or environment configuration files are an essential part of any system. For test automation, these files play an important role when you want to externalise the information such as application url, database hostname, database username etc.. These property files hold the information as a kay value pair and can be accessed when needed for the tests.

These are also vital when you want to run the tests against different environments, in this case you can have per environment property file and when running the tests you can pass an argument to load the specific property file so that the information about the environment is available to the tests.

Once you have per environment property file then the next step is to make sure that you only hold information about the specific environment in the respective property file and do not repeat the same set of properties in all these files. For example the following three files represent three different environments where you want to run your tests, and each environment will have a distinct application url, a username , a password and we are assuming that these are Selenium or WebDriver functional tests and you want to usefor your tests. Therefore the property files hold the information about Sauce Labs.

We have been using hamcrest matchers for some time now, and it is awesome to have more readable assertions in your tests. Recently we had a requirement to match lists for equality. This was simple enough, we used the following matcher to compare that the contents of the list are the same.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

@Test
public void listTestsWithOrder(){
  List<String> list1 = new ArrayList<String>();
  List<String> list2 = new ArrayList<String>();

  list1.add("red");
  list1.add("green");
  list1.add("orange");

  list2.add("red");
  list2.add("green");
  list2.add("orange");
  assertThat("List equality without order", list1, equalTo(list2));

}

How often a developer has come to you and asked the following questions about a test failure.

• When it started failing?
• Was it failing for the same reason before?

How do you then start looking at the previous test runs? Do you perform complex SQL to get this information out of a database if you are logging results there, or do you go through different builds in the CI tools to find this information?

Recently there was a question posted to the Tellurium user group about support for any of the BDD framework. We already have provided the support of running easyb stories as Tellurium tests. As I worked on the Cucumber and Selenium integration, I was intrigued to try using Cucumber with Tellurium. It turned out to be a pretty straight forward job.

I created a new tellurium maven project using one of the maven archetypes available. Once we have the project skeleton, I added the dependencies for Cucumber cuke4duke plugin, that will be used to run the Tellurium test.

Recently I started looking at ways of writing Selenium functional tests for a set up using BDD. There are different tools available to write the tests such as easyb, scalatest & cucumber. This blog post will focus on setting up a test project using Selenium 2, maven and Cucumber.

Once we have set up the Maven project then we need to set up the POM file so that all the required jar files for Selenium and Cucumber are available to the project.

We are using Spring 2.5, TestNG and Selenium RC for our functional tests. In out test base class we are using @BeforeClass to get the new browser session from Selenium Server. Today I wanted to replace @BeforeClass with @BeforeSuite annotation, so that we only launch browser before running the test suite and this browser session would be shared among the tests thus reducing the test execution time.

I thought it to be a straight forward task by just replacing the @BeforeClass with the @BeforeSuite annotation, because these annotations are already provided by TestNG.

But when I did that I got null pointer exceptions all over the place, the code was complaining about the classes instantiated via Spring Application Context being null, and when I looked at the AbstractTestNGSpringContextTests implementation I found that AbstractTestNGSpringContextTests’s springTestContextPrepareTestInstance() method is currently annotated with @BeforeClass, which does not allow @BeforeSuite methods in subclasses from interacting with the ApplicationContext.

To overcome this problem, you can override the springTestContextPrepareTestInstance() method in your subclass as follows.

1
2
3
4
5
6
7

@BeforeSuite(alwaysRun = true)
@BeforeClass(alwaysRun = true)
@BeforeTest(alwaysRun = true)
@Override
protected void springTestContextPrepareTestInstance() throws Exception {
        super.springTestContextPrepareTestInstance();
}

I was recently introduced to hamcrest (an open source matcher library). I was really impressed by the declarative nature of the match rules. We use TestNG to drive the Selenium Tests and naturally we are using TestNG assertions within our tests. Some of the TestNG assertions in our tests are as follows.

assertEqual (thePage.getLinkCount(), 10, "Link count on the page");
assertTrue (thePage.isLinkPresent(), "Link 1 is present.")

The equivalent assertions using hamcrest would be as follows.

assertThat("Link count on the page", thePage.getLinkCount(), equalTo(10));
assertThat("Link 1 is present", theBiscuit.isLinkPresent(), is(true));

You can see that the hamcrest matchers are more expressive and readable in the code. Another benfit of using hamcrest is that you can write custom matchers easily. e.g. you want to write an assertion in the test that checks that you have more than 20 items in your list and the item count is an even number.

assertThat ("Country item count", countryList.getCount(), allOf(greaterThan(20), evenNumber()));

It is easy to write and much more readable in the tests and if the test fails then it gives a clear description of the failure. Another benefit of using hamcrest is that you as a test developer does not have to worry about the different in the method signature for TestNG and JUnit.

Tellurium is now integrated with easyb framework. It is now very easy to write a Tellurium tests using easyb story/specification approach. easyb is a behavior driven development framework for the Java platform. By using a specification based Domain Specific Language, easyb aims to enable executable, yet readable documentation.

You can find more information about easyb on the following url http://www.easyb.org

I will be writing more about Tellurium in the near future. I have not been spending that much time on Tellurium these days as I would have liked to, and the team has done an incredible job in releasing Tellurium 0.7.0 with many exciting new features.

It was easy to pick this topic as a starting point to familiarize myself with the new features in Tellurium as I go along exploring.

We will be creating a new Tellurium project by making use of the Tellurium TestNG Archetype. This is an easy and quick way to get yourself started with Tellurium.

Tellurium includes two maven archetypes, i.e., tellurium-junit-archetype and tellurium-testng-archetype for Tellurium JUnit test project and Tellurium TestNG test project, respectively.