Lessons learnt from writing Automated Tests

The purpose of this article is to share some of the big lessons I’ve learnt in my 5+ years of writing automated tests for Campaign Monitor, an online service to send beautiful emails. If you are writing automated tests, it’s worth keeping these lessons in mind to ensure you end up with fast, maintainable and useful tests.

1 – Website testing is hard

Websites can be slow to load, both at a whole page level and an element level. This means you are constantly writing ‘waits’ into your code to make sure the element you want to test is available. As java-script techniques improve, there are more animations, hovers, fade-ins,  call-outs, etc… being added to websites which results in a much nicer experience for the user, but can be a nightmare for writing a matching automated test. With these things in mind, I suggest the following:

  1. Evaluate whether this test case is worth automating. You can test functionality via an automated test (eg does clicking button X cause action Y to happen?), but they are less useful at telling you whether a page animation looks correct visually, so you are better to keep that as a manual regression test.
  2. Do as much of the automated test outside of the UI as possible. If you can utilise an API call or make direct database updates to get your system into the state required or to verify the UI action performed in the app you will save a lot of time in executing the test and avoid possible test failures before you even get to the point of interest for the test.
  3. Use Smart Waits. As often as practical, you are better off doing your assert of an element state within a short loop. You usually don’t want your test to fail because your app took 4 seconds to login instead of the usual 2 seconds. Always opt for waiting for a specific element state rather than Thread.Sleep!

2 – Make your tests understandable

When you first write your test, you have all the context you need, you know why you chose each method over other methods, you know what you are attempting to do at each step along the way. But what about the next person to look at that test in a years time when the requirements of that test have changed? Writing a test that is easy to understand for all readers will save valuable time later on and will mean the test continues to provide value for a long time. How do we do this?

  1. Use meaningful variables and method names. You should be able to read through your test without diving down any deeper into the methods and know what is going on. To do this, you need your variables and methods to accurately and succinctly explain what they are for.. For example:
    int outputRowNumber = GetTableRowForCustomer(“customer name”);
    compared to:
    int x = GetCustomer(“customer name”);
  2. Add comments when necessary. Your colleagues aren’t idiots, so they won’t need comments on every line of code, but if, for example, you have a line of code calculating the number of seconds left in the day, it might be worth a few words to explain what you are doing.

3 – Run your tests often

The purpose of creating and running these tests is to find bugs in your application, so you should be running all of your tests at least once a week, and most tests daily or more. This will make it much easier to identify the change that occurred which has resulted in the failed test. How often you run the tests will depend on how often you are updating the code under test. The more often you update, the more often you should run the tests. You will probably have a mix of tests that take varying amount of times, so be smart about when you run them, run the fast ones during the day, but leave the slow ones to run overnight so you don’t have to wait for the results.

We’ve had great success in having a small and quick subset of tests that do a basic check of a broad range of features across the application. This gives us quick feedback to know that there aren’t any major issues with core functionality. We call them ‘Smoke tests’, because if there’s smoke, there’s fire!

4 – Be regular in maintenance

Any one who has tried automated testing will know that your tests will get out of date as the features they are testing update. But, if you keep an eye on the results of your tests regularly, and act on anything that needs updating in a speedy fashion, you will save yourself a lot of pain down the track. Much better to tackle the cause of 2 failing tests now then to deal with multiple causes of 30 failures in a months’ time. If you can address any maintenance tasks for updated feature behaviour within a week of it happening, the context will be fresh and you will be able to get on top of it quickly, before it spreads and merges with other issues and suddenly you have a sea of failed tests and don’t know where to start.

5 – Be technology independent

We previously used a technology called WatiN to drive our test automation and are now in the process of moving over to Selenium WebDriver. We’ve also used other technologies in the past and each time a change is needed, it takes a lot of time/effort to convert our solution. If/When the time comes where you feel the tool you are using is no longer working for your needs, the time/cost to change can be huge. It’s possible every method you’ve written is dependent on your current technological choices, so it either rules out ever changing, or means there is a steep curve involved in changing over.

So instead, add an interface layer on top of your code which just says “I want the ‘browser’ to ‘click’ on ‘this link’”. Then the implementation can figure out how to get the Browser to click on the given link. Then when you decide you want to try a new technology, just add a new implementation to the interface and nothing else needs to change.

I will probably add a follow up post as summarising 5 years of learning into 1 post is hard!

Bug Bash vs Usability Testing Session

The Scenario

Your project is nearing completion and you are getting close to release date. You think you’ve found the majority of the bugs there are to find in this product and are now ready to hand it over to others in the company to get their eyes over it. The question is, do you organise a ‘Bug Bash’ or an ‘Usability Testing Session’?

I think the answer depends on what you are seeking to get out of it. I’ll outline the differences between the two, when you should use each one and some tips for making them be useful.

Bug Bash

My Definition: A Group of people trying to find as many bugs as they can within a given feature in a short amount of time.


  1. Organise to get a group of people to meet in the same room for a designated amount of time (recommend 1 hour). Make sure there is a computer available to everyone, with any prior setup data ready to go.
  2. Plan what you are going to tell the group about the product, what areas you consider most risky, how to record any bugs they find and any further setup information they will need.
  3. Organise helpers from your team to verify and record bugs with you so that people aren’t all waiting for you.


  1. Welcome and thank everyone for their attendance
  2. Explain what the product is, and any particular areas you would like people to focus on.
  3. Explain how to record any bugs people find
  4. Start the timer (rec. 1 hour)
  5. Encourage interaction between everyone to help the test ideas flow, food and drinks on the table, light background music, etc..
  6. With your helpers, record any bugs and answer any questions raised. Resist giving too much information away, if the user can’t figure it out, it may be a usability bug.
  7. At the end, thank everyone for their participation and let them know you will send out the results soon (the follow up helps reinforce that you value their input)


  • Put people in teams of 2 working on the same machine, 1 person ‘drives’ and the other ‘navigates’ by offering suggestions of test cases to try out led by what they see the ‘driver’ doing.
  • Have prizes for the person/team that finds the most bugs and the best bug.

Who should I invite? Anyone and Everyone! – It’s good to get a mix of people, from different departments (QA, Design, Developer, Marketing…), some people from your team, some people from other teams. I like to get as many testers on board as possible, as realistically they will be most likely to find bugs, but at the same time, a developer or a designer will approach the problem from a different view point and is likely to find different types of bugs, so a mix is important.

Usability Testing Session

My Definition: Observing the user-interactions as a group of people attempt to complete a certain set of tasks with your product, whilst they gain familiarity with it.


  1. Organise to get a group of people to meet in the same room for a designated amount of time (recommend 1 hour). Make sure there is a computer available to everyone, with any prior setup data ready to go.
  2. Plan the tasks that the people must complete during the session. The tasks should be top-level only to allow the user freedom to explore the feature and figure out how to do it (even if it means getting lost along the way), eg. Submit a product review of your last purchased item.
  3. Organise helpers from your product team to record observations and answer questions from the group as they complete the challenge.


  1. Welcome and thank everyone for their attendance
  2. Explain the tasks that each person is to complete during the session
  3. Explain the purpose of the session, that it is to observe their interactions, and make note of any situations where they are confused by the product, or unclear of what to do next. (ie usability bugs)
  4. Start the session, best to keep a time-limit to the session to keep everyone on track.
  5. Create a relaxed vibe with food and drinks on the table, light background music, etc..
  6. Together with your helpers, observe the groups interactions with the feature, recording any time they became confused or were unclear on what to do. Answer any questions they have along the way as it’s also an opportunity for the group to get familiar with the product (since it’s often helpful for the whole company to understand the new product being developed).
  7. Finish up, thanks the participants and let them know that you will send out the results soon. (To help reinforce that you value their input)
  • Put people in teams, it will depend on your product for whether this is a likely scenario for actual end-users of your product. If end-users will be working solo, don’t use teams in the session either.
  • Have gifts for the participants as thanks, or perhaps prizes for the people who complete the tasks the fastest. (Being fast means there wasn’t anything that confusing for them)
Who should I invite? People who satisfy the following criteria
  • Have not worked closely on the development of the product (they will already know how to use it)
  • Are interested in learning about your new product OR
  • Are a good representation of who your end-users will be (whether that is marketers, developers, non-tech savvy, etc..)

How to choose

Now that we have looked at what each of our options are (of course there are more options out there, but just focusing on 2 here) how do we pick which one to use?

Pick ‘Bug Bash’ if…

  • You are trying to find as many issues with your product as you can to give greater confidence in releasing it
  • You are not trying to teach people how to use the product
  • You don’t have access to people who meet the ‘Usability Testing Session’ group criteria

Pick ‘Usability Testing Session’ if…

  • You are trying to familiarise people in your company with the product you are soon to release
  • You are trying to find usability bugs (perhaps you are worried your product is confusing to use at times)


This was a brief overview of 2 common techniques of getting other people in your company to look over the product your team has been working on to help find some issues you have been blinded to from working on the product every day. The technique you choose should be based on what you are hoping to get out of the session.


I’d love to hear from you if you’ve found either of these techniques helpful for you in the past or if you have any further tips to enhance the usefulness and enjoyment of these sessions. Or perhaps you have a different technique completely that you prefer? Put it all in the comments below :)