The Art of Problem Solving

Solving problems is something i love doing. My job presents me with problems to solve every day (e.g. how to test ‘invisible’ features or how to simulate a customer using our product), and outside of work I’m also faced with problems to solve on a regular basis (e.g. why the hot water tap doesn’t heat up or why the TV reception is bad for 1 channel only). Whatever the problem, there are ways of approaching it that will apply in most cases. Here’s the approach i use.

Step 1 – Gathering Intel

First up, we need to understand the problem we are trying to solve. So start by gathering all the information you can about the problem. Here are some starter questions worth asking:

  • What steps are needed to see the problem occurs? (So you can determine if your fix worked)
  • What is the expected outcome and what is the actual outcome? (So you can tell when it’s been fixed)
  • Is it something that happens every time or just sometimes? And if so, how often? (This might tell you something about why it’s happening)
  • Does anything you do seem to influence whether it happens or not or how severe it is? (Another potential cause to the problem)
  • When did the problem start? Is it recent or always? Can you trace it back to start with a particular event? (Did that event cause the problem? Maybe it just made it visible)

Once you’ve got these questions (or similar ones) answered, you should have a good idea of what the problem is that you are trying to solve and a way to test out if your fix for the problem has worked.

Step 2 – Creating a hypothesis

Now that you know the problem, step 2 is to create a hypothesis on how you might be able to fix the problem. This will be based on your findings in step 1 on what factors seem to contribute to the problem with an idea on what change then needs to be made to fix the problem.

For example, a simple case might be that last month someone changed the type of paper the photocopier was using and over the past few weeks, there appears to have been an increased in paper jams. So your hypothesis is that by changing the paper back to the previous paper, the occurrence of paper jams should decrease. You’ve seen the change in behaviour, you notice an event which possibly contributed to the problem and you came up with a method to fix the problem, including a way to tell if it worked.

Perhaps the problem is a bit trickier. You’ve been using your iPhone for months as a calendar and it sends you email notifications when you have an appointment coming up. You recently upgraded to a newer model iPhone, at the same time as changing your email address that you receive notifications on. The iPhone calendar app has been updated and you decide that instead of checking those emails on your iPhone, you are going to start using your new iPad to read them. But the emails have the wrong times for your appointment.

Lots of variations and no obvious cause to the change. But the same rules still apply, find out all you can, and create your best guess hypothesis of what might help fix the problem.

Step 3 – Test out your hypothesis

Finally, you test out your hypothesis, make the change(s) required and see if the problem still happens. This might take a lot of testing depending on how often and predictable the problem was to reproduce. Eventually, you’ll figure out whether your hypothesis was correct or not.

Hypothesis didn’t solve the problem? Repeat Steps 2 and 3 again with the new information you’ve just learnt until you find your answer.

Applying it to my job as a tester

How does this fit in with being a tester? Well there’s two main types of problems i come across in my work.

  1. How to test tricky to see/verify areas of the project?
  2. How to create automated tests to cover those areas that are also useful, readable and maintainable (see my previous blog post for more on writing good automated tests)

In both these cases, the same rules apply as i’ve outlined above. It might take a while to gather the right data and come up with a solution, but I’ve found following this simple and general approach quite helpful in my work. In my role as a tester for the Online SaaS company, Campaign Monitor (which is awesome by the way!) I have lots of things to consider in how to test our product and where appropriate, automate tests to cover our various UI elements and so I constantly find myself with problems to solve. From how to run automated tests in parallel to figuring out how customer feedback should shape the way i test and the things i look for.

I hope this simple approach might help others solving there problems too!

What makes a good automated test?

Testers are increasingly being asked to write, maintain and/or review automated tests for their projects. Automated tests can be very valuable in providing quick feedback on the quality of your project but they can also be misused or poorly written, reducing their value. This post will highlight a few factors that I think make the difference between a ‘bad’ automated test and a ‘good’ automated test.

1. The code

People vary in how they like to write their code, for example, what names they use and the structure they like. This is fine to be different, but there are still aspects that should be considered to make sure the test is going to be useful into the future.

Readable & Self-explanatory
Someone should be able to read through your code and easily figure out what is being done at each part of the process. Extracting out logical chunks of code into methods helps with this but be wary of having methods within methods within methods… Deep nesting can add unnecessary complexity and reduces maintainability since coupling of code is increased. Use comments sparingly as the methods and variables should be descriptive enough in most cases and comments quickly get outdated. Format your code so it’s easy to read, with logical line breaks and be wary of ‘clever code’, which might be more efficient and might get 4 lines of code into 1, but it must still be readable and understandable to the next person looking at it in 5 months time. Make your test readable and self-explanatory.

Clear purpose
In order to make it easier to get a picture of test coverage, it helps to have tests with a clear purpose of what they are and aren’t testing. This also means people reading through the test trying to understand it or fix it know exactly what it’s attempting to do. Similarly it helps to not have your tests covering multiple test cases at once. When the test fails, which test case did it fail on? You’ll have to investigate every time just to know where the bug is before you even start getting the reproduction steps. Having multiple test cases also makes naming tests harder and can result in these extra test cases being hidden underneath other tests and perhaps duplicated elsewhere or giving a false picture of test coverage. Make your test have a clear purpose.

2. Execution

When you actually run your tests, there are a few more attributes to look for that contribute to the usefulness of the test.

If your tests take too long to run, people won’t want to wait around for the results, or will avoid running them at all to save time, which makes them completely useless. What ‘too long’ looks like will vary in each context but speed is always important. You will reduce reluctance from using your tests by setting a realistic expectation of the duration for running your tests. There will also be times where you want a smaller and faster subset of tests for a quick overview, whereas other times you are happy for a longer, more thorough set of tests when you aren’t as worried about the time it takes. Make your test fast.

In most cases, depending on the context of your project, running your test 10 times in a row should give you the same 10 results. Similarly running it 100 times in a row. Flakey tests that pass 1 minute, then fail the next are the bane of automated tests. Make sure your tests are robust and will only fail for genuine failures. Add Wait loops for criteria to be met with realistic timeouts of what you consider a reasonable wait for an end-user. Do partial matches if a full match isn’t necessary. Make your test consistent.

The fastest way to run tests is in parallel instead of in sequence. This changes the way you write tests. Global variables and states can’t be trusted. Shared services likewise can’t be trusted. To run in parallel, your tests need to be independent, not relying on the output of other tests or data which is shared. Wherever possible, run your tests in parallel, finding the optimum number of streams to run in parallel and you will have your tests running much faster, giving you quicker feedback on the state of the system being tested. Make your test parralelisable.


There is plenty more that could be said about what makes a good Automated Test, but this should make a good start. Having code that is readable, self-explanatory and clear of purpose, and written in a way that it runs fast, consistently and can be run in parallel will get you a long way to having an effective, efficient and above all, useful set of automated tests.

I’d love to hear about what other factors you find most important in writing a good automated test as well in the comments below