How do you measure quality?  Number of defects? Customer happiness? Money earned? Developer smiles? That’s the question raised by @gojkoadzic in his keynote at the recent BDD and Agile Testing Exchange in London, to make us think and propose some elements of response.

We are used to ignore automatic alarms, even fire alarms:  we just don’t care, but we care when a person tells us about an issue or is shouting « fire in the building! ».

As in the story of the Gloves on the Boardroom Table,  making the problem tangible and visible help trigger reactions. The book Switch gives examples of such surprising techniques. Sharing information is not enough, a company is like an elephant, you don’t really drive it, you motivate it to go where you’d like to go. So visualization can help there.

This does not mean you should disable your functional tests, but it illustrate that what may seem nonsense today may be the next best practice tomorrow.

The absence of bugs is completely different than the presence of quality. Twitter has lots of bugs but most people are happy with it so it has quality; on the other hand, Nokia phones have no bug, yet if nobody likes them they have no quality either.

So how to measure quality? If you put 20 people in a room with 3 post-it’s each and ask them to write what attributes of the product they think are the most important, and if you get a small enough group of common answers then they are candidates of things you need to measure.

Here I have the feeling that each of the first two is just a proxy for the next, and only the last is the real one, though I’m not sure.

Taking an example of a UI, a window with 5 buttons on it it will likely requires 5 time more testing than one with only one button, even if the later brings a lot more value, which suggests that test coverage is crap.

If test coverage itself is crap, you may however monitor your risk coverage, that is the amount of testing compared to the associated risk. This metrics helps identify where you need to test more.

Visualize to make information actionable

Once you know what to measure, you want to visualize it. James Bach proposes his technique of low-tech testing dashboard. You divide your system into high-level subsystems, and for each you monitor the « progress of testing »: not yet started, half done or done, and the « level of confidence so far » with 3 smileys from happy to anxious. That’s better than a bug-tracking tool with 500 tickets mixed altogether.

Another visualization mode is the use of effects maps, or ACC matrix. You may also monitor in realtime the feed of sentiment of every Twitter message about your company or product as they do at FINN Technology in Oslo.

As a conclusion, Gojko introduces the brand new visualisingquality.org website where you can find many ways to visualize your measures of quality and where you can contribute additional ones. Since making things visible is so essential for action, let’s try ideas and share them through this website!