Advanced test scripts

All IT and most business organisations will be familiar with the idea that they have to test systems. A basic tool in this is the test script.

A test script sets out a succession of a step-by-step instructions to carry out a test. It is crucial that testing be scripted to an extent commensurate with the risks of not testing properly, but when looking at either corporate standards or individual projects it’s striking how seldom even the minimum standards are met. Do your scripts state the expected result, so you tell unequivocally whether the test has been passed or failed? In my experience, most don’t. Nor do they tell the user what preconditions must be satisfied before the test can begin (navigate to…, using these privileges…), or tell you how to check the result, and so on.

So here is my recipe for a truly complete test script. You won’t want to use it because it’s long and complicated and you can’t see the point of it, but if you can tell me which fields are not needed by anyone with a legitimate interest in your testing, feel free to take them out.

The script falls in five sections:

1. Document control
2. Setup information
3. Execution
4. Outcome
5. Result

1. Document control data
   
   • Identifier.
   • Test name.
   • Reference no.
   • Parent identifiers.
   • Author.
   • Authoriser.
   • Preparation date.
   • Version.
2. Set up information
   
   • Planned execution date.
   • Tester
   • Function.
   • Summary of test’s objective or purpose.
   • Test condition(s) implemented.
   • Positive/negative test?
   • Start point (i.e., navigating to appropriate screen/process/field.)
   • Set up/Initial conditions.
   • User ID/privileges required.
   • Preceding actions/tests to prepare the application.
   • File selection, parameter settings, etc.
   • The preconditions for the test as a whole (Eg, account no., currency, etc.)
3. Execution (probably in table format)
   
   • Step no. (if sequence is significant.)
   • Location (Screen/form/field name at which testing should begin enter, for example, “Go to screen…”, “Select ‘Reports’ menu”, etc.)
   • Input data.
   • Data to be entered.
   • Option(s) to be selected.
   • Test actions (Step by step, checklist-style – e.g., “Enter data”, “Select option A”, “Click on Submit”, etc.)
   
   
4. Outcome
   
   • Actual test time/date.
   • Actual result.
   • Checked boxes (against each test step, to confirm completion.)
   • Notes, with a general prompt to record anomalies, unexpected results, unplanned steps, & unusual system behaviour.
   • Narrative/commentary (to support re-runs & regression testing.)
   • Sign off.
   • Tester’s name & signature.



5. Test result
   
   • Expected result.
   • Method for checking actual against expected (if not just “Check actual vs expected results” - including automated file comparisons, etc., as appropriate. E.g., checking back-end systems, end-of-day report, messages, etc.)
   • Pass/fail.
   • Cause of failure.(Eg, “Comm320 failure”, “Data feed”)
   • Defect reference field (to locate defect reports, anomalies, etc. my be needed at both step and script levels.)

Try it. Really, it works.

The value of validation

I just attended a workshop on testing procedures. It was a bit worrying – not that anything was wrong with the procedures themselves (in fact they were well thought through) , but it was a bit shocking that testers still need telling.

One topic the workshop didn’t address was the distinction between verification and validation. These terms are used in different ways in different environments, so I should start by saying what I mean by each. Verification is making sure that a production meets its spec. Validation is checking that, even if it does, it will also fulfil the original requirement it is intended, to meet. Not at all the same thing, not only in the obvious sense but also in the sense that step-by-step verification from the requirements to (say) the code you are reviewing would not be equivalent to validating the code against the original requirement. There is just no substitute for asking which requirement a piece of code contributes to – and how each requirement is realised in the code.

There are lots of reasons for this, of which the fallibility of previous checking it one. But there is (a mathematician friend tells me) a much more compelling one that should convince even the most hardened reviewer and tester. This is that it is (apparently) possible to prove mathematically that no two languages can be translated into one another such that the semantics is exactly correct.

This may seem an abstruse point, so here is a practical example I have used in training courses. There is a well known saying in English that, if you translate it correctly into Russian (again, I am told) and then re-translate it back into a possible but correct English phrase. One of the possible outcomes is the following:

The vodka is acceptable but the meat is off.

So what was the original English phrase? Give yourself a few seconds before you look at the answer, which is at the foot of this post.

Now look. Not quite the same, is it? Now it is essential to recognise that both the translations, from English to Russian and from Russian to English, were both 100% correct. Just like a model may correctly represent a requirement, a design a model and code a design. And yet it is clear that if you came up with the second English phrase (the code, as it were) rather than the first (the requirement), it would leave something to be desired. The problem is, requirements, models, design and code are all in different languages (in every sense) and not two languages (let alone four) are exactly equivalent.

Hence the critical value of validation as well verification. Your just have to do it, not because your verification (reviews, testing, static analysis, etc.) isn’t good enough but because it does a different job.

Not that you should validate everything. It’s expensive, and like verification itself, not always the most productive thing you could be doing with your resources. Like everything else in good management, what you look at should be determined by the risk it represents. So only validate the items that represent a real threat if they are wrong. By and large, focus on the critical, the complex (at any level), the novel (to you). After that, either an error won’t matter much or you should be able to fix it relatively easily.

Answer: The spirit is willing but the flesh is weak. Or, more completely, ‘Watch and pray, that ye enter not into temptation: the spirit indeed is willing, but the flesh is weak’ (St Matthew 26:41).

Can non-technical reviewers review technical products?

In many (perhaps most) of the organisations I have worked in, the question of who reviews what has been either contentious, ambiguous or been given a quite unnatural answer. The reason for this unhappy situation is that it is often difficult for a non-technical reviewer to evaluate a project, especially its technical content.

In my own specialist area – IT – this might manifest itself in a pained question such as ‘How can the business approve a change in a database design?’ But there are similar questions in all complex management situations – can techies contribute usefully to business cases, for example?

Good question – and in my experience, people only say ‘good question’ when they mean that there’s no good answer. But in this case, there is an answer, and what is more once the answer is understood it leads to a more robust approach to reviewing generally.

The basic problem is to decide what objective reviewing is trying to achieve, and so to decide whether non-technical (non-business, etc.) reviewers have any role to play in achieving it. To put it concisely, the purpose of reviewing is to decide whether the item under review is meeting its requirements. I don’t mean this in the technical sense of ‘requirement’ – i.e., something the work is supposed to achieve for it to be considered a success. I just mean does it do what it is supposed to do? This might well mean ‘does it fulfil its requirements?’, but it could also mean ‘does it comply with this specification’ or ‘if we follow this plan, will we succeed?’, or any number of other things.

From that point of view, the right reviewers are the people who can – and need to – make that call. But that still doesn’t mean that they are technically capable of understanding the item they are reviewing. Or are they? In what sense do they need a precise technical understanding of the content of the item – for example, a design document - to be able to evaluate it? To put my complete argument in a nutshell, what I am getting at is the idea that reviewing is based not on what it says so much as on what it means.

To go back to my problem about business people reviewing a change to a database design. Can they understand what the change says? Probably not, if by that you mean their grasp of namespaces, indexing and denormalisation issues, and their opinion of such things, in strictly technical terms, is probably worthless.

But that isn’t necessarily all that the review is for. For behind every such technical change, there is a pyramid of managerial and business implications that non-technical reviewers can not only understand perfectly well but are probably better judged by non-technical people.


This is illustrated in the following diagram (click to expand):

Hopefully it is clear what the diagram implies. At the lowest level, where the database change itself occurs, there is probably little benefit to be had from asking non-technical people what they think of the change from a purely technical point of view. ‘Who knows, and who cares?’ is probably the right answer. But as soon as the wider implications of the change – the non-technical elements of what the change means rather than the details of what the change documents say – start coming to the fore, both their interest and their ability to judge should start to grow rapidly.

For example, assume that the database change in question is to move from a distributed to a centralise structure. Although the technical issues will be beyond the business’ grasp, and so will most of the implementation and operational issues, not much else should be beyond them. Looking at the diagram again, what are the changes in test requirements this database change will call for? To have all your database testers in one team, located centrally, rather than separate teams all around the business? What does that entail? Much lower costs? Great, we’ll have it. And a simplified roll-out that can now happen three months earlier? Even better. But what is the downside? The changes in platform mean that we will need to recruit a whole new database team? How long will that take? What will it cost? Oh... not such a no-brainer then. And there’s a small chance that we won’t be able to meet our delivery timescales after all? But at least the total development cost will be well down? Great! But the operating cost will in fact go up? Damn...

It’s a complicated business, as anyone who has been in such a situation will testify. But perhaps it should be – and perhaps excluding the business (and other non-technical people) from reviews on the grounds that they ‘won’t understand’ what they are reviewing is not only a very narrow interpretation of what ‘understanding’ means in such a situation but positively counter-productive. After all, if you don’t ask them now, when will you? When it’s too late?

Of course, it’s not easy to make sure a review like this is successfully executed. It’s very hard to work out the real implications of as subtle a thing as a database change. But if you are the project manager and you can’t tell your customers what the consequences of your project really are, perhaps you should be finding out. After all, it’s not as though they will never find out. But the alternative to telling them in an orderly and systematic manner like the above can only be finding out through missed milestones and blown budgets.

In a way none of this should need saying – anyone who does a change request nowadays will perform an impact analysis that covers most of these issues. But as so often in project management, this simple lesson simply has not spread in the systematic manner to areas like reviewing (product or project) as one would have hoped.