The value of validation

I just attended a workshop on testing procedures. It was a bit worrying – not that anything was wrong with the procedures themselves (in fact they were well thought through) , but it was a bit shocking that testers still need telling.

One topic the workshop didn’t address was the distinction between verification and validation. These terms are used in different ways in different environments, so I should start by saying what I mean by each. Verification is making sure that a production meets its spec. Validation is checking that, even if it does, it will also fulfil the original requirement it is intended, to meet. Not at all the same thing, not only in the obvious sense but also in the sense that step-by-step verification from the requirements to (say) the code you are reviewing would not be equivalent to validating the code against the original requirement. There is just no substitute for asking which requirement a piece of code contributes to – and how each requirement is realised in the code.

There are lots of reasons for this, of which the fallibility of previous checking it one. But there is (a mathematician friend tells me) a much more compelling one that should convince even the most hardened reviewer and tester. This is that it is (apparently) possible to prove mathematically that no two languages can be translated into one another such that the semantics is exactly correct.

This may seem an abstruse point, so here is a practical example I have used in training courses. There is a well known saying in English that, if you translate it correctly into Russian (again, I am told) and then re-translate it back into a possible but correct English phrase. One of the possible outcomes is the following:

The vodka is acceptable but the meat is off.

So what was the original English phrase? Give yourself a few seconds before you look at the answer, which is at the foot of this post.

Now look. Not quite the same, is it? Now it is essential to recognise that both the translations, from English to Russian and from Russian to English, were both 100% correct. Just like a model may correctly represent a requirement, a design a model and code a design. And yet it is clear that if you came up with the second English phrase (the code, as it were) rather than the first (the requirement), it would leave something to be desired. The problem is, requirements, models, design and code are all in different languages (in every sense) and not two languages (let alone four) are exactly equivalent.

Hence the critical value of validation as well verification. Your just have to do it, not because your verification (reviews, testing, static analysis, etc.) isn’t good enough but because it does a different job.

Not that you should validate everything. It’s expensive, and like verification itself, not always the most productive thing you could be doing with your resources. Like everything else in good management, what you look at should be determined by the risk it represents. So only validate the items that represent a real threat if they are wrong. By and large, focus on the critical, the complex (at any level), the novel (to you). After that, either an error won’t matter much or you should be able to fix it relatively easily.

Answer: The spirit is willing but the flesh is weak. Or, more completely, ‘Watch and pray, that ye enter not into temptation: the spirit indeed is willing, but the flesh is weak’ (St Matthew 26:41).