Advanced test scripts

All IT and most business organisations will be familiar with the idea that they have to test systems. A basic tool in this is the test script.

A test script sets out a succession of a step-by-step instructions to carry out a test. It is crucial that testing be scripted to an extent commensurate with the risks of not testing properly, but when looking at either corporate standards or individual projects it’s striking how seldom even the minimum standards are met. Do your scripts state the expected result, so you tell unequivocally whether the test has been passed or failed? In my experience, most don’t. Nor do they tell the user what preconditions must be satisfied before the test can begin (navigate to…, using these privileges…), or tell you how to check the result, and so on.

So here is my recipe for a truly complete test script. You won’t want to use it because it’s long and complicated and you can’t see the point of it, but if you can tell me which fields are not needed by anyone with a legitimate interest in your testing, feel free to take them out.

The script falls in five sections:

1. Document control
2. Setup information
3. Execution
4. Outcome
5. Result

1. Document control data
   
   • Identifier.
   • Test name.
   • Reference no.
   • Parent identifiers.
   • Author.
   • Authoriser.
   • Preparation date.
   • Version.
2. Set up information
   
   • Planned execution date.
   • Tester
   • Function.
   • Summary of test’s objective or purpose.
   • Test condition(s) implemented.
   • Positive/negative test?
   • Start point (i.e., navigating to appropriate screen/process/field.)
   • Set up/Initial conditions.
   • User ID/privileges required.
   • Preceding actions/tests to prepare the application.
   • File selection, parameter settings, etc.
   • The preconditions for the test as a whole (Eg, account no., currency, etc.)
3. Execution (probably in table format)
   
   • Step no. (if sequence is significant.)
   • Location (Screen/form/field name at which testing should begin enter, for example, “Go to screen…”, “Select ‘Reports’ menu”, etc.)
   • Input data.
   • Data to be entered.
   • Option(s) to be selected.
   • Test actions (Step by step, checklist-style – e.g., “Enter data”, “Select option A”, “Click on Submit”, etc.)
   
   
4. Outcome
   
   • Actual test time/date.
   • Actual result.
   • Checked boxes (against each test step, to confirm completion.)
   • Notes, with a general prompt to record anomalies, unexpected results, unplanned steps, & unusual system behaviour.
   • Narrative/commentary (to support re-runs & regression testing.)
   • Sign off.
   • Tester’s name & signature.



5. Test result
   
   • Expected result.
   • Method for checking actual against expected (if not just “Check actual vs expected results” - including automated file comparisons, etc., as appropriate. E.g., checking back-end systems, end-of-day report, messages, etc.)
   • Pass/fail.
   • Cause of failure.(Eg, “Comm320 failure”, “Data feed”)
   • Defect reference field (to locate defect reports, anomalies, etc. my be needed at both step and script levels.)

Try it. Really, it works.

Professionals and practice

Interesting discussion at my current client, about what to call the change management organisation. At the moment they planning to call themselves 'the Change Practise', but the desired effect - of being compared with legal, medical an other sorts of professional 'practise' - is being undermined by a barrage of ribald jokes about 'still having to practice' - exactly the opposite of what was intended.

The difficulty, as far as I can see, is two-fold. On the one hand, the business and IT managers I work with aren't professionals. They are often quite good, but they have none of the attributes of doctors or lawyers. There are few qualifications and none of any real substance. In they UK a doctor trains for five years and be formally qualified to a very high standard before they are permitted to treat people independently, but how many weeks does it take a modestly experienced manager to master Prince2? Nor are they obliged to join professional bodies exercising legal powers to strike them off if they aren't competent or are guilty of malpractice.

As for the values to which a manager is subject, there aren't any. Their only obligation is to do the job well enough not to get fired. No professional values, and absolutely none that transcend the interests of their employers - who in turn are under no obligation whatsoever to respect their managers' professional standards or concerns.

And last but by no means least, the quality and performance standards to which real professionals - especially doctors and nurses - are held simply do no apply. Just imagine what sort of state we'd all be in if the average doctor had as many failures and complications as the average project or programme manager!

On the other hand, businesses seem to be under the impression that selling something vigorously enough will somehow make the 'message' true. The discussion this all started from included a very senior member of the executive insisting that we could not call ourselves change 'management' because they wanted the name to convey not just management but also professionalism and leadership. But are they doing anything to empower their managers to lead? No. Are they inculcating a real professionalism? No. They like the sound of these words but, having no real idea what they mean, think that simply reciting them enough will somehow make them true.

So managers are not professionals. Is there any prospect that they could be? In the public sector, perhaps, though the erosion of the independence of civil service under the influence of consultants of all kinds makes that harder to imagine. As for business, absolutely no prospect at all. Managers are too in thrall to the interests, priorities and outrageously anti-professional powers of the businesses they work for.

Recruiting square pegs for round holes

Currently looking for a new client, and as usual most advertisements include the idea that the would-be employer will only consider consultants with a strong background in [insert name of business sector/activity/system here]. Looking at a potential client with a requirement to build IT management systems and processes in the London financial sector, they say that they will only consider candidates with a strong risk system background.

I have worked in lots of sectors – software development, credit cards, insurance, defence, manufacturing, insurance – and I can’ t say that knowing about how the business worked made any substantial difference to how they needed to build their IT development management systems – methods, tools, reporting, etc. Basically, this is one case where the nature of the solution that is being built has far more in common with other solutions of that technical nature (i.e., other IT systems) than it has with other kinds of solution in that sector. So all development methodologies tend to look the same, all project management tools, all reporting systems... And why not? 80% of the time, the code for a word process is indistinguishable from the code for a bank system or a helicopter command-and-control systems.

But of course the business knows best. So they ask someone who knows all about helicopters – or insurance policies, or whatever – to build their processes, and they get ... a lump of dead, mechanical ‘process’ that looks like it was written by Franz Kafka and goes down like a lead balloon with developers.

A related mistake is to recruit someone who is good at a job to build management systems that will help other people do that job just as well. Seems sensible until you ask yourself whether you recruit a racing driver – even a very good one – to design your car. I wouldn’t. I wouldn’t even care if they had a driving license, so long as they had a long track record in designing race-winning cars.

Management methods, models + theories

If, like me, you find management methods, models + theories a lot more interesting than actually managing, you could do worse than look here. It's an index of such things.

Successful managers vs effective managers

Trying to think a bit more clearly about the difference between effective managers (i.e., ones who solve problems) and successful managers (i.e., ones who get promoted), I find myself reading a 1988 paper by Professor Fred Luthans, who was/is George Holmes Distinguished Professor of Management at the University of Nebraska at Lincoln.

Professor Luthans 'found that communication and human resource management activities made by far the largest relative contribution to real managers' effectiveness and that traditional management and - especially - networking made by far the least relative contribution'.

By contrast, 'networking activity had by far the strongest relative relationship to success'.

And in summary, less than a tenth of managers made the top third of both 'successful' and 'effective' groups - which is what you would expect if there were no connection between the two.

Scary? Anyone have any ideas about how valid this finding is? Or how it is possible?

[Update - a quick email exchange with Professor Luthans confirms that, in his view, this is still the position.]

[Further update - I show this to various colleagues. They laugh. Basically, our bosses are blagging their way to the top (for non-London readers, that means getting to the top by less than legitimate means.) And no one is even faintly surprised.]

The objectives of maturity management

The purpose of maturity management achieve the following major objectives:

• To define a strategy for creating revolutionary change by means of evolutionary steps.
• To free leaders from the limitations of corporate management systems by creating management systems that enable leadership rather than constraining it.
• To define a truly manageable management system capable of supporting fundamental, strategic change.

Surprisingly, these are not stated objectives of other management models such as the Software Engineering Institute’s well known Capability Maturity Model or the Project Management Institute’s standards. Nor are they made any easier to achieve by the approach those standards adopt, which is basically pragmatic, eclectic and bound by convention.

These objectives are described in more detail below.

Objective 1: Revolution by evolution

The primary objective of maturity management is to deliver radical, even revolutionary change. That means not merely re-invigorating moribund management systems and staunching the haemorrhages caused by poor management practice, but creating genuinely world-class organisations.

But how is that objective to be achieved? Most approaches to organisational change share at least one assumption: that radical results could be delivered in a single heroic step. Maturity management is based on a quite different assumption: that realistically, radical change can only take place in well-defined, incremental steps, quite probably extending over many years and certainly requiring many discrete developmental steps.

Hence its first objective: to define a sequence of discrete, manageable stages through which radical change can be brought about. Revolution by evolution, in fact.

Objective 2: Freeing leadership from management

One way of conceptualising how maturity management works is in terms of the distinction many authors have drawn between management and leadership. To quote Stephen Covey's Seven Habits of Highly Effective People:

Management is efficiency in climbing the ladder of success; leadership determines whether the ladder is leaning against the right wall.

Other commentators have expressed similar sentiments in different ways, but it is striking that they all insist on this difference and on the importance of leading organisations rather than merely managing them. Leaders bring vision, inspiration and direction, and without it an organisation loses its impetus, its cultural integrity and its ability to take decisive action.

Yet many organisations seem determined to encumber their leaders with unnecessary or subordinate management tasks, even actively disabling them by failing to provide the basic information and decisions real leadership demands.

Of course, no organisation could succeed by completely replacing management by leadership. Conversely, where leadership is not supported by robust management, the ‘leadership’ and ‘empowerment’ routinely degenerates into senior management abdicating responsibility for the actions, accomplishments and performance of their subordinates, backed up by the usual blame and recrimination when things go wrong.

So a balance must be struck – but only the right balance:

• The ability to manage is quite commonplace, whereas leadership is notoriously rare.
• The ability of leaders to delivery results depends on the presence of management systems (including competent and empowered managers) capable of implementing their vision.
• Unbridled, universal ‘leadership’, if not backed up with clear control of the whole, will soon degenerate into chaos, and the whole becomes a great deal less than the sum of its parts.
• Once they have been applied to a range of assignments, many leadership skills can be translated into reliable methods, tools and techniques that can be taught to less inspired individuals.

Hence another aspect of maturity management: by continually upgrading management systems, activities that previously required that rare combination of inspiration and perspiration that defines genius can be done almost as effectively by any modestly capable individual who has been trained to use the appropriate methods, tools and techniques and is supported by the necessary flow of information and decisions. Indeed, the whole history of management consists very largely of the creation of management systems to do things that were previously done only by great leaders. That is one of the main reasons why great organisations – nations, teams, businesses and so on – can exist at all.

On the other hand, where will future leaders acquire the vision on which leadership so crucially depends? Where will they get that spark of insight leavened by sound practical experience? Surely the answer is, yet again, from the management systems in and through which they work. If these systems are bad, then any manager’s experience will be less than illuminating. If, on the other hand, the management systems they use are well designed, effective and properly directed and maintained, their experience of their work, the organisation and its goals will be clear, well-structured and informative. Its purposes, methods and underlying philosophy will be clear and reinforced throughout. Conversely, the better structured the system, the easier it will be to spot any residual problems. But most importantly of all from the point of view of inculcating leadership, the values, purpose and opportunities it faces will be clear.

Hence the maturity management approach: wherever possible it replaces leadership by management. This is not because we should prefer management to leadership after all, but because we should reserve the special talents involved in leadership for tasks where they are really needed. If some leadership skills can be made so straightforward that they happen as a matter of course and the same results can be reliably achieved by the routine use of a management system, this can only strengthen an organisation, and release its true leaders to focus on areas that demand real leadership.

To summarise the whole above argument in terms of a contemporary management buzz phrase, the trick is not to rely on those who can ‘think outside the box’, but to learn from them, and so make the box the rest of us work in bigger. Much, much bigger.

Objective 3: A manageable management structure

If the purpose of maturity management is to achieve radical change by incremental steps, and its principle instrument is the conversion of leadership into management, it is clear that its next objective must be to define a management system that drives change. More precisely, maturity management must tell us:

• To define why the assignment exists, and so ensure that the assignment contributes to strategic goals.
• To define what the assignment will do, and so keep the assignment is kept on track to its goals.
• To define how the assignment will do this, and so ensure that an effective technical solution is delivered.
• To handle all the data and decisions needed to reach the above goals.

To achieve this, a maturity management methodology defines a complete, generic management system consisting of three core components:

• A generic management task model.
• A generic management system model.
• A generic management maturity programme model.

Defining generic management components makes it much easier to define management in terms of discrete units of management activity that are easily understood, easy to implement and use, and easy to revise or replace in the face of new problems and changing circumstances. It also provides the bedrock of the principles of recursion and iteration. Furthermore, by breaking the implementation process into short-, medium- and long-term changes and by embedding the components in a well defined hierarchy of maturity levels, systems and tasks, it is easy to adapt the generic components to local needs and the most appropriate methods, tools and techniques.

Can non-technical reviewers review technical products?

In many (perhaps most) of the organisations I have worked in, the question of who reviews what has been either contentious, ambiguous or been given a quite unnatural answer. The reason for this unhappy situation is that it is often difficult for a non-technical reviewer to evaluate a project, especially its technical content.

In my own specialist area – IT – this might manifest itself in a pained question such as ‘How can the business approve a change in a database design?’ But there are similar questions in all complex management situations – can techies contribute usefully to business cases, for example?

Good question – and in my experience, people only say ‘good question’ when they mean that there’s no good answer. But in this case, there is an answer, and what is more once the answer is understood it leads to a more robust approach to reviewing generally.

The basic problem is to decide what objective reviewing is trying to achieve, and so to decide whether non-technical (non-business, etc.) reviewers have any role to play in achieving it. To put it concisely, the purpose of reviewing is to decide whether the item under review is meeting its requirements. I don’t mean this in the technical sense of ‘requirement’ – i.e., something the work is supposed to achieve for it to be considered a success. I just mean does it do what it is supposed to do? This might well mean ‘does it fulfil its requirements?’, but it could also mean ‘does it comply with this specification’ or ‘if we follow this plan, will we succeed?’, or any number of other things.

From that point of view, the right reviewers are the people who can – and need to – make that call. But that still doesn’t mean that they are technically capable of understanding the item they are reviewing. Or are they? In what sense do they need a precise technical understanding of the content of the item – for example, a design document - to be able to evaluate it? To put my complete argument in a nutshell, what I am getting at is the idea that reviewing is based not on what it says so much as on what it means.

To go back to my problem about business people reviewing a change to a database design. Can they understand what the change says? Probably not, if by that you mean their grasp of namespaces, indexing and denormalisation issues, and their opinion of such things, in strictly technical terms, is probably worthless.

But that isn’t necessarily all that the review is for. For behind every such technical change, there is a pyramid of managerial and business implications that non-technical reviewers can not only understand perfectly well but are probably better judged by non-technical people.


This is illustrated in the following diagram (click to expand):

Hopefully it is clear what the diagram implies. At the lowest level, where the database change itself occurs, there is probably little benefit to be had from asking non-technical people what they think of the change from a purely technical point of view. ‘Who knows, and who cares?’ is probably the right answer. But as soon as the wider implications of the change – the non-technical elements of what the change means rather than the details of what the change documents say – start coming to the fore, both their interest and their ability to judge should start to grow rapidly.

For example, assume that the database change in question is to move from a distributed to a centralise structure. Although the technical issues will be beyond the business’ grasp, and so will most of the implementation and operational issues, not much else should be beyond them. Looking at the diagram again, what are the changes in test requirements this database change will call for? To have all your database testers in one team, located centrally, rather than separate teams all around the business? What does that entail? Much lower costs? Great, we’ll have it. And a simplified roll-out that can now happen three months earlier? Even better. But what is the downside? The changes in platform mean that we will need to recruit a whole new database team? How long will that take? What will it cost? Oh... not such a no-brainer then. And there’s a small chance that we won’t be able to meet our delivery timescales after all? But at least the total development cost will be well down? Great! But the operating cost will in fact go up? Damn...

It’s a complicated business, as anyone who has been in such a situation will testify. But perhaps it should be – and perhaps excluding the business (and other non-technical people) from reviews on the grounds that they ‘won’t understand’ what they are reviewing is not only a very narrow interpretation of what ‘understanding’ means in such a situation but positively counter-productive. After all, if you don’t ask them now, when will you? When it’s too late?

Of course, it’s not easy to make sure a review like this is successfully executed. It’s very hard to work out the real implications of as subtle a thing as a database change. But if you are the project manager and you can’t tell your customers what the consequences of your project really are, perhaps you should be finding out. After all, it’s not as though they will never find out. But the alternative to telling them in an orderly and systematic manner like the above can only be finding out through missed milestones and blown budgets.

In a way none of this should need saying – anyone who does a change request nowadays will perform an impact analysis that covers most of these issues. But as so often in project management, this simple lesson simply has not spread in the systematic manner to areas like reviewing (product or project) as one would have hoped.

What makes a good manager?

Five minutes ago a junior project manager in the company I am currently consulting to came to me and asked me a simple question. What makes a good project manager?

I asked him why he wanted to know, and he said, ‘Because that’s what I want to be, and you’re an outsider who knows about this stuff’. So this is what I told him. I think it applies to other kinds of manager too.

Master the basics

There is no substitute for doing the basics and doing them well. Planning, risk management, dependencies, team-working, and all the rest. And make sure that you really do them well – that issues and actions are routinely tracked, that you write formal work packages rather than just assuming that what you have said is clear, and so on. Qualifications are nice because they provide a structured model for management, but just learning to do the basics properly is a matter of attention, responsibility and commitment – virtues without you will never be a good manager.

Be professional

You can acquire any number of professional qualifications and certificates. But for other people to think you a professional, you need to be worthy of the name.

For example, when I recently gave a course on delivery management and suggested that the project manager’s ultimate responsibility was to make sure that the benefits our colleagues hope for are really achieved, one very senior manager responded with ‘That’s not my job’.

Well, no, if you define your job by your formal job description, maybe it isn’t. But as Max Weber – the man who practically created the idea of professionalism – said, an employee is someone you pay so you can tell them what to do, whereas a professional is someone you pay to tell you what to do. I’m pretty sure that a good project manager is a professional, not just an employee, and I cannot imagine how a project manager could ‘tell me what to do’ without understanding what, ultimately I was trying to achieve.

So if my plan says ‘meet these requirements’ or ‘deliver this system’ then I don’t doubt that I will fail if I don’t. But I cannot see how I can really succeed unless I understand the benefits you think these requirements and deliverables will give you, and make sure that I work towards them too.

There are a million other things a professional manager does as a matter of routine. Many of them are not unlike being a consultant. Indeed, I don’t think there’s much difference between a good manager and a good consultant – they tend to merge in direct proportion to their maturity. Think big, take pains, even all do those clichéd things like ‘going the extra mile’. And do not think ‘outside the box’ – just recognise that there is no box unless, out of fear or ignorance, you put yourself in one.

Master the corporate management system

The third (and probably biggest) item is harder, as it is not in the individual manager’s power to do much about it. It’s the corporate project management system – that can make more difference to being a project manager than anything else. In fact I regard it as the litmus test of my own work building management systems – that managers don’t have to do anything but manage. All the rest – the petty bureaucracy and standards and elementary tools, techniques and templates – are all provided in easily used form by the policies and systems and training I create.

On the other hand, if your system isn’t quite that friendly, it is still open to the good manager to take a hand. Except in the kind (happily rare) organisation where authority is either so rigid and draconian that you would be far better off getting another job, the reason the management system doesn’t support you is usually either because you don’t understand what it is trying to do or the people who manage it aren’t telepathic and they did not know that it was causing you problems. If it’s the first (and regretfully, it often is) then once you understand the problem (hopefully) goes away, but if it is the latter, then what is stopping you getting them to change it?



Ultimately you will know when you are a really good manager – it is then that working really is more fun than fun.