When are your requirements ready?

A very common failing in all sorts of projects is being stuck with what are in fact quite inadequate requirements.

The fact is, most organisations are pretty bad at explaining exactly what it is they want a project to accomplish. There are lots of good reasons for this - the situation at the start of a project is often fluid or unclear, there are too many options to be precise, it's hard to define a truly innovative idea in detail until you have tried it out, conditions and priorities change as the project proceeds, better ideas surface, and so on.

But that's not the same as simply being bad at requirements. The above issues relate mainly to the content of requirements, which is very hard to nail down definitively; what I am talking about here is their quality - a different issue. Badly defined requirements area major cause of problems for projects and businesses alike, causing the routine delivery of the wrong thing and lots of unhappy stakeholders. Fortunately there are ways of ensuring that the requirements - such as they are - are at least defined well enough for the project to proceed reasonably comfortably.

The issue I have in mind is the requirements review process - how requirements get signed off. There are lots of things you can do to make this fairly robust, but one technique I have seldom seem defined clearly enough is that of asking the requirements' principle users to confirm that they are fit for purpose.

There are three key groups of people who have an interest in how well requirements are stated:
- the analysts who will have to translate the requirements into a functional solution.
- the (user and operational) acceptance testers who will have to check that the requirements have been met.
- the operations personnel who will have to convert the requirements into SLAs and OLAs.

These people are not generally very interested in the requirements contents. But give them poorly quality requirements - too vague, imprecise, unanalysed, inconsistent, with key areas missing, and so on - and they simply won't be able to do their job. Which means that the solution the project delivers is all but bound to leave everyone with a nasty taste in their mouths.

All I am advocating here is that requirements be signed off by these groups. As far as I am aware, although many organisations ask their analysts to approve requirements, most don't ask testers or operations staff, and this may be a major cause for project failure. It's not hard to arrange, and it should certainly be welcomed by the groups in question. If, on the other hand, you cannot get their approval, maybe you should be looking to the users to define what they want a little more clearly - and so avoid storing up trouble for the future.

Of course, there's also a case for weak requirements. Well, not exactly weak, but at least requirements that recognise that they may not be the last word. Organisations and businesses change. So do markets and so does good practise and the opportunity the requirement was originally designed to address. So if you're embarking on an 18-month project to deliver x, it's probably not too smart to try to nail down your definition of x too soon. But that is not to say that you should not try to meet the above test from the start - it's just that your project should also make a strong allowance for change. That could mean a large tolerance, but it should also mean setting the right expectations from the start. For example, the business and users should expect to have to re-validate their initial requirements at regular intervals, you might want to prioritise items that are pretty safe from future fluctuations (e.g., stable regulatory requirements or generic interface components), design for flexibility (loose coupling, modularity, etc.) and so on.

This is the case for agile, of course - but of that more than enough has already been said by anyone and everyone, including me!

Moving to Agile - Identifying the truly essential documentation

A crucial feature of migrating to Agile is identifying the irreducible documentation needed to support the process. For many Agile practitioners the very notion of a fixed documentation suite will raise their hackles, but there are many stakeholders, not all of whom are not directly involved in the delivery process, but all have interests that must be taken into account.

To name only the most obvious, there are three general classes of stakeholders, each with their own distinct information needs:

1. The delivery team itself
2. The business
3. Operations, compliance and support

These groups are already often poorly served by waterfall processes; the risk is that Agile’s understandable and largely justified desire to take an axe to the great mass of pointless documentation will result in them being served still worse. So…

The delivery team

Well, obviously. And generally speaking the Agile assumption that there is, by default at least, no need for internal project documentation, holds good. Why would I need to write down what I can tell everyone in a tenth of the time? Why write down anything at all if its lifetime is likely to average 15 days?

But less obvious, is it in fact the case that they need nothing? In very simple projects, yes, it usually is, but in a more complex programme – for example, a dozen Agile workstreams flowing into a common integration/release cycle – they may well need a good deal more. The PM is likely to be unavailable (doing programme-level duties at meetings and forums) and when they return to base, unable to convey everything by word of mouth.

Then the standard criterion applies – fitness for purpose. But in a project of any significance (size or importance), that rule is unlikely to result in their being no documentation at all. Add to that the other reason why documents exist – because the project is innovative, organisationally complex, heavy with technical or regulatory content, and ‘pure’ Agile starts to look a bit thin. Absolutely never create more documentation than are strictly needed, but don’t assume that the minimum set is zero. That is usually wishful thinking.

Business stakeholders

Yes, a properly empowered business rep should allow you to dispense with almost all external review and approval, but even in the most benign environment, there are likely to remain yet higher level reports. Much as most of us would like to dispense with this too (why don’t they just have faith in us?) but ultimately projects exist solely because they serve the interests of the organisation as a whole, and someone up there really does need to know what you are up to. So there is all but bound to be some form of reporting.

However, taking a constructive approach to this can also encourage those to whom reports are due – PMOs, line managers, HR, finance, direct reports to senior stakeholders, etc. – that they don’t really need the detail they are probably accustomed to. Reporting strictly by exception is the starting point, as it is that, rather than no reporting at all, which is the true corollary of empowerment. Using the move to Agile to rationalise reporting isn’t a bad idea either.

Operations, compliance and support

Now we come to the groups who are most in need to solid documentation and most likely to be neglected. In many organisations the needs of operations, service transition and support teams are already poorly met, and Agile is unlikely to do them any favours. Nevertheless, it is crucial that their needs are met, and these go far beyond touching base with them or even having them represented on the project. They will spend far more on the delivered system than the developers, and the better equipped they are to manage the delivered solution, the better for everyone.

Operations, compliance and support will make up the great majority of all but the most superficial IT projects (e.g., throwaway web pages or transient rate changes), and their needs are not only substantial but also penetrate deeply into the heart of what is being developed. They all need to be consulted about the original requirements (e.g., to define SLAs) and strategy (e.g., to ensure sustainability), they all need early warning of what, functionally and technically, is coming down the track, they all need to know what exceptions have arisen as the original requirements/backlog is progressively shaped and re-shaped, and they all need to be aware of the (now multiple) schedules for release.

Conclusion

So what does this all add up to?

1. There is a very great deal of documentation that genuinely can be thrown away – and good riddance. It adds nothing but dead weight to the project and should be discarded whenever possible. Which is remarkably often.
2. Quite a lot of other stuff needs to be transformed. It is very likely that groups such as administrative functions (PMOs, HR, finance) will have to change the way they think about projects and how they are reported on, but it is unlikely that they will eschew reporting altogether. Nor should they. Likewise for business stakeholders – they need to assign real power to their business representatives within the project, but they are unlikely abandon all visibility. And given their responsibilities, how wise would it be?
3. Finally, there are those who will need what they have always needed. But if you don’t give operations and support what they need, they probably won’t notice much: by and large they aren’t getting it now either.

Moving to Agile - Critical business factors

Some years ago I implemented DSDM (then the preferred flavour of Agile) at Churchill Insurance. As far as the core processes were concerned, there was no great difficulty (admittedly we threw everything at it), but the business was a problem for two key reasons: empowerment and availability.

The nub of the empowerment issue was asking the business to allow their own representatives on the project enough authority to approve of what was going on on the spot (e.g., prototypes, changes, reprioritisations, etc.), without constantly referring to senor management. In short, they could not let go of the strings. As anyone who has worked in IT projects for any length of time will know, there is a ludicrous irony in this, because quite a lot of the problem with delivering successful IT projects of any kind is the business’s ambivalent attitude the ownership. In all too many companies, the business want to be in control of the project (i.e., able to make make-or-break calls about it) without taking responsibility for its delivery. The implicit question is always, do you prefer delivery to control?, and all too often the effective answer is ‘control’. This is as much a problem for Agile as it ever was for waterfall projects.

The second problem was getting the required effort from the business representatives to the project. These individuals were necessarily very experienced and valuable people (who else would you empower?) but that was precisely what made them hard to replace in their day jobs. So we needed them to spend about three days a week on the project, but they still needed five (usually very long) days for their BAU work. No surprise, then, that project work soon started the back seat to day job, the quality and speed of decision-making plummeted and the project started to look pretty wonky.

So although the business was keen on an Agile approach, they could not participate effectively. It was a long struggle – only partially successful – to deal with this problem.

BPM, SOA and the relationship between business and IT

I am struck by the fact that discussions about the perennially fraught relationship between business and IT always seems to omit reference to the very different natures of business and IT. IT reality has always worked more or less at right-angles to business reality - an engineering discipline in the midst of business. The links between are generally limited to very high levels at which the relationship is mediated by very general language that does not really require either side to have any insight into the other (i.e., requirements), and at very low levels that actually have little influence over the relationship as a whole (i.e., day-to-day system users).

While this remains the case, there is no practical need for this relationship to get any better, and each side will continue to prioritise their owns engineering/business perspective over the other side’s concerns. And while IT technology always reverts to a strictly engineering mode of thought and the business side lacks a genuine ‘engineering’ element of its own, it is very unlikely that this impasse will be overcome. We can all agree that ‘something should be done’ and even agree what it is, but however important this is, it will never be urgent, and certainly not capable of being built into both sides’ day-to-day ‘common sense’.

However, my own impression is that there are developments in the pipeline that will overcome this. On the one hand, the emergence of service-oriented architectures in IT is forcing IT people to define what they do (even for themselves) in terms even the business can understand. On the other, the (rather slower) emergence of formal business process management is creating a kind of ‘business engineering’ that obliges businesses to thing of their own activity in quasi-technical terms that even an IT person can recognise. Between them, SOA and BPM are creating the language, the conceptual framework, and the technical toolkit business and IT need to create that ultimate goal, a unified business/IT worldview.

Not that this is either all that is wrong or likely to offer a complete solution. All the same, I doubt that there will be any solution until business and IT alike learn to think of themselves in such terms. This does not mean that IT people need to understand business or vice versa, but it does mean that they need to think of themselves in mutually compatible terms. Which is what, I think, a combination of SOA and BPM will achieve – without either side intending it.

Stakeholder management links

I have recently started to look at stakeholder management. As usual, I began by cruising the internet for useful sites. One helpful one I have found is the Project Stakeholder Management blog at http://www.projectstakeholder.com/. Its case studies and brief articles are especially helpful, as they include many substantial ideas and interesting examples, generally from outside management. I have certainly plagiarised it shamelessly (and I hope they will return the compliment).