Minimising bias of subject matter experts through effective project management

“Of all the causes which conspire to blind Man's erring judgment, and misguide the mind; What the weak head with strongest bias rules, - Is pride, the never-failing vice of fools”
Alexander Pope (English Poet, 1688-1744)

“Bias and prejudice are attitudes to be kept in hand, not attitudes to be avoided.”
Charles Curtis

The use of SME’s (Subject Matter Experts) is commonplace throughout the lifecycle of a project. The “experts”, as we will generally refer to them, are typically functional experts in their respective roles that the project manager relies on for making delivery estimations and identifying potential risks to a project. Depending on the dynamics of your particular organisation, such experts may come from the same functional team that will be responsible for the execution of the tasks for which the experts are providing input, or they may be in a specialist division that deals with project set-up. In either case, there are a several risks which the project manager needs to look out for to avoid being given an impossible or very difficult delivery task.

In many business environments, the experts available for a project are often within the same functional group that will be responsible for the execution of tasks on a project. Let’s consider for example, project Alpha’s objectives include designing a new car headlamp for Automotive Company XYZ. The experts used for this project are likely to include engineers, managers and/or senior personnel from various departments that will be responsible for the development of the product. The potential pitfall in using experts from the same group that will ‘implement it’ is that they accept the estimates without accounting for any bias that may be lurking in the estimates. This bias may be consciously or subconsciously included from the experts. Once the scope and requirements have been finalised for the project, the duration estimates are generally finalised with the input of the experts. In order to meet or exceed the duration estimates provided, the expert’s estimates may actually be “worst case” scenario unbeknown to the project manager, which has ensuing implications on project budget and schedule. The “Peter Principle” risks coming into play, by which the work expands to fill the time available and we do not get the optimum delivery result. Conversely, estimates from experts which are overly optimistic risk putting the Delivery team to the sword – and put the project budget at risk. The conclusion: Project managers should make sure that expert guidance should be “peer reviewed” by an external person to ensure it is reasonable.

Bias is not isolated to the work estimates of a project. It is also connected to the level of risk involved – whether for an estimate of work or generally assessing project risks. Risk, according to PMBOK, is calculated as Probability * Impact. Yet risk depends on the project team’s appetite for it – particularly the Client, Are they risk averse or risk-seeking? Depending on the environmental factors of your organisation, how you obtain the probability and impact may vary. However, most calculations of impact will be based on the subjectivity and experience of experts. Take the example below, where the project team has agreed that impacts should fall into one of five categories (see figure 1.1), and any risk rated over a 2.0 impact should require mitigation planning.

Which category a risk should fall into requires a degree of subjectively from those assigning the rating; which in most situations involves expert opinion. If the project manager relies only on a few experts to assess impact, potential identified risks may be understated, remain un-mitigated (not acted upon) and later cause issues for the project. Equally, true mitigation factors may not be properly assessed (the experts will not know as much about the project’s situation as other members of the team, or other stakeholders).

What tasks or steps should a project manager take to minimise expert bias? The project manager should, where possible, draw on a pool of experts, whom are not on the project team nor responsible for the project’s tasks, to validate the assumptions, estimates, and risks provided on a project. Using a pool of experts should be considered more carefully, especially if the project will employ contractors. If a pool of experts is not available, the project manager should use assumptions and constraints methods in calculating estimates to be included in the response, as well as having PERT three point estimation [ (O+(MLx4)+P)/6 ].

In the calculation of risk, the project manager should not limit the input to that of experts. Building on the risk example above, the probability of risk B was calculated using Monte Carlo analysis to be 47 per cent. What value should be assigned to the “Impact”? If the project manager accepts only limited expert input, there are added risks that the impact may not be accurate, thus not mitigated.

This article suggests that each project risk should be carefully assessed by the stakeholder group for the project. The ‘aggregated mean value’ of the assigned impact should be used to make a reasoned decision. Whether or not you weigh each of the stakeholder’s responses the same will be based on your organisational factors. If you do weight their responses, think carefully about whether the weighting should or should not be shared with stakeholders, as this can lead to arguments or disagreements. People may not like to know that their option is weighted less than another’s, but sometimes it may be necessary. See figure 1.3.

If we go back and use the aggregate impact, we find the risk rating as follows: See figure 1.4.

Summary Using Aggregate Risk Rating, and putting in place “Expert Pools” are just a few examples of how project managers can take steps to minimise the bias of expert opinion in making decisions for their project (throughout the various phases), to make sure the overall project risk and estimates of work are reasonable for the tasks required. Using a simple risk table (Probability x Impact) to validate your information can help you achieve this in a measured way.

Other articles by these authors:

Read more in CIO Management.

Copyright © 2010 IDG Communications, Inc.

7 secrets of successful remote IT teams