Lies, lies and damned performance measures
by Bob Palmer

A decade ago performance measurement in the public sector was all the rage. How is it faring today? The evidence points to street-wise managers manipulating performance measures to cover up poor performance.

In much of the public sector the bottom line of profit does not exist. The effectiveness of public services is therefore difficult to assess. Evaluation relies on a range of performance measures (PMs). Without these performance measures public sector managers are in danger of allocating valuable resources in the dark. In the early 1990s performance measures became highly fashionable and something of a cottage industry. Although local government had been users of PMs for some years they came to prominence again through various central government performance review initiatives and the advent of the Executive Agencies.

It is interesting a decade on to pause for a moment and consider how useful PMs have been in telling us whether our money has been spent wisely or not. My research shows that Local Government and the Executive Agencies remain effective users. However in Central Government, and in certain public sector organisations such as the Police and the National Health Service, the results have often been disappointing. At the very least many performance measures are diffuse, dull and ineffectual. Often systems have become an end in themselves. The "we may not have many bobbies on the beat but our PMs are red hot" syndrome. Success is often measured by whether the latest techniques are being used rather than by the results they may produce. More worrying is that street-wise managers have learned to manipulate measures to show apparently good performance when the opposite is the case.

Take a topical example. A major performance measure in the Health service is waiting lists. How easy to reduce the numbers on the list by concentrating resources on the easy and quick operations at the expense of difficult and lengthy operations. Another favourite Health Service measure is waiting time in A & E. Here again this measure can be made to look a lot better if the time of arrival is taken from when the patient is first seen by the doctor rather than the time of arrival in A&E. Another key measure "Average bed stay" can be equally misleading. Measures of clinical outcome such as "readmission rates" and "mortality rates" need to be considered in conjunction with ‘bed stay’.

Another measure open to manipulation is that old favourite used by the Police; "crime clear up rates", known by the FBI as detection rates. Crime clear up rates can be improved dramatically if you know how to play the game. First, as in the case of NHS waiting lists, you tackle the easy cases and leave the tricky ones for another time. Secondly, you decide to spend more time visiting prisons in order to persuade prisoners to have other crimes taken into account ( the infamous "TIAs"). This way you boost the clear up rate with a minimum of effort.

It is important therefore for PMs to differentiate between ‘primary’ clear up rates and ‘secondary’ clear ups: ie TIAs. Further it is necessary to measure the quality of the arrests using the percentage of arrests leading to conviction or indeed surviving preliminary hearings. It was interesting that in one US city a sample of 112 felony arrests showed that only 43% of arrests passed the preliminary hearing.

To further muddy the waters it is evident that crime is often seriously under reported, often by as much as 40%. Therefore some sampling of public opinion, perhaps using focus groups, is necessary to measure trends more accurately. A high under reporting of crime can suggest anything from fear of reprisal to lack of responsiveness by a Police Authority.

The foregoing are perhaps extreme examples of how measures may be manipulated. Just as misleading however are where bland intermediate measures are used rather than measures which are capable of testing incisively the final outcomes of public sector programmes. Take a programme introduced under the Tories some years ago called Estate Action. This was a well-meaning programme aimed at improving the quality of life on council housing estates in Inner City areas. The programme managers used a plethora of measures none of which really showed how successful the programme was. For example a key measure used was the number of dwellings renovated in the year compared to target.

This measure said nothing about the quality of life. More pragmatic measures such as ‘what percentage of residents had asked for a transfer to another estate?’ or ’what percentage of residents wished to purchase their own house on the estate?’ or ‘what percentage of finance is provided by private business?’ would have provided more useful management information.

An important aspect of performance measurement is to ask the ‘customer’ what they think of a particular service or programme. In the case of programmes like Estate Action the use of small focus groups drawn from residents proved useful together with sample customer surveys on such matters as fear of crime, vandalism and pollution. Imagination is a useful virtue in finding ways of designing performance measurement systems and often the off–beat measure can be the one which throws a revealing shaft of light on performance. For example asking the views of the concierge of an inner city estate can be worth a raft of performance measures. The incidence of bed sores in a ward may be good indicator of overall quality of nursing care and staffing levels. The street value of drugs can often be an effective measure of police enforcement.

A common problem faced by senior managers or politicians in interpreting performance measures is that they often see only misleading aggregates. Ministers have been embarrassed in the past when a parliamentary question has been asked about, say, a war widow who has not received her pension on time even though overall 99.99% of widows are paid promptly and accurately. In any performance report to top management the average figure should be accompanied by ‘outliers’, ie: the best and worst performance reported from the lowest levels say at individual DSS offices.

A few final thoughts on good practice in designing performance measures.

• PMs should be vivid and pragmatic and potentially poor performance should leap off the page.
• Pictures, graphs and brief narratives should be used.
• Performance reports should not be cluttered with too many measures.
• Although a rounded mix of efficiency and effectiveness measures should be used, no more than six or seven measures should be necessary for a specific programme or operation.
• Often PMs are designed to strain for unnecessary mathematical exactitude. It should be remembered that performance measures are only indicators (although academics will happily argue for hours about the arcane differences between measures and indicators) to alert management early to potential problems.

Below are some outline criteria for good performance measures.
Most importantly, performance measures should be designed to generate action. I have seen the most elaborate systems, with dozens of performance measures, lovingly nurtured and refined and presented in gilded reports yet these reports often show huge differences between comparative performance levels but with no evidence whatever of anybody asking questions let alone taking remedial action. A grim reminder of this was seen recently in national comparisons of child mortality rates. The tragedy here was that although the right measures were in place, extreme examples of bad practice were not addressed as promptly as they should have been.

There is little doubt that performance measurement has taken firm root in the public sector and overall provides useful information on services and programmes. However, the age of innocence has long gone and auditors and others with responsibility for performance review in particular should be alert to ‘gaming’ and the cynical manipulation of PMs in order to cover up poor service to the public. One needs to be even more alert where substantial performance bonuses are involved.

MEASURES FOR MEASURES

Evaluate each measure to see

That it is; If so, then it will be;
Objective-linked Directly related to clearly stated objectives
Responsibility-linked Matched to specific organisational units that are responsible for taking action to improve performance.
Organisationally acceptable Valued at all levels, used as a management tool, and ‘owned’ by those accountable for performance
Comprehensive Inclusive of all aspects of performance, economy, efficiency and above all effectiveness. There is no point in doing the wrong things well
Credible Based on accurate data sources, not open to manipulation or distortion
Cost-effective Acceptable in terms of cost to collect and process
Compatible Integrated with existing information systems
Comparable with other data Facilitates comparison from period to period, with peers, to targets etc. Includes "outliers"
Easy to interpret Presented graphically and accompanied by commentary.

Close window