WORLD HEALTH REPORT 2000
A brief, critical consumer
guide
Erik Nord, PhD, National Institute of Public Health,
Oslo, Norway. e-mail:erik.nord@folkehelsa.no
October 2000, published in Health
Policy 2002, 59, 183-191.
The paper briefly explains all
the methods behind the indices in the World Health Report.
The WHO has designed a rich set
of primary measurement tools by which different countries’ goal attainment and
performance in health care may be judged. The WHO has in my opinion gone too
far in compressing the results of these potentially useful primary measurements
in summary indices with unclear meaning, dubious validity and little practical
relevance.
The WHO has also gone too far in
applying the same measuring rods to countries with different histories and
values and different stages of development, and in encouraging international
comparisons that are of little practical relevance.
The WHO needs to add an indicator
of equality in access to its present indicator of fairness in financing.
Decision makers who, in spite of
the above criticisms, are inclined to regard the summary indices of the World
Health Report as useful, should be aware that the assignment of weights to
different aspects of responsiveness and overall goal attainment is difficult
and may be culturally dependent. No country should uncritically accept the
weights suggested by the WHO as being scientific or objective or correct. Every
country should judge carefully whether the WHO weights fit with the country’s
own values, and thereby judge the relevance of the various indicators and
indices for its own policy making.
Five main goals in health policy.
Operationalisation of attainment of five goals.
Assigning weights to the main goals.
Calculating overall goal attainment: The example of Norway.
A comparison of goal attainment in some developed countries.
The performance of a health system: Definition.
A comparison of countries with respect to performance.
Details about methods:
Disability adjustment of life years
How much does disability adjustment of life expectancy matter?
Some problems with the report.
The purpose of this paper is to give a brief summary of the World Health Report 2000, including explanations of the methods underlying its various indicators and indices.
While the laudable ambitions of the report are duly acknowledged, questions are raised about the validity and fruitfulness of a number of the report’s measures.
Text and tables are presented such as to allow the creation of overheads by some simple editing. Readers interested in using the paper for this purpose are welcome to do so.
Any feedback will be appreciated.
There has been a demand from member countries in the World Health Organisation for systematic comparative evaluation of different health systems. The WHO has responded by creating a set of indicators of goal attainment in health care. These indicators have in turn been used to create two main single index measures according to which all countries in the world are ranked. One is of ’overall goal attainment in health care’. The other is of ’health system performance’, which is overall goal attainment relative to available health care resources and human capital.
The indicators and indices are explained in the World Health Report 2000 (WHO, 2000) and a number of discussion papers, most of which are available on the WHO website.
The report is highly innovative in its various ways of operationalising goal attainment in health care. It is based on impressive data collection world wide, uses advanced statistical techniques to analyse the data and provides lots of interesting reading for researchers and health policy makers.
The report has nonetheless come about in a very short time and in the absence of peer review. Many questions may be raised about the validity of its various indicators and particularly the fruitfulness of evaluating and comparing complex, multifaceted systems in terms of single index statistics.
Five main goals
in health policy
The report defines goal attainment in health care in terms of five aspects (details in next section):
Good population health (measured as an average).
Equity in health.
Responsiveness to legitimate non-health expectations in the population.
Equity in responsiveness.
Fairness in financing. Back to contents
Operationalisation
of goal attainment
Life expectancy calculated such that years lived with illness count less
than 1, and less the more burdensome the illness is ( = Disability Adjusted
Life Expectancy = DALE).
Equality in probability of surviving the first 5 years of life in
children born by mothers with different characteristics (age, education, number
of children,etc).
Responsiveness of the health care system:
Index covering respect for patient’s dignity, confidentiality, patients’
autonomy, prompt attention, quality of amenities, access to social support
networks, freedom to choose provider.
The more subgroups that are treated with less responsiveness than the
majority, and the greater these subgroups are, the lower is the country’s score
on equity in responsiveness.
Defined as proportionality between a household’s total expenditure on
health care (taxwise and out of pocket) and its permanent income above
subsistence level (defined as total private expenditure plus direct tax payments
minus expenditure on food).
Assigning weights
to the five main goals
1000 people from 125 countries, half of them WHO staff,
were asked to judge the importance of the various goals
relative to each other.
According to the report, there was large agreement between
different groups.
The following weights were assigned:
|
Average health |
0,250 |
|
Equity in health |
0,250 |
|
Responsiveness (average) |
0,125 |
|
Equity in responsiveness |
0,125 |
|
Fair financing |
0,250 |
Calculating overall goal attainment:
Using Norway as an example.
|
Aspect |
Raw score |
Transformed to 0-100 scale |
Weight |
Contribution |
|
|
Health (DALE) |
71.7 |
86.2 (1) |
0.25 |
21.54 |
|
|
Equity |
0.999 |
99.9 |
0.25 |
24.98 |
|
|
Responsive- ness |
6.98 |
69.8 |
0.125 |
8.73 |
|
|
Equity in responsive- ness |
0.995 |
99.5 |
0.125 |
12.44 |
|
|
Fair finan- cing |
0.977 |
97.7 |
0.25 |
24.43 |
|
|
Overall score for goal attainment: |
92.2 |
||||
Margin of uncertainty (80 % confidence interval): 91.4-93.1
Transformation of DALE: (71,7-20)/(80-20), where 80 is a stipulated expected lifetime if there were only ’natural death’, and 20 is the ’least conceivable expected lifetime in any system’. The point is to count only those years that are achieved beyond those that come ’for free’. Back to table.
A
comparison of some countries with respect to
goal attainment
|
|
Japan |
Norway |
Finland |
USA |
Cuba |
|
Life exp. |
81.0 |
78.6 |
77.0 |
76.8 |
75.5 |
|
DALE |
74.5 |
71.7 |
70.5 |
70.0 |
68.4 |
|
Equity |
0.999 |
0.999 |
0.975 |
0.966 |
0.938 |
|
Respon- siveness |
7.00 |
6.98 |
6.76 |
8.10 |
4.97 |
|
Respons. equity |
0.995 |
0.995 |
0.995 |
0.995 |
0.920 |
|
Fair finan- cing |
0.977 |
0.977 |
0.977 |
0.954 |
0.972 |
|
Overall goal attainment |
93.4 |
92.2 |
90.8 |
91.1 |
84.2 |
Margin of uncertainty (80 % confidence interval): +/- 1
The performance
of a health system
Simplified explanation1:
Goal attainment tends to increase with the availability of resources. Performance is defined as overall goal attainment relative to resource availability.
Availability of resources is measured in terms of health care expenditures per capita and average years of schooling in the adult population.
Countries that are similar with respect to expenditure per capita and average years of schooling in the adult population are grouped together.
Within each group the overall goal attainment of the best country in the group is regarded as the maximally achievable at that group’s level of expenditure and education.
A country’s overall performance is defined as the ratio between actual goal attainment and the maximally achievable goal attainment in its group.
Performance is also measured with respect to DALE only, i.e. as actual DALE divided by maximum DALE in the group in question.
Examples
of performance scores
|
|
Japan |
Norway |
Finland |
USA |
Cuba |
|
Life expectancy |
81.0 |
78.6 |
77.0 |
76.8 |
75.5 |
|
DALE |
74.5 |
71.7 |
70.5 |
70.0 |
68.4 |
|
Goal attainment |
93.4 |
92.2 |
90.8 |
91.1 |
84.2 |
|
Health expen-diture per capita (USD) |
1.759 |
1.708 |
1.539 |
3.724 |
109 |
|
Performance on DALE |
0.945 |
0.897 |
0.829 |
0.774 |
0.849 |
|
Performance overall |
0.957 |
0.955 |
0.881 |
0.838 |
0.834 |
Details about
disability adjustment of life years
The conventional concept of life expectancy in a cohort is calculated by summing all lived person years in the cohort and dividing by the number of people in the cohort. Each person year counts as 1.
With disability adjusted life expectancy, years lived with illness count less than 1, and less the more burdensome the illness is.
How much less years with different illnesses should weigh has been determined on the basis of responses by international panels of health personnel to questions as to how they would prioritise between different preventive health programs affecting different numbers of people (the person trade-off technique).
Examples of weights for years with illness in calculations of DALE and corresponding disability weights in calculations of Burden of Disease:
|
Illness |
Value of year |
Disability weight |
|
Severe sore throat |
0.92 |
0.08 |
|
Rheumatoid arthritis |
0.79 |
0.21 |
|
Deafness |
0.67 |
0.33 |
|
Blindness |
0.38 |
0.62 |
The effect of
adjusting life expectancy for disability
|
Country |
Life exp. |
DALE |
Difference |
|
Japan |
81.0 |
74.5 |
6.5 |
|
Australia |
79.5 |
73.2 |
6.3 |
|
Sweden |
79.5 |
73.0 |
6.5 |
|
France |
79.2 |
73.1 |
6.1 |
|
Spain |
78.7 |
72.8 |
5.9 |
|
Norway |
78.6 |
71.7 |
6.9 |
|
Finland |
77.0 |
70.5 |
6.5 |
|
USA |
76.8 |
70.0 |
6.8 |
|
Denmark |
75.5 |
69.4 |
6.1 |
For most countries it has so far been possible to use only child mortality data. When more complete data are available on inequalities in adult mortality they will be used in future WHO estimates.
Operationalising
’responsiveness’
50 informants in each of 35 countries (all regions, none rich)
filled in a questionnaire regarding their own country.
For each of the seven aspects of responsiveness they answered 3-7 questions (items).
Having first answered the individual questions they proceeded to give a score between 10 and 0 for each aspect as a whole.
For each aspect, the mean of the scores given by the 50 national informants was calculated.
1000 subjects world wide ranked the seven aspects with respect to importance. On the basis of these rankings, the aspects received the following importance weights:
|
Dignity |
0.167 |
|
Confidentiality |
0.167 |
|
Autonomy |
0.167 |
|
Prompt attention |
0.200 |
|
Quality of amenities |
0.150 |
|
Access to social support network |
0.100 |
|
Freedom to choose provider |
0.050 |
Mean scores on the seven aspects (see above) were then multiplied by their respective weights and summed, to obtain an overall score for responsiveness.
Scores for other countries (than the 35) were estimated by means of variables that have been shown to correlate strongly with responsiveness in the 35 countries observed.
Operationalising
equity in responsiveness
50 informants in 35 countries indicated subgroups that they thought were treated with less responsiveness than others.
The number of times a particular subgroup was mentioned (by the 50 informants) was multiplied by its share of the population.
The products for all subgroups mentioned were summed and transformed to obtain an overall score for inequity in responsiveness.
Scores for other countries (than the 35) were estimated by means of variables that had been shown to correlate strongly with equity in responsiveness.
Some problems with the report
As noted in the introduction, the report is highly innovative in its various ways of operationalising goal attainment in health care. It is based on impressive data collection world wide, uses advanced statistical techniques to analyse the data and provides lots of interesting reading for researchers and health policy makers.
In spite of all this, I have serious doubts about the ways in which meaningful, valid and useful primary data on a large number of phenomena in health and health care world wide av been aggregated.and compressed into a small number of summary index scores for each WHO member country. At first glance, many are perhaps impressed by these summary statistics. But I think a closer look will show that they are simply a bit too summary.
The summary index of goal attainment
The authors of the report have an ambition of comparing (a) all countries in the world (b) on one single index of ’overall goal attainment’ (c) estimated in the same way in all countries. I think this is basically useless, for the following reasons:
First, the index covers phenomena that are not only difficult to measure (quantify) individually, but are also quite different in nature (population health, responsiveness, fair financing) and therefore difficult to compare. They are probably also valued somewhat differently in different cultures. The validity of international comparisons based on the summary index will therefore always be unclear, and many will for this reason disregard the index entirely.
Second, a summary index is of no practical use anyway. Different goals in health care are pursued in separate ’rooms’, by different people. For example, those responsible for promoting average health work quite independently of those responsible for securing responsiveness in terms of respecting patients’ dignity, autonomy, need for confidentiality etc. What may be useful and inspiring to each of these parties is information as to how their country is performing relative to other countries within their own area of responsibility.
Third, a global ranking is of no practical use anyway. Policy makers are interested in comparing the results of their country with those of comparable countries. For instance, Norwegian policy makers are interested in looking at Norway’s position relative to other OECD countries. It is not news, and of nil interest, for Norwegian policy makers to learn that goal attainment in Norway is way ahead of that of poor African countries. The converse is equally true.
In short, I believe a summary, universal index of goal attainment mainly caters to a widespread human attraction to competition as entertainment. The index does little more than to generate pleasant feelings in those labelled ’successful’ and frustration in those labelled ’unsuccessful’. This is certainly what happened in the Nordic countries: The Norwegian Minister of Health was happy with the Norwegian score, while the Finnish and Danish Ministers were frustrated with theirs. None of these would probably argue that the differences in index scores reflect real differences in goal attainment in the three countries beyond what differences in conventional life expectancy already tell them.
I thus believe that nothing would be lost, and much would be gained, by (a) placing countries at different levels of development in separate ’league tables’ and (b) comparing countries in each league table on a handful of indicators of goal attainment rather than on a single, summary index.
The summary index of health system
performance
Goal attainment in health depends on many other factors than health care expenditure per capita and level of education. Some of these other factors – like the AIDS epidemic in Africa – are strongly determined by history and cultural habit, and only to a limited degree within the control of authorities, even in the medium term. The concept of ’health system performance’ is therefore fuzzy and the term somewhat misleading.
The indicator for responsiveness
This is in fact a summary index, the components of which are extremely different in nature (e.g. autonomy versus quality of amenities) and difficult to quantify. Their relative importance is presumably also quite variable across cultures. Furthermore, performance on the various responsiveness items is measured by having panels of people judge their own country. There may be important cross-country differences in scale use and/or self-critical attitude. Altogether I find it very difficult to have much confidence in the resulting overall responsiveness scores. The position of the USA as a very clear winner is an odd result which certainly does not reduce such scepticism. I have the same kinds of objections regarding practical relevance for policy makers here as for the summary index of goal attainment.
The indicator for fair financing
The index of goal attainment does not include any measure of fairness in terms of equality in access to health services. Instead there is an indicator of ’fairness in financing’, which is constructed such that the US scores only 0.024 points less than for instance Denmark, Finland and Iceland. This is counterintuitive. The USA is known world-wide for the failure of its health care system to accommodate people with low income. In everyday language, this is financial unfairness. It is a kind of unfairness to which most people, including many Americans, assign very much weight. This basic value is not captured by the present operationalisation of overall goal attainment. (In fairness I add that this is acknowledged in the report itself (page 38, box 2.3)).
Missing data
Some weaknesses of the report are due to data missing at the moment, rather than problems with the proposed measures as such. As data collection improves, these weaknesses will become lesser worries. Examples are:
Equity in health is measured in terms of equity in survival of the first five years of life. This measure does not capture differences in equity between highly developed countries (where child mortality is low).
Responsiveness was measured in 35 countries, mainly developing ones. In other countries, responsiveness was estimated on the basis of variables found to be predictive of responsiveness in the first 35 countries. The accuracy of such predictions are of course limited. Also, it is far from evident that the relationships between dependent and predictive variables are the same in developing and developed countries.
The US score on responsiveness is purely a predicted one. It is high partly because several of the responsiveness items were found to be related to income in the 35 countries observed directly.
Fair financing was measured in only 21 countries. Scores for all other countries are projected from regressions using macro variables.
The indicator for average population health
(DALE)
Disability adjustment of life years is based on responses to a particular version of the person trade-off technique that has been heavily criticised for being based on an unethical premise and also for being difficult to understand (Arnesen and Nord, British Medical Journal 1999, 319, 1423-5). This specific version has consequently been rejected in an ongoing collaborative DALY project in six European countries. The WHO is presently considering various candidate procedures for determining disability weights in the future.
The WHO has designed a rich set of primary measurement tools by which different countries’ goal attainment and performance in health care may be judged. The WHO has in my opinion gone too far in compressing the results of these potentially useful primary measurements in summary indices with unclear meaning, dubious validity and little practical relevance.
The WHO has also gone too far in applying the same measuring rods to countries with different histories and values and different stages of development, and in encouraging international comparisons that are of little practical relevance.
The WHO needs to add an indicator of equality in access to its present indicator of fairness in financing.
Decision makers who, in spite of the above criticisms, are inclined to regard the summary indices of the World Health Report as useful, should be aware that the assignment of weights to different aspects of responsiveness and overall goal attainment is difficult and may be culturally dependent. No country should uncritically accept the weights suggested by the WHO as being scientific or objective or correct. Every country should judge carefully whether the WHO weights fit with the country’s own values, and thereby judge the relevance of the various indicators and indices for its own policy making.