How Bad Measurement Stymies Health Care Reform Efforts


The Strange Absence of Measurement Awareness in the Debate over Health Care Reform

It is not as though measurement never comes up as a topic when advocates of one or another approach to health care reform have their say. But awareness of what measurement can and should do is strangely absent. No one at all speaks to what is most important about measurement, and how the essentials are missing in what passes for measurement in health care. And I’m just talking basics here. We can save for another time some of the especially relevant capacities and features of metric technologies as they have evolved over the last 80 years, and as they have been in use for over 30 years.

To live up to the full meaning of the term, measures have to do some very specific things. To keep things simple, all we need to do is consider how we use measures in something as everyday as shopping in the grocery store. The first thing we expect from measures are numbers that stand for something that adds up the way they do. The second thing measures have to do is to stay the same no matter where we go.

Currently popular methods of measurement in health care do not meet either of these expectations. Ratings from surveys and assessments, counts of events, and percentages of the time that something happens are natural and intuitive places from which to begin measurement, but these numbers do not and cannot live up to our expectations as to how measures behave. To look and act like real measures, these kinds of raw data must be evaluated and transformed in specific ways, using widely available and mathematically rigorous methodologies.

None of this is any news to researchers. The scientific literature is full of reports on the theory and practice of advanced measurement. The philosopher, Charles Sanders Peirce, described the mathematics of rigorous measurement 140 years ago. Louis Thurstone, an electrical engineer turned psychologist, took major steps toward a practical science of rigorous measurement in the 1920s. Health care admissions, graduation, and professional licensure and certification examinations have employed advanced measurement since the 1970s. There are a great many advantages that would be gained if the technologies used in health care’s own educational measurement systems were applied within health care itself.

Though we rarely stop to think about it, we all know that fair measures are essential to efficient markets. When different instruments measure in different units, market transactions are encumbered by the additional steps that must be taken to determine the value of what is being bought and sold. Health care is now so hobbled by its myriad varieties of measures that common product definitions seem beyond reach.

And we have lately been alerted to the way in which innovation is more often a product of a collective cognitive effort than it is of any one individual’s effort. For the wisdom of crowds to reach a critical mass at which creativity and originality take hold, we must have in place a common currency for the exchange of value, i.e., a universal, uniform metric calibrated so as to be traceable to a reference standard shared by all.

Since the publication of a seminal paper by Kenneth Arrow in the early 1960s, many economists have taken it for granted that health care is one industry in which common product definitions are impossible. The success of advanced measurement applications in health care research over the last 30 years contradicts that assumption.

It’s already been 14 years since I myself published a paper equating two different instruments for assessing physical functioning in physical medicine and rehabilitation. Two years later I published another paper showing that 10 different published articles reporting calibrations of four different functional assessments all showed the same calibration results for seven or eight similar items included on each instrument. What many will find surprising about this research is that consensus on the results was obtained across different samples of patients seen by different providers and rated by different clinicians on different instruments. What we have in this research is a basis for a generalized functional assessment metric.

Simply put, in that research, I showed how our two basic grocery store assumptions about measurement could be realized in the context of ratings assigned by clinicians to patients’ performances of basic physical activities and mobility skills. With measures that really add up and are as universally available as the measures we take for granted in the grocery store, we could have a system in which health care purchasers and consumers can make more informed decisions about the relationship between price and value. With such a system, quality improvement efforts could be coordinated at the point of care, on the basis of observations expressed in a familiar language.

Some years ago, quality improvement researchers raised the question as to why there are no health care providers who have yet risen to the challenge and redefined the industry relative to quality standards, in the manner that Toyota did for the automobile industry. There have, in fact, been many who tried, both before and since that question was asked.

Health care providers have failed in their efforts to emulate Toyota in large part because the numbers taken for measures in health care are not calibrated and maintained the way the automobile industry’s metrics are. It is ironic that something as important as measurement, something that receives so much lip service, should nonetheless be so widely skipped over and taken for granted.

What we need is a joint effort on the part of the National Institutes of Health and the National Institute of Standards and Technology focused on the calibration and maintenance of the metrics health care must have to get costs under control. We need to put our money and resources where our mouths are. We will be very glad we did when we see the kinds of returns on investment (40%-400% and more) that NIST reports for metrological improvement studies in other industries.


Tags: , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: