Archive for June, 2009

Infrastructure and Health Care Reform

June 25, 2009

As an educator and researcher involved in the theory and application of advanced measurement methods, I am both encouraged by the (June 14) New York Times Sunday magazine’s focus on infrastructure, and chagrined at the uninformed level at which ongoing health care and economic reform discussions and analyses are taking place (as evident in the Sunday, June 21, Times editorial and business pages).

Socialistic solutions to problems in education, health care, and the economy at large are the inevitable outcome of our incomplete implementation and understanding of market capitalism. Take, for instance, the rancorous debate as to whether we should create a new public health insurance plan to compete with private plans. None of the proposals or counter proposals amount to anything more than alternate ways of manhandling health care resources toward one or another politically predetermined end. Accordingly, we find ourselves in the dilemma of choosing between equally real dangers. On the one hand, reduced payments and cost-cutting might do nothing but lower the quality and quantity of the available services, and, on the other hand, maintaining quality and quantity will eventually make health care completely unaffordable.

And here is what really gets me: apart from blind faith in the power of reduced payments to promote innovation, there is nary a word about how to set up a market infrastructure that will allow the invisible hand to do its work in bringing supply and demand efficiently into balance. Far from seeking ways in which costs can be reduced and profits enhanced at the same time, as they are in other industries, the automatic assumption in health care always seems to be that lower costs mean lower profits. We have always thought socialistically about health care, with economists, since Arrow, widely holding that health care is constitutionally incapable of sustaining a market economy. Hope that the economists are wrong appears to spring eternal, but who is doing the work to find a new way?

A new direction shows itself when we listen more closely to ourselves, and follow through on our basically valid intuitions. For instance, issues of sustainability, justice, and responsibility in the economic conversation employ the word “capital” to refer to a wide variety of resources essential to productivity, such as health, literacy, numeracy, community, and the air, water, and food services provided by nature.

The problem is that there seems to be little or no interest in figuring out how to transform this usage from an empty metaphor into a powerful tool. We similarly repeat ad nauseum the mantra, “you manage what you measure,” but almost nothing is being done to employ the highly advantageous features of advanced measurement theory and practice in the management of intangible forms of capital.

Better measurement of living capital is, however, absolutely essential to health care reform, entrepreneurial innovations in education, and to reinventing capitalism.  Instead of continuing to rely on highly variable local efforts at measuring and managing human, social, and natural capital, we need a broad program of capacity building focused on a metrological infrastructure of living capital, and its implementations.  If there is any one single blind spot that prevents us from fully learning the lessons of our recent economic disasters, it is the potential that new measurement technologies offer for reduced frictions and lower transaction costs in the intangible capital markets.

We know where to start, from two basic principles of market economics. First, we know the transaction costs are the most important costs in any market.  High transaction costs can strangle a market as the flow of capital is stifled. Second, we know that innovation, essential to product development, improvements, marketing, and enhanced profitability, is almost never accomplished by an individual working in isolation. Innovation requires an environment in which it is safe to play, to make mistakes, and through which new value can be immediately and decisively recognized for what it is.

How can living capital market frictions be reduced? For starters, we could focus on effecting order-of-magnitude improvements in the meaningfulness of the metrics we use for screening, diagnosis, research, and accountability. We can do whatever arithmetic we want with the numbers we have at hand, but most of the numbers that pass for measures of health, functionality, quality of life and care, etc. do not actually stand for something that adds up. The good news is that, again, the intuitions informing our efforts so far are largely valid, and have the ball rolling in the right direction.

How can better measurement advance the cause of innovation in health care? By providing a common language that all stakeholders can think and act in together, harmoniously. Research over the last 80 years has repeatedly proven the viability of a kind of a metric system for the things we measure with surveys, assessments, and tests. Such a system of universally uniform metrics would provide the common currency unifying the health care economy and establishing the basis for market self-organization. But contrary to our predominant metaphysical faith, scientifically proven results do not magically propagate themselves into the world. We have to invent and construct the systems we need.

Our efforts in this direction are stymied, as Tom Vanderbilt put it in the Times Sunday magazine on infrastructure, to the extent that we have “an inimical incuriosity” about the banal fundamentals of the systems that shape our world. We simply take dry technicalities for granted, and notice them only when they fail us. Our problem with intangibles measurement, then, is compounded by the fact that the infrastructure we are taking for granted is not just invisible or broken, it is nonexistent. Until we make the effort to build our capacity for managing health and other forms of living capital by creating reference standard common currencies for expressing, managing, and trading on their value, all of our efforts at health care reform–and at reinventing capitalism–will fall far short of what is possible.
William P. Fisher, Jr., Ph.D.
william@livingcapitalmetrics.com
http://www.LivingCapitalMetrics.com

We are what we measure.
It’s time we measured what we want to be.

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.

Go Ahead, Just Say NO to Socialized Health Care!

June 11, 2009

But please don’t say YES to more of the same old capitalist health care!!

Instead, we ought to remove the dead weight encumbering the health care market and figure out how to improve its efficiency. Transaction costs are by far the most important costs in a market, and transactions are defined, by and large, by the quality of the information available to the parties in the exchange.  We have the means in hand for making vast improvements in health care information quality. Advanced measurement technologies integrate mass customization with meaningful, universally uniform, and ubiquitously available reference standard metrics and common product definitions.

We desperately need investment in a new infrastructure of metrological standards for every important metric in health care (health, chronic disease, and functional status; quality of life and care; employee and organizational performance, etc.). Such standards could function as a fungible common currency for the exchange of individual and organizational health capital.

If we don’t create these measurement systems, it really doesn’t matter whether the government runs a single-payor health care system or if we stick with the current dysfunctional system. Consumers and payors need to be able to compare products’ value-per-dollar just like we do in grocery stores. Quality improvement specialists and researchers need to be able to compare outcomes across treatments in common metrics. And perhaps most importantly, front-line clinicians need to be able to evaluate a patient’s condition on the spot, at the point of care, in the terms of the same measure that will be used for accountability and quality improvement.

Our long history of successes in science, engineering, and market economics illustrate the astounding benefits that accrue from thinking together in common languages, and from acting together harmoniously, in coordinated ways that do not require choosing between uninformed decisions and negotiating every detail. Our challenge is to figure out how to extend those successes into new domains. The question is how to configure representations of human, social, and natural capital so that their markets work the same way that manufactured, liquid, and property capital work.

Answering this question, and successfully addressing this challenge, are not beyond us. Improved measurement will play a vital role in reducing transaction costs and in taking the health care market to a whole new level of efficiency. Without improved measurement, it really doesn’t matter if health care is socialized or not.

How Bad Measurement Stymies Health Care Reform Efforts

June 9, 2009

or

The Strange Absence of Measurement Awareness in the Debate over Health Care Reform

It is not as though measurement never comes up as a topic when advocates of one or another approach to health care reform have their say. But awareness of what measurement can and should do is strangely absent. No one at all speaks to what is most important about measurement, and how the essentials are missing in what passes for measurement in health care. And I’m just talking basics here. We can save for another time some of the especially relevant capacities and features of metric technologies as they have evolved over the last 80 years, and as they have been in use for over 30 years.

To live up to the full meaning of the term, measures have to do some very specific things. To keep things simple, all we need to do is consider how we use measures in something as everyday as shopping in the grocery store. The first thing we expect from measures are numbers that stand for something that adds up the way they do. The second thing measures have to do is to stay the same no matter where we go.

Currently popular methods of measurement in health care do not meet either of these expectations. Ratings from surveys and assessments, counts of events, and percentages of the time that something happens are natural and intuitive places from which to begin measurement, but these numbers do not and cannot live up to our expectations as to how measures behave. To look and act like real measures, these kinds of raw data must be evaluated and transformed in specific ways, using widely available and mathematically rigorous methodologies.

None of this is any news to researchers. The scientific literature is full of reports on the theory and practice of advanced measurement. The philosopher, Charles Sanders Peirce, described the mathematics of rigorous measurement 140 years ago. Louis Thurstone, an electrical engineer turned psychologist, took major steps toward a practical science of rigorous measurement in the 1920s. Health care admissions, graduation, and professional licensure and certification examinations have employed advanced measurement since the 1970s. There are a great many advantages that would be gained if the technologies used in health care’s own educational measurement systems were applied within health care itself.

Though we rarely stop to think about it, we all know that fair measures are essential to efficient markets. When different instruments measure in different units, market transactions are encumbered by the additional steps that must be taken to determine the value of what is being bought and sold. Health care is now so hobbled by its myriad varieties of measures that common product definitions seem beyond reach.

And we have lately been alerted to the way in which innovation is more often a product of a collective cognitive effort than it is of any one individual’s effort. For the wisdom of crowds to reach a critical mass at which creativity and originality take hold, we must have in place a common currency for the exchange of value, i.e., a universal, uniform metric calibrated so as to be traceable to a reference standard shared by all.

Since the publication of a seminal paper by Kenneth Arrow in the early 1960s, many economists have taken it for granted that health care is one industry in which common product definitions are impossible. The success of advanced measurement applications in health care research over the last 30 years contradicts that assumption.

It’s already been 14 years since I myself published a paper equating two different instruments for assessing physical functioning in physical medicine and rehabilitation. Two years later I published another paper showing that 10 different published articles reporting calibrations of four different functional assessments all showed the same calibration results for seven or eight similar items included on each instrument. What many will find surprising about this research is that consensus on the results was obtained across different samples of patients seen by different providers and rated by different clinicians on different instruments. What we have in this research is a basis for a generalized functional assessment metric.

Simply put, in that research, I showed how our two basic grocery store assumptions about measurement could be realized in the context of ratings assigned by clinicians to patients’ performances of basic physical activities and mobility skills. With measures that really add up and are as universally available as the measures we take for granted in the grocery store, we could have a system in which health care purchasers and consumers can make more informed decisions about the relationship between price and value. With such a system, quality improvement efforts could be coordinated at the point of care, on the basis of observations expressed in a familiar language.

Some years ago, quality improvement researchers raised the question as to why there are no health care providers who have yet risen to the challenge and redefined the industry relative to quality standards, in the manner that Toyota did for the automobile industry. There have, in fact, been many who tried, both before and since that question was asked.

Health care providers have failed in their efforts to emulate Toyota in large part because the numbers taken for measures in health care are not calibrated and maintained the way the automobile industry’s metrics are. It is ironic that something as important as measurement, something that receives so much lip service, should nonetheless be so widely skipped over and taken for granted.

What we need is a joint effort on the part of the National Institutes of Health and the National Institute of Standards and Technology focused on the calibration and maintenance of the metrics health care must have to get costs under control. We need to put our money and resources where our mouths are. We will be very glad we did when we see the kinds of returns on investment (40%-400% and more) that NIST reports for metrological improvement studies in other industries.

Real-life scenarios illustrating the value of better measurement

June 4, 2009

I’ve seen consultants work hospital employees through a game in which the object is to manage the care of various kinds of patients who enter into the system at different points. Patients might have the same conditions, prognosis, payor, and demographics but come in through the ED, a clinic, or emerge from the OR. Others will vary medically but enter at the same point. Real-world odds are used to simulate decisions and events as the game proceeds via random card draws. The variation in decisions and outcomes across groups of decision-maker/players is fascinating.

It just occurred to me to set up a game like this with two major scenarios contrasting around one single variable: the quality of measurement. One inning or half of the game is status quo, where existing ratings and percentages are contrived and set up within the actual constraints of real data to illustrate the dangers of relying on numbers that are not measures. (Contact me for examples of how percentages can and sometimes do mean exactly the opposite of what they appear to mean.)

In this part of the game, we see the kinds of normal and par for the course inefficiencies, errors, outcomes, and costs that everyone expects to see.

In the second half of the game, we set up the same kind of scenario, but this time decisions are informed by meaningfully calibrated and contextualized measures. Everyone in the system has the same frame of reference, and decisions are coordinated virtually by the way the information is harmonized.

I imagine the two parts of the game might be played simultaneously by two equally experienced groups of managers and clinicians. Each group might be given a systems perspective, and would be encouraged to innovate with comparative effectiveness studies. When they have both arrived at their outcomes, tracked on a scorecard, they are debriefed together, the results are compared, and they are informed about the inner workings of the data they worked from.

Part of the point here would be to show that evidence-based decision-making is only worth as much as the evidence in hand. Evidence that is not constructed on the basis of a scientific theory and that is not mediated by calibrated instrumentation is worth much less than evidence that is theoretically justified and read off calibrated instruments.

It might be useful to imagine a seminar or workshop in which these scenarios are explored as illustrations of the way fully formed metrological systems reduce transaction costs and market frictions by greasing the wheels of health care commerce with efficient lubricants. Maybe the contrast could also be brought out in terms of a survey or multiple choice test.

Variations on the scenarios could be constructed for education or human resource contexts, as well.

Just wanted to put this down in writing. What do you think?

The Measurement Twilight Zone

June 3, 2009

Measurement is everywhere. Our symbol for justice is a balance scale. We have technical standards for air, water, and food quality. Trade and commerce, from local to global markets, depend on quick and easy ways of knowing what and how much is for sale.

We all depend on measurement, but hardly anyone knows anything about how instruments are calibrated or how meaningful expressions of quantity are created and maintained.

So measurement exists in a kind of twilight zone between the clearest and most rigorous mathematics, on the one hand, and the darkest and most obscure ignorance, on the other. Take temperature, for instance. Virtually everyone over the age of five or so knows how to read a thermometer. But very few people can correctly describe the thermodynamic relationships that make a thermometer work.

We can rely on thermometer manufacturors to do the work of calibrating temperature measures for us. But what happens when we need to measure something for which there are no commercially available solutions?

As demand increases for measures of human and organizational performance, of social capital, and environmental impact, more and more managers, executives, entrepreneurs, accountants, philanthropists, and researchers unknowingly enter into the measurement twilight zone.

In the measurement twilight zone, things are not as they seem. Numbers add up the way they always do, but they no longer stand for constant amounts. We manage what we measure, and so we ask customers, employees, or patients to rate performances, we count right answers on tests, and we compute the percentage of time that some event happens.

But none of these numbers are measures. None of them add up. This is a very serious situation. It is not a rare, academic technicality of no practical consequence. Improving the quality of our measures is an urgent matter that ought to be the focus of a great deal more attention and interest than it currently is.

For instance, do you know that sometimes a 15% difference can stand for as much as or even a lot more than a 39% difference? Did you know that three markedly different percentage values–differences that vary by more than a standard error or even five– might actually stand for the same measured amount? Do you know that the difference between 1 percent and 2 percent can represent 4-8 times the difference between 49 percent and 50 percent?

Scores, ratings, and percentages are termed “ordinal” because, at best, they stand for a rank order of less and more. They do not stand for equal-interval amounts, though they can be a good start at creating real measures.

The general public doesn’t know much about all of this because the math is pretty intense, the software is hard to use, and we have an ingrained cultural prejudice that says all we have to do is come up with numbers of some kind, and–voila!– we have measurement. Nothing could be further from the truth.

My goal in all of this is to figure out how to put tools that work in the hands of the people who need them. You don’t need a PhD in thermodynamics to read a thermometer, so we ought to be able to calibrate similar instruments for other things we want to measure. And the way transparency and accountability demands are converging with economics and technology, I think the time is ripe for new ideas properly presented.

In my 25 years of experience in measurement, people often turn out to not understand what they think they understand. And they then also turn out to be amazed at what they learn when they take the trouble to put some time and care into crafting an instrument that really measures what they’re after.

For instance, did you know that there are mathematical ways of reducing data volume that not only involve no loss of information but that actually increase the amount of actionable value? We are swimming in seas of data that do not usually mean what we think they mean, so being able to ensure things add up properly at the same time we reduce the volume of numbers we have to deal with is an eminently practical aid to understanding and manageability.

Did you know that different sets of indicators or items can measure in a common metric? Or that a large bank of items can be adaptively administered, with the instrument individually tailored and customized for each respondent, organization, or situation, all without compromising the comparability of the measures?

These are highly practical things to be able to do. Markets live and die on shared product definitions and shared metrics. Innovation almost never happens as a result of one person’s efforts; it is almost always a result of activities coordinated through a network structured by a common language of reference standards. We are very far from having the markets and levels of innovation we need in large part because the quality of measurement in so many business applications is so poor.

And there’s lots more where that came from, but I’ll stop there. You can learn a lot more on these topics from a lot of sources. I’ll list a few below.

http://www.rasch.org
http://www.rasch.org/rmt
http://en.wikipedia.org/wiki/Rasch_model
http://www.lexile.com
http://www.winsteps.com
http://www.livingcapitalmetrics.com

William P. Fisher, Jr., Ph.D.

We are what we measure.
It’s time we measured what we want to be.