Archive for the ‘data’ Category

Three demands of meaningful measurement

September 28, 2009

The core issue in measurement is meaningfulness. There are three major aspects of meaningfulness to take into account in measurement. These have to do with the constancy of the unit, interpreting the size of differences in measures, and evaluating the coherence of the units and differences.

First, raw scores (counts of right answers or other events, sums of ratings, or rankings) do not stand for anything that adds up the way they do (see previous blogs for more on this). Any given raw score unit can be 4-5 times larger than another, depending on where they fall in the range. Meaningful measurement demands a constant unit. Instrument scaling methods provide it.

Second, meaningful measurement requires that we be able to say just what any quantitative amount of difference is supposed to represent. What does a difference between two measures stand for in the way of what is and isn’t done at those two levels? Is the difference within the range of error, and so random? Is the difference many times more than the error, and so repeatedly reproducible and constant? Meaningful measurement demands that we be able to make reliable distinctions.

Third, meaningful measurement demands that the items work together to measure the same thing. If reliable distinctions can be made between measures, what is the one thing that all of the items tap into? If the data exhibit a consistency that is shared across items and across persons, what is the nature of that consistency? Meaningful measurement posits a model of what data must look like to be interpretable and coherent, and then it evaluates data in light of that model.

When a constant unit is in hand, when the limits of randomness relative to stable differences are known, and when individual responses are consistent with one another, then, and only then, is measurement meaningful. Inconstant units, unknown amounts of random variation, and inconsistent data can never amount to the science we need for understanding and managing skills, abilities, health, motivations, social bonds, and environmental quality.

Managing our investments in human, social, and natural capital for positive returns demands that meaningful measurement be universalized in uniformly calibrated and accessible metrics. Scientifically rigorous, practical, and convenient methods for setting reference standards and making instruments traceable to them are readily available.

We have the means in hand for effecting order-of-magnitude improvements in the meaningfulness of the measures used in education, health care, human and environmental resource management, etc. It’s time we got to work on it.

We are what we measure. It’s time we measured what we want to be.

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.

Advertisements

Response to John Carney’s comments on Sept 25th’s Marketplace with Kai Ryssdal on NPR

September 25, 2009

Way to go, John Carney!! You’re my new hero!

Finally someone has said it right out loud: Even after a year in the economic trough, we are far from a consensus on what went wrong. Everyone is still fighting over the core problem, and so it is impossible to formulate a way forward.

Yes, it has been incredibly frustrating to watch everyone go round and round the central issue without ever really seeing it. Talk about the blind men and the elephant!

The first part of what needs to be done is on the table.  There does seem to be a fair degree of consensus on the idea that, somehow or other, we need to bring ALL forms of capital–human, social, and natural–into the econometric models, the financial spreadsheets, and the Genuine Progress Indicators or Happiness Indexes, so that the profit motive can be harnessed in sustainable, socially responsible ways to build communities, the environment, and human potential. As some have said, we need to transform socialized externalities into capitalized internalities.

I am hardly the first to suggest that. But what has been missing in previous proposals was the means by which we would devise transparent, fungible representations of each significant form of capital. How do we create common currencies for trading in each form of capital? By calibrating the instruments and deploying the reference standard common metrics we will use as those currencies. MEASUREMENT is the core problem. “You manage what you measure” is repeated like a mantra everywhere, but the quality of our measures of risk, opportunity, outcomes, and impacts–in short, of all measures of intangible forms of capital (human, social, intellectual, etc.)–is terrible!

You would never know it from most current measurement practice in business and government, but huge advances have been made over the last 50 years in scaling technology. The mathematical rigor, meaningfulness, practicality, and convenience of measures based in ratings, ability tests, surveys, and assessments could be an order of magnitude better if we just took advantage of existing technologies. We need universally uniform measures of health, skills and abilities, motivation, community life, governance, risk, and environmental quality akin to the measures of weight, volume, time, kilowatts, etc. that we take for granted in economic exchanges everyday.

It is essential to realize that universal uniformity in no way requires universal acceptance of exactly the same instruments, observations, content, questions, items, etc. Any brand tool that verifiably produces measures of the desired outcome, impact, process, etc. in the reference standard metric can compete in the measurement market, just as is the case with clocks or thermometers. Further, far from reducing rich complexities to a meaningless number, new measurement technologies open up new horizons for meaningful relationships by improving our understandings of ourselves and others.

The way forward centers on building a scientifically rigorous metric system that will make our stocks of human, social, and natural capital fungible. We need universally uniform metrics for the outcomes and impacts of schools, hospitals, social services, human, organizational, and natural resources, etc. When you appreciate the extent to which any economy lives and dies on its measurement standards, the need for national and global investments in quantitatively rigorous and qualitatively meaningful metrics becomes all too painfully obvious.

For more information, see https://livingcapitalmetrics.wordpress.com, http://www.livingcapitalmetrics.com, http://www.linkedin.com/in/livingcapitalmetrics, and other sources, such as the Wikipedia entry on Georg Rasch, http://www.rasch.org, and elsewhere.

These comments were also posted at http://marketplace.publicradio.org/display/web/2009/09/25/pm-weekly-wrap-q/.

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.