Measuring/Managing Social Value

From my December 1, 2008 personal journal, written not long after the October 2008 SoCap conference. I’ve updated a few things that have changed in the intervening years.

Over the last month, I’ve been digesting what I learned at the Social Capital Markets conference at Fort Mason in San Francisco, and at the conference I attended just afterward, Bioneers, in Marin county. Bioneers (www.Bioneers.org) could be called Natural Capital Markets. It was quite like the Social Capital Markets conference with only a slight shift in emphasis, and lots of discussion of social value.

The main thing that impressed me at both of these conferences, apart from what I already knew about the caring passion I share with so many, is the huge contrast between that passion and the quality of the data that so many are basing major decisions on. Seeing this made me step back and think harder about how to shape my message.

First, though it may not seem like it initially, there is incredible practical value to be gained from taking the trouble to construct good measures. We do indeed manage what we measure. So whatever we measure becomes what we manage. If we’re not measuring anything that has anything to do with our mission, vision, or values, then what we’re managing won’t have anything to do with those, either. And when the numbers we use as measures do not actually represent a constant unit amount that adds up the way the numbers do, then we don’t have a clue what we’re measuring and we could be managing just about anything.

This is not the way to proceed. First take-away: ask for more from your data. Don’t let it mislead you with superficial appearances. Dig deeper.

Second, to put it a little differently, percentages, scores, and counts per capita, etc. are not measures that have the same meaning or quality that measures of height, weight, time, temperature, or volts have. However, for over 50 years, we have been constructing measures mathematically equivalent to physical measures from ability tests, surveys, assessments, checklists, etc. The technical literature on this is widely available. The methods have been mainstream at ETS, ACT, state and national departments of education globally, etc for decades.

Second take-away: did I say you should ask for more from your data? You can get it. A lot of people already are, though I don’t think they’re asking for nearly as much as they could get.

Third, though the massive numbers of percentages, scores, and counts per capita are not the measures we seek, they are indeed exactly the right place to start. I have seen over and over again, in education, health care, sociology, human resource management, and most recently in the UN Millennium Development Goals data, that people do know exactly what data will form a proper basis for the measurement systems they need.

Third take-away: (one more time!) ask for more from your data. It may conceal a wealth beyond what you ever guessed.

So what are we talking about? There are methods for creating measures that give you numbers that verifiably stand for a substantive unit amount that adds up in the same way one-inch blocks do (probabilistically, and within a range of error). If the instrument is properly calibrated and administered, the unit size and meaning will not change across individuals or samples measured. You can reduce data volume dramatically, not only with no loss of information but also with false appearances of information either indicated as error or flagged for further attention. You can calibrate a continuum of less to more that is reliably and reproducibly associated with, annotated by, and interpreted through your own indicators. You can equate different collections of indicators that measure the same thing so that they do so in the same unit.

Different agencies using the same, different, or mixed collections of indicators in different countries or regions could assess their measures for comparability, and if they are of satisfactory quality, equate them so they measure in the same unit. That is, well-designed instruments written and administered in different languages routinely have their items calibrate in the same order and positions, giving the same meaning to the same unit of measurement. For instance, see the recent issue of the Journal of Applied Measurement ([link]) devoted to reports on the OECD’s Programme for International Student Assessment.

This is not a data analysis strategy. It is an instrument calibration strategy. Once calibrated, the instrument can be deployed. We need to monitor its structure, but the point is to create a tool people can take out into the world and use like a thermometer or clock.

I’ve just been looking at the Charity Navigator (for instance, [link]) and the UN’s Millenium Development Goals ([link]), and the databases that have been assembled as measures of progress toward these goals ([link]). I would suppose these web sites show data in forms that people are generally familiar with, so I’m working up analyses to use as teaching tools from the UN data.

You don’t have to take any of this at my word. It’s been documented ad nauseum in the academic literature for decades. Those interested can find out more than they ever wanted to know at http://www.Rasch.org, in the Wikipedia Rasch entry, in the articles and books at JAMPress.com, or in dozens of academic journals and hundreds of books. Though I’ve done my share of it, I’m less interested in continuing to add to that than I am in making a tangible contribution to improving people’s lives.

Sorry to go on like this. I meant to keep this short. Anyway, there it is.

PS, for real geeks: For those of you serious about learning about measurement as it is rigorously and mathematically defined, look into taking Everett Smith’s measurement course at Statistics.com ([link]) or David Andrich’s academic units at the University of Western Australia ([link]). Available software includes Mike Linacre’s Winsteps, Andrich’s RUMM, and Mark Wilson’s, at UC Berkeley, Conquest.

The methods Ev, Mike, David, and Mark teach have repeatedly been proven, both in mathematical theory and in real life, to be both necessary and sufficient in the construction of meaningful, practical measurement. Any number of ways of defining objectivity in measurement have been shown to reduce to the mathematical models they use. Why all the Chicago stuff? Because of Ben Wright. I’m helping (again) to organize a conference in his honor, to be held in Chicago next March. His work won him a Career Achievement Award from the Association of Test Publishers, and the coming conference will celebrate his foundational contributions to computerized measurement in health care.

As a final note, for those of you fearing reductionistic meaninglessness, look into my philosophical work.  But enough…

Advertisements

Tags: , , , , , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: