Real-life scenarios illustrating the value of better measurement

I’ve seen consultants work hospital employees through a game in which the object is to manage the care of various kinds of patients who enter into the system at different points. Patients might have the same conditions, prognosis, payor, and demographics but come in through the ED, a clinic, or emerge from the OR. Others will vary medically but enter at the same point. Real-world odds are used to simulate decisions and events as the game proceeds via random card draws. The variation in decisions and outcomes across groups of decision-maker/players is fascinating.

It just occurred to me to set up a game like this with two major scenarios contrasting around one single variable: the quality of measurement. One inning or half of the game is status quo, where existing ratings and percentages are contrived and set up within the actual constraints of real data to illustrate the dangers of relying on numbers that are not measures. (Contact me for examples of how percentages can and sometimes do mean exactly the opposite of what they appear to mean.)

In this part of the game, we see the kinds of normal and par for the course inefficiencies, errors, outcomes, and costs that everyone expects to see.

In the second half of the game, we set up the same kind of scenario, but this time decisions are informed by meaningfully calibrated and contextualized measures. Everyone in the system has the same frame of reference, and decisions are coordinated virtually by the way the information is harmonized.

I imagine the two parts of the game might be played simultaneously by two equally experienced groups of managers and clinicians. Each group might be given a systems perspective, and would be encouraged to innovate with comparative effectiveness studies. When they have both arrived at their outcomes, tracked on a scorecard, they are debriefed together, the results are compared, and they are informed about the inner workings of the data they worked from.

Part of the point here would be to show that evidence-based decision-making is only worth as much as the evidence in hand. Evidence that is not constructed on the basis of a scientific theory and that is not mediated by calibrated instrumentation is worth much less than evidence that is theoretically justified and read off calibrated instruments.

It might be useful to imagine a seminar or workshop in which these scenarios are explored as illustrations of the way fully formed metrological systems reduce transaction costs and market frictions by greasing the wheels of health care commerce with efficient lubricants. Maybe the contrast could also be brought out in terms of a survey or multiple choice test.

Variations on the scenarios could be constructed for education or human resource contexts, as well.

Just wanted to put this down in writing. What do you think?


Tags: , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: