Posts Tagged ‘education; measurement; standards; innovation; entrepreneurs;’

Go Ahead, Just Say NO to Socialized Health Care!

June 11, 2009

But please don’t say YES to more of the same old capitalist health care!!

Instead, we ought to remove the dead weight encumbering the health care market and figure out how to improve its efficiency. Transaction costs are by far the most important costs in a market, and transactions are defined, by and large, by the quality of the information available to the parties in the exchange.  We have the means in hand for making vast improvements in health care information quality. Advanced measurement technologies integrate mass customization with meaningful, universally uniform, and ubiquitously available reference standard metrics and common product definitions.

We desperately need investment in a new infrastructure of metrological standards for every important metric in health care (health, chronic disease, and functional status; quality of life and care; employee and organizational performance, etc.). Such standards could function as a fungible common currency for the exchange of individual and organizational health capital.

If we don’t create these measurement systems, it really doesn’t matter whether the government runs a single-payor health care system or if we stick with the current dysfunctional system. Consumers and payors need to be able to compare products’ value-per-dollar just like we do in grocery stores. Quality improvement specialists and researchers need to be able to compare outcomes across treatments in common metrics. And perhaps most importantly, front-line clinicians need to be able to evaluate a patient’s condition on the spot, at the point of care, in the terms of the same measure that will be used for accountability and quality improvement.

Our long history of successes in science, engineering, and market economics illustrate the astounding benefits that accrue from thinking together in common languages, and from acting together harmoniously, in coordinated ways that do not require choosing between uninformed decisions and negotiating every detail. Our challenge is to figure out how to extend those successes into new domains. The question is how to configure representations of human, social, and natural capital so that their markets work the same way that manufactured, liquid, and property capital work.

Answering this question, and successfully addressing this challenge, are not beyond us. Improved measurement will play a vital role in reducing transaction costs and in taking the health care market to a whole new level of efficiency. Without improved measurement, it really doesn’t matter if health care is socialized or not.

And Here It Is: The Next Major Technological Breakthrough

May 29, 2009

How It Will Transform Your Business and Your Life

We’ve all witnessed an amazing series of events in our lifetimes, and, hopefully, we’ve learned some important lessons over the years. In business, we’ve come to see that innovation is rarely the work of one person. When the crowd has the right tools and puts its mind to the task, nothing can stop it. We’re accordingly also learning the real truth of the fact that any firm’s greatest resource is its people—there is no more effective source of new efficiencies and whole new directions. Concern for social responsibility is no longer the exclusive domain of activists, since everyone is now attuned to the susceptibility of markets to unrestrained greed. And there are increasingly good reasons for thinking that perhaps we can reverse ongoing major environmental debacles and orient our systems to profits that are sustainable over the long term.

And in our personal lives, we’ve learned the vital importance of access to learning opportunities across the lifespan, access to health care, and caring relationships. Whether we call it spiritual or not, life is hardly worth living without a sense of wonder at the very existence of the universe and all the strange things inhabiting it.

We’ve learned a few things, then. Perhaps foremost among them is that we are going to have to adapt to the changes we ourselves bring about. And given the pace of change and the plain need to do better, we don’t hear anyone repeating Lord Kelvin’s famous opinion, from the end of the nineteenth century, that pretty much everything that can be discovered has been discovered. (Though isn’t there someone at Microsoft who could top the classics “No one will ever need more than 640k memory—or more than one browser tab”?) With everything that’s happened in the 100 years or so since Kelvin’s remark, one of the big lessons that has been learned is a certain humility, at least in that regard.

Change is in the air, that’s for sure, even if it doesn’t seem that there is any one particular form of it. But in fact there is an important new technology coming on line. It isn’t really new. Viewed narrowly, it has been taking shape for over 80 years, even though its root mathematical principles go back to Plato (like so many do). And, at least in retrospect, this new technology’s major features may seem very humdrum and mundane, they are so everyday.

So just what is going on? Speaking in Abu Dhabi on Monday, May 25, Nobel economist Paul Krugman suggested that economic recovery could come about in the wake of a new major technological breakthrough, one of the size and scope of the IT revolution of the 1990s. Other factors cited by Krugman as candidates for turning things around included more investment by major corporations, and new climate change regulations and policies.

Industry-wide systems of metrological reference standards for human, social, and natural capital fit the bill. They are a new technological breakthrough on the scale of the initial IT revolution. They would also be a natural outgrowth of existing IT systems and an extension of existing global trade standards. Such systems would also require large investments from major corporations, and would facilitate highly significant moves on climate change.

In addition, stepping beyond the solutions suggested by Krugman, systematic and objective methods of measuring living capital would help meet the widely recognized need for socially responsible and sustainable business practices. Better measurement will play a vital role in reducing transaction costs and making human, social, and natural capital markets more efficient. It will also be essential to fostering new forms of innovation, as the shared standards and common product definitions made possible by advanced measurement systems enable people to think and act together collectively in common languages.

Striking advances have been made in measurement practice in recent years. It is easy to assume that the assignment of numbers to observations suffices as measurement, and that there have been no developments worthy of note in measurement theory or practice for decades. Nothing could be further from the truth. You don’t know the first thing about what you don’t know about measurement.

I came into the study and use of mathematically rigorous measurement and instrument calibration methods from the history and philosophy of science. The principles that make rulers, weight scales, clocks, and thermometers as meaningful, convenient and practical as they are, and that drive engineering practices in high tech, for instance, are pretty well understood. What’s more, those principles have been successfully applied to tests, rating scales, and assessments for decades, primarily in high stakes graduation, admissions, and certification/licensure testing. Increasingly these principles are finding their way into health care and business.

The general public doesn’t know much about all of this because the math is pretty intense, the software is hard to use, and we have an ingrained cultural prejudice that says all we have to do is come up with numbers of some kind, and–voila!– we have measurement. Nothing could be further from the truth.

My goal in all of this is to figure out how to put tools that work in the hands of the people who need them. You don’t need a PhD in thermodynamics to read a thermometer, so we ought to be able to calibrate similar instruments for other things we want to measure. And the way transparency and accountability demands are converging with economics and technology, I think the time is ripe for new ideas properly presented.

A quick way to see the point is to recognize that fair and just measures have to represent something that adds up the way the numbers do. Numbers don’t just automatically do that. We invest huge resources in crafting good instruments in the natural sciences, but we assume anyone at all can put together a measure using counts of right answers or sums of ratings or percents of the time some event occurs. But none of these are measures. Numbers certainly always add up in the same way, but whether they are meaningful or not is a question that is rarely asked. The numbers we often take as measures of outcomes or results or processes almost never stand for something that adds up the way everyone thinks they do.

So, yes, I know we need metrics that are manageable, understandable, and relevant. And I know how quickly people’s eyes glaze over in face of what they think are irrelevant technicalities. But eyes also tend to glaze over when something unexpected and completely different is offered. True originality is not easily categorized or recognized for what it is. And when something is fundamentally different from what people are used to, it can be rejected just because it is more trouble to to make the transition to a new system than it is to remain with the existing system, no matter how dysfunctional it is.

And boy is the current way of developing and deploying business metrics dysfunctional! Do you know that the difference between 1 percent and 2 percent can represent 4-8 times the difference between 49 percent and 50 percent? Did you know that sometimes a 15% difference can stand for as much as or even a lot more than a 39% difference? Did you know that three markedly different percentage values—differences that vary by more than a standard error or even five—might actually stand for the same measured amount?

In my 25 years of experience in measurement, people often turn out to not understand what they think they understand. And they then also turn out to be amazed at what they learn when they take the trouble to put some time and care into crafting an instrument that really measures what they’re after.

For instance, did you know that there are mathematical ways of reducing data volume that not only involve no loss of information but that actually increase the amount of actionable value? Given the way we are swimming in seas of data that do not usually mean what we think they mean, being able to experimentally make sure things add up properly at the same time we reduce the volume of numbers we have to deal with seems to me to be an eminently practical aid to understanding and manageability.

Did you know that different sets of indicators or items can measure in a common metric? Or that a large bank of items can be adaptively administered, with the instrument individually tailored and customized for each respondent, organization, or situation, all without compromising the comparability of the measures?

These are highly practical things to be able to do. Markets live and die on shared product definitions and shared metrics. Innovation almost never happens as a result of one person’s efforts; it is almost always a result of activities coordinated through a network structured by a common language of reference standards. We are very far from having the markets and levels of innovation we need in large part because the quality of measurement in so many business applications is so poor. But that is going to change in very short order as those most banal of subjects, measurement and metrological systems, catch fire.

Stimulating Excellence in Education

May 22, 2009

Comments on

Stimulating Excellence:

Unleashing the Power of Innovation in Education

published in May 2009 by

The Center for American Progress

The American Enterprise Institute

The Broad Foundation

New Profit Inc.

PublicImpact

and

The Annie E. Casey Foundation

Available at

http://www.americanprogress.org/issues/2009/05/entrepreneurs_event.html

Comments by

William P. Fisher, Jr., Ph.D.

7 May 2009

New Orleans

Submitted for publication to the Chronicle of Higher Education

I’ve just had a look at the new report, “Stimulating Excellence: Unleashing the Power of Innovation in Education.” I agree wholeheartedly with its focus on creating the conditions for entrepreneurial innovation and reward in education. I, too, deplore the lack of a quality improvement culture in education, and the general failure to recognize the vital importance of measuring performance for active management.

The specific recommendations concerning power metrics and drastic improvements in information quality really resonate with me. But two very significant shortcomings in the report demand attention.

First, only on page 34, in the report’s penultimate paragraph, do the authors briefly touch on what all educators know is absolutely the most important thing to understand about teaching and learning: it always starts from where the student is at, growing out of what is already known. This is doubly important in the context of the report’s focus, teaching and learning about how to institute a new culture of power metrics and innovation. To try to institute fundamental changes with little or no concern for what is already in place is a sure recipe for failure.

Second, there is one feature of the educational system as it currently exists that will be of particular value as we strive to improve the quality of the available information. That feature concerns tests and measurement. Many of the report’s recommendations would be quite different if its authors had integrated their entrepreneurial focus with the technical capacities of state-of-the-art educational measurement.

The obvious recommendation with which to start concerns the reason why public education in the United States is such a fragmented system: because outcome standards and product definitions are expressed (almost) entirely in terms of locally-determined content and expert opinion. Local content, standards, and opinions are essential, but to be meaningful, comparable, practical, and scientific they have to be brought into a common scale of comparison.

The technology for creating such scales is widely available. For over 40 years, commercial testing agencies, state departments of education, school districts, licensure and certification boards, and academic researchers have been developing and implementing stable metrics that transcend the local particulars of specific tests. The authors of the “Stimulating Excellence” report are right to stress the central importance of comparable measures in creating an entrepreneurial environment in education, but they did not do enough to identify existing measurement capabilities and how they could help create that environment.

For instance, all three of the recommendations made at the bottom of page 12 and top of page 13 address capabilities that are already in place in various states and districts around the country. The examples that come easiest to mind involve the Lexile Framework for Reading and Writing, and the Quantile Framework for Mathematics, developed by MetaMetrics, Inc., of Durham, NC (www.lexile.com).

The Lexile metric for reading ability and text readability unifies all major reading tests in a common scale, and is used to report measures for over 28 million students in all 50 states. Hundreds of publishers routinely obtain Lexile values for their texts, with over 115,000 books and 80 million articles (most available electronically) Lexiled to date.

Furthermore, though one would never know from reading the “Stimulating Excellence” report, materials on the MetaMetrics web site show that the report’s three recommendations concerning the maximization of data utility have already been recognized and acted on, since

*    many standardized assessments are already aligned with state learning standards,

*   available products already quickly incorporate assessment results into the process of teaching and learning (and a lot more quickly than “a day or two after testing”!), and

*   several states already have years of demonstrated commitment to keeping their standards and assessments relevant to the changing world’s demands on students.

That said, a larger issue concerns the need to create standards that remain invariant across local specifics. A national curriculum and national testing standards seem likely to fall into the trap of either dictating specific content or fostering continued fragmentation when states refuse to accept that content. But in the same way that computer-adaptive testing creates a unique examination for each examinee—without compromising comparability—so, too, must we invest resources in devising a national system of educational standards that both takes advantage of existing technical capabilities and sets the stage for improved educational outcomes.

That is what the report’s key recommendation ought to have been. An approximation of it comes on page 35, with the suggestion that now is the time for investment in what is referred to as “backbone platforms” like the Internet. Much more ought to have been said about this, and it should have been integrated with the previous recommendations, such as those concerning information quality and power metrics. For instance, on page 27, a recommendation is made to“build on the open-source concept.” Upon reading that, my immediate thought was that the authors were going to make an analogy with adaptively administered item banks, not literally recommend actual software implementation processes.

But they took the literal road and missed the analogical boat. That is, we ought to build on the open-source concept by creating what might be called crowd-sourced wikitests—exams that teachers and researchers everywhere can add to and draw from, with the qualification that the items work in practice to measure what they are supposed to measure, according to agreed-upon data quality and construct validity standards. This process would integrate local content standards with global construct standards in a universally uniform metric not much different from the reference standard units of comparison we take for granted in measuring time, temperature, distance, electrical current, or weight.

And this is where the real value of the “backbone platform” concept comes in. The Internet, like phones and faxes before it, and like alphabetic, phonetic and grammatical standards before them, provides the structure of common reference standards essential to communication and commerce. What we are evolving toward is a new level of complexity in the way we create the common unities of meaning through which we achieve varying degrees of mutual understanding and community.

In addition, measurement plays a fundamental role in the economy as the primary means of determining the relation of price to value. The never-ending spiral of increasing costs in education is surely deeply rooted in the lack of performance metrics and an improvement culture. We ought to take the global infrastructure of measurement standards as a model for what we need as a “backbone platform” in education. We ought to take the metaphor of transparency and the need for “clear metrics” much more literally. We really do need instruments that we can look right through, that bring the thing we want to see into focus, without having to be primarily concerned with which particular instrument it is we are using.

Finally, as no one will be surprised to learn, the existing measurement technology ignored in the “Stimulating Excellence” report is also unknown to those seeking to improve higher education, health care, social services, human resource management, government and governance, and environmental management. Decades of research show that the advances made in educational measurement transfer quite well from tests to surveys, rating scales, assessments, rankings, and checklists. A great deal needs to be done, and the challenges are huge, but taking them on will enable us to expand the domains in which we insist on fair dealing, and in which the balance scale applies as a symbol of justice.

When the entrepreneurial vision presented in the “Stimulating Excellence” report is situated in a context better informed by what educators are already doing and what they already know, the stage will be set for a new culture of performance improvement in education, a culture that explicitly articulates, tests, and acts on its educational values. At that point, we can expect great things!