Posts Tagged ‘performance assessment’

Question Authority: Queries In the Back of the Wall Street Demonstrators’ Minds

October 2, 2011

I think the Wall Street demonstrators’ lack of goals and the admission of not having a solution is very important. All solutions offered so far are band-aids at best, and most are likely to do more harm than good.

I think I have an innovative way of articulating the questions people have on their minds. I thought of scattering small pieces of paper anywhere there are these demonstrations going on, with questions like these on them:

Feeling robbed of the trust, loyalty, and commitment you invested?

Unable to get a good return on your investment in your education?

Feeling robbed of your share of the world’s natural resources?

How many shares of social capital do you own?

How many shares of literacy capital do you have on the market?

How many shares of health capital do you own?

How many shares of natural capital do you own?

Wishing there was an easy way to know what return rate you get on your health investments?

Wishing there was an easy way to know what return rate you get on your education investments?

Why don’t you have legal title to your literacy capital shares?

Why don’t you have legal title to your social capital shares?

Why don’t you have legal title to your health capital shares?

Why don’t you have legal title to your natural capital shares?

Why don’t you know how many literacy capital shares are rightfully yours?

Why don’t you know how many social capital shares are rightfully yours?

Why don’t you know how many health capital shares are rightfully yours?

Why don’t you know how many natural capital shares are rightfully yours?

Why is there no common currency for trading on your literacy capital?

Why is there no common currency for trading on your health capital?

Why is there no common currency for trading on your social capital?

Why is there no common currency for trading on your natural capital?

Why aren’t corporations accountable for their impacts on your literacy capital investments?

Why aren’t corporations accountable for their impacts on your natural capital investments?

Why aren’t corporations accountable for their impacts on your social capital investments?

Why aren’t corporations accountable for their impacts on your health capital investments?

Why aren’t governments accountable for their impacts on your literacy capital investments?

Why aren’t governments accountable for their impacts on your natural capital investments?

Why aren’t governments accountable for their impacts on your social capital investments?

Why aren’t governments accountable for their impacts on your health capital investments?

Why are educational outcomes not comparable in a common metric?

Why are health care outcomes not comparable in a common metric?

Why are social program outcomes not comparable in a common metric?

Why are natural resource management program outcomes not comparable in a common metric?

Why do accounting and economics focus on land, labor, and manufactured capital instead of putting the value of ecosystem services, and health, literacy, and social capital, on the books and in the models, along with property and manufactured capital?

If we truly do manage what we measure, why don’t we have a metric system for literacy capital?

Can we effectively manage literacy capital if we don’t have a universally recognized and accepted metric for it?

If we truly do manage what we measure, why don’t we have a metric system for health capital?

Can we effectively manage health capital if we don’t have a universally recognized and accepted metric for it?

If we truly do manage what we measure, why don’t we have a metric system for social capital?

Can we effectively manage social capital if we don’t have a universally recognized and accepted metric for it?

If we truly do manage what we measure, why don’t we have a metric system for natural capital?

Can we effectively manage natural capital if we don’t have a universally recognized and accepted metric for it?

How is our collective imagination being stifled by the lack of a common language for literacy capital?

How is our collective imagination being stifled by the lack of a common language for health capital?

How is our collective imagination being stifled by the lack of a common language for social capital?

How is our collective imagination being stifled by the lack of a common language for natural capital?

How can the voice of the people be heard without common languages for things that are important to us?

How do we know where we stand as individuals and as a society if we can’t track the value and volume of our literacy, health, social, and natural capital shares?

Why don’t NIST and NSF fund new research into literacy, health, social, and natural capital metrics?

Why aren’t banks required to offer literacy, health, social, and natural capital accounts?

If we want to harmonize relationships between people, within and between societies, and between culture and nature, why don’t we tune the instruments on which we play the music of our lives?

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.

Advertisements

Subjectivity, Objectivity, Performance Measurement and Markets

April 23, 2011

Though he attributes his insight to a colleague (George Baker), Michael Jensen has once more succinctly stated a key point I’ve repeatedly tried to convey in my blog posts. As Jensen (2003, p. 397) puts it,

…any activity whose performance can be perfectly measured objectively does not belong inside the firm. If its performance can be adequately measured objectively it can be spun out of the firm and contracted for in a market transaction.

YES!! Though nothing is measured perfectly, my message has been a series of variations on precisely this theme. Well-measured property, services, products, and commodities in today’s economy are associated with scientific, legal and financial structures and processes that endow certain representations with meaningful indications of kind, amount, value and ownership. It is further well established that the ownership of the products of one’s creative endeavors is essential to economic advancement and the enlargement of the greater good. Markets could not exist without objective measures, and thus we have the central commercial importance of metric standards.

The improved measurement of service outcomes and performances is going to create an environment capable of supporting similar legal and financial indications of value and ownership. Many of the causes of today’s economic crises can be traced to poor quality information and inadequate measures of human, social, and natural value. Bringing publicly verifiable scientific data and methods to bear on the tuning of instruments for measuring these forms of value will make their harmonization much simpler than it ever could be otherwise. Social and environmental costs and value have been relegated to the marginal status of externalities because they have not been measured in ways that made it possible to bring them onto the books and into the models.

But the stage is being set for significant changes. Decades of research calibrating objective measures of a wide variety of performances and outcomes are inexorably leading to the creation of an intangible assets metric system (Fisher, 2009a, 2009b, 2011). Meaningful and rigorous individual-level universally available uniform metrics for each significant intangible asset (abilities, health, trustworthiness, etc.) will

(a) make it possible for each of us to take full possession, ownership, and management control of our investments in and returns from these forms of capital,

(b) coordinate the decisions and behaviors of consumers, researchers, and quality improvement specialists to better match supply and demand, and thereby

(c) increase the efficiency of human, social, and natural capital markets, harnessing the profit motive for the removal of wasted human potential, lost community coherence, and destroyed environmental quality.

Jensen’s observation emerges in his analysis of performance measures as one of three factors in defining the incentives and payoffs for a linear compensation plan (the other two being the intercept and the slope of the bonus line relating salary and bonus to the performance measure targets). The two sentences quoted above occur in this broader context, where Jensen (2003, pp. 396-397) states that,

…we must decide how much subjectivity will be involved in each performance measure. In considering this we must recognize that every performance measurement system in a firm must involve an important amount of subjectivity. The reason, as my colleague George Baker has pointed out, is that any activity whose performance can be perfectly measured objectively does not belong inside the firm. If its performance can be adequately measured objectively it can be spun out of the firm and contracted for in a market transaction. Thus, one of the most important jobs of managers, complementing objective measures of performance with managerial subjective evaluation of subtle interdependencies and other factors is exactly what most managers would like to avoid. Indeed, it is this factor along with efficient risk bearing that is at the heart of what gives managers and firms an advantage over markets.

Jensen is here referring implicitly to the point Coase (1990) makes regarding the nature of the firm. A firm can be seen as a specialized market, one in which methods, insights, and systems not generally available elsewhere are employed for competitive advantage. Products are brought to market competitively by being endowed with value not otherwise available. Maximizing that value is essential to the viability of the firm.

Given conflicting incentives and the mixed messages of the balanced scorecard, managers have plenty of opportunities for creatively avoiding the difficult task of maximizing the value of the firm. Jensen (2001) shows that attending to the “managerial subjective evaluation of subtle interdependencies” is made impossibly complex when decisions and behaviors are pulled in different directions by each stakeholder’s particular interests. Other research shows that even traditional capital structures are plagued by the mismeasurement of leverage, distress costs, tax shields, and the speed with which individual firms adjust their capital needs relative to leverage targets (Graham & Leary, 2010). The objective measurement of intangible assets surely seems impossibly complex to those familiar with these problems.

But perhaps the problems associated with measuring traditional capital structures are not so different from those encountered in the domain of intangible assets. In both cases, a particular kind of unjustified self-assurance seems always to attend the mere availability of numeric data. To the unpracticed eye, numbers seem to always behave the same way, no matter if they are rigorous measures of physical commodities, like kilowatts, barrels, or bushels, or if they are currency units in an accounting spreadsheet, or if they are percentages of agreeable responses to a survey question. The problem is that, when interrogated in particular ways with respect to the question of how much of something is supposedly measured, these different kinds of numbers give quite markedly different kinds of answers.

The challenge we face is one of determining what kind of answers we want to the questions we have to ask. Presumably, we want to ask questions and get answers pertinent to obtaining the information we need to manage life creatively, meaningfully, effectively and efficiently. It may be useful then, as a kind of thought experiment, to make a bold leap and imagine a scenario in which relevant questions are answered with integrity, accountability, and transparency.

What will happen when the specialized expertise of human resource professionals is supplanted by a market in which meaningful and comparable measures of the hireability, retainability, productivity, and promotability of every candidate and employee are readily available? If Baker and Jensen have it right, perhaps firms will no longer have employees. This is not to say that no one will work for pay. Instead, firms will contract with individual workers at going market rates, and workers will undoubtedly be well aware of the market value of their available shares of their intangible assets.

A similar consequence follows for the social safety net and a host of other control, regulatory, and policing mechanisms. But we will no longer be stuck with blind faith in the invisible hand and market efficiency, following the faith of those willing to place their trust and their futures in the hands of mechanisms they only vaguely understand and cannot control. Instead, aggregate effects on individuals, communities, and the environment will be tracked in publicly available and critically examined measures, just as stocks, bonds, and commodities are tracked now.

Previous posts in this blog explore the economic possibilities that follow from having empirically substantiated, theoretically predictable, and instrumentally mediated measures embodying broad consensus standards. What we will have for human, social, and natural capital will be the same kind of objective measures that have made markets work as well as they have thus far. It will be a whole new ball game when profits become tied to human, social, and environmental outcomes.

References

Coase, R. (1990). The firm, the market, and the law. Chicago: University of Chicago Press.

Fisher, W. P., Jr. (2009a, November). Invariance and traceability for measures of human, social, and natural capital: Theory and application. Measurement, 42(9), 1278-1287.

Fisher, W. P.. Jr. (2009b). NIST Critical national need idea White Paper: metrological infrastructure for human, social, and natural capital (Tech. Rep. No. http://www.livingcapitalmetrics.com/images/FisherNISTWhitePaper2.pdf). New Orleans: LivingCapitalMetrics.com.

Fisher, W. P., Jr. (2010, 22 November). Meaningfulness, measurement, value seeking, and the corporate objective function: An introduction to new possibilities. Available at http://ssrn.com/abstract=1713467.

Fisher, W. P., Jr. (2011). Bringing human, social, and natural capital to life: Practical consequences and opportunities. Journal of Applied Measurement, 12(1), in press.

Graham, J. R., & Leary, M. T. (2010, 21 December). A review of empirical capital structure research and directions for the future. Available at http://ssrn.com/abstract=1729388.

Jensen, M. C. (2001, Fall). Value maximization, stakeholder theory, and the corporate objective function. Journal of Applied Corporate Finance, 14(3), 8-21.

Jensen, M. C. (2003). Paying people to lie: The truth about the budgeting process. European Financial Management, 9(3), 379-406.

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.

Parameterizing Perfection: Practical Applications of a Mathematical Model of the Lean Ideal

April 2, 2010

To properly pursue perfection, we need to parameterize it. That is, taking perfection as the ideal, unattainable standard against which we judge our performance is equivalent to thinking of it as a mathematical model. Organizations are intended to realize their missions independent of the particular employees, customers, suppliers, challenges, products, etc. they happen to engage with at any particular time. Organizational performance measurement (Spitzer, 2007) ought to then be designed in terms of a model that posits, tests for, and capitalizes on the always imperfectly realized independence of those parameters.

Lean thinking (Womack & Jones, 1996) focuses on minimizing waste and maximizing value. At every point at which resources are invested in processes, services, or products, the question is asked, “What value is added here?” Resources are wasted when no value is added, when they can be removed with no detrimental effect on the value of the end product. In their book, Natural Capitalism: Creating the Next Industrial Revolution, Hawken, Lovins, and Lovins (1999, p. 133) say

“Lean thinking … changes the standard for measuring corporate success. … As they [Womack and Jones] express it: ‘Our earnest advice to lean firms today is simple. To hell with your competitors; compete against perfection by identifying all activities that are muda [the Japanese term for waste used in Toyota’s landmark quality programs] and eliminating them. This is an absolute rather than a relative standard which can provide the essential North Star for any organization.”

Further, every input should “be presumed waste until shown otherwise.” A constant, ongoing, persistent pressure for removing waste is the basic characteristic of lean thinking. Perfection is never achieved, but it aptly serves as the ideal against which progress is measured.

Lean thinking sounds a lot like a mathematical model, though it does not seem to have been written out in a mathematical form, or used as the basis for calibrating instruments, estimating measures, evaluating data quality, or for practical assessments of lean organizational performance. The closest anyone seems to have come to parameterizing perfection is in the work of Genichi Taguchi (Ealey, 1988), which has several close parallels with Rasch measurement (Linacre, 1993).  But meaningful and objective quantification, as required and achieved in the theory and practice of fundamental measurement (Andrich, 2004; Bezruczko, 2005; Bond & Fox 2007; Smith & Smith, 2004; Wilson, 2005; Wright, 1999), in fact asserts abstract ideals of perfection as models of organizational, social, and psychological processes in education, health care, marketing, etc. These models test the extent to which outcomes remain invariant across examination or survey questions, across teachers, students, schools, and curricula, or across treatment methods, business processes, or policies.

Though as yet implemented only to a limited extent in business (Drehmer, Belohlav, James, & Coye, 2000; Drehmer & Deklava, 2001;  Lunz & Linacre, 1998; Salzberger, 2009), advanced measurement’s potential rewards are great. Fundamental measurement theory has been successfully applied in research and practice thousands of times over the last 40 years and more, including in very large scale assessments and licensure/certification applications (Adams, Wu, & Macaskill, 1997; Masters, 2007; Smith, Julian, Lunz, et al., 1994). These successes speak to an opportunity for making broad improvements in outcome measurement that could provide more coherent product definition, and significant associated opportunities for improving product quality and the efficiency with which it is produced, in the manner that has followed from the use of fundamental measures in other industries.

Of course, processes and outcomes are never implemented or obtained with perfect consistency. This would be perfectly true only in a perfect world. But to pursue perfection, we need to parameterize it. In other words, to raise the bar in any area of performance assessment, we have to know not only what direction is up, but we also need to know when we have raised the bar far enough. But we cannot tell up from down, we do not know how much to raise the bar, and we cannot properly evaluate the effects of lean experiments when we have no way of locating measures on a number line that embodies the lean ideal.

To think together collectively in ways that lead to significant new innovations, to rise above what Jaron Lanier calls the “global mush” of confused and self-confirming hive thinking, we need the common languages of widely accepted fundamental measures of the relevant processes and outcomes, measures that remain constant across samples of customers, patients, employees, students, etc., and across products, sales techniques, curricula, treatment processes, assessment methods, and brands of instrument.

We are all well aware that the consequences of not knowing where the bar is, of not having product definitions, can be disastrous. In many respects, as I’ve said previously in this blog, the success or failure of health care reform hinges on getting measurement right. The Institute of Medicine report, To Err is Human, of several years ago stresses the fact that system failures pose the greatest threat to safety in health care because they lead to human errors. When a system as complex as health care lacks a standard product definition, and product delivery is fragmented across multiple providers with different amounts and kinds of information in different settings, the system becomes dangerously cumbersome and over-complicated, with unacceptably wide variations and errors in its processes and outcomes, not to even speak of its economic inefficiency.

In contrast with the widespread use of fundamental measures in the product definitions of other industries, health care researchers typically implement neither the longstanding, repeatedly proven, and mathematically rigorous models of fundamental measurement theory nor the metrological networks through which reference standard metrics are engineered. Most industries carefully define, isolate, and estimate the parameters of their products, doing so in ways 1) that ensure industry-wide comparability and standardization, and 2) that facilitate continuous product improvement by revealing multiple opportunities for enhancement. Where organizations in other industries manage by metrics and thereby keep their eyes on the ball of product quality, health care organizations often manage only their own internal processes and cannot in fact bring the product quality ball into view.

In his message concerning the Institute for Healthcare Improvement’s Pursuing Perfection project a few years ago, Don Berwick, like others (Coye, 2001; Coye & Detmer, 1998), observed that health care does not yet have an organization setting new standards in the way that Toyota did for the auto industry in the 1970s. It still doesn’t, of course. Given the differences between the auto and health care industries uses of fundamental measures of product quality and associated abilities to keep their eyes on the quality ball, is it any wonder then, that no one in health care has yet hit a home run? It may well be that no one will hit a home run in health care until reference standard measures of product quality are devised.

The need for reference standard measures in uniform data systems is crucial, and the methods for obtaining them are widely available and well-known. So what is preventing the health care industry from adopting and deploying them? Part of the answer is the cost of the initial investment required. In 1980, metrology comprised about six percent of the U.S. gross national product (Hunter, 1980). In the period from 1981 to 1994, annual expenditures on research and development in the U.S. were less than three percent of the GNP, and non-defense R&D was about two percent (NIST Subcommittee on Research, National Science and Technology Council, 1996). These costs, however, must be viewed as investments from which high rates of return can be obtained (Barber, 1987; Gallaher, Rowe, Rogozhin, et al., 2007; Swann, 2005).

For instance, the U.S. National Institute of Standards and Technology estimated the economic impact of 12 areas of research in metrology, in four broad areas including semiconductors, electrical calibration and testing, optical industries, and computer systems (NIST, 1996, Appendix C; also see NIST, 2003). The median rate of return in these 12 areas was 147 percent, and returns ranged from 41 to 428 percent. The report notes that these results compare favorably with those obtained in similar studies of return rates from other public and private research and development efforts. Even if health care metrology produces only a small fraction of the return rate produced in physical metrology, its economic impact could still amount to billions of dollars annually. The proposed pilot projects therefore focus on determining what an effective health care outcomes metrology system should look like. What should its primary functions be? What should it cost? What rates of return could be expected from it?

Metrology, the science of measurement (Pennella, 1997), requires 1) that instruments be calibrated within individual laboratories so as to isolate and estimate the values of the required parameters (Wernimont, 1978); and 2) that individual instruments’ capacities to provide the same measure for the same amount, and so be traceable to a reference standard, be established and monitored via interlaboratory round-robin trials (Mandel, 1978).

Fundamental measurement has already succeeded in demonstrating the viability of reference standard measures of health outcomes, measures whose meaningfulness does not depend on the particular samples of items employed or patients measured. Though this work succeeds as far as it goes, it being done in a context that lacks any sense of the need for metrological infrastructure. Health care needs networks of scientists and technicians collaborating not only in the first, intralaboratory phase of metrological work, but also in the interlaboratory trials through which different brands or configurations of instruments intended to measure the same variable would be tuned to harmoniously produce the same measure for the same amount.

Implementation of the two phases of metrological innovation in health care would then begin with the intralaboratory calibration of existing and new instruments for measuring overall organizational performance, quality of care, and patients’ health status, quality of life, functionality, etc.  The second phase takes up the interlaboratory equating of these instruments, and the concomitant deployment of reference standard units of measurement throughout a health care system and the industry as a whole. To answer questions concerning health care metrology’s potential returns on investment, the costs for, and the savings accrued from, accomplishing each phase of each pilot will be tracked or estimated.

When instruments measuring in universally uniform, meaningful units are put in the hands of clinicians, a new scientific revolution will occur in medicine. It will be analogous to previous ones associated with the introduction of the thermometer and the instruments of optometry and the clinical laboratory. Such tools will multiply many times over the quality improvement methods used by Brent James, touted as holding the key to health care reform in a recent New York Times profile. Instead of implicitly hypothesizing models of perfection and assessing performance relative to them informally, what we need is a new science that systematically implements the lean ideal on industry-wide scales. The future belongs to those who master these techniques.

References

Adams, R. J., Wu, M. L., & Macaskill, G. (1997). Scaling methodology and procedures for the mathematics and science scales. In M. O. Martin & D. L. Kelly (Eds.), Third International Mathematics and Science Study Technical Report: Vol. 2: Implementation and Analysis – Primary and Middle School Years (pp. 111-145). Chestnut Hill, MA: Boston College.

Andrich, D. (2004, January). Controversy and the Rasch model: A characteristic of incompatible paradigms? Medical Care, 42(1), I-7–I-16.

Barber, J. M. (1987). Economic rationale for government funding of work on measurement standards. In R. Dobbie, J. Darrell, K. Poulter & R. Hobbs (Eds.), Review of DTI work on measurement standards (p. Annex 5). London: Department of Trade and Industry.

Berwick, D. M., James, B., & Coye, M. J. (2003, January). Connections between quality measurement and improvement. Medical Care, 41(1 (Suppl)), I30-38.

Bezruczko, N. (Ed.). (2005). Rasch measurement in health sciences. Maple Grove, MN: JAM Press.

Bond, T., & Fox, C. (2007). Applying the Rasch model: Fundamental measurement in the human sciences, 2d edition. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Coye, M. J. (2001, November/December). No Toyotas in health care: Why medical care has not evolved to meet patients’ needs. Health Affairs, 20(6), 44-56.

Coye, M. J., & Detmer, D. E. (1998). Quality at a crossroads. The Milbank Quarterly, 76(4), 759-68.

Drehmer, D. E., Belohlav, J. A., & Coye, R. W. (2000, Dec). A exploration of employee participation using a scaling approach. Group & Organization Management, 25(4), 397-418.

Drehmer, D. E., & Deklava, S. M. (2001, April). A note on the evolution of software engineering practices. Journal of Systems and Software, 57(1), 1-7.

Ealey, L. A. (1988). Quality by design: Taguchi methods and U.S. industry. Dearborn MI: ASI Press.

Gallaher, M. P., Rowe, B. R., Rogozhin, A. V., Houghton, S. A., Davis, J. L., Lamvik, M. K., et al. (2007). Economic impact of measurement in the semiconductor industry (Tech. Rep. No. 07-2). Gaithersburg, MD: National Institute for Standards and Technology.

Hawken, P., Lovins, A., & Lovins, H. L. (1999). Natural capitalism: Creating the next industrial revolution. New York: Little, Brown, and Co.

Hunter, J. S. (1980, November). The national system of scientific measurement. Science, 210(21), 869-874.

Linacre, J. M. (1993). Quality by design: Taguchi and Rasch. Rasch Measurement Transactions, 7(2), 292.

Lunz, M. E., & Linacre, J. M. (1998). Measurement designs using multifacet Rasch modeling. In G. A. Marcoulides (Ed.), Modern methods for business research. Methodology for business and management (pp. 47-77). Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.

Mandel, J. (1978, December). Interlaboratory testing. ASTM Standardization News, 6, 11-12.

Masters, G. N. (2007). Special issue: Programme for International Student Assessment (PISA). Journal of Applied Measurement, 8(3), 235-335.

National Institute for Standards and Technology (NIST). (1996). Appendix C: Assessment examples. Economic impacts of research in metrology. In C. o. F. S. Subcommittee on Research (Ed.), Assessing fundamental science: A report from the Subcommittee on Research, Committee on Fundamental Science. Washington, DC: National Standards and Technology Council [http://www.nsf.gov/statistics/ostp/assess/nstcafsk.htm#Topic%207; last accessed 18 February 2008].

National Institute for Standards and Technology (NIST). (2003, 15 January). Outputs and outcomes of NIST laboratory research. Retrieved 12 July 2009, from http://www.nist.gov/director/planning/studies.htm#measures.

Pennella, C. R. (1997). Managing the metrology system. Milwaukee, WI: ASQ Quality Press.\

Salzberger, T. (2009). Measurement in marketing research: An alternative framework. Northampton, MA: Edward Elgar.

Smith, R. M., Julian, E., Lunz, M., Stahl, J., Schulz, M., & Wright, B. D. (1994). Applications of conjoint measurement in admission and professional certification programs. International Journal of Educational Research, 21(6), 653-664.

Smith, E. V., Jr., & Smith, R. M. (2004). Introduction to Rasch measurement. Maple Grove, MN: JAM Press.

Spitzer, D. (2007). Transforming performance measurement: Rethinking the way we measure and drive organizational success. New York: AMACOM.

Swann, G. M. P. (2005, 2 December). John Barber’s pioneering work on the economics of measurement standards [Electronic version]. Retrieved http://www.cric.ac.uk/cric/events/jbarber/swann.pdf from Notes for Workshop in Honor of John Barber held at University of Manchester.

Wernimont, G. (1978, December). Careful intralaboratory study must come first. ASTM Standardization News, 6, 11-12.

Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Womack, J. P., & Jones, D. T. (1996, Sept./Oct.). Beyond Toyota: How to root out waste and pursue perfection. Harvard Business Review, 74, 140-58.

Wright, B. D. (1999). Fundamental measurement for psychology. In S. E. Embretson & S. L. Hershberger (Eds.), The new rules of measurement: What every educator and psychologist should know (pp. 65-104 [http://www.rasch.org/memo64.htm]). Hillsdale, New Jersey: Lawrence Erlbaum Associates.

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.

Review of Spitzer’s Transforming Performance Measurement

January 25, 2010

Everyone interested in practical measurement applications needs to read Dean R. Spitzer’s 2007 book, Transforming performance measurement: Rethinking the way we measure and drive organizational success (New York, AMACOM). Spitzer describes how measurement, properly understood and implemented, can transform organizational performance by empowering and motivating individuals. Measurement understood in this way moves beyond quick fixes and fads to sustainable processes based on a measurement infrastructure that coordinates decisions and actions uniformly throughout the organization.

Measurement leadership, Spitzer says, is essential. He advocates, and many organizations have instituted, the C-suite position of Chief Measurement Officer (Chapter 9). This person is responsible for instituting and managing the four keys to transformational performance measurement (Chapters 5-8):

  • Context sets the tone by presenting the purpose of measurement as either negative (to inspect, control, report, manipulate) or positive (to give feedback, learn, improve).
  • Focus concentrates attention on what’s important, aligning measures with the mission, strategy, and with what needs to be managed, relative to the opportunities, capacities, and skills at hand.
  • Integration addresses the flow of measured information throughout the organization so that the covariations of different measures can be observed relative to the overall value created.
  • Interactivity speaks to the inherently social nature of the purposes of measurement, so that it embodies an alignment with the business model, strategy, and operational imperatives.

Spitzer takes a developmental approach to measurement improvement, providing a Measurement Maturity Assessment in Chapter 12, and also speaking to the issues of the “living company” raised by Arie de Geus’ classic book of that title. Plainly, the transformative potential of performance measurement is dependent on the maturational complexity of the context in which it is implemented.

Spitzer clearly outlines the ways in which each of the four keys and measurement leadership play into or hinder transformation and maturation. He also provides practical action plans and detailed guidelines, stresses the essential need for an experimental attitude toward evaluating change, speaks directly to the difficulty of measuring intangible assets like partnership, trust, skills, etc., and shows appreciation for the value of qualitative data.

Transforming Performance Measurement is not an academic treatise, though all sources are documented, with the endnotes and bibliography running to 25 pages. It was written for executives, managers, and entrepreneurs who need practical advice expressed in direct, simple terms. Further, the book does not include any awareness of the technical capacities of measurement as these have been realized in numerous commercial applications in high stakes and licensure/certification testing over the last 50 years (Andrich, 2005; Bezruczko, 2005; Bond & Fox, 2007; Masters, 2007; Wilson, 2005). This can hardly be counted as a major criticism, since no books of this kind have yet to date been able to incorporate the often highly technical and mathematical presentations of advanced psychometrics.

That said, the sophistication of Spitzer’s conceptual framework and recommendations make them remarkably ready to incorporate insights from measurement theory, testing practice, developmental psychology, and the history of science. Doing so will propel the strategies recommended in this book into widespread adoption and will be a catalyst for the emerging re-invention of capitalism. In this coming cultural revolution, intangible forms of capital will be brought to life in common currencies for the exchange of value that perform the same function performed by kilowatts, bushels, barrels, and hours for tangible forms of capital (Fisher, 2009, 2010).

Pretty big claim, you say? Yes, it is. Here’s how it’s going to work.

  • First, measurement leadership within organizations that implements policies and procedures that are context-sensitive, focused, integrated, and interactive (i.e., that have Spitzer’s keys in hand) will benefit from instruments calibrated to facilitate:
    • meaningful mapping of substantive, additive amounts of things measured on number lines;
    • data volume reductions on the order of 80-95% and more, with no loss of information;
    • organizational and individual learning trajectories defined by hierarchies of calibrated items;
    • measures that retain their meaning and values across changes in item content;
    • adapting instruments to people and organizations, instead of vice versa;
    • estimating the consistency, and the leniency or harshness, of ratings assigned by judges evaluating performance quality, with the ability to remove those effects from the performance measures made;
    • adjusting measurement precision to the needs of the task at hand, so that time and resources are not wasted in gathering too much or too little data; and
    • providing the high quality and uniform information needed for networked collective thinking able to keep pace with the demand for innovation.
  • Second, measurement leadership sensitive to the four keys across organizations, both within and across industries, will find value in:
    • establishing industry-wide metrological standards defining common metrics for the expression of the primary human, social, and natural capital constructs of interest;
    • lubricating the flow of human, social, and natural capital in efficient markets broadly defined so as to inform competitive pricing of intangible assets, products, and services; and
    • new opportunities for determining returns on investments in human, community, and environmental resource management.
  • Third, living companies need to be able to mature in a manner akin to human development over the lifespan. Theories of hierarchical complexity and developmental stage transitions that inform the rigorous measurement of cognitive and moral transformations (Dawson & Gabrielian, 2003) will increasingly find highly practical applications in organizational contexts.

Leadership of the kind described by Spitzer is needed not just to make measurement contextualized, focused, integrated, and interactive—and so productive at new levels of effectiveness—but to apply systematically the technical, financial, and social resources needed to realize the rich potentials he describes for the transformation of organizations and empowerment of individuals. Spitzer’s program surpasses the usual focus on centralized statistical analyses and reports to demand the organization-wide dissemination of calibrated instruments that measure in common metrics. The flexibility, convenience, and scientific rigor of instruments calibrated to measure in units that really add up fit the bill exactly. Here’s to putting tools that work in the hands of those who know what to do with them!

References

Andrich, D. (2005). Georg Rasch: Mathematician and statistician. In K. Kempf-Leonard (Ed.), Encyclopedia of Social Measurement (Vol. 3, pp. 299-306). Amsterdam: Academic Press, Inc.

Bezruczko, N. (Ed.). (2005). Rasch measurement in health sciences. Maple Grove, MN: JAM Press.

Bond, T., & Fox, C. (2007). Applying the Rasch model: Fundamental measurement in the human sciences, 2d edition. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Dawson, T. L., & Gabrielian, S. (2003, June). Developing conceptions of authority and contract across the life-span: Two perspectives. Developmental Review, 23(2), 162-218.

Fisher, W. P., Jr. (2009, November). Invariance and traceability for measures of human, social, and natural capital: Theory and application. Measurement (Elsevier), 42(9), 1278-1287.

Fisher, W. P., Jr. (2010). Bringing human, social, and natural capital to life: Practical consequences and opportunities. Journal of Applied Measurement, 11, in press [Pre-press version available at http://www.livingcapitalmetrics.com/images/BringingHSN_FisherARMII.pdf%5D.

Masters, G. N. (2007). Special issue: Programme for International Student Assessment (PISA). Journal of Applied Measurement, 8(3), 235-335.

Spitzer, D. (2007). Transforming performance measurement: Rethinking the way we measure and drive organizational success. New York: AMACOM.

Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, New Jersey: Lawrence Erlbaum Associates.

Creative Commons License
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.