William P. Fisher, Jr.
Living Capital Metrics LLC, BEAR Center, Graduate School of Education, UC Berkeley, and
the Research Institute of Sweden, Gothenburg
4 January 2021
I. Summary
As was observed by Reginald McGregor in the STEM learning ecosystems Zoom call today preparing for the Biden-Harris Town Hall meetings, past policies addressing equity, quality programming, funding, professional development, after school/school alignment, and other issues in education have not had the desired impacts on outcomes. McGregor then asked, what must we do differently to obtain the results we want and need? In short, what we must do differently is to focus systematically on how to create a viral contagion of trust–not just with each other but with our data and our institutions. Trust depends intrinsically on verifiable facts, personal ownership, and proven productive consequences–and we have a wealth of untapped resources for systematically building trust in mass scalable ways, for creating a social contagion of trust that disseminates the authentic wealth of learning and valued relationships. This proposal describes those resources, where they can be found, who the experts in these areas are, which agencies have historically been involved in developing them, what is being done to put them to work, and how we should proceed from here. Because it will set the tone for everything that follows, and because there is no better time for such a seismic shift in the ground than at the beginning, a clear and decisive statement of what needs to be done differently ought to be a Day One priority for the Biden-Harris administration. Though this memo was initiated in response to the STEM learning ecosystems town hall meetings, its theme is applicable across a wide range of policy domains, and should be read as such.
II. Challenge and Opportunity
What needs to be done differently hinges on the realization that a theme common to all of the issues identified by McGregor concerns the development of trusting relationships. Igniting viral contagions of trust systematically at mass scales requires accomplishing two apparently contradictory goals simultaneously: creating communications and information standards that are both universally transparent and individually personalized. It may appear that these two goals cannot be achieved at the same time, but in actual fact they are integrated in everyday language. The navigable continuity of communications and information standards need not be inconsistent with the unique strengths, weaknesses, and creative improvisations of custom tailored local conversations. Standards do not automatically entail pounding square pegs into round holes.
Transparent communications of meaningful high quality information cultivate trust by inspiring confidence in the repeated veracity and validity of what is said. Capacities for generalizing lessons learned across localities augment that trust and support the spread of innovations. Personalized information applicable to unique individual circumstances cultivates trust as students, teachers, parents, administrators, researchers, employers, and others are each able (a) to recognize their own special uniqueness reflected in information on their learning outcomes, (b) to see the patterns of their learning and growth reflected in that information over time, and (c) to see themselves in others’ information, and others in themselves. Systematic support and encouragement for policies and practices integrating these seemingly contradictory goals would constitute truly new approaches to old problems. Given that longstanding and widespread successes in combining these goals have already been achieved, new hope for resounding impacts becomes viable, feasible, and desirable.
III. Plan of Action
To stop the maddening contradiction of expecting different results from repetitions of the same behaviors, decisive steps must be taken toward making better use of existing models and methods, ones that coherently inform new behaviors leading to new outcomes. We are not speaking here of small incremental gains produced via intensive but microscopically focused efforts. We are raising the possibility that we may be capable of igniting viral contagions of trust. Just as the Arab Spring was in many ways fostered by the availability of new and unfettered technologically mediated social networks like Facebook and Twitter, so, also, will the creation of new outcomes communications platforms in education, healthcare, social services, and environmental resource management unleash powerful social forces. In the same way that smartphones are both incredibly useful for billions of people globally and are also highly technical devices involving complexities beyond the ken of the vast majority of those using them, so, too, do the complex models and methods at issue here have similar potentials for mass scaling.
To efficiently share transferable lessons as to what works, we need the common quantitative languages of outcome measurement standards, where (a) quantities are defined not in the ordinal terms of test scores but in the interval terms of metrologically traceable units with associated uncertainties, and (b) where those quantities are estimated not from just one set of assessment questions or items but from linked collections of diverse arrays of different kinds of self, observational, portfolio, peer, digital, and other assessments (or even from theory). To support individuals’ creative improvisations and unique circumstances, those standards, like the alphabets, grammars, and dictionaries setting the semiotic standards of everyday language, must enable new kinds of qualitative conversations negotiating the specific hurdles of local conditions. Custom tailored individual reports making use of interval unit estimates and uncertainties have been in use globally for decades.
Existing efforts in this area have been underway since the work of Thurstone in the 1920s, Rasch and Wright in the period from the 1950s through the 1990s, and of thousands of others since then. Over the course of the last several decades, the work of these innovators has been incorporated into hundreds of research studies funded by the Institute for Education Sciences, the National Science Foundation, and the National Institutes of Health. Most of these applications have, however, been hobbled by limited conceptualizations restricting expectations to the narrow terms of statistical hypothesis testing instead of opening onto the far more expansive possibilities offered by an integration of metrological standards and individualized reporting. This is a key way of expressing the crux of the shift proposed here. We are moving away from merely numeric statistical operations conducted via centrally planned and controlled analytic methods, and we are moving toward fully quantitative quality-assured measurement operations conducted via widely distributed and socially self-organized methods.
Because history shows existing institutions rarely successfully alter their founding principles, it is likely necessary for a government agency previously not involved in this work to now take the lead. That agency should be the National Institute of Standards and Technology (NIST). This recommendation is supported by the recent emergence of new alliances of psychometricians and metrologists clarifying the theory and methods needed for integrating the two seemingly opposed goals of comparable standards and custom tailored applications. The International Measurement Confederation (IMEKO) of national metrology institutes has provided a forum for reports in this area since 2008, as has, since 2017, the International Metrology Congress, held in Paris. An international meeting bringing together equal numbers of metrologists and psychometricians was held at UC Berkeley in 2016 (NIST’s Antonio Possolo gave a keynote), dozens of peer-reviewed journal articles in this new area have appeared since 2009, two authoritative books have appeared since 2019, and multiple ongoing collaborations internationally focused on the development of new unit standards and traceable instrumentation for education, health care, and other fields are underway.
Important leaders in this area capable of guiding the formation of the measurement-specific policies for research and practice include David Andrich (U Western Australia, Perth), Matt Barney (Leaderamp, Vacaville, CA), Betty Bergstrom (Pearson VUE, Chicago), Stefan Cano (Modus Outcomes, UK), Theo Dawson (Lectica, Northampton, MA), Peter Hagell (U Kristianstad, Sweden), Martin Ho (FDA), Mike Linacre (Winsteps.com), Larry Ludlow (Boston College), Luca Mari (U Cattaneo, Italy), Robert Massof (Johns Hopkins), Andrew Maul (UC Santa Barbara), Jeanette Melin (RISE, Sweden), Janice Morrison (TIES, Cleveland), Leslie Pendrill (RISE, Sweden), Maureen Powers (Gemstone Optometry, Berkeley), Andrea Pusic (Brigham & Women’s, Boston), Matthew Rabbitt (USDA), Thomas Salzberger (U Vienna, Austria), Karen Schmidt (U Virginia), Mark Wilson (UC Berkeley), and many others.
Partnerships across economic sectors are essential to the success of this initiative. Standards provide the media by which different groups of stakeholders can advance their unique interests more effectively in partnership than they can in isolation. Calls for proposals should stress the vital importance of establishing the multidisciplinary functionality of boundary objects residing at the borders between disciplines. Just as has been accomplished for the SI Unit metrological standards in the natural sciences, educators’ needs for comparable but customized information must be aligned with the analogous needs of stakeholders in other domains, such as management, clinical practice, law, accounting, finance, economics, etc. Of the actors in this domain listed above, at this time, the Research Institute of Sweden (RISE) is most energetically engaged in forming the needed cross-disciplinary collaborations.
Though the complexity and cost of such efforts appear almost insurmountable, beginning the process of envisioning how to address the challenges and capitalize on the opportunities is far more realistic and productive than continuing to flounder without direction, as we currently are and have been for decades. Estimates of the cost of creating, maintaining, and improving existing standards come to about 8% of GDP, with returns on investment estimated by NIST to be in the range of about 40% to over 400%, with a mean of about 140%. The levels of investment needed in the new metrological efforts, and the returns to be gained from those investments, will not likely differ significantly from these estimates.
IV. Conclusion
This proposal is important because it offers a truly original response to the question of what needs to be done differently in STEM education and elsewhere to avoid continuing to reproduce the same tired and ineffective results. The originality of the proposal is complemented by the depth at which it taps the historical successes of the natural sciences and the economics of standards: efficient markets for trading on trust in productive ways could lead to viral contagions of caring relationships. The proposal is also supported by the intuitive plausibility of taking natural language as a model for the creation of new common languages for the communication and improvement of learning, healthcare, employment, and other outcomes. As is the case for any authentic paradigm shift, opposition to the proposal is usually rooted in assumptions that existing expertise, methods, and tools are sufficient to the task, even when massive amounts of evidence point to the need for change. Simple, small, and inexpensive projects can be designed as tests of the concept and as means of attracting interest in the paradigm shift. Convening cross-sector groups of collaborators for the purposes of designing and conducting small demonstration projects may be an effective way of beginning. Finally, the potential for creating economically self-sustaining cycles of investments and returns could be an attractive way of incentivizing private sector participation, especially when this is expressed in terms of the alignment of financial wealth with the authentic wealth of trusting relationships.
V. About the author
William P. Fisher, Jr., Ph.D. received his doctorate from the University of Chicago, where he was mentored by Benjamin D. Wright and supported by a Spencer Foundation Dissertation Research Fellowship. He has been on the staff of the BEAR Center in the Graduate School of Education at UC Berkeley since 2011, and has consulted independently via Living Capital Metrics LLC since 2009. In 2020, Dr. Fisher joined the staff of the Research Institute of Sweden as a Senior Research Scientist. Dr. Fisher is recognized for contributions to measurement theory and practice that span the full range from the philosophical to the applied in fields as diverse as special education, mindfulness practice, nursing, rehabilitation, clinical chemistry, metrology, health outcomes, and survey research.
VI. Supporting literature
Andrich, David. “A Rating Formulation for Ordered Response Categories.” Psychometrika 43, no. 4, December 1978: 561-73.
Andrich, David. Rasch Models for Measurement. Sage University Paper Series on Quantitative Applications in the Social Sciences, vol. series no. 07-068. Beverly Hills, California: Sage, 1988.
Andrich, David, and Ida Marais. A Course in Rasch Measurement Theory: Measuring in the Educational, Social, and Health Sciences. Cham, Switzerland: Springer, 2019.
Barber, John M. “Economic Rationale for Government Funding of Work on Measurement Standards.” In Review of DTI Work on Measurement Standards, ed. R. Dobbie, J. Darrell, K. Poulter and R. Hobbs, Annex 5. London: Department of Trade and Industry, 1987.
Barney, Matt, and William P. Fisher, Jr. “Adaptive Measurement and Assessment.” Annual Review of Organizational Psychology and Organizational Behavior 3, April 2016: 469-90.
Cano, Stefan, Leslie Pendrill, Jeanette Melin, and William P. Fisher, Jr. “Towards Consensus Measurement Standards for Patient-Centered Outcomes.” Measurement 141, 2019: 62-69, https://doi.org/10.1016/j.measurement.2019.03.056.
Chien, Tsair-Wei, John Michael Linacre, and Wen-Chung Wang. “Examining Student Ability Using KIDMAP Fit Statistics of Rasch Analysis in Excel.” In Communications in Computer and Information Science, ed. Honghua Tan and Mark Zhou, 578-85. Berlin: Springer Verlag, 2011.
Chuah, Swee-Hoon, and Robert Hoffman. The Evolution of Measurement Standards. Tech. Rept. no. 5. Nottingham, England: Nottingham University Business School, 2004.
Fisher, William P., Jr. “The Mathematical Metaphysics of Measurement and Metrology: Towards Meaningful Quantification in the Human Sciences.” In Renascent Pragmatism: Studies in Law and Social Science, ed. Alfonso Morales, 118-53. Brookfield, VT: Ashgate Publishing Co., 2003.
Fisher, William P., Jr. “Meaning and Method in the Social Sciences.” Human Studies: A Journal for Philosophy and the Social Sciences 27, no. 4, October 2004: 429-54.
Fisher, William P., Jr. “Invariance and Traceability for Measures of Human, Social, and Natural Capital: Theory and Application.” Measurement 42, no. 9, November 2009: 1278-87.
Fisher, William P., Jr. NIST Critical National Need Idea White Paper: Metrological Infrastructure for Human, Social, and Natural Capital. Tech. Rept. no. http://www.nist.gov/tip/wp/pswp/upload/202_metrological_infrastructure_for_human_social_natural.pdf. Washington, DC: National Institute for Standards and Technology, 2009.
Fisher, William P., Jr. “Measurement, Reduced Transaction Costs, and the Ethics of Efficient Markets for Human, Social, and Natural Capital,” Bridge to Business Postdoctoral Certification, Freeman School of Business, Tulane University, 2010, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2340674.
Fisher, William P., Jr. “What the World Needs Now: A Bold Plan for New Standards [Third Place, 2011 NIST/SES World Standards Day Paper Competition].” Standards Engineering 64, no. 3, 1 June 2012: 1 & 3-5 [http://ssrn.com/abstract=2083975].
Fisher, William P., Jr. “Imagining Education Tailored to Assessment as, for, and of Learning: Theory, Standards, and Quality Improvement.” Assessment and Learning 2, 2013: 6-22.
Fisher, William P., Jr. “Metrology, Psychometrics, and New Horizons for Innovation.” 18th International Congress of Metrology, Paris, September 2017: 09007, doi: 10.1051/metrology/201709007.
Fisher, William P., Jr. “A Practical Approach to Modeling Complex Adaptive Flows in Psychology and Social Science.” Procedia Computer Science 114, 2017: 165-74, https://doi.org/10.1016/j.procs.2017.09.027.
Fisher, William P., Jr. “Modern, Postmodern, Amodern.” Educational Philosophy and Theory 50, 2018: 1399-400. Reprinted in What Comes After Postmodernism in Educational Theory? ed. Michael Peters, Marek Tesar, Liz Jackson and Tina Besley, 104-105, New York: Routledge, DOI: 10.1080/00131857.2018.1458794.
Fisher, William P., Jr. “Contextualizing Sustainable Development Metric Standards: Imagining New Entrepreneurial Possibilities.” Sustainability 12, no. 9661, 2020: 1-22, https://doi.org/10.3390/su12229661.
Fisher, William P., Jr. “Measurements Toward a Future SI.” In Sensors and Measurement Science International (SMSI) 2020 Proceedings, ed. Gerald Gerlach and Klaus-Dieter Sommer, 38-39. Wunstorf, Germany: AMA Service GmbH, 2020, https://www.smsi-conference.com/assets/Uploads/e-Booklet-SMSI-2020-Proceedings.pdf.
Fisher, William P., Jr. “Wright, Benjamin D.” In SAGE Research Methods Foundations, ed. P. Atkinson, S. Delamont, A. Cernat, J. W. Sakshaug and R.A. Williams. Thousand Oaks, CA: Sage Publications, 2020, https://methods.sagepub.com/foundations/wright-benjamin-d.
Fisher, William P., Jr., and A. Jackson Stenner. “Theory-Based Metrological Traceability in Education: A Reading Measurement Network.” Measurement 92, 2016: 489-96, http://www.sciencedirect.com/science/article/pii/S0263224116303281.
Fisher, William P., Jr., and Mark Wilson. “Building a Productive Trading Zone in Educational Assessment Research and Practice.” Pensamiento Educativo: Revista de Investigacion Educacional Latinoamericana 52, no. 2, 2015: 55-78, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2688260.
Gallaher, Michael P., Brent R. Rowe, Alex V. Rogozhin, Stephanie A. Houghton, J. Lynn Davis, Michael K. Lamvik, and John S. Geikler. Economic Impact of Measurement in the Semiconductor Industry. Tech. Rept. no. 07-2. Gaithersburg, MD: National Institute for Standards and Technology, 2007.
He, W., and G. G. Kingsbury. “A Large-Scale, Long-Term Study of Scale Drift: The Micro View and the Macro View.” Journal of Physics Conference Series 772, 2016: 012022, https://iopscience.iop.org/article/10.1088/1742-6596/772/1/012022/meta.
Holster, Trevor A., and J. W. Lake. “From Raw Scores to Rasch in the Classroom.” Shiken 19, no. 1, April 2015: 32-41.
Hunter, J Stuart. “The National System of Scientific Measurement.” Science 210, no. 21, 1980: 869-74.
Linacre, John Michael. “Individualized Testing in the Classroom.” In Advances in Measurement in Educational Research and Assessment, ed. Geofferey N Masters and John P. Keeves, 186-94. New York: Pergamon, 1999.
Mari, Luca, and Mark Wilson. “An Introduction to the Rasch Measurement Approach for Metrologists.” Measurement 51, May 2014: 315-27, http://www.sciencedirect.com/science/article/pii/S0263224114000645.
Mari, Luca, Mark Wilson, and Andrew Maul. Measurement Across the Sciences [in Press]. Springer Series in Measurement Science and Technology. Cham: Springer, 2021.
Massof, Robert W. “Editorial: Moving Toward Scientific Measurements of Quality of Life.” Ophthalmic Epidemiology 15, 1 August 2008: 209-11.
Masters, Geofferey N. “KIDMAP – a History.” Rasch Measurement Transactions 8, no. 2, 1994: 366 [http://www.rasch.org/rmt/rmt82k.htm].
Morrison, Jan, and William P. Fisher, Jr. “Connecting Learning Opportunities in STEM Education: Ecosystem Collaborations Across Schools, Museums, Libraries, Employers, and Communities.” Journal of Physics: Conference Series 1065, no. 022009, 2018, doi:10.1088/1742-6596/1065/2/022009.
Morrison, Jan, and William P. Fisher, Jr. “Measuring for Management in Science, Technology, Engineering, and Mathematics Learning Ecosystems.” Journal of Physics: Conference Series 1379, no. 012042, 2019, doi:10.1088/1742-6596/1379/1/012042.
National Institute for Standards and Technology. “Appendix C: Assessment Examples. Economic Impacts of Research in Metrology.” In Assessing Fundamental Science: A Report from the Subcommittee on Research, Committee on Fundamental Science, ed. Committee on Fundamental Science Subcommittee on Research. Washington, DC: National Standards and Technology Council, 1996, https://wayback.archive-it.org/5902/20150628164643/http://www.nsf.gov/statistics/ostp/assess/nstcafsk.htm#Topic%207.
National Institute for Standards and Technology. Outputs and Outcomes of NIST Laboratory Research. 18 December 2009. NIST. Last visited 18 April 2020 <https://www.nist.gov/director/outputs-and-outcomes-nist-laboratory-research>.
North, Douglass C. Structure and Change in Economic History. New York: W. W. Norton & Co., 1981.
Pendrill, Leslie. Quality Assured Measurement: Unification Across Social and Physical Sciences. Cham: Springer, 2019.
Pendrill, Leslie, and William P. Fisher, Jr. “Counting and Quantification: Comparing Psychometric and Metrological Perspectives on Visual Perceptions of Number.” Measurement 71, 2015: 46-55, doi: http://dx.doi.org/10.1016/j.measurement.2015.04.010.
Poposki, Nicola, Nineta Majcen, and Philip Taylor. “Assessing Publically Financed Metrology Expenditure Against Economic Parameters.” Accreditation and Quality Assurance: Journal for Quality, Comparability and Reliability in Chemical Measurement 14, no. 7, July 2009: 359-68.
Rasch, Georg. Probabilistic Models for Some Intelligence and Attainment Tests. Reprint, University of Chicago Press, 1980. Copenhagen, Denmark: Danmarks Paedogogiske Institut, 1960.
Rasch, Georg. “On General Laws and the Meaning of Measurement in Psychology.” In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability: Volume IV: Contributions to Biology and Problems of Medicine, ed. Jerzy Neyman, 321-33 [http://www.rasch.org/memo1960.pdf]. Berkeley: University of California Press, 1961.
Solloway, Sharon, and William P. Fisher, Jr. “Mindfulness in Measurement: Reconsidering the Measurable in Mindfulness.” International Journal of Transpersonal Studies 26, 2007: 58-81 [http://digitalcommons.ciis.edu/ijts-transpersonalstudies/vol26/iss1/8 ].
Stenner, A. Jackson, William P. Fisher, Jr., Mark H. Stone, and Don S. Burdick. “Causal Rasch Models.” Frontiers in Psychology: Quantitative Psychology and Measurement 4, no. 536, August 2013: 1-14 [doi: 10.3389/fpsyg.2013.00536].
Sumner, Jane, and William P. Fisher, Jr. “The Moral Construct of Caring in Nursing as Communicative Action: The Theory and Practice of a Caring Science.” Advances in Nursing Science 31, no. 4, 2008: E19-36.
Swann, G. M. P. The Economics of Metrology and Measurement. Report for the National Measurement Office and Department of Business, Innovation and Skills. London, England: Innovative Economics, Ltd, 2009.
Williamson, Gary L. “Exploring Reading and Mathematics Growth Through Psychometric Innovations Applied to Longitudinal Data.” Cogent Education 5, no. 1464424, 2018: 1-29.
Wilson, Mark, Ed. Towards Coherence Between Classroom Assessment and Accountability. National Society for the Study of Education, vol. 103, Part II. Chicago: University of Chicago Press, 2004.
Wilson, Mark R. Constructing Measures. Mahwah, NJ: Lawrence Erlbaum Associates, 2005.
Wilson, Mark R. “Seeking a Balance Between the Statistical and Scientific Elements in Psychometrics.” Psychometrika 78, no. 2, April 2013: 211-36.
Wilson, Mark. “Making Measurement Important for Education: The Crucial Role of Classroom Assessment.” Educational Measurement: Issues and Practice 37, no. 1, 2018: 5-20.
Wilson, Mark, and William P. Fisher, Jr. “Preface: 2016 IMEKO TC1-TC7-TC13 Joint Symposium: Metrology Across the Sciences: Wishful Thinking?” Journal of Physics Conference Series 772, no. 1, 2016: 011001, http://iopscience.iop.org/article/10.1088/1742-6596/772/1/011001/pdf.
Wilson, Mark, and William P. Fisher, Jr., Eds. Psychological and Social Measurement: The Career and Contributions of Benjamin D. Wright. Springer Series in Measurement Science and Technology, ed. M. G. Cain, G. B. Rossi, J. Tesai, M. van Veghel and K.-Y Jhang. Cham, Switzerland: Springer Nature, 2017, https://link.springer.com/book/10.1007/978-3-319-67304-2.
Wilson, Mark, and William P. Fisher, Jr. “Preface of Special Issue, Psychometric Metrology.” Measurement 145, 2019: 190, https://www.sciencedirect.com/journal/measurement/special-issue/10C49L3R8GT.
Wilson, Mark, and Kathleen Scalise. “Assessment of Learning in Digital Networks.” In Assessment and Teaching of 21st Century Skills: Methods and Approach, ed. Patrick Griffin and Esther Care, 57-81. Dordrecht: Springer Netherlands, 2015.
Wilson, Mark, and Y. Toyama. “Formative and Summative Assessments in Science and Literacy Integrated Curricula: A Suggested Alternative Approach.” In Language, Literacy, and Learning in the STEM Disciplines, ed. Alison L. Bailey, Carolyn A. Maher and Louise C. Wilkinson, 231-60. New York: Routledge, 2018.
Wright, Benjamin D. “Sample-Free Test Calibration and Person Measurement.” In Proceedings of the 1967 Invitational Conference on Testing Problems, 85-101 [http://www.rasch.org/memo1.htm]. Princeton, New Jersey: Educational Testing Service, 1968.
Wright, Benjamin D. “Solving Measurement Problems with the Rasch Model.” Journal of Educational Measurement 14, no. 2, 1977: 97-116 [http://www.rasch.org/memo42.htm].
Wright, Benjamin D. “Despair and Hope for Educational Measurement.” Contemporary Education Review 3, no. 1, 1984: 281-88 [http://www.rasch.org/memo41.htm].
Wright, Benjamin D. “Additivity in Psychological Measurement.” In Measurement and Personality Assessment, ed. Edward Roskam, 101-12. North Holland: Elsevier Science Ltd, 1985.
Wright, Benjamin D. “A History of Social Science Measurement.” Educational Measurement: Issues and Practice 16, no. 4, Winter 1997: 33-45, 52. https://doi.org/10.1111/j.1745-3992.1997.tb00606.x.
Wright, Benjamin D., and G N Masters. Rating Scale Analysis. Chicago: MESA Press, 1982. Full text: https://www.rasch.org/BTD_RSA/pdf%20%5Breduced%20size%5D/Rating%20Scale%20Analysis.pdf.
Wright, Benjamin D., R. J. Mead, and L. H. Ludlow. KIDMAP: Person-by-Item Interaction Mapping. Tech. Rept. no. MESA Memorandum #29. Chicago: MESA Press [http://www.rasch.org/memo29.pdf], 1980.
Wright, Benjamin D., and Mark H Stone. Best Test Design. Chicago: MESA Press, 1979, Full text: https://www.rasch.org/BTD_RSA/pdf%20%5Breduced%20size%5D/Best%20Test%20Design.pdf.
LivingCapitalMetrics Blog by William P. Fisher, Jr., Ph.D. is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
Based on a work at livingcapitalmetrics.wordpress.com.
Permissions beyond the scope of this license may be available at http://www.livingcapitalmetrics.com.