Table Comparing Scores, Ratings, and Percentages with Real Measures

(Documentation to be posted tomorrow.)

Characteristics	Raw Scores and/or Percentages	Rasch Measurement
Quantitative hypothesis	Neither formulated nor tested	Formulated and tested
Criteria for falsifying quantitative hypothesis	None	Additivity, conjoint transitivity, parameter separation, unidimensionality, invariance, statistical sufficiency, monotonicity, homogeneity, infinite divisibility, etc.
Relation to sample distribution	Dependent	Independent
Paradigm	Descriptive statistics	Prescriptive measurement
Model-data relation	Models describe data, models fit to data, model with best statistics chosen	Models prescribe data quality needed for objective inference, data fit to models, GIGO principle
Relation to structure of natural laws	None	Identical
Statistical tests of quantitative hypothesis	None	Information-weighted and outlier-sensitive model fit, Principal Components Analysis, many other fit statistics available
Reliability coefficients	Cronbach’s alpha, KR-20, etc.	Cronbach’s alpha, KR-20, etc. and Separation, Strata
Reliability error source	Unexplained portion of variance	Mean square of individual error estimates
Range of measurement	Arbitrary, from minimum to maximum score	Nonarbitrary, infinite
Unit status	Ordinal, nonlinear	Interval, linear
Unit status assumed in statistical comparisons	Interval, linear	Interval, linear
Proofs of unit status	Correlational	Axiomatic; reproduced physical metrics; graphical plots; independent cross-sample recalibrations; etc.
Error theory for individual scores/measures	None	Derived from sampling theory
Architecture (capacity to add/delete items)	Closed	Open
Supports adaptive administration and mass customization	No (changes to items change meaning of scores)	Yes (changes to items do not change meaning of measure)
Supports traceability to metrological reference standard	No	Yes
Domains scored	Either persons or items but rarely both	All facets in model (persons, items, rating categories, judges, tasks, etc.)
Comparability of domains scored	Would be incomparable if scored	Comparable; each interpreted in terms of the other
Unscored domain characteristics	Assumed all same score or random (though probably not)	No unscored domain
Relation with other measures of same construct	Incommensurable	Commensurable and equatable
Construct definition	None	Consistency, meaningfulness, interpretability, and predictability of calibration/measure hierarchies
Focus of interpretation	Mean scores or percentages relative to demographics or experimental groups	Measures relative to calibrations and vice versa; measures relative to demographics or experimental groups
Relation to qualitative methods	Stark difference in philosophical commitments	Rooted in same philosophical commitments
Quality of research dialogue	Researchers’ expertise elevated relative to research subjects	Research subjects voice individual and collective perspectives on coherence of construct as defined by researchers’ questions
Source of narrative theme	Researcher	Object of unfolding dialogue

Tags: Meaning, measurement, ordinal, performance metrics