Measurement variation in method comparison studies: An empirical examination

Procedures commonly used in rehabilitation research to analyze data from method comparison studies were examined. A simulation study was conducted in which different types of measurement variation were introduced to a reference set of range of motion (ROM) values. The test ROM data included individual data sets with random variation, alternate variation, constant variation, and proportional variation. Comparisons were made between original ROM reference values and simulated test ROM data. The statistical comparisons involved Pearson product-moment correlation, least square regression, t tests, and intraclass correlation coefficients. The individual data sets were also presented graphically using a method referred to as limits of agreement. The analyses revealed Pearson product-moment correlations, t tests, and other commonly used statistical procedures were not sensitive to different types of measurement variation in evaluating the results of method comparison investigations. The findings indicate that the limits of agreement technique, based on plotting difference scores against mean values, provided the most accurate estimation of measurement error. The advantages and limitations of various procedures to analyze agreement between rehabilitation assessment methods are described.