In modern psychometric analysis of cognitive assessment, there
is a choice between
psychometric vs.
cognitive science
paradigms for modeling the latent scale. The first involves as few as
one continuous latent ability parameter, while the second focuses on a
set of binary latent skills. When the expert cognitive model is
qualitatively specified (eg. paragraphs describing general trends in
observable behavior for a set of developmental stages), interpretation
of responses and latent variables is flexible, and may even be
ambiguous. Models incorporating aspects from both psychometric and
cognitive science paradigms can help in exploring response patterns, and
refining both the exam design and the cognitive model. Here we present two
such analyses of a pilot study of proportional reasoning. The first is a
Rasch model using binary response coding with a continuous latent trait,
which can approximate a set of developmental stages through careful item
design and milestones. The second is a Bayes net model using polytomous
response coding linked compensatorily to a set of latent skills, which
allows for a factor-analytic approach to skill interpretation. We explore
each scheme's usefulness in inferring a student's current state of
knowledge, directing program planners, and directing assessment developers
toward refining either the qualitative cognitive model or exam items.