From: brian [brian@stat.cmu.edu]
Sent: Friday, March 16, 2007 2:14 PM
To: XXXXX
Cc: YYYYY
Subject: Pmka decision on your ms 06-1572 

Dear Dr. XXXXX,

Thank you for your letter of Feb 19 responding to the review of
Psychometrika ms \#05-1572, "Multidimensional adaptive testing with
optimal design criteria for item selection".  I am sorry that I did
not find the time before now to react to your response, and I
appreciate your patience.

I can understand your frustration with a negative result such as this.
One thing that you must understand generally is that a journal such as
Psychometrika receives many more mss. than can be published, and most
of what I do as an editor is reject mss.  When your ms. is rejected,
it does not mean that there is no worth to your work, it means that in
one way or another it is my judgement, based on my own reading and the
reports of the referees and associate editor, that the paper is not
suitable for publication in its current form, even after minor
revision.

Indeed, quite often I reject mss. in which the key ideas may be good,
and the technical quality of the work may be high, but the form of the
paper is not appropriate, or the argument that the work might be
relevant to psychometric theory or practice is not convincing, or etc.
In such cases, an author may be able to see from the referees' and
editors' comments what went wrong with the paper, and develop a new
paper that receives a better hearing at the journal.

It may be that your ms. is an example of one of these types.  Indeed I
can see in your letter to me that you are already thinking of several
ways to improve the ms. that might make it "review better", and hence
read better for readers of the journal if it is eventually published.

All three referees and the Associate Editor recognized the technical
quality of the work.  Reading "between the lines" of the written
comments of the AE and referees, there seemed to be two key
difficulties with the paper:

1. The paper did not adequately provide an argument, and evidence, for
   the potential relevance/utility of the work.

2. Despite the technical quality of the work, the paper did not have a
   clear methodological message (either theoretical methodology or
   practical methodology).  Some less crucial or less successful
   material (e.g. on E-optimality) could have been removed or greatly
   reduced to make the main points of the ms. stand out better; and
   the final conclusions of the paper seem tentative and conditional.

As a result of these two difficulties, the paper reads more like an
incremental technical contribution than like a full development of a
new direction or new insight in psychometric methodology.  (This is
the source of the AE's comments "esoteric" and "usefulness and
importance are not clear".)  It is also the reason that the first
referee thought the paper was narrow and the second referee thought
the research was not well justified.

As you point out, letter-optimality for CAT is not very well
established in the psychometric literature (with only the Segall 1996
paper as an antecedent apparently).  When a methodology is already
established in the psychometric literature, Psychometrika does publish
some incremental work in the area; but for a methodology still new to
the psychometric literature, more is required.

Two possibilities for moving forward were raised by the Associate
Editor: first, a less technical version of the paper could be
submitted to a journal focused on educational testing practice, where
the practical impact of the work might be more readily evaluated;
and/or second, the key technical parts of the ms. could be extracted
into a shorter technical note for Psychometrika.

While both of these approaches would be valuable, another possibility
might be to try to address the two weaknesses above directly: Develop
a new ms. for Psychometrika that does provide a clearer motivation for
considering letter-optimality.  Give us the big picture that makes us
see that letter-optimality is the right thing to do, and show us some
(theoretical/methodological, since it is Psychometrika) success
stories where letter-optimality design in multidimensional CAT beats
other established/plausible methods, along with an example or two.
Some of the more detailed technical developments can be summarized,
placed in an appendix, and/or moved to a separate paper.

Some of the ideas in the five points of your main letter to me,
suitably developed and efficiently presented, could be the basis of
useful and informative introduction to a new paper that properly
motivates and provides evidence for the relevance of multidimensional
CAT research, and for the utility and appropriateness of
letter-optimality in multidimensional CAT.  Additional connections to
the letter-optimality literature would also be useful.

Many of the remaining comments in the attachments to your letter
correctly deduce that the referees or AE misunderstood various points
of the ms., or were impatient with figures or other material that was
not adequately prepared for submission.  In the case of your ms., the
referees and AE represent some of the most accomplished readers of
Psychometrika.  If they are missing a point or becoming impatient,
then surely so would other readers of the journal, if the paper were
to be published.  You would have lost the opportunity to communicate,
and we would have wasted the space in the journal for your paper.
Referees are not always able to say precisely (or even correctly) why
they fell off the tracks reading your paper, but it is a fact that
they did, and it is your job as an author to improve the writing and
presentation so that they are less likely to fall off the tracks next
time.

In developing a new paper (for Psychometrika or elsewhere) I hope that
you try to take into account the comments and suggestions made in the
reviews of 06-1572. Inevitably a very few review comments may be off
the mark and can be ignored (and here is where the insights of an
experienced colleague like YYYYY become very useful), but the vast
majority of them provide valuable hints on how to improve the paper.

I think that the underlying research for this paper is potentially
very interesting.  I hope that out of this review process comes
another paper in which the relevance and value of your work is more
readily recognized by readers, and which can therefore have the impact
it deserves.  Of course you are free to submit anywhere, but I would
welcome such a ms. as a new submission to Psychometrika.

Best regards,

-BJ
 for Psychometrika

Brian Junker                    (412) 268 - 2718
Department of Statistics        brian@stat.cmu.edu
232 Baker Hall                  FAX: (412) CMU-STAT
Carnegie Mellon University        or (412) 268-7828
Pittsburgh PA 15213 USA

WWW: http://www.stat.cmu.edu/~brian/