28 June, 2007
This was sent as an email to the members of the editorial board of Software Testing, Verification and Reliability.
Wiley has recently moved us to Manuscript Central's (MC) phase two. We are using version 3.6, which means we are now using the most advanced version of MC.
This allows members of the Editorial Board to "handle papers". The process is as follows:
STVR only allows one major revision. If a paper needs a second major revision, it should be rejected. If the paper has potential, the authors can be encouraged to re-work the paper and resubmit a different paper.
Revisions will normally be assigned to the same EB member who handled the original paper. Major revisions should go to the same reviewers if possible; if not, new reviewers should be given the original reviews. Reviewers should be encouraged to comment on other reviews; for example, to help the editor recognize mistakes by reviewers. Minor revisions can be assessed by the EB member, or the EB member can ask any or all of the original reviewers to look at the revision. It is common, for example, to ask a reviewer who raised key concerns to look at a minor revision.
When analyzing reviews, your job is to make a fair, unbiased decision. This means that you have to arbitrate among differences, and when reviewers disagree, understand why and find the truth about the paper. Remember that reviews are ONLY inputs and acceptance is not based on a voting system. The editor makes the decision and can deliberately choose to ignore some comments. We are trying to strike a balance between not rejecting good (or potentially good) papers and not accepting poor papers. The journal has high standards, but do try to help the authors find a good paper if it's there. Authors deserve the chance to improve papers.
It is perfectly acceptable, and even encouraged, to ask reviewers to modify their reviews. Reviews should be respectful, professional, impersonal, and objective, no matter how poor the paper is. It is also okay to go back to a reviewer and ask a question such as "You said A, reviewer 2 said B, do you think it's possible that reviewer 2 may have seen something you did not?". (Be sure to protect the anonymity of reviewers!)
Your assessment needs to be more than a simple recommendation of reject, major revision, minor revision, or accept. In particular, you need to specify the precise set of expectations to be fulfilled for the editor to accept the paper (for a revision), or the critical problems that led to rejection. This summary of the reviews is very helpful to the authors and very important to the journal's reputation.
I would prefer your recommendations to me to be very direct, clear, and explicit. This is a private, confidential communication, and any suggestions will be appropriately edited before anything is sent to the authors.
Choosing reviewers is an art. Again, some of you have experience with this, but some do not, so please bear with me. STVR requires each paper to be reviewed by at least 3 people. It's best if the 3 have some balance in terms of levels of expertise and experience. A true expert on the topic needs to evaluate every paper, but it also helps to have a reviewer to represent the rest of STVR's readers to ensure that the paper is understandable outside of the small circle of experts. While experience certainly helps with reviewers, I have found younger researchers to be more responsive and energetic.
You can find reviewers among your circle of professional colleagues, from the references in the paper, among authors (and editorial board members) from STVR, and from a web search on key words. I will also recommend reviewers when possible. I strongly encourage each of you to develop a system to track how responsive and helpful each reviewer is. As many of you know, one of the greatest frustrations among editors is chasing after reviewers. Try to allow for failure by doing things like selecting 4 or 5 names and initially inviting 3, and having backup emergency reviewers.
Although it is not wrong to add yourself as a reviewer for a paper, I caution against it. Your role is to independently and objectively arbitrate among three reviewers, and if you are one of the reviewers it becomes very hard to be impartial.
Time to making decisions on papers is a perennial problem for all journals and we are no different, but it's certainly a problem that we can work together to solve.
Reducing time to decision is about managing the steps in the process above. MY GOAL IS TO HAVE DECISIONS ON PAPERS WITHIN 3 MONTHS OF SUBMISSION. This is an ambitious goal, which I will bracket by stating that any paper that takes more than 6 months represents a serious problem. Satisfying this goal is up to you. And it's not that hard, we just need to manage the process.
I expect to assign papers to EB members within 48 hours of submission. It is very important that you respond as quickly as possible, especially if for some reason you cannot handle the paper. The biggest source of delay is people who wait to say "no"--that creates two delays; the delay to respond and the delay to find someone else.
I would like you to try to get papers to reviewers within one week. It takes a couple of hours to select potential reviewers, and most reviewers will respond within a day or two. Again, the biggest source of delay is when potential reviewers take the 5 days we allocate, then say "no". The second source of delay is when you have to contact 4, 5, or 6 reviewers to get three (my maximum is 8).
We ask reviewers to respond within 6 weeks, which is very reasonable and gives plenty of time. I strongly suggest that after 8 weeks, you contact a 4th reviewer and ask for a quick turnaround. Once you get the third review, be sure to inform the slow reviewer that the review is no longer necessary. We are all busy, but that person made a commitment and should be able to honor it. Telling the slow reviewer is a professional courtesy (he or she can remove the task from the list), and also sends a message that we are serious. If both the 4th and 3rd reviewer submits a review simultaneously, then the author will benefit from extra help.
Once 3 reviews are in, I hope you can send me a recommendation within 3 days. I will plan to make a final decision within 3 additional days.
This is a total of 71 days, or about 10 weeks. Even with some small delays, there is enough built-in slack so that we should reasonably be able to inform the authors within 3 months of submission. Unusual delays should not do more than double the time--hence my 6 month rule.
I hope you all realize that deciding which papers are published in a major international journal is a big responsibility. The decisions we make impact people's careers, just as previous decisions impacted our careers. As decision-makers for the major journal in software testing, we are care-takers for the field. We have a professional obligation to take this responsibility seriously and respectfully.
I look forward to continuing to work with each of you.
Jeff Offutt
Editor in Chief, STVR
28 June, 2007
=======================================
Wiley's Software Testing, Verification and Reliability
Useful research in making better software.
http://www.interscience.wiley.com/journal/softwaretesting
=======================================