The 'gold standard' editorial process defined by the Cochrane Collaboration aims to produce studies that are appropriate to the publication medium, important, useful, original, methodologically sound, ethical and accurate . This is the first study to investigate journal editor demographics and the factors influencing orthopaedic journal editors during the manuscript review process. Evidence based medicine relies on the journal review process to provide an unbiased representation of high quality studies . Deleterious effects on health policy and clinical decisions can result from publication and related biases. A recent Health Technology Assessment found that studies with significant or positive results were more likely to be published and published earlier than those with non-significant or negative results . Exclusion of non-English language studies was also found to result in a high risk of bias in certain research areas such as complementary and alternative medicine .
The questionnaire used was based on a previously validated questionnaire designed by Radford et al . Additional questions and a review of journal websites and instructions to authors were used to provide additional information on journal demographics. The response rate was 42% with no significant difference in impact factor between responders and non-responders' journal. Whilst the error level was 11.4% with 90% confidence limits, this is the first study of its kind and represents an indication of journal editors' preferences for manuscript acceptance as well as their characteristics which has not previously been documented in the literature. Poor response rates are a potential source of bias since the confidence with which the survey findings can be accepted are a function of the achieved sample size .
A small sample size resulting from poor response rates exists if the responders differ from the non-responders in terms of key characteristics . A comparison of impact factor between responders and non-responders showed no significant difference with p = 0.67. However comparison of language of publication showed 96% of responders edited publications in English with only 76% of non-responders editing publications in English (p = 0.03). This may indicated that non-response was due to an inability to interpret the questionnaire which was in English. This is a weakness in the study design indicating that the results may be subject to language bias.
Analysis of the demographics of orthopaedic journal editors showed that 90% were male. Gender bias in terms of editors, reviewers and authors has been studied in the literature. Gilbert et al retrospectively reviewed 1851 research articles submitted to JAMA in 1991 and found no bias based on reviewers' or editors' gender on the decision to accept or reject a manuscript . A further study conducted by Caelleigh et al of 50 female and 50 male reviewers showed no bias when assessing an empirical study with two versions, one attributing a lower income in women to gender issues and the other attributing it to social learning factors [12, 15]. However, other studies have confirmed the presence of gender bias in the publication process [16, 17]. A Scandinavian study used a sham paper with either male or female authors to assess gender bias in a 1637 randomly selected Swedish physicians. Female authors were ranked higher than male authors. Male assessors were found to reflect no gender bias but female assessors upgraded female authors more than male authors . A study of grant proposals has also shown gender bias to the detriment of female applicants .
Eighty-three percent of editors spoke English as a first language with 50% of journals published in the US and 40% in the UK. Geographical bias has been shown to influence publication. A recent study of non-scientific factors influencing acceptance of manuscripts to the American Journal of Bone and Joint Surgery showed that manuscripts submitted from countries other than the US or Canada were significantly less likely to be accepted (odds ratio 0.51, 95% confidence interval, 0.28 to 0.92; p = 0.026) . A Scandinavian study sent a methodologically flawed sham study in English and Scandinavian versions to 180 Scandinavian reviewers. The reviewers considered the English version significantly better than the Scandinavian version (p < 0.05) .
Fifty-seven percent of editors were over 60 years of age with the remainder being 41-60 years of age. Seven percent of editors had retired. Data were not collected on editorial experience. On searching Medline there is no published study in the literature investigating the effect of editor's age on the quality of peer review and this is an area that would benefit from further study. A study of the natural history of peer reviewers has found that the majority of peer reviewers deteriorated over time at a gradual rate but it is unclear whether this can be extrapolated to editors .
Analysis of publishers showed that 23% of orthopaedic journals are published by Elsevier, 18% were independently published and 14% published by Lipincott. The effect of publishers on the editorial process is unknown and deserves further study as only 18% of orthopaedic journals are published independently. A previous survey of editorial independence at medical journals owned by professional associations showed that 70% of editors reported having complete or near complete editorial freedom, although many received modest pressure from their owners over editorial content. Forty-eight percent of journal's board of directors surveyed could hire and 55% could fire the editor indicating that editors may not be entirely independent from the influence of publishers .
Publishers can influence the speed of publication of a manuscript according to the system of review. The introduction of online systems for peer review is seen as advantageous by authors and editors rather than reviewers . Some publishers provide support to reviewers with online websites and surveys to improve knowledge and quality of the peer review process which may in turn improve the quality of publications .
Our study showed that 59% of journals used a review proforma, with 52% anonymising manuscripts and 76% using a statistician to routinely review all studies. Jefferson et al studied the effect of guidelines on peer review at the BMJ and Lancet and found there was no impact of guidelines on the quality of submissions but they did help editors manage submissions . Gardner et al investigated the statistical assessment of manuscripts submitted to the BMJ using a checklist and found the statistical quality of the papers improved during the peer review process . Seventy-six percent of surveyed orthopaedic editors ranked appropriate statistical analysis as very important when deciding to accept a paper yet only 76% had a statistician routinely review papers. Previous studies have shown that a routine statistical review can improve the quality of methodology of studies . The use of a statistical checklist may therefore be beneficial in improving the quality of papers during the review process in journals without a routine statistical review.
Blinding of reviewers and editors has been extensively investigated in the literature with conflicting results. Some studies have shown blinding improves the quality of reviews and provides more consistent results [28–30]. More recent studies have shown that blinding of reviewers makes no difference to the quality of reviews although there is a significant increase in reviewers refusing to give their opinions [31–33]. Furthermore the process of blinding is not infallible with reviewers able to guess the identity of authors in small highly specialised areas of study and those authors who self cite . Our study showed that 36% of editors could guess the author or institution in less than 10% of submitted anonymised manuscripts.
Our study found that in 59% of orthopaedic journals 2 referees were routinely used with 35% using 3 referees routinely. This is consistent with the review process of other biomedical journals with the majority using 2 reviewers and seeking the opinion of a third reviewer if the 2 reviewers contradicted each other or if a review was significantly delayed .
In 89% of orthopaedic journals surveyed the editor could over-rule the reviewer's final decision. Little is known about the editorial review process with previous surveys of editors of biomedical journals finding great diversity in editorial policy and procedures . Recent studies of the editorial decision process have shown conflicting results. A study in the Netherlands found that individual editor's decisions were far from consistent with individual editor's decisions complying poorly to team decisions, editor's ratings not predicting reviewers' ratings and editor's ratings poorly predicting future citation . A study from the US of editors at Obstetrics and Gynaecology was more encouraging with editors found to accurately assess the quality of manuscripts . Early editorial screening of manuscripts without peer review can however decrease the time to publication and reduces the workload of peer reviewers without an effect on the quality of accepted manuscripts .
When orthopaedic journal editors were surveyed, the most important factors for acceptance of a manuscript were: justified study conclusions, appropriate statistical analysis, study findings that could change practice, an understandable and well written manuscript and that the instructions to authors were correctly followed. Studies of editors of biomedical journals have shown that manuscripts are more likely to be rejected if the topic lacks originality, is not suited to the journal's readership, has weak methodological quality and lacks impact to change current clinical practice [2, 35]. Poor use of English is also cited as a reason for rejection . Our study found no evidence of studies with positive results being more likely to be accepted for publication over those with negative or non-significant results which is in agreement with the current literature . A study of editorial meetings at JAMA showed that clarity of a manuscript and response to referees' comments as well as journalism goals such as importance to medicine and strategic emphasis for the journal were also important in ensuring acceptance .