A survey of orthopaedic journal editors determining the criteria of manuscript selection for publication
© Hing et al; licensee BioMed Central Ltd. 2011
Received: 6 December 2010
Accepted: 28 April 2011
Published: 28 April 2011
To investigate the characteristics of editors and criteria used by orthopaedic journal editors in assessing submitted manuscripts.
Between 2008 to 2009 all 70 editors of Medline listed orthopaedic journals were approached prospectively with a questionnaire to determine the criteria used in assessing manuscripts for publication.
There was a 42% response rate. There was 1 female editor and the rest were male with 57% greater than 60 years of age. 67% of the editors worked in university teaching hospitals and 90% of publications were in English.
The review process differed between journals with 59% using a review proforma, 52% reviewing an anonymised manuscript, 76% using a routine statistical review and 59% of journals used 2 reviewers routinely. In 89% of the editors surveyed, the editor was able to overrule the final decision of the reviewers.
Important design factors considered for manuscript acceptance were that the study conclusions were justified (80%), that the statistical analysis was appropriate (76%), that the findings could change practice (72%). The level of evidence (70%) and type of study (62%) were deemed less important. When asked what factors were important in the manuscript influencing acceptance, 73% cited an understandable manuscript, 53% cited a well written manuscript and 50% a thorough literature review as very important factors.
The editorial and review process in orthopaedic journals uses different approaches. There may be a risk of language bias among editors of orthopaedic journals with under-representation of non-English publications in the orthopaedic literature.
Clinicians in orthopaedic practice are increasingly the subject of revalidation and are expected to be able to critically appraise new innovations with evidence based medicine the cornerstone of medical practice . The clinician relies on published literature and conferences to expand his knowledge base but there is little known on the criteria used by journal editors in selecting manuscripts for publication [2, 3]. All clinicians exhibit bias in some form with the ethical principal of 'equipoise' difficult to uphold during the review process [4, 5].
There is little published in the orthopaedic literature on the subject of publication bias. Okike et al investigated the influence of non-scientific factors on the acceptance for publication in the Journal of Bone and Joint Surgery (American Volume) and found that manuscripts were more likely to be accepted if they were from the United States or Canada, reported a conflict of interest related to a non-profit entity or were authored by an individual with 10 or more prior publications in frequently cited orthopaedic journals . Editors play a vital role in the peer review process but little evidence exists to support the quality of published research due to the use of editorial peer review [7–9].
We developed a questionnaire to establish which factors influence an editor's decision to accept a manuscript and the criteria used in each journal's review process.
A Medline search was used to identify all English and non-English language orthopaedic journals that were listed on Index Medicus in 2008. Seventy journals were identified and approached between December 2008 and March 2009 with a questionnaire in English via an email link or by post if no email address was listed for the principal editor [see additional file 1]. The questionnaire was designed in English as only 10% of journals were not published in English (Czech, French, Spanish, Polish, German and Italian). We anticipated that most of these journal editors would either speak English as a second language or have access to a translator. Non-respondents were approached again 2 months later then 6 months later. The editors were informed that the survey was confidential and that their responses would be anonymised.
The questionnaire design was based on a validated questionnaire designed by Radford et al to determine the criteria used by dental journals in assessing manuscripts . A set of additional questions were used to determine the demographics of the editors questioned and the journals they represented. The additional questions included questions on age, gender, qualifications, primary institution of work and country of residence. The questionnaire was designed to determine which factors in the study, the manuscript and which author characteristics influenced an editor's decision to accept or reject a manuscript. Questions were also asked regarding the review process and the journal characteristics. If an editor represented more than one journal he was sent a questionnaire specifically for each journal that he edited. Each journals' website was also examined to determine the demographics of editors and journals that had not responded. Journal impact factors for 2008 were also compiled from the Thomson Reuters Journal Citation report published in 2009.
Responses were collected on an online database http://www.surveymonkey.com (Portland, USA) and analysed using Excel spreadsheets (Microsoft, Seattle, USA) and SPSS version 11.5 (Chicago, Illinois). Statistical significance was set at p < 0.05. The impact factor and language of responders was compared to non-responders using a Students t test. A post hoc analysis of error level for completed surveys was also performed (survey sample random calculator, http://www.custominsight.com).
Thirty editors responded representing a 42% response rate (error level 11.4%, 90% confidence intervals). One American editor sent his response from France and one sent his response from a different state. One editor expressed an interest in feedback from the survey and one editor noted he did not like the questions. The mean impact factor for responding journal editors was 1.42 and for non-responders 1.53 which was not significantly different (p = 0.67).
Ninety percent of editors of orthopaedic journals were male, 1% female and 9% could not be determined. Eighty-three percent of editors spoke English as a first language. Fifty-seven percent of editors were over 60 years of age with the remainder being 41 - 60 years old. Sixty-seven percent worked in a University Hospital with only 3% working in a district hospital. Seven percent of editors had retired.
Analysis of the country of origin of publications showed 50% of journals were published in the US, 40% in the UK, 7% were from the East and 3% from Latin America. Analysis of publishers showed that 23% of orthopaedic journals are published by Elselvier, 18% were independent, 17% by Springer-Verlag, 14% by Lipincott, 6% by SLACK and 3% by Saunders, 3% SAGE and the remaining 16% by other smaller publishers. Thirty-seven percent of journals publish 12 issues per year and 37% publish 6 issues per year. The remainder varied from 4 to 10 issues per year.
Analysing the review process showed that 59% used a review proforma, 52% anonymised manuscripts before reviewing and 76% had a statistician routinely review all studies. In 89% of journals the editor could over-rule the reviewer's final decision with 78% of referees allowed to see each other's reports. In 59% of journals 2 referees routinely reviewed a manuscript with 35% of journals using 3 referees routinely. Of those journals that anonymised the manuscripts prior to peer review, 36% of editors could guess the authors or institutions in less than 10% of cases.
Ranked design factors that influenced an editor's decision to accept a study for publication.
Study design factors
Percentage score (very unimportant/slightly unimportant/indifferent/slightly important/very important)
The study conclusions are justified
80% very important
The statistical analysis is appropriate
76% very important
The study findings could change practice
72% very important
The level of evidence
70% very important
The study design (e.g. randomised controlled trial)
62% very important
The study is a 'hot topic
53% slightly important
The study complies with the journal's aim
48% very important
The study has a large sample size
45% slightly important
The study reinforces my beliefs
41% very unimportant
The study findings are unexpected
39% slightly important
The study findings are statistically significant
38% very important
Ranked manuscript factors that influenced an editor's decision to accet a study for publication.
Manuscript design factors
Percentage score (very unimportant/slightly unimportant/indifferent/slightly important/very important)
The manuscript is understandable
73% very important
The manuscript is well written
53% very important
The literature review is thorough
50% very important
The references include papers from my journal
There is no financial conflict of interest
37% slightly important
When asked to rate what factors regarding the authors most influenced an editor to accept a manuscript for publication, 48% of editors regarded an author who correctly followed the instructions to authors as a slightly important factor. They were indifferent to how distinguished the senior author was (41%) or whether the author was from a high quality institution (45%) and rated knowledge of the authors or authors' work as very unimportant (45%).
The 'gold standard' editorial process defined by the Cochrane Collaboration aims to produce studies that are appropriate to the publication medium, important, useful, original, methodologically sound, ethical and accurate . This is the first study to investigate journal editor demographics and the factors influencing orthopaedic journal editors during the manuscript review process. Evidence based medicine relies on the journal review process to provide an unbiased representation of high quality studies . Deleterious effects on health policy and clinical decisions can result from publication and related biases. A recent Health Technology Assessment found that studies with significant or positive results were more likely to be published and published earlier than those with non-significant or negative results . Exclusion of non-English language studies was also found to result in a high risk of bias in certain research areas such as complementary and alternative medicine .
The questionnaire used was based on a previously validated questionnaire designed by Radford et al . Additional questions and a review of journal websites and instructions to authors were used to provide additional information on journal demographics. The response rate was 42% with no significant difference in impact factor between responders and non-responders' journal. Whilst the error level was 11.4% with 90% confidence limits, this is the first study of its kind and represents an indication of journal editors' preferences for manuscript acceptance as well as their characteristics which has not previously been documented in the literature. Poor response rates are a potential source of bias since the confidence with which the survey findings can be accepted are a function of the achieved sample size .
A small sample size resulting from poor response rates exists if the responders differ from the non-responders in terms of key characteristics . A comparison of impact factor between responders and non-responders showed no significant difference with p = 0.67. However comparison of language of publication showed 96% of responders edited publications in English with only 76% of non-responders editing publications in English (p = 0.03). This may indicated that non-response was due to an inability to interpret the questionnaire which was in English. This is a weakness in the study design indicating that the results may be subject to language bias.
Analysis of the demographics of orthopaedic journal editors showed that 90% were male. Gender bias in terms of editors, reviewers and authors has been studied in the literature. Gilbert et al retrospectively reviewed 1851 research articles submitted to JAMA in 1991 and found no bias based on reviewers' or editors' gender on the decision to accept or reject a manuscript . A further study conducted by Caelleigh et al of 50 female and 50 male reviewers showed no bias when assessing an empirical study with two versions, one attributing a lower income in women to gender issues and the other attributing it to social learning factors [12, 15]. However, other studies have confirmed the presence of gender bias in the publication process [16, 17]. A Scandinavian study used a sham paper with either male or female authors to assess gender bias in a 1637 randomly selected Swedish physicians. Female authors were ranked higher than male authors. Male assessors were found to reflect no gender bias but female assessors upgraded female authors more than male authors . A study of grant proposals has also shown gender bias to the detriment of female applicants .
Eighty-three percent of editors spoke English as a first language with 50% of journals published in the US and 40% in the UK. Geographical bias has been shown to influence publication. A recent study of non-scientific factors influencing acceptance of manuscripts to the American Journal of Bone and Joint Surgery showed that manuscripts submitted from countries other than the US or Canada were significantly less likely to be accepted (odds ratio 0.51, 95% confidence interval, 0.28 to 0.92; p = 0.026) . A Scandinavian study sent a methodologically flawed sham study in English and Scandinavian versions to 180 Scandinavian reviewers. The reviewers considered the English version significantly better than the Scandinavian version (p < 0.05) .
Fifty-seven percent of editors were over 60 years of age with the remainder being 41-60 years of age. Seven percent of editors had retired. Data were not collected on editorial experience. On searching Medline there is no published study in the literature investigating the effect of editor's age on the quality of peer review and this is an area that would benefit from further study. A study of the natural history of peer reviewers has found that the majority of peer reviewers deteriorated over time at a gradual rate but it is unclear whether this can be extrapolated to editors .
Analysis of publishers showed that 23% of orthopaedic journals are published by Elsevier, 18% were independently published and 14% published by Lipincott. The effect of publishers on the editorial process is unknown and deserves further study as only 18% of orthopaedic journals are published independently. A previous survey of editorial independence at medical journals owned by professional associations showed that 70% of editors reported having complete or near complete editorial freedom, although many received modest pressure from their owners over editorial content. Forty-eight percent of journal's board of directors surveyed could hire and 55% could fire the editor indicating that editors may not be entirely independent from the influence of publishers .
Publishers can influence the speed of publication of a manuscript according to the system of review. The introduction of online systems for peer review is seen as advantageous by authors and editors rather than reviewers . Some publishers provide support to reviewers with online websites and surveys to improve knowledge and quality of the peer review process which may in turn improve the quality of publications .
Our study showed that 59% of journals used a review proforma, with 52% anonymising manuscripts and 76% using a statistician to routinely review all studies. Jefferson et al studied the effect of guidelines on peer review at the BMJ and Lancet and found there was no impact of guidelines on the quality of submissions but they did help editors manage submissions . Gardner et al investigated the statistical assessment of manuscripts submitted to the BMJ using a checklist and found the statistical quality of the papers improved during the peer review process . Seventy-six percent of surveyed orthopaedic editors ranked appropriate statistical analysis as very important when deciding to accept a paper yet only 76% had a statistician routinely review papers. Previous studies have shown that a routine statistical review can improve the quality of methodology of studies . The use of a statistical checklist may therefore be beneficial in improving the quality of papers during the review process in journals without a routine statistical review.
Blinding of reviewers and editors has been extensively investigated in the literature with conflicting results. Some studies have shown blinding improves the quality of reviews and provides more consistent results [28–30]. More recent studies have shown that blinding of reviewers makes no difference to the quality of reviews although there is a significant increase in reviewers refusing to give their opinions [31–33]. Furthermore the process of blinding is not infallible with reviewers able to guess the identity of authors in small highly specialised areas of study and those authors who self cite . Our study showed that 36% of editors could guess the author or institution in less than 10% of submitted anonymised manuscripts.
Our study found that in 59% of orthopaedic journals 2 referees were routinely used with 35% using 3 referees routinely. This is consistent with the review process of other biomedical journals with the majority using 2 reviewers and seeking the opinion of a third reviewer if the 2 reviewers contradicted each other or if a review was significantly delayed .
In 89% of orthopaedic journals surveyed the editor could over-rule the reviewer's final decision. Little is known about the editorial review process with previous surveys of editors of biomedical journals finding great diversity in editorial policy and procedures . Recent studies of the editorial decision process have shown conflicting results. A study in the Netherlands found that individual editor's decisions were far from consistent with individual editor's decisions complying poorly to team decisions, editor's ratings not predicting reviewers' ratings and editor's ratings poorly predicting future citation . A study from the US of editors at Obstetrics and Gynaecology was more encouraging with editors found to accurately assess the quality of manuscripts . Early editorial screening of manuscripts without peer review can however decrease the time to publication and reduces the workload of peer reviewers without an effect on the quality of accepted manuscripts .
When orthopaedic journal editors were surveyed, the most important factors for acceptance of a manuscript were: justified study conclusions, appropriate statistical analysis, study findings that could change practice, an understandable and well written manuscript and that the instructions to authors were correctly followed. Studies of editors of biomedical journals have shown that manuscripts are more likely to be rejected if the topic lacks originality, is not suited to the journal's readership, has weak methodological quality and lacks impact to change current clinical practice [2, 35]. Poor use of English is also cited as a reason for rejection . Our study found no evidence of studies with positive results being more likely to be accepted for publication over those with negative or non-significant results which is in agreement with the current literature . A study of editorial meetings at JAMA showed that clarity of a manuscript and response to referees' comments as well as journalism goals such as importance to medicine and strategic emphasis for the journal were also important in ensuring acceptance .
This study represents the first to survey orthopaedic journal editors for their views on manuscript acceptance and the demographics of editors. Whilst the response rate was 42%, there was no significant difference in impact factor between responders and non-responders. Reasons cited by editors for manuscript acceptance are similar to those of other biomedical journals. Editor demographics indicate that most are male, over 40 years of age and speak English. There is an over-representation of UK and US based publications in the literature which increases the risk of language bias in the orthopaedic literature since poor English is cited as a reason to reject a manuscript and deserves further study.
- Poolman RW, Kerkhoffs GM, Struijis PAA, Bhandari M: Don't be misled by the orthopaedic literature. Tips for critical appraisal. Acta Orthopaedica. 2007, 78 (2): 162-171. 10.1080/17453670710013636.View ArticlePubMedGoogle Scholar
- Radford DR, Smillie L, Wilson RF, Grace AM: The criteria used by editors of scientific dental journals in the assessment of manuscripts submitted for publication. British Dental Journal. 1999, 187 (7): 376-379.PubMedGoogle Scholar
- Senturia SD: How to avoid the reviewer's axe: one editor's view. IEEE transactions on ultrasonics, ferroelectrics, and frequency control. 2004, 51 (1): 127-130.PubMedGoogle Scholar
- Fries JF, Krishnan E: Equipoise, design bias, and randomised controlled trials: the elusive ethics of new drug development. Arthritis Res & Ther. 2004, 6: R250-255. 10.1186/ar1170.View ArticleGoogle Scholar
- Cain DM, Detsky AS: Everyone's a little bit biased (even physicians). JAMA. 2008, 299 (24): 2893-2895. 10.1001/jama.299.24.2893.View ArticlePubMedGoogle Scholar
- Okike K, Kocher MS, Mehlmn CT, Heckman JD, Bhandari M: Nonscientific factors associated with acceptance for publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Jt Surg. 2008, 90: 2432-7. 10.2106/JBJS.G.01687.View ArticleGoogle Scholar
- Jefferson T, Alderson P, Wager E, Davidoff F: Effects of editorial peer review. A systematic review. JAMA. 2002, 287 (21): 2784-2786. 10.1001/jama.287.21.2784.View ArticlePubMedGoogle Scholar
- Purcell GP, Donovan SL, Davidoff F: Changes to manuscripts during the editorial process. JAMA. 1998, 280 (3): 227-228. 10.1001/jama.280.3.227.View ArticlePubMedGoogle Scholar
- Jefferson T, Rudin M, Brodney Folse S, Davidoff F: Little evidence to support the use of editorial peer review to ensure quality of published research. Evidence Based Dentistry. 2007, 8 (3): 88-89. 10.1038/sj.ebd.6400516.View ArticleGoogle Scholar
- Jefferson T, Rudin M, brodney Folse S, Davidoff F: Editorial peer review for improving the qulity of reports of biomedical studies (Review). The Cochrane Library. 2008, 4: 5-Google Scholar
- Begg CB, Berlin JA: Publication bias; a problem in interpreting medical data. J Roy Stat Soc A. 1988, 151: 445-63.Google Scholar
- Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing CB, Kwok CS, Pang C, Harvey I: Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment. 2010, 14 (8): iii-View ArticlePubMedGoogle Scholar
- McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, Thomas R, Harvey E, Garratt A, Bond J: Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment. 2001, 5 (31): 101-View ArticleGoogle Scholar
- Gilbert JR, Williams ES, Lundberg GD: Is there gender bias in JAMA's peer review process?. Journal of the American Medical Association. 1994, 272: 139-142. 10.1001/jama.272.2.139.View ArticlePubMedGoogle Scholar
- Caelleigh AS, Hojat M, Steinecke A, Gonnella JS: Effects of reviewers' gender on assessments of a gender-related standardized manuscript. 4th International Congress on Peer Review in Biomedical Publication, 14-16th. 2001, September , Barcelona, SpainGoogle Scholar
- Eagerly AH: The science and politics of comparing women and men. American Psychologist. 1995, 50: 145-158.View ArticleGoogle Scholar
- Riger S: Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist. 1992, 47: 730-740.View ArticleGoogle Scholar
- Johansson EE, Risberg G, Hamberg K, Westman G: Gender bias in female physician assessments. Women considered better suited for qualitative research. Scand J Prim Health Care. 2002, 20: 79-84. 10.1080/02813430215553.View ArticlePubMedGoogle Scholar
- Wenneras C, Wold A: Nepotism and sexism in peer-review. Nature. 1997, 387: 341-343. 10.1038/387341a0.View ArticlePubMedGoogle Scholar
- Nylenna M, Riis P, Karlsson Y: Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. JAMA. 1994, 272: 149-51. 10.1001/jama.272.2.149.View ArticlePubMedGoogle Scholar
- Callahan M: The natural history of peer reviewers: the decay of quality. The 6th International Conference on Peer Review and Biomedical Publication, September 10-12, Vancouver, Canada.
- Davis RM, Mullner M: Editorial independence at medical journals owned by professional associations: a survey of editors. Sci Eng Ethics. 2002, 8: 513-28. 10.1007/s11948-002-0004-7.View ArticlePubMedGoogle Scholar
- Mulligan A: Is peer review in crisis?. Oral Oncology. 2005, 41: 135-141. 10.1016/j.oraloncology.2004.11.001.View ArticlePubMedGoogle Scholar
- Mulligan A: Peer review survey 2009: preliminary findings.Senseaboutscience.org.uk.
- Jefferson T, Smith R, Yee Y, Drummond M, Prat M, Gale R: Evaluating the BMJ guidelines for economic submissions: prospective audit of economic submissions to BMJ and The Lancet. JAMA. 1998, 280 (3): 275-7. 10.1001/jama.280.3.275.View ArticlePubMedGoogle Scholar
- Gardner MJ, Bond J: An exploratory study of statistical assessment of papers published in the British Medical Journal. JAMA. 1990, 263 (10): 1355-7. 10.1001/jama.263.10.1355.View ArticlePubMedGoogle Scholar
- Day FC, Schriger DL, Todd C, Wears RL: The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers. Annals of Emergency medicine. 2002, 40 (3): 329-33. 10.1067/mem.2002.127326.View ArticlePubMedGoogle Scholar
- McNutt RA, Evans AT, Fletcher RH, Fletcher SW: The effects of blinding on the quality of peer review. JAMA. 1990, 263 (10): 1371-6. 10.1001/jama.263.10.1371.View ArticlePubMedGoogle Scholar
- Fisher M, Freidman SB, Strauss B: The effects of blinding on acceptance of research papers by peer review. JAMA. 1994, 272: 143-6. 10.1001/jama.272.2.143.View ArticlePubMedGoogle Scholar
- Jadad AR, Moore A, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomised clinical trials: is blinding necessary?. Controlled Clinical Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.View ArticlePubMedGoogle Scholar
- Van Rooyen S, Godlee F, Evans S, Black N, Smith R: Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ. 1999, 318: 23-7.PubMed CentralView ArticlePubMedGoogle Scholar
- Godlee F, Gale CR, Martyn CN: Effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports. JAMA. 1998, 280 (3): 237-40. 10.1001/jama.280.3.237.View ArticlePubMedGoogle Scholar
- Walsh E, Rooney M, Appleby L, Wilkinson G: Open peer review: a randomised controlled trial. British Journal of Psychiatry. 2000, 176: 47-51. 10.1192/bjp.176.1.47.View ArticlePubMedGoogle Scholar
- Justice AC, Cho MK, Winker MA: Does masking author identity improve peer review quality. JAMA. 1998, 280: 240-2. 10.1001/jama.280.3.240.View ArticlePubMedGoogle Scholar
- Lee K, Boyd E, Bero L: A look inside the black box: a description of the editorial process at the three leading biomedical journals [abstract]. 12th Cochrane Colloquium, 2-6. 2004, October ; Ottawa, CanadaGoogle Scholar
- Opthof T, Coronel R: Consistency of decision making by editors: the relation between reviewers' and editors' ratings and future (10 years) citation. The 6th International Conference on Peer Review and Biomedical Publication, September 10-12, Vancouver, Canada.
- Logothetti H, Martin S, Benner R, Scott J, Queenan J, Spong C: Evaluation of editors' judgement on quality of articles. The 6th International Conference on Peer Review and Biomedical Publication, September 10-12, Vancouver, Canada.
- Claiborne Johnston S, Lowenstein DH, Ferriero DM, Messing RO, Oksenberg JR, Hauser SL: Early editorial manuscript screening versus obligate peer review: a randomised trial. Ann Neurol. 2007, 61: A10-A12.PubMedGoogle Scholar
- Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. JAMA. 2002, 287 (21): 2825-10.1001/jama.287.21.2825.View ArticlePubMedGoogle Scholar
- Dickersin K, Ssemanda E, Mansell C, Rennie D: What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Medical Methodology. 2007, 7: 44-10.1186/1471-2288-7-44.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.