A survey of orthopaedic journal editors determining the criteria of manuscript selection for publication
Journal of Orthopaedic Surgery and Research volume 6, Article number: 19 (2011)
To investigate the characteristics of editors and criteria used by orthopaedic journal editors in assessing submitted manuscripts.
Between 2008 to 2009 all 70 editors of Medline listed orthopaedic journals were approached prospectively with a questionnaire to determine the criteria used in assessing manuscripts for publication.
There was a 42% response rate. There was 1 female editor and the rest were male with 57% greater than 60 years of age. 67% of the editors worked in university teaching hospitals and 90% of publications were in English.
The review process differed between journals with 59% using a review proforma, 52% reviewing an anonymised manuscript, 76% using a routine statistical review and 59% of journals used 2 reviewers routinely. In 89% of the editors surveyed, the editor was able to overrule the final decision of the reviewers.
Important design factors considered for manuscript acceptance were that the study conclusions were justified (80%), that the statistical analysis was appropriate (76%), that the findings could change practice (72%). The level of evidence (70%) and type of study (62%) were deemed less important. When asked what factors were important in the manuscript influencing acceptance, 73% cited an understandable manuscript, 53% cited a well written manuscript and 50% a thorough literature review as very important factors.
The editorial and review process in orthopaedic journals uses different approaches. There may be a risk of language bias among editors of orthopaedic journals with under-representation of non-English publications in the orthopaedic literature.
Clinicians in orthopaedic practice are increasingly the subject of revalidation and are expected to be able to critically appraise new innovations with evidence based medicine the cornerstone of medical practice . The clinician relies on published literature and conferences to expand his knowledge base but there is little known on the criteria used by journal editors in selecting manuscripts for publication [2, 3]. All clinicians exhibit bias in some form with the ethical principal of 'equipoise' difficult to uphold during the review process [4, 5].
There is little published in the orthopaedic literature on the subject of publication bias. Okike et al investigated the influence of non-scientific factors on the acceptance for publication in the Journal of Bone and Joint Surgery (American Volume) and found that manuscripts were more likely to be accepted if they were from the United States or Canada, reported a conflict of interest related to a non-profit entity or were authored by an individual with 10 or more prior publications in frequently cited orthopaedic journals . Editors play a vital role in the peer review process but little evidence exists to support the quality of published research due to the use of editorial peer review [7–9].
We developed a questionnaire to establish which factors influence an editor's decision to accept a manuscript and the criteria used in each journal's review process.
A Medline search was used to identify all English and non-English language orthopaedic journals that were listed on Index Medicus in 2008. Seventy journals were identified and approached between December 2008 and March 2009 with a questionnaire in English via an email link or by post if no email address was listed for the principal editor [see additional file 1]. The questionnaire was designed in English as only 10% of journals were not published in English (Czech, French, Spanish, Polish, German and Italian). We anticipated that most of these journal editors would either speak English as a second language or have access to a translator. Non-respondents were approached again 2 months later then 6 months later. The editors were informed that the survey was confidential and that their responses would be anonymised.
The questionnaire design was based on a validated questionnaire designed by Radford et al to determine the criteria used by dental journals in assessing manuscripts . A set of additional questions were used to determine the demographics of the editors questioned and the journals they represented. The additional questions included questions on age, gender, qualifications, primary institution of work and country of residence. The questionnaire was designed to determine which factors in the study, the manuscript and which author characteristics influenced an editor's decision to accept or reject a manuscript. Questions were also asked regarding the review process and the journal characteristics. If an editor represented more than one journal he was sent a questionnaire specifically for each journal that he edited. Each journals' website was also examined to determine the demographics of editors and journals that had not responded. Journal impact factors for 2008 were also compiled from the Thomson Reuters Journal Citation report published in 2009.
Responses were collected on an online database http://www.surveymonkey.com (Portland, USA) and analysed using Excel spreadsheets (Microsoft, Seattle, USA) and SPSS version 11.5 (Chicago, Illinois). Statistical significance was set at p < 0.05. The impact factor and language of responders was compared to non-responders using a Students t test. A post hoc analysis of error level for completed surveys was also performed (survey sample random calculator, http://www.custominsight.com).
Thirty editors responded representing a 42% response rate (error level 11.4%, 90% confidence intervals). One American editor sent his response from France and one sent his response from a different state. One editor expressed an interest in feedback from the survey and one editor noted he did not like the questions. The mean impact factor for responding journal editors was 1.42 and for non-responders 1.53 which was not significantly different (p = 0.67).
Ninety percent of editors of orthopaedic journals were male, 1% female and 9% could not be determined. Eighty-three percent of editors spoke English as a first language. Fifty-seven percent of editors were over 60 years of age with the remainder being 41 - 60 years old. Sixty-seven percent worked in a University Hospital with only 3% working in a district hospital. Seven percent of editors had retired.
Analysis of the country of origin of publications showed 50% of journals were published in the US, 40% in the UK, 7% were from the East and 3% from Latin America. Analysis of publishers showed that 23% of orthopaedic journals are published by Elselvier, 18% were independent, 17% by Springer-Verlag, 14% by Lipincott, 6% by SLACK and 3% by Saunders, 3% SAGE and the remaining 16% by other smaller publishers. Thirty-seven percent of journals publish 12 issues per year and 37% publish 6 issues per year. The remainder varied from 4 to 10 issues per year.
Analysing the review process showed that 59% used a review proforma, 52% anonymised manuscripts before reviewing and 76% had a statistician routinely review all studies. In 89% of journals the editor could over-rule the reviewer's final decision with 78% of referees allowed to see each other's reports. In 59% of journals 2 referees routinely reviewed a manuscript with 35% of journals using 3 referees routinely. Of those journals that anonymised the manuscripts prior to peer review, 36% of editors could guess the authors or institutions in less than 10% of cases.
The editors' opinion regarding the most important factors in a submitted study likely to influence a decision to accept a manuscript are summarised in table 1. The most important factor likely to influence acceptance was that the study conclusions were justified (80%). An appropriate statistical analysis (76%), study findings that could change practice (72%) and the level of evidence of a study (70%) were also deemed important factors in the study design. The editors surveyed felt that the most important factors in the manuscript that influenced acceptance were that the manuscript was understandable (73%), well written (53%) and that the literature review was thorough (50%), table 2.
When asked to rate what factors regarding the authors most influenced an editor to accept a manuscript for publication, 48% of editors regarded an author who correctly followed the instructions to authors as a slightly important factor. They were indifferent to how distinguished the senior author was (41%) or whether the author was from a high quality institution (45%) and rated knowledge of the authors or authors' work as very unimportant (45%).
The 'gold standard' editorial process defined by the Cochrane Collaboration aims to produce studies that are appropriate to the publication medium, important, useful, original, methodologically sound, ethical and accurate . This is the first study to investigate journal editor demographics and the factors influencing orthopaedic journal editors during the manuscript review process. Evidence based medicine relies on the journal review process to provide an unbiased representation of high quality studies . Deleterious effects on health policy and clinical decisions can result from publication and related biases. A recent Health Technology Assessment found that studies with significant or positive results were more likely to be published and published earlier than those with non-significant or negative results . Exclusion of non-English language studies was also found to result in a high risk of bias in certain research areas such as complementary and alternative medicine .
The questionnaire used was based on a previously validated questionnaire designed by Radford et al . Additional questions and a review of journal websites and instructions to authors were used to provide additional information on journal demographics. The response rate was 42% with no significant difference in impact factor between responders and non-responders' journal. Whilst the error level was 11.4% with 90% confidence limits, this is the first study of its kind and represents an indication of journal editors' preferences for manuscript acceptance as well as their characteristics which has not previously been documented in the literature. Poor response rates are a potential source of bias since the confidence with which the survey findings can be accepted are a function of the achieved sample size .
A small sample size resulting from poor response rates exists if the responders differ from the non-responders in terms of key characteristics . A comparison of impact factor between responders and non-responders showed no significant difference with p = 0.67. However comparison of language of publication showed 96% of responders edited publications in English with only 76% of non-responders editing publications in English (p = 0.03). This may indicated that non-response was due to an inability to interpret the questionnaire which was in English. This is a weakness in the study design indicating that the results may be subject to language bias.
Analysis of the demographics of orthopaedic journal editors showed that 90% were male. Gender bias in terms of editors, reviewers and authors has been studied in the literature. Gilbert et al retrospectively reviewed 1851 research articles submitted to JAMA in 1991 and found no bias based on reviewers' or editors' gender on the decision to accept or reject a manuscript . A further study conducted by Caelleigh et al of 50 female and 50 male reviewers showed no bias when assessing an empirical study with two versions, one attributing a lower income in women to gender issues and the other attributing it to social learning factors [12, 15]. However, other studies have confirmed the presence of gender bias in the publication process [16, 17]. A Scandinavian study used a sham paper with either male or female authors to assess gender bias in a 1637 randomly selected Swedish physicians. Female authors were ranked higher than male authors. Male assessors were found to reflect no gender bias but female assessors upgraded female authors more than male authors . A study of grant proposals has also shown gender bias to the detriment of female applicants .
Eighty-three percent of editors spoke English as a first language with 50% of journals published in the US and 40% in the UK. Geographical bias has been shown to influence publication. A recent study of non-scientific factors influencing acceptance of manuscripts to the American Journal of Bone and Joint Surgery showed that manuscripts submitted from countries other than the US or Canada were significantly less likely to be accepted (odds ratio 0.51, 95% confidence interval, 0.28 to 0.92; p = 0.026) . A Scandinavian study sent a methodologically flawed sham study in English and Scandinavian versions to 180 Scandinavian reviewers. The reviewers considered the English version significantly better than the Scandinavian version (p < 0.05) .
Fifty-seven percent of editors were over 60 years of age with the remainder being 41-60 years of age. Seven percent of editors had retired. Data were not collected on editorial experience. On searching Medline there is no published study in the literature investigating the effect of editor's age on the quality of peer review and this is an area that would benefit from further study. A study of the natural history of peer reviewers has found that the majority of peer reviewers deteriorated over time at a gradual rate but it is unclear whether this can be extrapolated to editors .
Analysis of publishers showed that 23% of orthopaedic journals are published by Elsevier, 18% were independently published and 14% published by Lipincott. The effect of publishers on the editorial process is unknown and deserves further study as only 18% of orthopaedic journals are published independently. A previous survey of editorial independence at medical journals owned by professional associations showed that 70% of editors reported having complete or near complete editorial freedom, although many received modest pressure from their owners over editorial content. Forty-eight percent of journal's board of directors surveyed could hire and 55% could fire the editor indicating that editors may not be entirely independent from the influence of publishers .
Publishers can influence the speed of publication of a manuscript according to the system of review. The introduction of online systems for peer review is seen as advantageous by authors and editors rather than reviewers . Some publishers provide support to reviewers with online websites and surveys to improve knowledge and quality of the peer review process which may in turn improve the quality of publications .
Our study showed that 59% of journals used a review proforma, with 52% anonymising manuscripts and 76% using a statistician to routinely review all studies. Jefferson et al studied the effect of guidelines on peer review at the BMJ and Lancet and found there was no impact of guidelines on the quality of submissions but they did help editors manage submissions . Gardner et al investigated the statistical assessment of manuscripts submitted to the BMJ using a checklist and found the statistical quality of the papers improved during the peer review process . Seventy-six percent of surveyed orthopaedic editors ranked appropriate statistical analysis as very important when deciding to accept a paper yet only 76% had a statistician routinely review papers. Previous studies have shown that a routine statistical review can improve the quality of methodology of studies . The use of a statistical checklist may therefore be beneficial in improving the quality of papers during the review process in journals without a routine statistical review.
Blinding of reviewers and editors has been extensively investigated in the literature with conflicting results. Some studies have shown blinding improves the quality of reviews and provides more consistent results [28–30]. More recent studies have shown that blinding of reviewers makes no difference to the quality of reviews although there is a significant increase in reviewers refusing to give their opinions [31–33]. Furthermore the process of blinding is not infallible with reviewers able to guess the identity of authors in small highly specialised areas of study and those authors who self cite . Our study showed that 36% of editors could guess the author or institution in less than 10% of submitted anonymised manuscripts.
Our study found that in 59% of orthopaedic journals 2 referees were routinely used with 35% using 3 referees routinely. This is consistent with the review process of other biomedical journals with the majority using 2 reviewers and seeking the opinion of a third reviewer if the 2 reviewers contradicted each other or if a review was significantly delayed .
In 89% of orthopaedic journals surveyed the editor could over-rule the reviewer's final decision. Little is known about the editorial review process with previous surveys of editors of biomedical journals finding great diversity in editorial policy and procedures . Recent studies of the editorial decision process have shown conflicting results. A study in the Netherlands found that individual editor's decisions were far from consistent with individual editor's decisions complying poorly to team decisions, editor's ratings not predicting reviewers' ratings and editor's ratings poorly predicting future citation . A study from the US of editors at Obstetrics and Gynaecology was more encouraging with editors found to accurately assess the quality of manuscripts . Early editorial screening of manuscripts without peer review can however decrease the time to publication and reduces the workload of peer reviewers without an effect on the quality of accepted manuscripts .
When orthopaedic journal editors were surveyed, the most important factors for acceptance of a manuscript were: justified study conclusions, appropriate statistical analysis, study findings that could change practice, an understandable and well written manuscript and that the instructions to authors were correctly followed. Studies of editors of biomedical journals have shown that manuscripts are more likely to be rejected if the topic lacks originality, is not suited to the journal's readership, has weak methodological quality and lacks impact to change current clinical practice [2, 35]. Poor use of English is also cited as a reason for rejection . Our study found no evidence of studies with positive results being more likely to be accepted for publication over those with negative or non-significant results which is in agreement with the current literature . A study of editorial meetings at JAMA showed that clarity of a manuscript and response to referees' comments as well as journalism goals such as importance to medicine and strategic emphasis for the journal were also important in ensuring acceptance .
This study represents the first to survey orthopaedic journal editors for their views on manuscript acceptance and the demographics of editors. Whilst the response rate was 42%, there was no significant difference in impact factor between responders and non-responders. Reasons cited by editors for manuscript acceptance are similar to those of other biomedical journals. Editor demographics indicate that most are male, over 40 years of age and speak English. There is an over-representation of UK and US based publications in the literature which increases the risk of language bias in the orthopaedic literature since poor English is cited as a reason to reject a manuscript and deserves further study.
Poolman RW, Kerkhoffs GM, Struijis PAA, Bhandari M: Don't be misled by the orthopaedic literature. Tips for critical appraisal. Acta Orthopaedica. 2007, 78 (2): 162-171. 10.1080/17453670710013636.
Radford DR, Smillie L, Wilson RF, Grace AM: The criteria used by editors of scientific dental journals in the assessment of manuscripts submitted for publication. British Dental Journal. 1999, 187 (7): 376-379.
Senturia SD: How to avoid the reviewer's axe: one editor's view. IEEE transactions on ultrasonics, ferroelectrics, and frequency control. 2004, 51 (1): 127-130.
Fries JF, Krishnan E: Equipoise, design bias, and randomised controlled trials: the elusive ethics of new drug development. Arthritis Res & Ther. 2004, 6: R250-255. 10.1186/ar1170.
Cain DM, Detsky AS: Everyone's a little bit biased (even physicians). JAMA. 2008, 299 (24): 2893-2895. 10.1001/jama.299.24.2893.
Okike K, Kocher MS, Mehlmn CT, Heckman JD, Bhandari M: Nonscientific factors associated with acceptance for publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Jt Surg. 2008, 90: 2432-7. 10.2106/JBJS.G.01687.
Jefferson T, Alderson P, Wager E, Davidoff F: Effects of editorial peer review. A systematic review. JAMA. 2002, 287 (21): 2784-2786. 10.1001/jama.287.21.2784.
Purcell GP, Donovan SL, Davidoff F: Changes to manuscripts during the editorial process. JAMA. 1998, 280 (3): 227-228. 10.1001/jama.280.3.227.
Jefferson T, Rudin M, Brodney Folse S, Davidoff F: Little evidence to support the use of editorial peer review to ensure quality of published research. Evidence Based Dentistry. 2007, 8 (3): 88-89. 10.1038/sj.ebd.6400516.
Jefferson T, Rudin M, brodney Folse S, Davidoff F: Editorial peer review for improving the qulity of reports of biomedical studies (Review). The Cochrane Library. 2008, 4: 5-
Begg CB, Berlin JA: Publication bias; a problem in interpreting medical data. J Roy Stat Soc A. 1988, 151: 445-63.
Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing CB, Kwok CS, Pang C, Harvey I: Dissemination and publication of research findings: an updated review of related biases. Health Technology Assessment. 2010, 14 (8): iii-
McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, Thomas R, Harvey E, Garratt A, Bond J: Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment. 2001, 5 (31): 101-
Gilbert JR, Williams ES, Lundberg GD: Is there gender bias in JAMA's peer review process?. Journal of the American Medical Association. 1994, 272: 139-142. 10.1001/jama.272.2.139.
Caelleigh AS, Hojat M, Steinecke A, Gonnella JS: Effects of reviewers' gender on assessments of a gender-related standardized manuscript. 4th International Congress on Peer Review in Biomedical Publication, 14-16th. 2001, September , Barcelona, Spain
Eagerly AH: The science and politics of comparing women and men. American Psychologist. 1995, 50: 145-158.
Riger S: Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist. 1992, 47: 730-740.
Johansson EE, Risberg G, Hamberg K, Westman G: Gender bias in female physician assessments. Women considered better suited for qualitative research. Scand J Prim Health Care. 2002, 20: 79-84. 10.1080/02813430215553.
Wenneras C, Wold A: Nepotism and sexism in peer-review. Nature. 1997, 387: 341-343. 10.1038/387341a0.
Nylenna M, Riis P, Karlsson Y: Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. JAMA. 1994, 272: 149-51. 10.1001/jama.272.2.149.
Callahan M: The natural history of peer reviewers: the decay of quality. The 6th International Conference on Peer Review and Biomedical Publication, September 10-12, Vancouver, Canada.
Davis RM, Mullner M: Editorial independence at medical journals owned by professional associations: a survey of editors. Sci Eng Ethics. 2002, 8: 513-28. 10.1007/s11948-002-0004-7.
Mulligan A: Is peer review in crisis?. Oral Oncology. 2005, 41: 135-141. 10.1016/j.oraloncology.2004.11.001.
Mulligan A: Peer review survey 2009: preliminary findings.Senseaboutscience.org.uk.
Jefferson T, Smith R, Yee Y, Drummond M, Prat M, Gale R: Evaluating the BMJ guidelines for economic submissions: prospective audit of economic submissions to BMJ and The Lancet. JAMA. 1998, 280 (3): 275-7. 10.1001/jama.280.3.275.
Gardner MJ, Bond J: An exploratory study of statistical assessment of papers published in the British Medical Journal. JAMA. 1990, 263 (10): 1355-7. 10.1001/jama.263.10.1355.
Day FC, Schriger DL, Todd C, Wears RL: The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers. Annals of Emergency medicine. 2002, 40 (3): 329-33. 10.1067/mem.2002.127326.
McNutt RA, Evans AT, Fletcher RH, Fletcher SW: The effects of blinding on the quality of peer review. JAMA. 1990, 263 (10): 1371-6. 10.1001/jama.263.10.1371.
Fisher M, Freidman SB, Strauss B: The effects of blinding on acceptance of research papers by peer review. JAMA. 1994, 272: 143-6. 10.1001/jama.272.2.143.
Jadad AR, Moore A, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomised clinical trials: is blinding necessary?. Controlled Clinical Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.
Van Rooyen S, Godlee F, Evans S, Black N, Smith R: Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ. 1999, 318: 23-7.
Godlee F, Gale CR, Martyn CN: Effect on the quality of peer review of blinding peer reviewers and asking them to sign their reports. JAMA. 1998, 280 (3): 237-40. 10.1001/jama.280.3.237.
Walsh E, Rooney M, Appleby L, Wilkinson G: Open peer review: a randomised controlled trial. British Journal of Psychiatry. 2000, 176: 47-51. 10.1192/bjp.176.1.47.
Justice AC, Cho MK, Winker MA: Does masking author identity improve peer review quality. JAMA. 1998, 280: 240-2. 10.1001/jama.280.3.240.
Lee K, Boyd E, Bero L: A look inside the black box: a description of the editorial process at the three leading biomedical journals [abstract]. 12th Cochrane Colloquium, 2-6. 2004, October ; Ottawa, Canada
Opthof T, Coronel R: Consistency of decision making by editors: the relation between reviewers' and editors' ratings and future (10 years) citation. The 6th International Conference on Peer Review and Biomedical Publication, September 10-12, Vancouver, Canada.
Logothetti H, Martin S, Benner R, Scott J, Queenan J, Spong C: Evaluation of editors' judgement on quality of articles. The 6th International Conference on Peer Review and Biomedical Publication, September 10-12, Vancouver, Canada.
Claiborne Johnston S, Lowenstein DH, Ferriero DM, Messing RO, Oksenberg JR, Hauser SL: Early editorial manuscript screening versus obligate peer review: a randomised trial. Ann Neurol. 2007, 61: A10-A12.
Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. JAMA. 2002, 287 (21): 2825-10.1001/jama.287.21.2825.
Dickersin K, Ssemanda E, Mansell C, Rennie D: What do the JAMA editors say when they discuss manuscripts that they are considering for publication? Developing a schema for classifying the content of editorial discussion. BMC Medical Methodology. 2007, 7: 44-10.1186/1471-2288-7-44.
The authors declare that they have no competing interests.
LH, FS and STD contributed to the study design. DH and CBH designed the questionnaire. CBH analysed the data and wrote the manuscript. All authors contributed to the manuscript.
Deborah Higgs, Lee Hooper, Simon T Donell and Fujian Song contributed equally to this work.
Electronic supplementary material
About this article
Cite this article
Hing, C.B., Higgs, D., Hooper, L. et al. A survey of orthopaedic journal editors determining the criteria of manuscript selection for publication. J Orthop Surg Res 6, 19 (2011). https://doi.org/10.1186/1749-799X-6-19