Skip to main content

Accuracy of institutional orthopedic trauma databases: a retrospective chart review

Abstract

Introduction

Academic trauma institutions rely on fracture databases as research and quality control tools. Frequently, these databases are populated by trainees, but the completeness and accuracy of such databases has not yet been evaluated. The purpose of this study is to determine the capture rate of a resident-populated database in collecting extremity fractures and to determine the accuracy of assigned Orthopaedic Trauma Association (OTA) classifications.

Materials and methods

A retrospective study was performed at a level 1 trauma center of all adult patients who underwent treatment for extremity fractures after an emergency department or inpatient consultation. A 20% random sample was taken from these entries and compared to a resident-populated fracture database designed to capture the same patients. For all matching records containing a resident-assigned OTA classification, relevant imaging was blindly reviewed by a trauma fellowship-trained orthopedic attending surgeon for fracture pattern classification. Resident OTA classifications were compared to this gold standard to determine overall accuracy rate.

Results

Three hundred eighteen (80%) out of 400 entries were captured by the resident-populated database. Two hundred thirty-one of these 318 entries contained an OTA classification. One hundred fifty-three (66%) of these 231 entries demonstrated concordance between resident and attending assigned OTA classifications. On subgroup analysis, 133 (70%) of the 190 lower extremity classifications were accurately identified as compared to just 20 (49%) of the 41 upper extremity classifications (p = 0.009). Seventy-nine (65%) of the 121 end segment fractures showed agreement versus 42 (67%) of the 63 diaphyseal injury patterns (p = 0.85). Accuracy of classification did not significantly vary by resident year of training (p = 0.142).

Conclusion

Trainee generated databases at academic institutions may be subject to incomplete data entry and inaccurate fracture classifications. Quality control measures should be instituted to ensure accuracy in such databases if efforts are invested with the expectation of useful information.

Introduction

The development and maintenance of trauma registries has been essential in the advancement of high-quality trauma care delivery. These extensive acute care databases capture detailed information on epidemiology, mechanism of injury, and trauma care outcomes [1,2,3,4]. Trauma systems have relied on trauma registries for over 30 years and such databases have proven to be extremely valuable for research purposes and quality control [5].

The effectiveness of trauma registries to improve clinical research and patient outcomes depends on the quality of data recorded. If the data captured in the registry is incomplete or inaccurate, comparing trauma care systems will be ineffective and utilizing trauma care outcome data will not be beneficial [6]. In addition to poor data quality, high costs associated with managing a trauma database is also an important consideration. In 2015, the cost associated with managing single hospital trauma registries globally was valued between $10,000 and $1 million (USD) [7]. Trauma registry development and maintenance requires substantial financial costs and well-trained staff to gather and analyze data. The inadequate number of staff members with vast knowledge in trauma epidemiology and outcome research further limits the utility of such a database [8].

Utilizing orthopedic surgery resident physicians as a means of capturing accurate and complete data on trauma patients and outcomes can be a cost-effective way to build a valuable institutional trauma registry for research purposes. The completeness and accuracy of such trainee-populated databases at academic trauma institutions has not yet been evaluated. The purpose of this study is to determine the completeness of a resident-populated trauma database in collecting information on all upper and lower extremity fractures treated at a single institution and to determine the accuracy of resident-assigned Orthopedic Trauma Association (OTA) classifications.

Materials and methods

This retrospective cohort study was conducted at a single university-affiliated public level 1 trauma center and was approved by the institutional review board. Using appropriate Current Procedural Terminology (CPT) codes, we queried our institution’s main orthopedic billing database for adult patients greater than 18 years of age who sustained an upper extremity or lower extremity fracture and received definitive treatment at our hospital between April 2012 and February 2017. A 20% random sample was obtained from these entries, and medical chart review of inpatient documentation and radiographs was undertaken to confirm eligibility. All nonunions, peri-prosthetic fractures, and dislocations were excluded from analysis. Included entries were compared to a resident-populated orthopedic trauma database designed to capture the same patient population and record all trauma injuries presenting to our institution. Residents were tasked with capturing incoming patients with upper or lower extremity fractures and classifying injuries based on the Orthopaedic Trauma Association (OTA) system for bone, segment, and fracture type. The residents participating in the study were either on their orthopedic trauma rotation or taking trauma call at the academic level 1 trauma center. Participating residents were trained on orthopedic trauma database utilization by the clinical research coordinator who regularly reminded participants to record patient extremity fractures during weekly fracture conferences. To ensure that patients from the billing and resident databases were identical, matching was done based on medical record number, date of admission, and affected bone.

For all matched records containing a resident-assigned OTA classification, relevant imaging was reviewed blindly by a trauma fellowship-trained orthopedic surgeon for correct classification of fracture patterns. Resident OTA classifications were compared to this gold standard to determine overall accuracy rate of the resident database. Subgroup analyses were also performed with chi-squared tests to determine whether accuracy rates for upper extremity versus lower extremity and end segment versus diaphysis classifications were significantly different. Additional subgroup analysis was also performed to determine accuracy of classification by comparing resident year levels of training, which included postgraduate years one to three.

Results

Between April 2012 and February 2017, 2002, unique fracture records were identified from the main orthopedic billing database. Representing a 20% sample, 400 entries were randomly selected and compared to the orthopedic database. Three hundred eighteen (80%) out of 400 entries, comprised of emergency department visits and inpatient orthopedic consultations, were captured by residents (Fig. 1). Two hundred forty-two (76%) out of these 318 resident entries contained a completed OTA classification, and imaging for these patients was blindly reviewed for appropriate OTA classification by an experienced surgeon. The remaining 24% of entries had incomplete or absent OTA classifications assigned by residents. Prior to accuracy analysis, 11 entries were excluded because injury patterns were described as nonunions, peri-prosthetic fractures, or dislocations. One hundred fifty-three of the remaining 231 entries demonstrated congruence in OTA classification between resident and attending assignments for an overall accuracy rate of 66% (Fig. 1).

Fig. 1
figure1

Flow chart of 400 patient records analyzed for completeness and accuracy

On subgroup analysis, 133/190 (70%) lower extremity classifications were accurately identified compared to just 20/41 (49%) upper extremity classifications (p = 0.009). Seventy-nine (65%) of the 121 end segment fractures showed agreement versus 42 (67%) of the 63 diaphyseal injury patterns (p = 0.85) (Fig. 2). Accuracy of classification did not significantly vary by resident year of training (p = 0.142).

Fig. 2
figure2

Percentage of accurate OTA classifications for different fracture subgroups

Discussion

Trauma registries serve as an important research and quality control tool for academic hospitals. Physician residents can aid in data collection for trauma databases, with the aim of reducing registry operating costs and errors in data entry. Both the completeness and accuracy of trainee-populated databases has not yet been investigated. The purpose of this study is twofold: (1) to determine the capture rate of a resident-populated database at collecting information on all upper and lower extremity fractures treated at a single institution and (2) to determine the accuracy of resident-assigned Orthopaedic Trauma Association (OTA) classifications. The residents at our institution were able to capture 80% of a randomly selected cohort of 400 cases treated at the level 1 trauma center, while correctly assigning OTA classification for 66% of the captured cases. Specifically, 70% of lower extremity fractures were correctly classified as opposed to 49% of the upper extremity fractures. Resident year did not show a significant correlation with accuracy of classification.

O’Reilly and colleagues conducted a literature review on 69 publications discussing trauma registries to determine the different methods used in classification, measurement, and improvement of data quality in trauma databases. Most studies addressed at least one of the 3 main data quality dimensions—capture rate, accuracy, and completeness, while our study discussed all factors to some degree [9]. Literature on registries discuss the importance of professional paid staff to abstract and validate patient data [7, 10,11,12]. Wynn et al. compared data from trauma patients from both trauma and administrative databases based on matching ICD-9 codes at a level 1 trauma center. The medical record and billing personnel group underreported mechanism of injury, diagnoses, diagnostic interventions, surgical procedures, and complications from orthopedic injuries as compared to trained trauma registry staff (1243 correct entries vs. 1101 correct entries) [11]. This suggests that trauma registries populated by staff well-versed in clinical aspects of trauma care can potentially be more complete and detailed than using administrative staff to record data for research and quality control purposes.

O’Reilly et al. published the first survey of trauma registry custodians, working for single hospital and multi-hospital registries across the globe. From the 40 single hospital registries, more than half employed at least one person for each job description: director, manager, data manager, trauma nurse coordinator, and data collector (primarily nurses with no clinical duties). Thirty of these registries had a total operating cost between $10,000 and $1 million per year (USD) to manage entry of more than 100 data elements [7]. Our study shows how a trainee-populated registry can be more accurate and complete than a database populated by administrative staff specialized in medical records and billing. Further, utilizing a resident database is likely to be less costly to maintain for a single institution.

Most of the research on registries specific to orthopedic trauma has been investigated in the context of the US military. Specifically, The Military Orthopedic Trauma Registry (MOTR), which began live data abstraction in 2013, compiles detailed information on war injuries with the aim of improving clinical practice guidelines for extremity injury combat casualty care [13,14,15]. Rivera et al. conducted a quality assurance survey of MOTR entrants for lower extremity injuries to determine if this data could provide robust orthopedic trauma information for late amputation causes. The study found that 80% of 45 entries listed the direct contributing factor (infection, nonunion, etc.) to late amputation, suggesting that this fracture database is detailed enough to answer clinically relevant questions for orthopedic trauma surgeons [16]. Similar to how this study showed 80% data completeness for amputation causes, our study showed 231/318 (73%) complete OTA classifications for both upper and lower extremity injuries, suggesting that an institutional trauma database run by trainees can be just as complete as those run by trauma registrars from MOTR.

This study does have limitations. First, the study was conducted at only one institution and therefore it is uncertain whether the results are generalizable to other trauma centers. Second, the data was abstracted and interpreted by physician residents. The trainee population mainly included first, second, and third year residents, and thus may have reduced the accuracy and completeness for data entry due to resident inexperience. Fourth and fifth year residents were mainly unavailable to participate in the study due to time constraints. Lastly, a sample size of only 400 out of 2002 original entries was analyzed and therefore could have impacted the results of the capture rate and OTA classification accuracy. In a future study, more of the original entries should be analyzed to confirm relative accuracy and completeness. Moving forward, a large multi-center study in which the resident population is equally distributed across all five resident levels may be required to validate study findings. The current study highlights the necessity to implement a regimented process of reviewing the registry data to ensure quality. Institutions that maintain large internal databases should allocate resources to have a clinical research coordinator confirm data completeness and have an experienced orthopedic surgeon consistently evaluate the accuracy of the abstracted data.

Conclusion

Trainee-generated databases at academic institutions may be subject to incomplete data entry and inaccurate fracture classifications. Quality control measures should be instituted to ensure accurate data in such databases if time and effort are invested with the expectation of a useful databank and research tool.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

OTA:

Orthopedic trauma classifications

USD:

United States dollars

CPT:

Current procedural terminology

ICD-9:

International Classifications of Disease, Ninth Revision

MOTR:

Military Orthopedic Trauma Registry

References

  1. 1.

    Moore L, Clark DE. The value of trauma registries. Injury. 2008;39(6):686–95. https://doi.org/10.1016/j.injury.2008.02.023.

    Article  PubMed  Google Scholar 

  2. 2.

    Morabito DJ, Proctor SM, May CM. Overview of trauma registries. J AHIMA. 1992;63(2):39–44.

    CAS  PubMed  Google Scholar 

  3. 3.

    MacKenzie EJ, Hoyt DB, Sacra JC, Jurkovich GJ, Carlini AR, Teitelbaum SD, et al. National inventory of hospital trauma centers. JAMA. 2003;289(12):1515–22. https://doi.org/10.1001/jama.289.12.1515.

    Article  PubMed  Google Scholar 

  4. 4.

    O’Reilly GM, Cameron PA, Joshipura M. Global trauma registry mapping: a scoping review. Injury. 2012;43(7):1148–53. https://doi.org/10.1016/j.injury.2012.03.003.

    Article  PubMed  Google Scholar 

  5. 5.

    O’Reilly GM, Joshipura M, Cameron PA, Gruen R. Trauma registries in developing countries: a review of the published experience. Injury. 2013;44(6):713–21. https://doi.org/10.1016/j.injury.2013.02.003.

    Article  PubMed  Google Scholar 

  6. 6.

    Zehtabchi S, Nishijima DK, McKay MP, Mann NC. Trauma registries: history, logistics, limitations, and contributions to emergency medicine research. Acad Emerg Med. 2011;18(6):637–43. https://doi.org/10.1111/j.1553-2712.2011.01083.x.

    Article  PubMed  Google Scholar 

  7. 7.

    O’Reilly GM, Gabbe B, Cameron PA. Trauma registry methodology: a survey of trauma registry custodians to determine current approaches. Injury. 2015;46(2):201–6. https://doi.org/10.1016/j.injury.2014.09.010.

    Article  PubMed  Google Scholar 

  8. 8.

    Shapiro MJ, Cole KE Jr, Keegan M, Prasad CN, Thompson RJ. National survey of state trauma registries − 1992. J. Trauma. 1994;37:835–40.

    CAS  Article  Google Scholar 

  9. 9.

    O’Reilly GM, Gabbe B, Moore L, Cameron PA. Classifying, measuring, and improving the quality of data in trauma registries: a review of the literature. Injury. 2016;47(3):559–67. https://doi.org/10.1016/j.injury.2016.01.007.

    Article  PubMed  Google Scholar 

  10. 10.

    O’Reilly GM, Gabbe B, Braaf S, Cameron PA. An interview of trauma registry custodians to determine lessons learnt. Injury. 2016;47(1):116–24. https://doi.org/10.1016/j.injury.2015.06.032.

    Article  PubMed  Google Scholar 

  11. 11.

    Wargo C, Bolig N, Hixson H, McWilliams N, Rummerfield H, Stratton E, et al. Trauma registry reengineered. J Trauma Nurs. 2014;21(6):287–90. https://doi.org/10.1097/JTN.0000000000000087.

    Article  PubMed  Google Scholar 

  12. 12.

    Wynn A, Wise M, Wright MJ, Rafaat A, Wang YZ, Steeb G, et al. Accuracy of administrative and trauma registry databases. J Trauma Acute Care Surg. 2001;51(3):464–8. https://doi.org/10.1097/00005373-200109000-00007.

    CAS  Article  Google Scholar 

  13. 13.

    Owens BD, Kragh JF, Wenke JC, Macaitis J, Wade CE, Holcomb JB. Combat wound in Operation Iraqi Freedom and Operation Enduring Freedom. J Trauma. 2008;64(2):295–9. https://doi.org/10.1097/TA.0b013e318163b875.

    Article  PubMed  Google Scholar 

  14. 14.

    Masini BD, Waterman SM, Wenke JC, Owens BD, Hsu JR, Ficke JR. Resource utilization and disability outcome assessment of combat casualties from Operation Iraqi Freedom and Operation Enduring Freedom. J Orthop Trauma. 2009;23(4):261–6. https://doi.org/10.1097/BOT.0b013e31819dfa04.

    Article  PubMed  Google Scholar 

  15. 15.

    Cross JD, Ficke JR, Hsu JR, Masini BD, Wenke JC. Battlefield orthopaedic injuries cause the majority of long-term disabilities. J Am Acad Orthop Surg. 2011;19(Suppl 1):S1–7. https://doi.org/10.5435/00124635-201102001-00002.

    Article  PubMed  Google Scholar 

  16. 16.

    Rivera JC, Greer RM, Spott MA, Johnson AE. The Military Orthopedic trauma registry: the potential of a specialty specific process improvement tool. J Trauma Acute Care Surg. 2016;81(Suppl 1):S100–3. https://doi.org/10.1097/TA.0000000000001145.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This project was supported by the Fogarty International Center of the National Institutes of Health (NIH) under Award Number D43TW009343 and the University of California Global Health Institute (UCGHI). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or UCGHI.

Funding

The authors did not receive support from any organization for the submitted work. No funding was received to assist with the preparation of this manuscript. No funding was received for conducting this study. No funds, grants, or other support was received.

Author information

Affiliations

Authors

Contributions

AC and AC both collected and analyzed data, and drafted the manuscript. AN and AD both helped with study design and data analysis. SM edited the manuscript and prepared final materials for submission. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Abigail C. Cortez.

Ethics declarations

Ethics approval and consent to participate

This is a retrospective study. The UCSF Research Ethics Committee has confirmed that no ethical approval is required. Informed consent was obtained from all individual participants included in the study.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chopra, A., Cortez, A.C., El Naga, A. et al. Accuracy of institutional orthopedic trauma databases: a retrospective chart review. J Orthop Surg Res 16, 363 (2021). https://doi.org/10.1186/s13018-021-02478-3

Download citation

Keywords

  • Trauma database
  • Trauma registry
  • Database accuracy
  • Resident database