- Research article
- Open Access
Inter-rater variability of three-dimensional fracture reduction planning according to the educational background
Journal of Orthopaedic Surgery and Research volume 16, Article number: 159 (2021)
Computer-assisted three-dimensional (3D) planning is increasingly delegated to biomedical engineers. So far, the described fracture reduction approaches rely strongly on the performance of the users. The goal of our study was to analyze the influence of the two different professional backgrounds (technical and medical) and skill levels regarding the reliability of the proposed planning method. Finally, a new fragment displacement measurement method was introduced due to the lack of consistent methods in the literature.
3D bone models of 20 distal radius fractures were presented to nine raters with different educational backgrounds (medical and technical) and various levels of experience in 3D operation planning (0 to 10 years) and clinical experience (1.5 to 24 years). Each rater was asked to perform the fracture reduction on 3D planning software.
No difference was demonstrated in reduction accuracy regarding rotational (p = 1.000) and translational (p = 0.263) misalignment of the fragments between biomedical engineers and senior orthopedic residents. However, a significantly more accurate planning was performed in these two groups compared with junior orthopedic residents with less clinical experience and no 3D planning experience (p < 0.05).
Experience in 3D operation planning and clinical experience are relevant factors to plan an intra-articular fragment reduction of the distal radius. However, no difference was observed regarding the educational background (medical vs. technical) between biomedical engineers and senior orthopedic residents. Therefore, our results support the further development of computer-assisted surgery planning by biomedical engineers. Additionally, the introduced fragment displacement measure proves to be a feasible and reliable method.
Level of Evidence
Diagnostic Level II
The restoration of the joint surface by anatomical reduction is the primary goal in the surgical treatment of intra-articular distal radius fractures [1,2,3,4,5]. Through preoperative planning, a better understanding of the fracture pathology, the order of reduction of multiple fragments, and consequently, a more accurate choice of the surgical approach or a better choice of the implant could be eventually achieved . Preoperative planning is commonly performed by surgeons on two-dimensional (2D) images from plain radiographs . However, 2D preoperative planning often provides insufficient information to understand the three-dimensional (3D) complexity of the fracture morphology [7,8,9,10,11]. Due to recent developments in computer-aided design (CAD) software and rapid prototyping technology, accurate 3D preoperative simulations became widely accessible [12,13,14,15,16]. So far, 3D preoperative simulations are applied in the planning of osteotomies and show advantages over plain radiographs or 2D computed tomography (CT) in visualization and quantification of rotational malunions and intra-articular steps and gaps [17,18,19,20,21,22]. The biggest advantage of 3D preoperative simulation in fractures is the possibility to gain an exact understanding of all fracture lines and surfaces and to simulate the reduction of all fragments. Different approaches are described in the literature and used in daily life for simulating fracture reduction: (1) free-hand visual alignment, (2) the incorporation of the mirrored contralateral side to facilitate reduction [23,24,25,26,27], (3) the use of a statistical shape model (SSM) , or (4) attempts to use automatic alignment algorithms . None of these methods is fully automated or yet applicable in the clinical use and hence all rely strongly on the performance of the users itself, which could influence treatment decisions and outcome. Influencing factors can reach from surgical, respectively anatomical, knowledge, the amount of practice in using 3D programs as far as to experience in 3D operation planning itself [23,24,25,26,27,28]. In general, surgeons lack training in 3D planning, and in our experience, few training opportunities are available to them. Also because of the technical complexity, the 3D planning is increasingly transferred to biomedical engineers. This way, the technical expertise becomes easily available for the (not technology affine) surgeons but thereof depend on a close cooperation between them and the engineer to transfer medical knowledge.
However, it is still unclear if enough medical, in particular anatomical, knowledge is existent in this rather new subgroup of biomedical engineers for independent 3D planning. Therefore, we investigated the difference in fracture reduction accuracy regarding the overall experience in 3D planning of the user (in years) as well as the educational background (medical versus technical). We hypothesize that trained biomedical engineers have enough anatomical knowledge to perform 3D fracture reduction and perform no worse than surgeons in training. To be able to measure the variability between different raters in a standardized and automatic way, we developed a method for validating the displacement of fragments in 3D.
In this retrospective study, we included CT data of 20 patients with a distal radius fracture acquired between August 2016 and September 2017. The age range was 15 to 70 years with a mean age of 41.9 years (SD 15.5). The included radii were 10 times a right radius and 10 times a left radius. The 20 fractures were classified using the AO/OTA classification system (2× 2R3A, 4× 2R3B, 14× 2R3C) . Each of the 20 fractures consisted of 1 to 6 fragments (mean number of fragments = 3.15), from which only the intra-articular fragments were included in the study. Overall, 51 fragments from 20 different cases of distal radius fractures (an average of 2.55 fragments per case) were included in the evaluation. More detailed information of the demographics is provided in Table 1.
The image data had been acquired using a CT device (Siemens SOMATOM Definition AS, Siemens Healthcare, Erlangen, Germany) with a slice thickness of 1.0 mm (120 kV). 3D bone models were extracted from the CT data with a commercial segmentation software (Mimics 19.0; Materialise NV, Leuven, Belgium) using thresholding, region growing, and the marching cubes algorithm as described before . Each bone fragment of the fracture was segmented separately to 3D bone models and imported into a preoperative planning software (CASPA, CARD AG, Zurich, Switzerland). Nine raters were selected according to their profession, clinical experience, and 3D operation planning experience, including fractures and osteotomies of hand and forearm bones, as shown in Table 2. The clinical experience was interpreted as a measure for medical, in particular anatomical, knowledge of orthopedic surgeons. The raters included seven medical doctors and two biomedical engineers (BE). Three medical doctors were junior orthopedic residents (JOR), three medical doctors were senior orthopedic residents (SOR), and one was a senior orthopedic surgeon (SS). The medical doctors had clinical experience in a range of 1.5 up to 24 years and from marginal (< 1 year) experience in 3D operation planning up to 10 years. The senior orthopedic surgeon (SS) was defined as the gold standard with its 24 years of clinical and 10 years of 3D planning experience. The biomedical engineers had 2, respectively 3 years of experience in 3D operation planning. Each rater was familiar to the planning software due to previous work. All raters were introduced to the study goal and to the reduction task equally. The reduction task comprised the simulation of the reduction of each 3D bone model from the initial position to an optimal anatomical alignment with articular congruency. Fracture reduction was performed by interactive displacement and rotation of the fragments with the computer mouse from multiple viewpoints. The viewpoints were freely defined by the raters with the mouse. There was no constraint of time nor a restriction in the order of placing the fragments. Figure 1 shows a bone model in the initial position and the different reduction plans of one rater per group. The reduction plans of all raters were then compared with the plan of the SS (gold standard). To calculate the difference of the reduction plans, we introduced a new fragment displacement measure. The proposed fragment displacement measure permits standardized measurements in a completely automatic procedure. The 3D displacement of each fragment is represented by only two parameters: a pure 3D shift and a pure 3D rotation. To describe the displacement of a fragment from position p1 (pre-reduction) to position p2 (post-reduction), a center point of the fragment has to be calculated in a reproducible, automated way. A so-called oriented bounding box  was automatically calculated, which is the uniquely defined box with minimal volume covering the fragment. The center of this bounding box was defined as the center of the fragment. The transformation shift (TFS) was then calculated as the length of the 3D displacement vector from the center point of the fragment in pre-reduced position to the center point of the fragment in post-reduced position. The second value, the transformation angle (TFA), was defined as the pure rotational difference of a fragment from pre- to post-reduction position by angle φ around the center point. This angle is calculated automatically by using quaternions derived from the Horn transformation . TFS and TFA are independent variables. Figure 2 shows an example of the proposed fragment displacement measure.
Deviations from the gold standard planning of each bone fragment were averaged per rater and case (n = 20). These average planning differences were further aggregated to yield an average difference per rater (n = 8) and the standard deviation thereof. These two parameters (TFA and TFA) were statistically analyzed separately with the intention to represent the rater’s average performance and consistency, respectively. The effect of profession, clinical, and 3D planning experience was assessed with an ANOVA and subsequent Bonferroni-corrected post hoc tests for the first, and with a linear regression model for the latter two factors. p values < 0.05 were considered statistically significant. Statistical analysis was performed with SPSS (IBM SPSS Statistics for Windows, Version 26.0. Armonk, NY: IBM Corp.).
The distribution of the TFS per rater are illustrated in Fig. 3a. The performances among the three profession groups differed significantly (F(2, 7) = 296.686, p < 0.01). Post hoc analyses revealed a significant better fragment reduction of SOR (p < 0.01) and BE (p < 0.01) than JOR. In contrast, the difference between SOR and BE was not significantly different (p = 0.263). The analysis of the consistency of TFS (standard deviation) showed a significant difference between the profession groups (F(2, 7) = 6.208, p = 0.044). The post hoc analyses revealed no significant difference between specific groups. The linear regression models show a significant influence of the experience in 3D planning (F(1, 6) = 7.515, p = 0.034), with a R2 of 0.556) as well as a significant influence of the clinical experience (F(1, 4) = 31.282, p < 0.01, with a R2 of 0.887) to the reduction performance. The analysis shows an improvement of the TFS by 0.943 mm per year of experience in 3D planning and by 0.560 mm per year of clinical experience. More details can be seen in Figs. 4a and 5a.
The distribution of the TFA per rater are illustrated in Fig. 3b. The TFA among the three profession groups differs significantly (F(2, 7) = 17.795, p < 0.01). Post hoc analyses show a significant better fragment reduction of SOR (p = 0.013) and BE (p = 0.011) than JOR. In contrast, the difference between SOR and BE was not significantly different (p = 1.000). The analysis of the consistency of TFA (standard deviation) showed no significant difference between the profession groups (F(2, 7) = 1.688, p = 0.275). The linear regression models show a significant influence of the experience in 3D planning (F(1, 6)= 6.216, p = 0.047), with a R2 of 0.509) as well as a significant influence of the clinical experience (F(1, 4) = 11.066, p = 0.029, with a R2 of 0.735) to the reduction performance. The analysis shows an improvement of the TFA by 2.472° per year of experience in 3D planning and by 1.394° per year of clinical experience. More details can be seen in Figs. 4b and 5b.
3D preoperative planning with its advantages over 2D planning becomes more and more adopted in orthopedic surgery. However, preoperative 3D fracture reduction is currently performed manually by surgeons or engineers due to the lack of clinical-ready automated algorithms. Even though some of the computer-assisted reduction methods give the impression to be standardized and reproducible, all algorithms described in the literature [23,24,25,26,27,28] strongly rely on medical, in particular anatomical, knowledge of the user. In this study, we investigated whether trained biomedical engineers can accurately plan reductions of distal radius fractures compared with resident orthopedic surgeons with various clinical and 3D planning experience.
Our results demonstrate that the experience of 3D planning and clinical experience of the users are relevant factors for the performance of preoperative fracture reduction planning. The most important finding is that we observed no difference in the planning accuracy between senior orthopedic residents and biomedical engineers, neither in TFS nor in TFA, but a significant difference to junior orthopedic residents in both variables. This outcome is surprising, considering the very different medical knowledge (measured in years of clinical experience) and educational background, but emphasizes this new profession in computer-assisted surgical methods. The analyses of the consistency of the performances by TFA show no difference between the three groups, which underlines the abovementioned finding as well. The consistency of the performance by TFA shows a significant difference between the groups, but no difference in the post hoc analyses, which does not allow further interpretation.
A prerequisite for measuring variability between different raters is a standardized and objective measurement method. So far, little data exists about reliable and validated displacement or transformation measures, which can be applied for 3D preoperative planning in orthopedic surgery [34, 35]. We developed such a method particularly for outcome analysis of the preoperative operation planning, but this method can also be used in analysis of navigation accuracy of computer-assisted surgeries. Compared with the other in the literature describing 3D displacement measurements [18, 22, 25, 36,37,38,39], our measurement method has several advantages. It is a mathematical method including coordinate system independence and, consequently, it is user-independent. The results are reproducible which allows a better comparison between future studies. Both measures, TFS and TFA, are statistically independent and thus, the results can easily be processed in statistics. Finally, the measures are intuitive and enable user-friendly use in the clinical routine.
A limitation of the present study is the small numbers of raters and the relatively small sample size of 20 planning cases. Therefore, an analysis of the reliability of the performance of the raters within the different groups, as for example an ICC analysis, could not be realized. This would be very interesting for the further investigation of preoperative 3D planning accuracy. Another limitation in the evaluation is that the time used for preoperative reduction planning was not recorded, which in our opinion is not relevant as there are few indications where a distal radius fracture needs to be operated on within a few hours (e.g., open fractures, neurological deficits, ...). In an unpublished work, we already demonstrated the feasibility of 3D planning and navigation by patient-specific instruments (PSI) for the treatment of distal radius fractures. The entire process from acquiring the CT image data until being able to perform the navigated surgery required comfortably 2–3 days. A technical shortcoming of current 3D preoperative reduction planning methods is the absence of cartilage models in order to include articular congruity in the reduction task. Moreover, only the reduction of the bone fragments was assessed, without further clinical knowledge such as the surgical approach and biomechanical considerations such as of ligaments. Finally, it would be interesting to investigate the accuracy of the reduction plans compared with a fully automated fragment reduction method. A further interest persists in the postoperative outcome (radiological and clinical) dependent on the method of preoperative planning.
In conclusion, we found that experience in 3D operation planning and clinical experience are relevant factors to accurately plan an intra-articular reduction of the distal radius. No difference regarding the educational background (medical vs. technical) was observed and therefore supports the further development of computer-assisted surgical planning by biomedical engineers. The hereby used fragment displacement measure is a basic and easy tool to compare fragment transformation in 3D bone models. Consequently, we suggest using more standardized measurement methods for all the future work for comparison of fragment transformations in 3D bone models in order to make future studies more comparable.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Junior orthopedic residents
Senior orthopedic residents
Senior orthopedic surgeon
- SSM :
Statistical shape model
Fernández DL. Malunion of the distal radius: current approach to management. Instr Course Lect. 1993;42:99–113.
Fernandez DL. Reconstructive procedures for malunion and traumatic arthritis. Orthop Clin North Am. 1993;24(2):341–63.
Matthews LS, Kaufer H, Garver DF, et al. The effect on supination-pronation of angular malalignment of fractures of both bones of the forearm. J Bone Joint Surg Am. 1982;64(1):14–7.
Sarmiento A, Ebramzadeh E, Brys D, et al. Angular deformities and forearm function. J Orthop Res. 1992;10(1):121–33.
Trumble TE, Schmitt SR, Vedder NB. Factors affecting functional outcome of displaced intra-articular distal radius fractures. J. Hand Surg. Am. 1994;19(2):325–40.
Rüedi TP, Murphy WM. AO-Prinzipien des Frakturmanagements. 2. Auflage. Stuttgart: Thieme; 2008.
Bilić R, Zdravković V, Boljević Z. Osteotomy for deformity of the radius. Computer-assisted three-dimensional modelling. J Bone Joint Surg Br. 1994;76(1):150–4.
Jupiter JB, Ruder J, Roth DA. Computer-generated bone models in the planning of osteotomy of multidirectional distal radius malunions. J. Hand Surg. Am. 1992;17(3):406–15.
Creasman C, Zaleske DJ, Ehrlich MG. Analyzing forearm fractures in children. The more subtle signs of impending problems. Clin Orthop Relat Res. 1984;188:40–53.
Usui M, Ishii S, Miyano S, et al. Three-dimensional corrective osteotomy for treatment of cubitus varus after supracondylar fracture of the humerus in children. J Shoulder Elb Surg. 1995;4(1 Pt 1):17–22.
Uchida Y, Ogata K, Sugioka Y. A new three-dimensional osteotomy for cubitus varus deformity after supracondylar fracture of the humerus in children. J Pediatr Orthop. 1991;11(3):327–31.
Brown GA, Firoozbakhsh K, DeCoster TA, et al. Rapid prototyping: the future of trauma surgery? J Bone Joint Surg Am. 2003;85-A Suppl:49–55.
Ellis RE, Tso CY, Rudan JF, et al. A surgical planning and guidance system for high tibial osteotomy. Comput Aided Surg. 1999;4(5):264–74.
Perry M, Banks P, Richards R, et al. The use of computer-generated three-dimensional models in orbital reconstruction. Br J Oral Maxillofac Surg. 1998;36(4):275–84.
Shimizu T, Fujioka F, Gomyo H, et al. Three-dimensional starch model for simulation of corrective osteotomy for a complex bone deformity: a case report. Foot Ankle Int. 2003;24(4):364–7.
Radermacher K, Portheine F, Anton M, et al. Computer assisted orthopaedic surgery with image based individual templates. Clin Orthop Relat Res. 1998;354:28–38.
Schweizer A, Fürnstahl P, Harders M, et al. Complex radius shaft malunion: osteotomy with computer-assisted planning. Hand (N Y). 2010;5(2):171–8.
Schweizer A, Fürnstahl P, Nagy L. Three-dimensional correction of distal radius intra-articular malunions using patient-specific drill guides. J Hand Surg Am. 2013;38(12):2339–47.
Miyake J, Murase T, Moritomo H, et al. Distal radius osteotomy with volar locking plates based on computer simulation. Clin Orthop Relat Res. 2011;469(6):1766–73.
Farshad M, Hess F, Nagy L, et al. Corrective osteotomy of distal radial deformities: a new method of guided locking fixed screw positioning. J Hand Surg Eur Vol. 2013;38(1):29–34.
Fürnstahl P, Wirth S, Nagy L, et al. Advantages and pitfalls in computer assisted orthopedic surgery planning using rapid-prototyped guides. RTejournal - Forum für Rapid Technol. 2014;2014(11).
Roner S, Vlachopoulos L, Nagy L, et al. Accuracy and early clinical outcome of 3-dimensional planned and guided single-cut osteotomies of malunited forearm bones. J Hand Surg Am. 2017;42(12):1031.e1–8.
Murase T, Oka K, Moritomo H, et al. Three-dimensional corrective osteotomy of malunited fractures of the upper extremity with use of a computer simulation system. J Bone Jt Surgery-American Vol. 2008;90(11):2375–89.
Vlachopoulos L, Unner CD, Gass T, et al. Computer algorithms for three-dimensional measurement of humeral anatomy: analysis of 140 paired humeri. J Shoulder Elb Surg. 2016;25(2):e38–48.
Vlachopoulos L, Schweizer A, Meyer DC, et al. Three-dimensional corrective osteotomies of complex malunited humeral fractures using patient-specific guides. J Shoulder Elb Surg. 2016;25(12):2040–7.
Bicknell RT, DeLude JA, Kedgley AE, et al. Early experience with computer-assisted shoulder hemiarthroplasty for fractures of the proximal humerus: development of a novel technique and an in vitro comparison with traditional methods. J Shoulder Elb Surg. 2007;16(3):S117–25.
Omori S, Murase T, Oka K, et al. Postoperative accuracy analysis of three-dimensional corrective osteotomy for cubitus varus deformity with a custom-made surgical guide based on computer simulation. J Shoulder Elb Surg. 2015;24(2):242–9.
Vlachopoulos L, Lüthi M, Carrillo F, et al. Restoration of the patient-specific anatomy of the proximal and distal parts of the humerus. J Bone Jt Surg. 2018;100(8):e50.
Jiménez-Delgado JJ, Paulano-Godino F, PulidoRam-Ramírez R, et al. Computer assisted preoperative planning of bone fracture reduction: simulation techniques and new trends. Med Image Anal. 2016;30:30–45.
Meinberg E, Agel J, Roberts C, et al. Fracture and dislocation classification compendium—2018. J Orthop Trauma. 2018;32:S1–S10.
Schweizer A, Mauler F, Vlachopoulos L, et al. Computer-assisted 3-dimensional reconstructions of scaphoid fractures and nonunions with and without the use of patient-specific guides: early clinical outcomes and postoperative assessments of reconstruction accuracy. J. Hand Surg. Am. 2016;41(1):59–69.
Li S, Zhang Z, Li B, et al. Multiscale rotated bounding box-based deep learning method for detecting ship targets in remote sensing images. Sensors (Basel). 2018;18(8):2702.
Horn BKP. Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A. 1987;4(4):629–42.
Kim JS, Park TS, Park SB, et al. Measurement of femoral neck anteversion in 3D. Part 2: 3D modelling method. Med Biol Eng Comput. 2000;38(6):610–6.
Kawanishi Y, Moritomo H, Omori S, et al. A comparison of 3-D computed tomography versus 2-D radiography measurements of ulnar variance and ulnolunate distance during forearm rotation. J Hand Surg Eur Vol. 2014;39(5):526–32.
de Roo MGA, Dobbe JGG, Peymani A, et al. Accuracy of manual and automatic placement of an anatomical coordinate system for the full or partial radius in 3D space. Sci Rep. 2020;10(1):8114.
Teunis T, Bosma NH, Lubberts B, et al. Melone’s concept revisited: 3D quantification of fragment displacement. J Hand Microsurg. 2016;8(1):27–33.
Fürnstahl P, Schweizer A, Graf M, et al. Surgical treatment of long-bone deformities: 3D preoperative planning and patient-specific instrumentation. Lect Notes Comput Vis Biomech. 2016;23:123–49.
De Muinck Keizer RJO, Meijer DT, Van Der Gronde BATD, et al. Articular gap and step-off revisited: 3D quantification of operative reduction for posterior malleolar fragments. J Orthop Trauma. 2016;30(12):670–5.
The authors would like to thank all members of the Research in Orthopedic Computer Science Team and the Institute of Diagnostic and Interventional Radiology, University Hospital Zurich, for assistance in many belongings. Moreover, the authors would like to thank Olivia Zindel-Geisseler for the critical revision.
This work was partially supported by the Swiss Canton of Zurich through a highly-specialized medicine grant. The employment of Simon Roner in the Research in Orthopedic Computer Science Team was funded by a grant of the Balgrist Stiftung.
Ethics approval and consent to participate
Institutional review board approval was obtained from the local ethical committee (BASEC 2016-01925) prior to the start of the study.
Consent for publication
All patients signed a general consent.
The authors declare no competing interests
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zindel, C., Fürnstahl, P., Hoch, A. et al. Inter-rater variability of three-dimensional fracture reduction planning according to the educational background. J Orthop Surg Res 16, 159 (2021). https://doi.org/10.1186/s13018-021-02312-w
- Distal radius fracture
- Computer-assisted planning
- Fracture reduction
- Three-dimensional displacement analysis
- Planning experience